Search for a command to run...
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
The first time a bank made me wait, I was annoyed. I was trying to send a fairly large transfer, tapped confirm, and instead of the usual "done" the app said the payment was "under review." Nothing moved. My balance sat exactly where it was. A few hours later it went through. At the time it felt like the app was being slow. Later I understood it was being careful: somebody, or some rule, had decided that a payment that size does not just happen because one person tapped a button.
That is an approval workflow, and this week I gave the ledger one. A transaction over a configured size does not post. It waits. Someone with the authority to approve it either lets it through or turns it down, and only then does the money move, or not. This is post 13 of the build.
The whole feature turns on one number, per currency. If the largest single posting in a transaction is bigger than the threshold you set for its currency, the transaction is held instead of posted. Under the threshold, nothing changes: it posts exactly like it did last week.
Here is a concrete one. Say the USD threshold is 1,000.00 (the ledger stores money in minor units, so that is 100000 cents). Someone tries to move 2,000.00 between two accounts:
| Posting | Account | Amount (cents) |
|---|---|---|
| 1 | ops:checking | -200000 |
| 2 | vendor:payable | +200000 |
| Largest leg | 200000 |
200000 is over 100000, so this one does not post. It is held. The person who submitted it gets back a "held for approval" response, not a posted transaction. And here is the part that matters most: nothing is written to the postings table. No balance anywhere reflects that 2,000.00, because in this ledger a balance is never a stored number. It is the sum of an account's postings. If I want the held money to not exist yet, the simplest, safest thing is to make sure not a single posting exists yet.
POST /v1/transactions (largest leg = 200000 USD)
|
v
approval gate: leg > threshold for USD?
| |
no | | yes
v v
post normally write a pending_transactions row
(postings written, (the intended request, as JSON.
201 Created) NO postings. NO balance change.)
|
v
202 Accepted, held for approval
So a held transaction is stored as intent, not as a ledger entry. A row in a new pending_transactions table holds the original request (the legs, the reference, the effective date) as a JSON payload, plus which threshold it tripped and who submitted it. Postings, transactions, balances: untouched.
When an approver approves the held transaction, what should happen? The tempting answer is "flip a status flag and mark it posted." That answer is wrong for this ledger, and seeing why is the interesting part.
The held row is just intent. To actually post it, I take that stored payload and run it back through the normal posting path, the exact same code an ordinary transaction uses. Same validation, same balance and currency database triggers, same audit trail. Approval is a replay of the original request, not a special second way to write a transaction.
That buys two things. First, there is only one place in the codebase that ever writes a transaction, so there is only one place that can be wrong. Second, and this surprised people I described it to: the replay validates against current balances, not the balances from when the transaction was first submitted. If the account had enough money on Monday but has been drained by Wednesday when the approval finally happens, the approval fails. It should. The pending row is a request to move money now, checked now, not a promise cashed against a stale snapshot.
held pending --approve--> replay the stored payload through
(intent JSON) the normal CreateTransaction path
|
+-------------+-------------+
| |
validates now? validates now?
yes no
| |
v v
postings written, still pending, error
pending -> approved, returned, nothing
linked to the new tx changed. try later.
I want to be honest about the parts that were not clean the first time, because that is where the real engineering lived this week.
The first: what happens if two people approve the same held transaction at the same instant? Or one approves while another rejects? My first version took a database row lock, checked the status, released the lock, and then did the write in a separate step. That gap is a door. Two decisions could both walk through it and race, and you could end up with a posted transaction whose pending row says "rejected," a real payment orphaned from any record that it was ever allowed. The fix was to hold the lock across the whole decision, and for the one case that cannot (the approve has to post in its own transaction), to re-check under the lock afterward and let the already-posted money win, so a real transaction is never left orphaned.
The second was subtler and I did not catch it myself. My AI pair, running a final review across the whole week's diff, flagged it: the held path never recorded the caller's idempotency key. Idempotency keys are the thing that makes a retry safe. The client sends a payment, the network eats the response, the client retries with the same key, and the server is supposed to recognize the retry and not do the work twice. But a held transaction skipped the code that stores the key. So a retried large payment created a second pending row, and approving both would post the money twice. That is the worst class of bug this project can have: it makes money out of nothing. The fix was to dedupe the hold on the caller's key, so a retry returns the same pending instead of minting a new one. I will come back to the AI-pair honesty at the end, but that one is worth flagging here: the bug was on the main retry path, not some exotic race, and a per-task review would never have seen it because the seam only exists when you look at the whole feature at once.
There is also a switch for the classic "four eyes" rule: require that the approver is a different person than the submitter. It is off by default (the public demo runs on a single key, so it has to be able to approve its own), but the control exists and is one config flag away.
Now the second half of the week, and the part I am most pleased with. Every one of these approval moments (requested, approved, rejected, cancelled, expired) should be an event. Something the outside world can subscribe to. When a big payment gets held, a system somewhere probably wants to ping a human. When it is approved, a downstream service wants to know.
The reflex reach here is Kafka, or some event broker. I do not have one, and I do not want one. Here is the thing though: I already had a durable, ordered, append-only event stream sitting in the database. I built it weeks ago and called it the audit log.
The way it works (the outbox pattern) is that when a transaction posts, it writes a row to an outbox table in the same database transaction as the postings. Either both land or neither does. A background worker then drains that outbox in commit order and builds the tamper-evident, hash-chained audit log, and a webhook worker reads that log by sequence number and delivers events to subscribers. That is already an event stream with exactly-once semantics off a committed transaction. Approval events just need to ride the same rails.
The catch: that audit chain was built to record transactions. Every row pointed at a transaction id. But a rejected approval never becomes a transaction. There is nothing to point at. So I had to teach the chain to carry events that are about something other than a transaction.
That meant three new columns and a careful choice. The transaction_id became nullable. Two new columns, subject_type and subject_id, let a row say "this event is about a pending transaction, id such-and-such" instead of naming a transaction. And a hash_version column, because the audit log is tamper-evident: every row's hash is computed over its own fields, and if I changed how the hash was computed, every existing row from the last twelve weeks would fail to verify.
So version 1 is the old preimage, exactly as it was, and every transaction row stays version 1 and re-hashes to the same value it always did. Version 2 folds in the subject and tolerates a missing transaction id, and only the new lifecycle events use it. The verifier reads each row's version and checks it with the matching recipe. I wrote a test that builds a chain mixing v1 transaction rows and v2 approval rows, verifies the whole thing, then tampers with one v2 row's subject and confirms the verifier catches it. The crown jewel of the project (a ledger history you can prove was not altered) had to keep holding while the chain learned a new trick.
And because approval events now live in the same log the webhook worker already reads, they get delivered to subscribers for free. No new stream, no broker, no second delivery path. The webhook that used to carry "a transaction was posted" now also carries "a large payment is waiting for you," and it did not cost a single new moving part.
Pulling it together, here is the shape of what shipped this week:
A transaction over its currency's threshold is held as intent, with no postings, so no balance ever reflects unapproved money.
Approving it replays the real posting path against current balances. It can fail, and failing leaves the request pending, not half-applied.
The decision is serialized under a row lock, and a posted transaction can never be orphaned from its record.
A retried large payment returns the same held request, so an approval can never double-post.
Every step is an event on the existing tamper-evident chain, delivered by the existing webhook worker, with zero new infrastructure.
The whole feature is off by default. If you never set a threshold, the ledger behaves exactly as it did last week.
Same deal as every week: I own the decisions, Claude accelerates the work, and I am honest about the split. This week the split was interesting.
I made the calls that mattered: hold intent instead of flipping a status, replay through the one real posting path, reuse the audit chain as the event stream rather than add a broker, keep the old hash version frozen. Those are in the architecture decision record, ADR-025, written before any code, as usual.
But two of the bugs above were caught by the review loop, not by me writing carefully. The concurrency gap where a decision could orphan a posted transaction, and the idempotency gap where a retried large payment could double-post, both surfaced when a reviewer looked at the finished work with fresh eyes. The idempotency one in particular came out of a review that read the entire week's diff at once, precisely the view that no single task ever has. I have said before that "I watched 100 goroutines try to corrupt my ledger and fail" is a different kind of knowledge than reading about it. This week's version of that is: an adversarial second read of your own money-moving code will find the bug you were too close to see. Build that step in.
The repo is open at github.com/sohag-pro/go-ledger. Next week is the last of this stretch: webhooks in full, the launch, and a look back at the whole build.