Search for a command to run...
A double-entry ledger that passed every test still had a bug that turned 100 dollars into 15 dollars, a chain that forked the moment you ran two copies, and no way to ever forget a customer. Here is the teardown, and the fix.
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
There is a specific kind of fear that shows up once a side project starts looking finished.
The go-ledger has passed every test for months. Double-entry postings that sum to zero, idempotency keys, a tamper-evident audit chain, multi-currency with FX conversion, a live demo at go.sohag.pro. On a good day I would call it production-grade. Then one evening I asked the question that ruins good days: if a real payments company wanted to run this as their own white-label ledger, and put it in front of a due-diligence auditor, what would that auditor tear apart in the first hour?
So I ran the audit. On myself. I sat down and reviewed the whole codebase the way a skeptical fintech engineer would, looking for the things that do not show up in a green test suite: the ways it loses money quietly, the ways it breaks when a second copy starts up, the ways it cannot satisfy a regulator. I wrote the findings down as if I were handing a client a report they paid for.
It failed. Not a little. Three findings were the kind you label "Blocker: do not run this with real money," and one of them was a bug that produced the wrong amount of money and no test had ever noticed.
This post is the teardown and the fix. It is long, because the audit was thorough and the remediation was thirty tasks and four architecture decision records. But the interesting part is not the count. It is that most of the failures were invisible from inside a passing test suite, and every one of them teaches something about what "production-grade" actually costs.
Before the findings, the bar. A hobby ledger and a white-label money core look identical in a demo. They diverge on four questions:
Can it lose money without anyone noticing? Not "does it crash," but "does it ever record the wrong amount and keep going."
Can it lose your data? If the disk dies, is the ledger gone, and how much of it.
Can it run as more than one process? The moment you want zero-downtime deploys or more throughput, a second instance starts. Does the invariant survive that.
Can one customer ever see or affect another? Multi-tenant isolation is not a feature you add later. It is a property every query must have.
The demo answered all four with a confident "sure." The audit answered all four with "no."
This is the one that still bothers me.
When I added multi-currency, I did the money math the careful way everyone tells you to: integers only, never floating point, amounts stored as minor units. One dollar is 100 cents. A conversion multiplies the source amount by a scaled-integer rate and rounds once, using banker's rounding, in big integers so nothing overflows. I wrote property tests. I wrote a reconciliation test that proved money was conserved across a round trip. Green.
Here is what I missed. "Minor units" is not the same number for every currency. The US dollar has 2 decimal places, so 100 dollars is 10,000 minor units. The Japanese yen has zero decimal places: one yen is the smallest unit, there are no sub-yen. Bahraini dinar has three. My conversion code assumed every currency had two, baked in, invisibly, forever.
Watch what that does. Convert 100.00 USD to JPY at a rate of 150 yen per dollar. The right answer is 15,000 yen.
source: 100.00 USD -> 10,000 minor units (USD has 2 dp)
rate: 150 yen per 1 dollar
what the code did: what it should have done:
10,000 * 150 = 1,500,000 10,000 * 150 * 10^(0 - 2)
"1,500,000 yen" = 1,500,000 / 100
= 15,000 yen
off by 100x. Every JPY conversion. Silently.
The code produced 1,500,000 yen for a 100 dollar conversion. A hundred times too much. And nothing failed, because every test I had written used USD and EUR, which both happen to have two decimal places, so the difference in exponents was zero and the bug was mathematically invisible. My "money is conserved" reconciliation test passed because it converted USD to EUR and back, and two wrongs of the same magnitude cancel.
The fix was a currency registry that knows each currency's minor-unit exponent, and one term added to the conversion: multiply by ten to the power of (destination exponent minus source exponent), folded into the same single rounding step so it stays integer-only and unbiased. Then tests that actually use JPY (0 dp) and BHD (3 dp), asserting the exact expected integer.
The lesson is not "handle currency exponents," although, please handle currency exponents. The lesson is that a test suite proves the cases you thought of. It says nothing about the cases you did not, and the case you did not think of is exactly where the money leaks. The only defense I have found is adversarial review by someone who does not share your blind spots, even when that someone has to be you in a different chair.
Every transaction in the ledger extends a per-tenant tamper-evident hash chain. Each audit row stores the hash of the row before it, so if anyone edits history, every hash after the edit stops matching and verification catches it. It is a nice property. It was also a trap.
To keep two concurrent posts for the same tenant from both grabbing the same "previous hash" and forking the chain, I serialized them with an in-process mutex. A lock, in memory, in one Go process. It worked perfectly, on one process.
The audit finding: that lock lives in one process's memory. Start a second instance of the service (which you will, the first time you want a rolling deploy), and the two instances share nothing. Both read the same chain head, both append a row claiming to follow it, and the chain forks into two branches that both think they are canonical.
ONE process (mutex works): TWO processes (mutex is blind):
post A --\ instance 1: read head H --> write B (prev=H)
mutex --> H -> A -> B instance 2: read head H --> write C (prev=H)
post B --/ \
two rows both claim prev=H
chain forked, verify breaks
This is a "single-instance correctness cliff." The service is correct right up until the moment you scale it, and then it is silently corrupt. For a system whose entire pitch is "you can trust the history," that is disqualifying.
The fix was the biggest architectural change in the remediation: take the chain off the hot path entirely. A posting now writes an event to an append-only outbox inside the same transaction, atomically, with no chain read. A single background worker, elected leader through a Postgres advisory lock held on one dedicated connection, drains the outbox in commit order and builds the chain by itself. One writer, so no fork, no matter how many instances are posting. To get commit order right across concurrent transactions I used the database's own transaction-visibility watermark, processing only events whose transaction id is below the oldest still-running transaction, so a slow commit can never sneak in behind a row that already got chained.
Getting that worker right took two rounds of review. The first version ran its queries on a pooled connection instead of the lock-holding one, which meant that if the leader lost its lock (a database failover, a dropped connection), it would not notice and would keep draining while a new leader also started, reintroducing the exact fork the design existed to prevent. Concurrency code has a way of looking finished long before it is.
Here is a genuine conflict, not a bug. A ledger is append-only and immutable, by law and by design: you do not get to rewrite financial history. A person also has, by a different law, the right to be forgotten: ask for their personal data to be erased. A posting's free-text description can carry personal data ("rent to Jane Doe, 12 Elm Street"). You cannot delete the row, it would break the hash chain and the append-only invariant. You cannot keep the plaintext forever either.
The resolution is a technique called crypto-shredding, and it is genuinely elegant. You do not erase the data. You encrypt it, and erase the key.
Each tenant gets a data key. Descriptions are encrypted with it before they are stored, and the same ciphertext goes into both the posting and the audit snapshot, so the hash chain hashes ciphertext. "Erasing" a subject means destroying the key. Every byte stays exactly where it was, so the chain still verifies bit for bit. But the ciphertext is now unreadable by anyone, forever. The data is gone without a single row changing.
normal read: ciphertext --[tenant key]--> "rent to Jane Doe"
after shred: ciphertext --[key destroyed]--> "[redacted: erased]"
the row never changed. the hash still matches. the PII is gone.
The subtle part, which the review caught, is that destroying the key must not brick the tenant. A conversion leg is labeled "convert: debit source account," which is not personal data, but it still goes through the same encryption. If shredding just refused to encrypt anything afterward, every future conversion for that tenant would fail forever. So the keys are versioned: a shred destroys the current key (erasing everything under it) and the next write mints a fresh one. The tenant keeps operating, the erased history stays erased.
Those three were the headliners. The full audit had forty-one findings across correctness, security, durability, operations, and compliance. A sampling of what else was wrong, and what it turned into:
Durability was a lie. The only backup was a nightly pg_dump to the same disk as the database. If the disk died, the ledger and its backup died together, and up to a full day of transactions was gone. Now: continuous write-ahead-log archiving and encrypted base backups shipped off the box, point-in-time recovery, and a job that actually restores the backup into a throwaway database and re-verifies every balance and every audit chain, because a backup you have never restored is a hope, not a backup.
A forgotten WHERE tenant_id could cross tenants. Every query scoped by tenant id, but "every query, forever, without exception" is not a guarantee a human can make. Now Postgres row-level security enforces it in the database: a request runs with its tenant id set, and a query that forgets the filter returns nothing across tenants instead of everything. Defense in depth, because the app-level filter will eventually have a bug.
No tenant entity, no key lifecycle. "Tenant" was a bare id scattered across tables. Now there is a real tenants table with status (you can suspend one), API keys with scopes and expiry and rotation, and an admin surface plus a CLI to onboard a tenant and issue keys without touching SQL.
No way to reverse a transaction, no webhooks, no way to erase or shred, no rate limit on gRPC, no alerts. All added, each the boring-but-necessary connective tissue of a system someone else is supposed to integrate with.
Thirty tasks touching money code is how you turn a working ledger into a broken one. The process mattered as much as the fixes.
Every significant decision became an architecture decision record first, written before the code, so the "why this and not that" was captured while it was fresh: one master remediation record and three deep ones for durability, the multi-instance chain, and crypto-shredding. Then each task was implemented in isolation, reviewed against its spec, and, for the risky ones (the FX math, the chainer, the encryption), reviewed a second time by a reviewer whose only job was to try to break it. That second reviewer earned its keep: it found the chainer's lost-lock fork, a deploy script that would have run migrations as root, and a fingerprint that silently ignored a field.
When it was done, I did the thing you are supposed to do and almost never do: I ran the audit again, from scratch, as if I had never seen the code, to check that every finding was actually fixed and not just marked fixed. Then I ran it a third time with a fresh reviewer. Both came back the same. Every blocker closed in code, the money bug closed with correct arithmetic and the JPY and BHD tests that should have existed from the start, no regressions. What remains is not engineering: it is operational provisioning, the offsite backup bucket and the production monitoring stack, both written and waiting on credentials.
The uncomfortable takeaway is that "all tests pass" and "production-grade" are nearly unrelated statements. My tests passed for months while the code turned 100 dollars into 15,000 yen and would have forked its own audit trail the first time it scaled. Tests prove the failures you imagined. An audit, done adversarially, by someone willing to assume you are wrong, finds the failures you did not.
You do not need to hire a fintech expert to get that second set of eyes, although it helps. You need to sit in a different chair, stop defending the code, and ask the four questions a skeptic would ask: can it lose money, can it lose data, can it run twice, can one customer touch another. Then believe the answers, even when they ruin a good evening.
The go-ledger is genuinely a white-label money core now. It took admitting it was not.