Search for a command to run...
I asked my code to add two numbers and it quietly answered zero. This week I stopped trusting the ledger and started trying to break it, and it broke.
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
The accounting math was bulletproof. Then I asked a harder question: what could someone do if they could just reach the API? The answer was, unfortunately, everything.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
Here is a small program. It adds two amounts of money together.
a := MinInt64 // the most negative number an int64 can hold
b := MinInt64
sum, err := a.Add(b)
// sum is 0. err is nil.
Two enormous negative numbers, added together, and my ledger said the answer was zero and nothing went wrong. No error, no warning, just a clean, confident, completely wrong result. On a payment system, "we added two things and silently got zero" is the exact shape of a bug that makes money disappear.
I did not find this by staring at the code. I found it because this week I stopped asking "do my tests pass?" and started asking "what would it take to make this thing lie to me?" That is a different question, and it is the whole point of a testing week.
This is post 9 of 12. The ledger already has a domain model, a Postgres schema, concurrency control, REST and gRPC APIs, idempotency, and tracing. What it did not have was proof that any of it holds up when things go wrong. So this week is five kinds of test, each doing a job the others cannot.
The obvious place to start is coverage: what percentage of my lines does the test suite actually run? I set a gate in CI: the build fails if coverage on the core code drops below 80%.
But here is the trap. Coverage counts lines touched, not behavior checked. I can hit 80% by testing every getter and constructor while never once testing the branch that rejects an unbalanced transaction. On a ledger, the dangerous lines, the rollback path, the overflow check, the currency mismatch, are exactly the ones that are annoying to reach and easy to skip. A high number can mean "well tested" or it can mean "well padded," and the number itself will not tell you which.
So I did two things. First, the gate is not one flat number. The money-critical packages carry higher floors than the plumbing:
package floor why
------------------------------------------------------------
internal/domain 90% the double-entry invariant lives here
internal/ledger 85% posting, idempotency, rollback
internal/postgres 80% the SQL and the DB constraints
everything else 80% transport, wiring, plumbing
------------------------------------------------------------
generated code excluded it tests the code generator, not me
Second, and more important, I treated the coverage number as a floor to clear on the way to real tests, not the goal itself. Hitting 90% on the domain package was worth doing only because getting there meant writing tests for the branches that actually matter. The percentage is a smoke alarm: useful for telling you a whole room is untested, useless for telling you the tests you do have are any good.
For that second question, you need a different tool.
Most tests are examples. You write down an input, you write down the answer you expect, you check they match. That is fine, but you only ever check the handful of cases you thought of. The bug that bites you is the one you did not think of.
Property-based testing flips this around. Instead of "for input 5, expect 10," you state a property that must hold for all inputs, and the library throws hundreds of random cases at it trying to find one that breaks it. For a ledger, the properties write themselves:
Add any two amounts in the same currency, and either you get the exact mathematical sum, or you get a loud overflow error. Never a silent wrong answer.
Apply any random sequence of balanced transactions, and every account's balance always equals the sum of its own postings, and the total across all accounts is always zero.
Adding two amounts in different currencies always fails.
I wrote the first one as: for any two int64 amounts, if their true sum fits in an int64, Add returns it exactly; if it does not, Add returns an overflow error. Then I let the library hunt.
It found MinInt64 + MinInt64 in a few thousand tries.
The bug was in my overflow check. I was detecting overflow by looking at signs: two positives that produce a negative overflowed, two negatives that produce a positive overflowed. Reasonable, and wrong at one point. MinInt64 + MinInt64 in two's complement wraps all the way around and lands on exactly zero. Zero is not positive, so my "two negatives became positive" check missed it, and the function returned zero with no error. The fix is the textbook signed-overflow test: adding a negative number should never make the result larger.
I want to be clear about what happened here, because it is the best argument for property testing I have. I wrote that money type weeks ago. I tested it. It passed. It shipped. It had a bug that could destroy money, sitting quietly in the one input I never thought to try, and no example-based test I would have written by hand was ever going to catch it. The randomness caught it in seconds.
Property tests check the logic. They do not check what happens when the infrastructure fails halfway through. For that, you have to actually break something, on purpose, at a precise moment.
The scenario I care about is the one that keeps ledger engineers up at night. A client sends a payment. The server commits it to Postgres. And then, before the "success" reply gets back to the client, the network drops. The commit happened. The money moved. But the client has no idea. All it knows is that its request died. So it does the natural thing: it retries.
If your system is not careful, that retry posts the payment a second time, and now your friend paid for dinner twice.
I tested two versions of this. In the first, the connection dies before the commit reaches Postgres. The correct outcome is a clean rollback: nothing persists, not a single half-written posting, because a payment that only half-happened is not allowed to exist. In the second, the commit lands but the acknowledgement is lost, the diagram above, and the correct outcome is that the retry recognizes the idempotency key and returns the original transaction instead of creating a new one.
The tricky part is making "the connection dies at exactly the right moment" happen reliably. My first instinct was to just kill the Postgres container mid-request. That is a bad idea: it is a race. Sometimes you kill it before the commit, sometimes after, sometimes you miss the window entirely and test nothing, and occasionally it fails in CI for reasons nobody can reproduce. A flaky chaos test is worse than no chaos test, because it teaches you to ignore red builds.
So instead I put a programmable network proxy (a tool called toxiproxy) between the app and the database, and I cut the connection at a controlled point in the protocol, driven by the code, not by timing. For the rollback case, I sever the link the instant the app tries to send its COMMIT, so the commit never reaches the server. For the ambiguous case, I cut only the reply path: the COMMIT flows to Postgres and durably commits, but the acknowledgement coming back gets dropped. That second one is the money-losing scenario made real and repeatable, and the test proves the retry does not double-post. Same failure, every run, no luck involved.
The last question is boring but real: does it hold up under load? The target I set was 500 payments per second with a 99th-percentile latency under 100 milliseconds, run locally against the full stack in Docker.
Load testing a ledger is easy to fake, in two specific ways, and I was determined not to.
The first fake: idempotency keys. If every request in your load test reuses the same key, you are not testing the payment path at all. After the first one, every request is a cheap "oh, I've seen this key, here's the cached answer" lookup that never touches the real write logic. You would get a gorgeous throughput number that measures the wrong thing entirely. So every request in my test carries a unique key, with a deliberate small slice of duplicates mixed in to exercise the replay path honestly.
The second fake, and this one is a genuine temptation, is durability. That 100-millisecond target is dominated by one thing: how long Postgres takes to physically flush each commit to disk. There is a setting that makes commits return before they are safely on disk, and flipping it would make my latency numbers look fantastic. It would also mean that a power cut could lose a committed payment. On a ledger, that is not a performance tuning knob, it is a lie. Durability stays on. The number is what it is: 500 requests a second, p99 well under the target, with every commit actually on disk.
I also run two shapes of load, not one. A "fan-out" test spreads payments across many accounts to measure raw throughput. A "hot account" test slams a single account, because that is where real systems fall over: everyone contends for the same row lock, and the tail latency blows up. The system that looks fast when work is spread out is not the same system as the one under contention, and only testing the easy shape tells you nothing about the hard one.
Five kinds of test, each catching what the others structurally cannot:
A unit test would never have found the ambiguous-commit double-post, that needs a real database and a real network failure. A chaos test would never have found the MinInt64 overflow, that needs thousands of random inputs. A load test would never have told me the invariant holds, only that it is fast. Each layer sees a different failure. Skip one and you have a blind spot exactly the size of the thing it would have caught.
As with every week, I built this with Claude in my terminal, and the honest split is worth describing, because this was a week where the division of labor really mattered.
I owned the decisions and the skepticism. When Claude drafted the chaos test by literally killing the Postgres container, I pushed back: that is a race, use a proxy and cut the connection deterministically. When it wrote the ambiguous-commit case as a plain retry with no actual failure injected, an automated review I ran flagged it, and I sent it back to inject the real ack-loss, because the whole value of that test is the failure. I made the calls on the coverage floors, on keeping durability on, and on a coverage badge that uses no long-lived secrets.
Claude owned the execution, and it was genuinely good at it: writing the several thousand lines of test code across the packages, wiring up toxiproxy, reconciling my plan against the actual code when my assumptions about environment variables and API shapes were slightly off, and running the whole thing task by task with a review after each one. And, satisfyingly, one of those reviews is what caught the weakened chaos test before it could ship. The overflow bug in my money type was found by the property test it wrote, running against the code I wrote weeks ago. Neither of us gets to feel too clever.
The invariant that this whole project is built on, money never appears or disappears, is now something I have watched survive an overflow, a mid-commit database death, a lost acknowledgement, and 500 requests a second. That is a different kind of confidence than "the tests pass." It is closer to "I tried to break it in the specific ways it could break, and it did not."
The full strategy is written up in ADR-011, and the repo, with a coverage badge that is finally honest, is at github.com/sohag-pro/go-ledger.
Next week the project leaves the laptop for good: Docker and the real infrastructure that puts this thing on a server.