Search for a command to run...
The accounting math was bulletproof. Then I asked a harder question: what could someone do if they could just reach the API? The answer was, unfortunately, everything.
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
My ledger has run in production for weeks on a box I set up by hand one evening and never wrote down properly. If it died, I would rebuild it from memory. That is not a plan, so I turned the memory into code.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
There is a moment in every project where you stop admiring what you built and start trying to break it. I hit that moment with go-ledger last week. The tests were green, the invariant held under 10,000 concurrent transactions, the load test sustained 500 requests a second. I felt good.
So I did the thing that ruins a good mood on purpose. I sat down and reviewed my own code as if I were a hostile senior engineer being paid to find the way in, and as if real money, not demo data, flowed through it. Not "do the tests pass." A different question: if someone could reach this API, what could they do?
The answer took the good mood with it. This is a special edition, a bonus post between the numbered weeks, about what that audit found and how I fixed it.
The accounting core was genuinely solid, and I want to say that plainly because it is the part that took months to get right. Money is stored as integer minor units, never floating point, so no rounding ghost ever creeps in. The double-entry invariant (every transaction's postings sum to zero) is enforced in two independent places: the Go domain and a Postgres constraint trigger, so even a direct psql write cannot create money from nothing. Balances are summed from immutable, append-only postings, never a mutable number someone could overwrite. Every write is atomic under SERIALIZABLE. Idempotency keys share the exact transaction that posts the money, so a retry cannot double-post.
None of that was the problem. The problem was everything in front of it.
Here is the single line that summed up the whole risk. Every request in the service acted as one tenant, read from a config value:
deps.Transactions.Post(ctx, deps.DefaultTenant, txn, idem)
There was no authentication. None. Anyone who could send an HTTP request to /v1/transactions could move money between accounts, and anyone who could reach /v1/accounts/{id}/balance could read anyone's balance. The tenant was not derived from who you were, because there was no "who you were." It was a constant.
I had shipped a bank vault with a flawless internal mechanism and left the front door propped open with a rock. And I had done it on purpose, sort of: the comments even said "until an auth layer lands." That is a fine trade-off for a demo. It is a catastrophe the instant the words "real money" enter the sentence.
The audit found five things, ranked the way I would rank them if this were a real pentest report:
# severity finding
------------------------------------------------------------
1 Critical no authentication or authorization at all
2 High idempotency was opt-in, so a retry could double-post
3 High unbounded postings array and no request body limit
4 Medium no rate limiting, incomplete server timeouts
5 Medium audit log was guarded but not tamper-evident
------------------------------------------------------------
I'll walk through the fixes, but two of them are worth slowing down for: the first, because it had a constraint that made it interesting, and the last, because fixing it broke something else.
The obvious fix for "no auth" is "require auth." Add API keys, check them, done. But go-ledger has a public try-it console at go.sohag.pro/console that anyone can use with no login, and a live demo that resets every four hours. If I required authentication everywhere, I would break the one thing that makes the project fun to show people.
The tension is real: authentication has to be mandatory for the thing to be safe, and optional for the demo to work. The trick is to notice those are not actually in conflict. I made authentication uniform (every /v1 request needs a bearer API key, the tenant is derived only from that key, and a key for tenant A physically cannot touch tenant B's data), and then I gave the demo a real key that happens to be public:
request ---> [ auth middleware ] ---> handler
|
| reads "Authorization: Bearer <key>"
| looks up the SHA-256 hash in api_keys
| puts the key's tenant in the request context
v
tenant comes ONLY from the key, never from the request body
The demo key is scoped to the demo tenant, rate-limited harder than a normal key, shipped right there in the console's HTML, and wiped-adjacent (the four-hour reset clears the tenant's data but never the keys, so the demo keeps working). Exposing it is safe on purpose, because it can only ever touch a tenant that gets erased every four hours. One code path, no special cases, and the demo survives.
The keys themselves are stored as SHA-256 hashes, never plaintext. A leaked database dump leaks no usable keys. And because the tenant is derived only from the key, the authorization question ("can this caller touch this account?") collapses into a question the database already answers for free: is the account in the key's tenant? The foreign keys from week three enforce that. I did not have to write a single access-control rule.
The audit log already rejected updates and deletes with a database trigger, so it was append-only. But append-only is not the same as tamper-evident. A privileged database user could still, in principle, rewrite a row's contents in place if they could get past the trigger. I wanted the log to be able to prove it had not been altered.
The standard tool for that is a hash chain, the same idea underneath a blockchain minus all the noise. Each audit row stores a hash of its own contents plus the hash of the previous row. Change any row and its hash no longer matches; change it and try to fix the hash, and now the next row's stored "previous hash" is wrong. The tampering cannot hide.
row 1 row 2 row 3
+----------+ +----------+ +----------+
| data | | data | | data |
| prev: 00 | | prev: h1 | <-- | prev: h2 |
| hash: h1 | --> | hash: h2 | | hash: h3 |
+----------+ +----------+ +----------+
tamper with row 2's data -> recomputed hash != h2 -> chain breaks,
and row 3's "prev: h2" no longer points at anything valid.
I built it per tenant (so the four-hour demo wipe restarts one tenant's chain from scratch without touching anyone else's), added a GET /v1/audit/verify endpoint that walks the chain and reports the first break, and wrote tests that raw-edit a row and confirm the verifier catches it. Building it surfaced two genuinely nasty determinism bugs, the kind a hash chain is uniquely allergic to: Postgres stores timestamps at microsecond precision, so a nanosecond timestamp hashed on the way in did not match what came back out, and the jsonb type silently reformats your JSON, so the bytes you hashed were not the bytes you stored. Both would have made the chain report "tampered" on data nobody touched. Fixed by truncating the timestamp and storing raw json.
And then the load test went red.
Here is the thing about a hash chain. To append row N, you must first read row N minus one, because you need its hash. That read-then-write, done inside a SERIALIZABLE transaction, means two transactions posting to the same tenant at the same time both read the same "latest" row and both try to append. Postgres notices the conflict and aborts one of them. Under 500 requests a second to a single tenant, that is not an occasional retry. It is a stampede. The retry budget exhausted and the ledger started returning 503s.
I had traded a security win for a throughput regression, and the load test caught it honestly. The audit chain, by its nature, serializes writes within a tenant. The question was how to make that serialization cheap instead of catastrophic.
My first instinct was a database advisory lock. It made things worse, and understanding why is the interesting part. Under SERIALIZABLE, a transaction's view of the database is frozen at its first query. If a second transaction takes a lock and waits, its frozen view was already taken before the first one committed, so when it finally runs it still reads a stale chain tail and still aborts. The lock delayed the collision without preventing it.
The fix that worked moves the serialization out of the database entirely and into the application. Since the service runs as a single instance, I put a tiny in-memory lock per tenant, held only while a post runs:
posts to tenant A: A1 --wait--> A2 --wait--> A3 (one at a time)
posts to tenant B: B1 --wait--> B2 (its own queue)
A and B run fully in parallel. Only same-tenant posts queue,
and they queue on a Go mutex, not on a held database connection.
The distinction that matters: transactions waiting for their turn block on an in-process lock, not on a checked-out database connection, so a busy tenant can never starve everyone else of connections. Different tenants run in parallel. Same-tenant posts serialize, which is exactly what a per-tenant hash chain requires anyway. And the SERIALIZABLE retry loop stays underneath as the correctness backstop, in case this ever runs as more than one instance.
Then I made the load test honest about all this by spreading its 500 requests a second across eight tenants, which is how a real ledger's traffic actually looks. The result: 38,417 requests, zero failures, 500 requests a second sustained at a 99th-percentile latency under 5 milliseconds, and not a single audit-chain conflict. The security feature and the throughput target both hold, at the same time, with everything switched on.
The middle three are less dramatic but no less necessary. Idempotency went from optional to mandatory: a POST /v1/transactions with no Idempotency-Key header is now a 400, because "retry safely" should be the default on an endpoint that moves money, not an opt-in a client can forget. The postings array got a maximum and the request body got a size cap, so one request can no longer become a multi-million-row transaction that eats the server's memory. And every API key now gets its own rate limit, with the demo key throttled tighter than the rest.
The gap between "my tests pass" and "this is safe for real money" is not a gap in the hard, interesting core. I had spent months making the accounting bulletproof. The gap was in the boring perimeter: the lock on the door, the cap on the input, the limit on the rate. The unglamorous stuff that does not show up in a demo and absolutely shows up in an incident.
If you are building something that moves money, or moves anything that matters, do the hostile read of your own code. Assume the attacker can reach every endpoint, because they can. Ask what they could do, not what your tests check. The list you get back is uncomfortable, and it is the most useful list you will write all quarter.
Every fix here is recorded in ADR-012, and the whole ledger, front door now firmly locked, is at github.com/sohag-pro/go-ledger.
The numbered series continues next week with Docker and real infrastructure. This one was a detour worth taking.