I Built a Production Go Payment Ledger in 14 Weeks. Here's What I'd Do Differently.

Near the end of this build I did something I'd recommend to anyone who has just finished a big project and feels good about it: I handed the whole thing to a reviewer who had no stake in my feelings and asked them to be brutal. Not "does this look nice." More like "pretend this is moving real money and try to lose some."

The review came back with three critical findings. One of them was about a feature I had shipped weeks earlier, written a whole blog post about, and quietly been proud of. It turned out the feature did not do the thing it claimed to do. At all.

This is the last post in the series, so instead of a highlight reel, here is the honest version: the mistakes I'd fix if I started over, and the few decisions I'd make again without blinking.

The control that did nothing

Week 13 added approval workflows. The idea is simple and every payments shop has some version of it: a transaction over a certain size does not just post because one person asked. It gets held, and a second person has to approve it. Two sets of eyes. In the industry it's called four-eyes, or maker-checker: the person who makes the request cannot be the person who checks it.

I built the gate, held the over-threshold transaction as pending intent, and added a flag, APPROVAL_REQUIRE_DIFFERENT_ACTOR, that was supposed to enforce exactly that: the approver must be a different actor than the creator. I tested it. It passed. I wrote it up as a real control.

Here's the concrete scenario. Alice, using her own API key, submits a 500,000-unit transfer. It's over the threshold, so it's held. Bob, using his own key, should be able to approve it, and Alice should not. That's the whole point.

The problem was in one word: actor. Everywhere in the codebase, the "actor" recorded on an event was the tenant id, not the individual API key that made the request. So when the approval code checked "is the approver a different actor than the creator," it was comparing the tenant to the tenant. Same tenant on both sides. Always equal.

  WHAT I SHIPPED (actor = tenant id)

    Alice's key  ┐                       Bob's key   ┐
    creates ─────┼─▶ actor = tenant-42   approves ───┼─▶ actor = tenant-42
                 ┘                                    ┘
                        creator == approver ?  tenant-42 == tenant-42  ->  ALWAYS TRUE

    flag OFF: no check at all, one key can create AND approve
    flag ON : every approval is "your own", so NOTHING can ever be approved


  WHAT IT SHOULD HAVE BEEN (actor = API-key id)

    Alice's key  ┐                       Bob's key   ┐
    creates ─────┼─▶ actor = key-alice   approves ───┼─▶ actor = key-bob
                 ┘                                    ┘
                        creator == approver ?  key-alice == key-bob  ->  FALSE, Bob may approve
                                               key-alice == key-alice ->  TRUE, Alice may not

With the flag off, there was no segregation of duties at all: a single admin key could create the transaction and approve it. With the flag on, it was worse in a funny way: every approval looked like self-approval, so no over-threshold transaction could ever be approved by anyone. The feature was either a no-op or a deadlock, and I had confidently documented it as neither.

The fix was not hard once I saw it: carry the individual API-key id as the acting principal through the whole path, instead of collapsing everyone in a tenant into one identity. Now maker-checker compares two actual keys. But I want to be clear about why I missed it: my tests used the tenant as the actor too, so they agreed with the bug. The test and the code shared the same wrong assumption, so of course they got along.

The lesson I keep relearning: a green test proves the code matches the test, not that either matches reality. The only thing that caught this was a fresh pair of eyes that did not already believe the feature worked.

The footgun I'm leaving in on purpose (and why that's a decision, not an accident)

The live demo resets every few hours. To do that, a background job deletes the demo tenant's data and repopulates it. Fine. But the way I wrote the reset, it deletes every tenant that isn't the demo tenant, across a dozen tables, including the encryption keys and the audit anchors, and it deliberately switches off the append-only guard to do it.

The only thing standing between that job and a real database is two environment variables. Point it at the wrong DATABASE_URL once and there is no undo.

For a demo box that holds nothing but fake, regenerated data, I decided that's acceptable, and I'm leaving it. But I want to name it honestly, because "it's fine for the demo" is a judgment I made, not a property of the code. In anything holding real money I would never let a destructive, cross-tenant, guard-bypassing delete be gated by two env vars and good intentions. It would need an allowlist, a row-count ceiling, and a refusal to run at all if it saw a tenant it didn't recognize. The difference between a demo tool and a production tool is often exactly this: not what it does, but how hard it is to make it do the wrong thing.

Breaking my own rule about ADRs

Early on I made a rule for myself: every significant decision gets an architecture decision record written before the code. The reasoning is that the "why" is freshest during the design, and a decision you have to write down is a decision you actually make instead of drift into.

I kept that rule for thirteen weeks. Then I built webhooks and just... didn't. The code shipped, it worked, it had signing and retries and a delivery log, and there was no ADR explaining any of the choices. I only noticed the gap in this final week, when I went to link the webhooks ADR from the README and there wasn't one.

I wrote it retroactively (it's ADR-027, and it says right at the top that it was written after the fact). Writing it late is better than not writing it, but it's not the same. A retroactive ADR documents what you did; a real one shapes what you're about to do. By the time I wrote this one, all the interesting alternatives were already closed. The value of the rule was in the timing, and I gave that up for one week because the feature felt "obvious." Nothing in a payment system is obvious enough to skip the paper trail.

The cloud I planned and never used

My original plan, written in Week 0, said the deployment would be AWS ECS Fargate, RDS, and Terraform. It sounded appropriately serious.

When I actually got to the deployment week, I looked hard at that and changed my mind. For a portfolio-scale service that resets every few hours and serves a demo, a single well-run VPS with Postgres on the same box and a GitHub Actions pipeline that ships on every push is simpler, cheaper, and teaches the same operational discipline: migrate before you swap, health-check, roll back on failure. The cloud-console tax was not buying me anything at this size.

I don't regret the switch. I regret how confidently the original plan asserted the opposite. "AWS ECS/Terraform" was in the plan because it sounded like what a real system uses, not because I had weighed it against the actual requirements. A plan should bound your choices, but it shouldn't pretend to have made decisions you haven't earned yet. The honest version of that plan line would have been "deploy target: TBD, decide at Week 10 against real constraints."

The week that actually hurt

If you're going to build a ledger, Week 4 is where you find out whether you're serious. That's the week two requests try to post to the same account at the same time.

Getting the balance invariant to hold under concurrency meant SERIALIZABLE isolation, a retry loop for the serialization failures Postgres throws when it detects a conflict, and a genuinely nasty debugging session where the thing that was serializing my transactions was not the row I expected but a predicate lock on a range of rows I was reading to compute a balance. The fix was real work, and the stress test that proves it (thousands of goroutines racing on a handful of accounts, and the ledger never once going wrong) is the test I'm proudest of.

I wouldn't do that week differently. I'm mentioning it because everything else in the build was tractable, and this was the one place where the problem was as hard as the domain promised it would be. If you're planning your own version of this project, budget double for the concurrency week. It's the one that earns the "production" adjective.

The alerts nobody is watching

The code emits a lovely set of signals. There's a Prometheus gauge that goes non-zero the instant the balance invariant is ever violated in production, and an alert pack with a severity-one page wired to it.

Nothing scrapes it. There is no Prometheus running in front of the live service, and the offsite backups are configured but not yet provisioned, so the durability story today is a same-disk dump. I built the instrument panel and never plugged in the gauges.

This is the gap between "the code is production-grade" and "the deployment is production-grade," and they are not the same claim. For a demo, I've accepted it. But it's the item that would keep me up at night if this held real money, precisely because it's invisible: a decorative alert doesn't warn you that it's decorative. It just quietly never fires.

What I'd keep, exactly as it is

For all of that, there's a shorter list of things I'd do again without changing a line of the approach.

• Enforce the invariant in the database, not just the code. The zero-sum rule lives in the domain types and in a Postgres constraint trigger on the postings table. Every write path, including a raw psql insert or a buggy seeder, has to go through it. The reviewer tried to find a way to create or destroy money and could not, and that's because the guarantee doesn't depend on any code path remembering to check.

• Property tests and chaos tests, not just examples. The invariant gets tested by throwing hundreds of randomized transactions at it, including the mean case where one currency goes up and another goes down so a naive global sum still looks balanced. And a chaos test kills the database connection at the exact moment of commit to prove rollback and idempotent retry actually hold. Those two caught more real bugs than any hand-written example.

• An ADR per decision (the twelve weeks I actually did it). Reading them back, they're the most useful artifact in the repo. Not the code, the reasoning.

• Building in public. Every week's post forced me to explain the week's work to a stranger, and explaining a design is the fastest way to find the part you don't actually understand. More than once the blog post is where I caught the bug, not the code.

The honest close

Fourteen weeks ago I paid a friend back for dinner and got curious about the system that recorded it. What I have now is a real, deployed, open-source payment ledger that I can explain every line of, including the lines that were wrong.

The version of this retrospective that only listed wins would have been easier to write and worth less. The self-audit chapters, the ones where I document a failure with numbers, are the parts of this series I'd point a hiring manager at first, because anyone can show you working code. What's harder, and rarer, is showing you the bug you shipped, how you found it, and exactly what you changed. That's the whole project, really: not that the ledger is perfect, but that its history is honest, and everything else is a cache.

Thanks for reading along. The code is open, the book is free, and the demo is live (at least until the next reset wipes it).