# Idempotency Keys and the Immutable Audit Log: Two Patterns That Make Fintech APIs Boring (In a Good Way) You are paying for something on your phone, you tap Pay, and the spinner just sits there. No confirmation, no error, nothing. Did it go through? Your thumb hovers over the button. Tap it again and maybe you pay twice. Do nothing and maybe you did not pay at all. We have all been in that half second of doubt, and the reason most of us tap again without really panicking is that somewhere a backend engineer already solved this for us. The second tap does not charge you twice. That property has a name, idempotency, and this week I built it into go-ledger, along with its close cousin: an audit log that can record what happened and never let anyone quietly rewrite it. This is post 6 of 12. Last week the ledger got a REST API and went live at `go.sohag.pro`. This week is about making that API safe to retry and easy to trust. ## The retry problem is not rare, it is the default Here is the uncomfortable truth about networks: a request that times out did not necessarily fail. Your `POST /v1/transactions` left the phone, reached the server, posted the transaction, and then the response got lost on the way back. From the client's point of view that looks identical to a request that never arrived. So the client retries, because retrying is the correct thing to do when you are not sure. And now the server sees the same payment twice. Without protection, that is two transactions, two movements of money, one angry customer. The fix is an idempotency key: the client generates a unique string once per intended action and sends it on every retry of that action. ```plaintext POST /v1/transactions Idempotency-Key: 7f3a9c2e-pay-dinner-share { "currency": "BDT", "postings": [ ... -600 ... +600 ... ] } ``` The contract is simple to state and surprisingly interesting to build: * First time the server sees this key: do the work, remember the result. * Same key again, same request: do not do the work again, return the original result. * Same key, but a different request body: refuse, because the client is confused and reusing a key for two different things is a bug worth shouting about (HTTP 409). ## The naive version has a hole in it The obvious implementation is: look up the key in a table, if it is there return the saved response, if not then post the transaction and save the response. Read, then decide, then write. That works fine until two copies of the same request arrive at nearly the same instant, which is exactly what a double tap or an aggressive retry produces. Both requests look up the key. Neither finds it, because neither has finished yet. Both decide "new key, go ahead." Both post the transaction. You are back to charging twice, except now it only happens under load, which is the worst kind of bug because your tests never see it. The check and the write have to be one atomic step, and the database is the only thing in this picture that can make that guarantee. So instead of checking first, I let the database's uniqueness constraint be the referee. The idempotency key row goes into a table whose primary key is the tenant plus the key. Inserting it inside the same database transaction that posts the ledger transaction means the whole thing commits together or not at all: ![](https://cdn.hashnode.com/uploads/covers/62d6bdce3060d03288d9e0ed/b435f358-9f13-4d12-b13c-bcefd57ee8c9.png align="center") ```plaintext 100 identical requests, same Idempotency-Key | +--------+--------+--------+ ... +--------+--------+ | | | | | v v v v v [ each opens a SERIALIZABLE transaction and tries: ] insert transaction + postings insert idempotency_key (tenant, key) <-- unique, this is the referee insert audit_log row commit | the FIRST to commit the key wins ---+---> commits everything | the other 99 hit a duplicate key ---+---> whole transaction rolls back, nothing they wrote survives | v losers look up the winner's row and REPLAY its response | v result: 1 transaction, 1 audit row, 99 replays ``` The key insight is that the losing requests do not just get rejected. When a request loses the race, its entire transaction rolls back, so the transaction and postings it optimistically wrote vanish too. Then it reads back the row the winner committed, loads that original transaction, and returns it as if it had done the work. From the client's side, all 100 retries got a clean, identical answer. Under the hood, money moved exactly once. To make the "same key, different body" case work, the stored row also holds a fingerprint of the original request: a hash of the currency and every posting. On a repeat, if the incoming request hashes to the same fingerprint it is a genuine retry and gets the replay. If it hashes differently, someone reused a key for a different payment, and the ledger returns 409 rather than silently doing the wrong thing. I spent a little care making that fingerprint framing collision proof, so that two different transactions can never accidentally hash the same, but that is a detail for the repo. ## The part I got wrong on the first pass My first version of the fingerprint glued the fields together with a separator byte, which is the kind of thing that looks fine and passes every test you think to write. Then I pictured a hostile input: what if a field itself contains that separator byte? You could, in principle, craft two different transactions that produce the same glued-together string, collide their fingerprints, and slip a different payment through on a reused key. The fix was to length-prefix every field so the boundaries can never be faked. Nobody would hit this by accident, but "nobody would hit this by accident" is not a sentence you want anywhere near the code that decides whether a payment is a duplicate. I only caught it because I went looking for it, which is the whole argument for building these things yourself. ## The audit log: write it once, never touch it again The second pattern this week is the audit log. A ledger already keeps immutable postings, so why a separate log? Because the postings tell you *what the balances are*, and the audit log tells you *what happened and who did it*. When an auditor or an incident review asks "show me every action that touched this account, in order, with the full before and after," you want a single, honest, tamper-evident answer. Two design decisions make it trustworthy. First, the audit row is written inside the same database transaction as the posting it describes. There is no separate "log this later" step that could fail on its own and leave you with a transaction nobody recorded. If the posting commits, its audit row committed with it. If anything fails, both roll back. They are one fact. ```plaintext one database transaction (all of it, or none of it) +----------------------------------------------------+ | insert transaction + postings | | insert idempotency key (if one was supplied) | | insert audit_log row: action, actor, snapshot | +----------------------------------------------------+ | commit succeeds | posting and its audit record are now the same indivisible piece of history ``` Second, "append-only" is not just a promise in the application code, it is enforced by the database. A Postgres trigger rejects any attempt to UPDATE or DELETE a row in the audit log. Even a direct `DELETE FROM audit_log` from a psql prompt bounces off it. The only sanctioned exception is the demo seeder that resets `go.sohag.pro` every few hours, and it has to explicitly flip a per-transaction flag to be allowed through. The application's normal path can never set that flag, so in production the log is genuinely immutable, not immutable-until-someone-writes-the-wrong-migration. You can read the log back by transaction or by account: ```plaintext GET /v1/transactions/{id}/audit what happened to this transaction GET /v1/accounts/{id}/audit every audited action touching this account ``` The by-account view is paginated with the same keyset cursor as last week's statement endpoint, because "list everything that ever happened to a busy account" is exactly the kind of query that needs a ceiling before it faces a real production account. ## Proving it, not hoping it The definition of done for this week was a specific, slightly mean test: fire 100 requests with the same idempotency key at the same time, then check the database. Not "it returned 200 a hundred times," but the thing that actually matters: exactly one transaction row exists, exactly one audit row exists, and 99 of the responses were replays of the first. That test runs against a real Postgres in a container, with the race actually happening, not mocked away. Watching it come back green, with the counts landing on exactly one, is a different feeling than reading that unique constraints work. That gap, between knowing a thing and having watched it hold under fire, is the entire reason I am building this in the open. ## My AI companion this week Same deal as always: I own the decisions, Claude does a lot of the typing. This week the collaboration was interesting because the work was mostly judgment, not code volume. I decided the key had to be inserted inside the posting transaction rather than checked beforehand, and Claude implemented that cleanly and wrote the 100-goroutine hammer test. But the fingerprint collision I described above surfaced during review: a reviewer pass (also Claude, wearing a different hat) flagged the separator-byte weakness, I agreed it was real, and we hardened it before it ever shipped. That is roughly the rhythm I have settled into. I set the invariants and the definition of done, the machine moves fast inside those rails, and the interesting moments are the disagreements and the "wait, what about this input" catches. A payment duplicate check is precisely the code you do not hand over without understanding every line, and I do. ## Where this leaves us go-ledger now survives retries the way a real payment API has to, and keeps a record of itself that nobody can quietly edit. The API got no bigger in surface area, just harder to misuse, which is the good kind of boring. Next week the ledger grows a second front door: a gRPC layer sitting on the exact same domain services, so REST and gRPC are two protocols over one source of truth rather than two implementations drifting apart. The repo is open source at [github.com/sohag-pro/go-ledger](https://github.com/sohag-pro/go-ledger), and the live API is at `go.sohag.pro`. Go tap Pay twice on something and trust the ledger to have your back.