Search for a command to run...
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Last week I sent a little money across a border. The app showed me a rate, I tapped confirm, and the euros landed a moment later. Out of habit I checked the rate against a search engine, and the app's number was a hair worse than the "real" one. Not a lot. Enough to notice. That small gap between the true mid-market rate and the rate you actually get is where the business lives. It has a name: the spread, or the markup. It is how the company on the other side of the screen earns its cut for moving your money between two currencies.
go-ledger has been able to do currency conversion for a while (that was the multi-currency work back in Week 11). It stores a mid rate for a pair, applies a spread against the customer, and posts the four legs of the conversion atomically. What it could not do until this week was let the operator running the ledger actually change any of that without SSH-ing into the box.
Here is the embarrassing part. Every FX rate in the system came from a single environment variable read once at startup:
FX_RATES="USD:EUR=0.9200/25,USD:BDT=110.50/50"
Each entry is a pair, a mid rate, and a spread in basis points (25 bps is a quarter of one percent). At boot, the server parsed that string and wrote one row per entry into an append-only fx_rates table. Simple, and fine for a demo. But think about what changing a rate actually took: edit /etc/go-ledger/env on the server, restart the service, and hope you got the syntax right. An operator with no shell access could not do it at all. Anyone using the API or the console had no idea the setting existed.
There was a second, quieter problem. The spread was welded to every rate row. If you wanted "half a percent on everything," you had to repeat the same number on every single pair, and to change it you had to re-enter every pair. There was no notion of a default markup that just applied across the board. That is a strange way to run a pricing business.
So this week's job: move rates and markup out of the environment and into a live admin API, add a real default markup, and put both in the operator console. Then, because this is money, audit the result until it stops lying to me.
Say a customer converts 100.00 USD into euros. The operator has set the USD to EUR mid rate to 0.9200 and a markup of 50 basis points (0.50%). The ledger never gives the customer the mid rate; it always widens it against them by the spread:
converted = 100.00 * 0.9200 * (1 - 0.0050)
= 100.00 * 0.9200 * 0.9950
= 91.54 EUR
At the pure mid rate the customer would have received 92.00 EUR. They got 91.54. That 0.46 EUR difference is the operator's markup, and the whole point of this feature is letting the operator set that 50 without redeploying anything, for one currency pair or for every pair at once, for one tenant or for the whole platform.
One detail that matters for a ledger: none of that math touches a floating-point number. Amounts are integer minor units, and the rate is an integer scaled by a hundred million. The console lets you type 0.9200, but it converts that to the integer 92000000 using string math before it ever hits the API, so the money path stays exact from the browser to the database.
The neat trick that made a default markup possible was to let a rate row carry no spread at all. The spread_bps column on a rate became nullable. A concrete value is a per-pair override. NULL means "use whatever the default markup is."
Then a new append-only table, fx_markup_defaults, holds the default itself: one value for the whole platform (the global default), and optionally one per tenant that overrides the global for that tenant. Both are append-only, exactly like the rates: you never update a row, you write a new one, so the full history of what the markup ever was stays on record.
When a conversion runs, the ledger resolves the actual spread to apply by walking a short chain of preferences:
A conversion needs a spread for tenant T, pair USD/EUR
|
v
Does the USD/EUR rate row carry its own spread?
| |
yes | | no (it is NULL)
v v
use that spread Does tenant T have a markup default?
(per-pair override) | |
yes | | no / cleared
v v
use the tenant's Is there a global
markup default markup default?
| |
yes | | no
v v
use the global use 0
markup default (no markup)
The important word is "resolved." The chain is walked at the moment a conversion happens, not when the rate is saved. So if you set a new global default of 50 bps, every pair that does not override it immediately starts charging 50, without you touching a single rate row. That is what a default is supposed to do.
And because a recorded conversion snapshots the exact mid and spread it used into an immutable detail row, changing a default tomorrow can never rewrite what a conversion did yesterday. The history stays reproducible even though the config is now live and mutable.
The API is four endpoints under /v1/admin/fx: set a rate, list current rates, set a markup default, read the markup defaults. All behind the same admin scope as the rest of the operator API. The console grew two cards in Settings, one for exchange rates and one for the markup fee, each with a toggle for "global default" versus "this tenant." A rate change is now a form and a button.
A feature that moves money is not done when the tests pass. It is done when someone who is paid to be suspicious cannot find a way to make it lie. So I ran the whole change past an audit with a simple brief: you are a fintech engineer doing due diligence, assume real money flows through this, find the ways it loses or mis-prices money. It found six things. Two of them mattered a lot.
The seed silently undid every live change. This is the one that would have quietly ruined the entire feature. Remember that the environment variable still seeds rates at boot, as a bootstrap. The seeder's logic was: for each entry in FX_RATES, if the current rate for that pair differs from the environment value, insert the environment value. That was correct back when the seeder was the only thing that ever wrote a rate. But now the admin API is a second writer. Picture the sequence:
Monday: FX_RATES says USD:EUR = 0.9200
Tuesday: operator uses the console, sets USD:EUR = 0.9500 (live)
Tuesday night: routine deploy, server restarts
Boot: seeder sees current rate (0.9500) differs from env (0.9200)
-> it "helpfully" re-inserts 0.9200 with a fresh timestamp
Wednesday: every conversion prices at 0.9200 again. No log. No error.
The operator's change survived exactly until the next deploy, then vanished with no trace. The feature's whole promise, live rate edits, was being undone by the deploy pipeline. The fix was to make the seeder compare against the last rate the seeder itself wrote (the last row whose source was env), not against whatever the current winning rate is. Now a change to FX_RATES still lands, but an operator's live edit is never clobbered.
The public demo let anyone set a 99.99% markup. The live demo at go.sohag.pro resets every four hours and exposes the admin panels publicly, because there is nothing durable to protect between resets. Except now there was. A global markup is not scoped to the demo tenant; it is platform-wide. The demo reset only ever deleted the demo tenant's own rows, so an anonymous visitor could set a global markup of 9,999 basis points (99.99%, perfectly legal to the validation), and it would survive every reset and mis-price every other visitor's conversions until the process restarted, forever for markup since the boot seeder never touched it. The fix was to have the demo reset also clear the API-set global FX config each cycle, back to a clean baseline, while leaving the environment-seeded rates alone.
The other four were smaller: a console number field that accepted 1e3 and quietly saved 1, a rate input that could lose precision above a couple hundred million, a down-migration that flattened default-following rates to zero markup on a rollback, and a way for a tenant markup override to get stuck because you could never clear it back to following the global. All real, all fixed, and then I audited the fixes, which found two more residual issues the first round of fixes had introduced (the demo cleanup missed a tenant-scoped variant of the same tampering, and the rollback logic resurrected a markup you had explicitly cleared). Fixed those, audited a third time, clean.
That loop is the actual lesson. The first audit found the design bug. The second audit found the bugs in the fix. Money-adjacent code earns that paranoia. A CRUD app with this class of bug shows a stale number on a page. A ledger with this class of bug charges the wrong people the wrong amount and does it silently, which is the worst way for a payment system to be wrong.
An operator opens the console, goes to Settings, and sets USD to EUR to 0.9500 with no override spread. They set a global markup of 50 bps. A conversion for any tenant on that pair now applies 0.95 and 50 bps, resolved at conversion time. They give one big customer a break: a tenant markup default of 20 bps, and that tenant alone now gets the cheaper rate, everyone else still 50. Later they clear that tenant override and it falls back to the global 50 again. No file edits, no restart, and every one of those changes is a new row in an append-only history you can read back.
The rate stopped hiding in an environment variable. And the thing that quietly reset it every deploy is gone.
The repo is open source at github.com/sohag-pro/go-ledger, and the design is written up in ADR-020.