Search for a command to run...
My ledger has run in production for weeks on a box I set up by hand one evening and never wrote down properly. If it died, I would rebuild it from memory. That is not a plan, so I turned the memory into code.
WhoAmI => notes.sohag.pro/author
No comments yet. Be the first to comment.
Kicking off a 12-week build of an open-source, production-grade payment ledger in Go: double-entry accounting, Postgres, gRPC, OpenTelemetry, and a real deployment.
No SSH, no checklist, no remembering the one command I always forget. Just git push, and production updates itself. This is exactly how that pipeline is wired, and how you can build the same one for a plain server.
I had a solid list of reasons my life wasn't moving faster, until a grainy old lecture pointed out the one name missing from it
The finale isn't a victory lap. It's the story of the control I shipped that did nothing, the footgun still sitting in my demo, and the handful of things I'd keep exactly as they are.
How do you hold a large payment for a second pair of eyes without ever letting the unapproved money touch a balance, and how do you stream that decision to the outside world without standing up a broker?
How do you show the total under a parent account when the whole system refuses to store a balance? A recursive query, a trigger that refuses to draw a circle, and a rule about what actually has to sum to zero.
Every time I wanted to change an FX rate I had to edit a file on the server and restart the app. So I moved rates and markup into a live admin API, and then audited it hard enough to find the bug that quietly undid the whole thing.
My ledger has been live at go.sohag.pro for a few weeks now. Real HTTPS, real Postgres, real deploys on every push to main. And there is a thing I have been not thinking about, the way you do not think about a smoke alarm with a dead battery.
The server was set up by hand. One evening, over SSH, as root, I created the users, wrote the systemd unit, configured nginx, installed Postgres, ran certbot, and tuned the memory so the 1 GB box would not fall over. Then I wrote most of it down in a text file so I would not forget. Most of it.
Here is the uncomfortable question I finally asked myself: if that box died tonight, how would I rebuild it? And the honest answer was, from that text file and from memory, at whatever hour it happened, hoping I did not miss the one chmod that matters. That is not a disaster recovery plan. That is a disaster with a paragraph of notes.
Week 10 was supposed to be "Docker and Terraform for AWS." I did the Docker part, dropped the Terraform part, and spent the real effort turning the server that already exists into code that can rebuild it. This is post 10 of 12.
The Dockerfile had been a skeleton since week one, with a comment that literally said "fleshed out in Week 10." So I fleshed it out. Multi-stage build, distroless base, non-root, build cache mounts, and the binary stripped down. The point of a multi-stage build is that the heavy toolchain never ships. You compile in a fat image and copy only the finished binary into a tiny one.
The final image is about 10 MB, and there is a make image-size target that fails the build if it ever creeps past 20 MB, so the budget is enforced, not aspirational. There is also a docker compose up that brings up the whole local world at once: the app, Postgres, Jaeger for traces, and Prometheus scraping the metrics. Clone, one command, and you have the full stack running locally.
But here is the thing I want to be honest about: none of that Docker work touches production. My server does not run Docker. It runs the bare binary.
This surprises people, so let me defend it. The box has 1 GB of RAM, and it is shared: it also serves my portfolio site. On a machine that small, a container runtime is pure overhead for no benefit I would actually use. The app is already a single static Go binary. It already runs under systemd with a genuinely strict sandbox: no new privileges, a read-only filesystem, no access to devices, memory that cannot be both writable and executable. That is strong isolation at zero memory cost.
Docker earns its place for local development, where "works on my machine" is a real problem worth solving, and for the load-test stack in CI. It does not earn a daemon on my production box. So the image is a dev and CI artifact, and I wrote that decision down in an ADR instead of letting the plan file quietly imply otherwise. Which brings me to the AWS I did not build.
The original plan said Terraform modules for AWS: a VPC, ECS Fargate, RDS, a load balancer, the works. I had already decided weeks ago to deploy to a plain VPS instead, and it has worked great. So writing Terraform now would mean building infrastructure code for a cloud I will never provision, which then slowly drifts away from the real box because nobody runs it. That is worse than no code. It is code that lies.
Terraform is for creating cloud resources. I do not have cloud resources to create. I have one server that already exists and needs to be configured correctly and reproducibly. That is a different job, and it has a different tool.
The tool for "configure an existing machine, the same way, every time" is Ansible. So I took my prose runbook, the one with the gaps, and rewrote it as an Ansible playbook: a set of roles that each describe one slice of the server as a desired state.
base: the firewall (only SSH, HTTP, HTTPS open), fail2ban, automatic security updates.
postgres: Postgres 16, a dedicated database and role, the memory tuning for a 1 GB box, and a daily backup.
app: the two users, the app directory, the root-owned environment file, and a sudoers rule that lets the deploy user restart exactly one service and nothing else.
systemd: the hardened service unit.
nginx: the site config that proxies to the app.
tls: certbot for the certificate.
The difference between the text file and the playbook is not just format. A runbook is a list of commands I ran once. A playbook is a description of how the box should be, that I can run again and again, and each run only changes what has drifted. Run it on a fresh machine and it builds the whole thing. Run it on the live box and it reports "nothing to change," which is its own kind of proof that the code and reality agree.
There is one detail I am a little proud of, because it is the kind of thing a runbook cannot capture but code can enforce.
Remember that the box is shared. My ledger and my portfolio live on the same nginx. And the portfolio uses HSTS preload, which is a browser rule that says, roughly: every subdomain here must always serve valid HTTPS, forever, no exceptions, no "click to proceed anyway." Which means if I ever push a broken nginx config while fiddling with the ledger's site, I do not just break the ledger. I can take down the portfolio too, with no override button for visitors.
A prose runbook can say "be careful with nginx." Code can actually be careful. So the nginx role does this:
write new vhost to sites-available (never overwrite the cert block)
|
v
link it into sites-enabled
|
v
run `nginx -t` (validate the WHOLE config)
|
+----+-----------------------------+
| passes | fails
v v
reload nginx (graceful, not remove the link again,
a restart, so live connections abort with a clear error,
are never dropped) leaving the working config live
|
v
assert the portfolio still returns 200
If the new config does not validate, the playbook tears its own change back out and stops, so the box is never left holding a broken config that some later reload would activate. And after every change, it makes an HTTP request to the portfolio and refuses to call itself successful unless that other, unrelated site is still answering 200. The reload is graceful, never a restart, because a restart drops in-flight connections. These are the safeguards I would tell a careful human to remember. Now they are not something to remember. They just happen.
One catch: the repo is public, and a playbook that configures my server naturally wants to know my server's address, my database password, and my deploy key. None of that can be committed.
So the split is simple. The roles and the playbook are generic and tracked: they describe the shape of the box with no secrets in them. The actual inventory (the server's address) and the vault (the password, the key) are gitignored, exactly like the runbook always was. What gets committed are example files that show the shape without the contents. Anyone can read exactly how my box is built. Nobody can read where it is or how to log in.
The Docker image is nice. The one-command local stack is genuinely useful. But the thing that changed how I feel about this project is smaller and quieter: the server is no longer a thing that only exists as a running process and a half-remembered evening. It is described, in code, in the repo, in a form I could hand to a stranger, or to myself at 2am after the box died.
That is the actual product of infrastructure-as-code. Not automation for its own sake. It is that the knowledge stops living in one person's head, where it decays, and starts living in a file, where it does not.
The repo, playbook and all, is at github.com/sohag-pro/go-ledger, and the reasoning is in ADR-013. Next week is the last real build week: tightening the deploy pipeline and the observability around it, before the launch-and-retrospective finale.