How it's built

The problem

A simulator is a room full of computers that has to be working when someone books it. Before the portal there was no way to answer "is it up?" other than asking a person who was standing next to it — which meant every answer was hours stale, and the first sign of trouble was usually somebody's phone call.

What was needed wasn't a dashboard for its own sake. It was one picture of the whole fleet, and the difference between finding out in minutes and finding out tomorrow.

The constraints

Every decision below follows from four things that were true before a line was written, and none of which I could change:

• The machines get switched off. They are powered up for the working day and shut down at the end of it. "Offline" is the normal state, not an incident.
• The link is not reliable. A monitoring system that quietly loses an hour of history whenever a network hiccups is worse than none, because you stop believing the parts that are there.
• Nothing new to install. A control panel already ran on each machine. Anything I added had to ship inside it, or it would never be kept up to date.
• Closed service. No public access, and no third-party requests from the browser — the page talks to us or to nobody.

Five decisions

1. The link drops

   The agent keeps its own queue

   Rather than send and hope, the agent writes every reading to a small embedded database first and treats the network as a detail. When the link comes back it back-fills everything it collected while nobody was listening.

   Cost: the server now receives the same record more than once as a matter of course, so ingest had to become idempotent — each item carries an id and a repeat is a no-op rather than a duplicate row.

2. A machine that dies says nothing

   Sessions, closed by a sweeper

   A clean shutdown sends a message. A power cut sends silence, and silence looks exactly like a healthy machine that hasn't spoken recently. So uptime is modelled as sessions: a boot opens one, a shutdown closes it, and a background sweep closes anything that has stopped reporting for a few minutes.

   Cost: that sweep is a guess, and sometimes it is wrong — a blip closes a session on a machine that never went anywhere. So recovery had to be quiet and self-correcting: the record is repaired without ceremony. A monitor that flaps between states teaches people to stop reading it.

3. The clocks lie

   Everything in UTC, normalised at the edge

   A simulator's clock is not a clock — it gets moved deliberately as part of running a session, so machine-local time is not something you can build an axis on. Every timestamp is stored, sliced and displayed in UTC, and the label says so.

   Cost: the database driver hands timestamps back in the server's local zone, which is the kind of thing that reads correctly for months and then shifts an entire day's chart by an hour. Every value is normalised where it is read, and the connection pins its own timezone rather than trusting the default.

4. A day showed more hours than it had

   Merge the intervals, don't add them up

   Daily uptime started life as the sum of session lengths. That is right until two sessions overlap, and then the overlap is counted twice — eight hours where six actually happened. The fix is to merge the intervals for a day before measuring them.

   Cost: this moved the calculation out of one clever query and into plain code, which does more work. The number is the entire point of the screen, and a fast wrong number is worthless.

5. Charts, without a chart library

   The SVG is drawn by hand

   Every graph on the portal is SVG the code writes itself. Partly this falls straight out of the no-third-party-requests rule. Mostly it is that a fleet screen is read at a glance, in a light room or a dark one, and I wanted both themes to look decided rather than inherited.

   Cost: axes, ticks, bucketing and hit-testing are all mine to write and mine to fix. It is more code than importing a library, and none of it has ever broken because somebody else shipped a new version.

What it costs to run

over 150,000

requests a day, averaged over a month

2 million+

records taken in since launch

600+

requests a minute at peak

20+

releases, and counting

The record count is the one I care about. Monitoring earns its keep only if the history is complete — every reading kept at full resolution rather than thinned out to save room.

What I'd do differently

The overlap bug is the one that stings, and not because it was subtle — it was arithmetic. It survived because it lived inside a query that was hard to read and easy to trust, and because I had made the number fast before I had made it obviously right. It was wrong quietly, for weeks, on a screen people were using to make decisions.

What I took from it is an order of operations: get it correct in code I can follow, and only reach for the clever version once something is actually too slow. It is a duller way to write software, and I have not been caught out like that since.

← Back to work