Writing ·

The Tests Were Green and the Job Was Gone

A live derivatives platform stopped force-closing intraday positions for seven weeks and nothing anywhere failed. The unit tests passed, the config was populated, the docstring still described the feature. A case study in the bugs that only live behind a green suite.

For about seven weeks, a trading platform I built and run stopped force-closing intraday positions at the end of the session. Positions that were supposed to be squared off automatically carried overnight instead, with intraday leverage still applied to them. Nothing failed. No alert fired, no test went red, no log line looked wrong.

The feature had unit tests. Four of them, all passing. The config table still carried a square-off time for every exchange. The module docstring still listed the job as "job 1" of the lifecycle scheduler. Everything that describes a feature still described it. The only thing missing was anything that called it.

This is a writeup of that incident, a second one with the same shape found the same week, and what I changed afterwards. The system is client work, so the code is not public and the product is not named here. The failures are the interesting part anyway.

The system

A paper-trading platform for Indian equity and commodity derivatives, NSE and MCX futures and options. Users trade a simulated book against a live market feed, with the full order lifecycle behind it: margining, position limits, mark to market, intraday square-off, weekly settlement.

Three surfaces on one backend. A Flutter mobile app, a React web terminal, and a Python worker that holds the vendor market feed, the matching engine, the lifecycle scheduler and a websocket relay in one process. Postgres underneath, with row level security doing the authorization rather than application code.

Roughly 46,000 lines across the three, 95 test files, 396 passing worker tests, 128 commits between 13 June and 16 August 2026, currently on v1.0.4. It is a supervised pilot with real users, not an open rollout, and the gap between those two things is most of what the rest of this piece is about.

Failure one: the job nobody called

A large refactor renamed a quantity column from a raw-units representation to lots. The square-off job read the old column, so it broke. What happened next is the part worth looking at: instead of being migrated, it was replaced in the scheduler's job list by a different function entirely, an at-the-open gap guard, which happened to be adjacent in the file and did compile against the new schema.

So the scheduler kept running the right number of jobs. The suite kept passing, because the tests covered in_square_off_window(), a pure predicate that takes a time and a config and returns a boolean. That function was still correct. It was just no longer reachable from anything that runs.

Three artifacts kept the illusion alive, and all three are things a reviewer normally treats as evidence:

  • The tests. Four green tests on the predicate.
  • The config. Every exchange still had a square_off entry in its lifecycle config, so the database looked like a system that squares off.
  • The docs. The module docstring still enumerated the job.

A tested, configured, documented, uncalled feature. I found it by noticing that the predicate had zero production callers, then running git log -S on the symbol to see where it went.

Restoring it produced a second lesson immediately. The original keyed square-off off the static lifecycle config, which is wrong on any day the exchange overrides its session: MCX trades 09:00 to 17:00 on 1 January against a configured 23:20 square-off. Porting the deleted code back verbatim would have restored a bug along with the feature, so the restored version derives the square-off minute from the day's actual session close.

Failure two: the guard that fails closed, except in production

The same week, adding a quote-freshness gate, I wrote a method with this signature:

def get_quote(self, token, exchange=None, ...):
    ...
    if exchange:
        # per-exchange feed liveness check

Its docstring said it fails closed on an unknown exchange. A unit test asserted exactly that, by passing an exchange the state had never seen. Eight tests passed.

The production wrapper calls it as exchange=_exchange_by_token.get(token). A map miss there is not an unknown exchange, it is None, and if exchange: skips the entire liveness check for None. The token stayed tradable on token age alone, with no check that the feed for its exchange was still alive.

That map miss is reachable, not theoretical. The token-to-exchange map is pruned on every instrument reload, while the engine's instrument specs are built once at startup and never pruned, so the engine can hold a spec for a token the feed has already dropped.

Every one of my eight tests called the inner method directly, with an explicit exchange. Not one of them went through the wrapper the callers are actually handed. The branch that ships was the one branch never exercised.

An optional parameter on a safety check is a fail-open default wearing a fail-closed docstring. if arg: around a guard means "skip the guard when the caller cannot supply the argument", which is precisely backwards for a guard.

What the two have in common

A test binds to a function. Production binds to a call site. Nothing in a green suite says those two are the same object.

Both failures are invisible to coverage tools as well, and in opposite directions. The square-off predicate had 100% line coverage and zero production callers. The freshness guard had a covered function and an uncovered production path into it. Coverage measures whether test code reaches a line, never whether shipped code does.

Four checks came out of it, and they are the ones I actually run now:

  1. After any refactor that renames a column or type, git log -S"<old_symbol>" every function that touched it and confirm each one landed somewhere. Do not trust the suite to notice a deletion.
  2. Periodically grep helper predicates for production callers. A tested but uncalled predicate is a deletion nobody noticed.
  3. Delete superseded helpers rather than leaving them. The leftover is what made the deletion invisible.
  4. Test guards through the exact callable the callers are handed, with the argument absent. A test that always supplies the argument can never exercise the branch that ships.

The blind spot underneath both: nothing reports its own absence

Both bugs are a special case of a broader problem. Monitoring reports events. A thing that has stopped happening produces no event.

The platform's alerting is built around that. Push alerts go to a Telegram ops channel for feed silence during market hours, vendor auth failure, a stuck engine or scheduler cycle, failed settlement jobs, and worker start and stop. That channel deliberately does not depend on the database, so it still fires when the database is what is broken.

But a push alerter cannot report its own death, so there is also a dead-man's switch: the worker pings an external healthchecks endpoint every 60 seconds, and the external service alerts when the pings stop. The ping asserts that the engine and scheduler threads are alive, not merely that some loop is running, and is withheld if either stalls. Otherwise the switch would confirm liveness of the one component least likely to be the problem.

Per-account business events, stop-outs, profit alerts, concentration warnings, deliberately do not page. They go to the admin screen. An alert channel that fires on normal user behaviour stops being read, which is a slower version of not having one.

What is still wrong

Worth saying plainly, because a case study that ends on the fixes is a sales page.

The feed, matching engine and lifecycle scheduler are one process on one host. That is a real single point of failure and the only honest fix is a second host. The unclean-shutdown half of it is handled: the worker now drains on SIGTERM in about seven seconds instead of being killed at the timeout, after finding that the vendor websocket SDK ignores its own disconnect flag on error and reconnects the socket you just asked it to close.

The worker suite has 11 order-dependent failures. They pass file by file and fail in the full run, because the Postgres fixture is session-scoped and earlier modules leave rows that later whole-database assertions trip over. That is test isolation, not product behaviour, but it is exactly the kind of standing yellow that trains you to ignore a real red.

And it is a supervised pilot, not an open rollout, which is the correct posture for a system whose last two interesting bugs were both things the tests structurally could not see.