Writing ·
The Headline Win Rate Is Not the Result: Decomposing P&L
A trading bot went live at 75.3% win rate, almost exactly matching its backtest. That should have been the end of the story. Decomposing the aggregate found two things hiding underneath it: a subsystem quietly losing money, and an edge that was decaying day over day.
A Polymarket trading bot I built and paper-run went live, and six days in, a status check showed 75.3% win rate against a 77.4% backtest. Close enough to call it validated. If I'd stopped at the headline number, that's exactly what I would have written up: backtest holds, ship it.
I didn't stop there, and decomposing the aggregate found two things the win rate was hiding. One subsystem was quietly losing money while the overall number stayed green. And the edge that made the number green in the first place was decaying by the day, invisible in a single six-day average.
This is a piece about why an aggregate performance number is the wrong place to stop, not just for this bot, for any system where a single metric (win rate, accuracy, Sharpe) gets treated as the verdict. The bot is the worked example.
Why an aggregate hides more than it shows
A win rate, an accuracy score, a Sharpe ratio: these are all weighted averages over time and over subsystems. Averages are exactly the statistic that hides two specific failure modes.
Concentration. If a handful of outlier events carry most of the result, the aggregate describes those outliers, not the typical case. A 75% win rate built from 100 small wins and 3 catastrophic losses reads identically to a 75% win rate built from 100 uniform wins, until you look inside it.
Composition. A system is rarely one mechanism. It's usually several, an entry rule, an exit rule, a sizing rule, blended into one reported number. If one subsystem is strongly positive and another is mildly negative, the blend can still look fine. The negative piece doesn't show up until you split the number back into its parts.
Both failure modes produce the same symptom: a good aggregate. Neither is visible without deliberately taking the number apart.
The mechanism, briefly
The bot trades Polymarket's 5-minute BTC up/down markets. The edge is latency, not prediction: when BTC moves at least 5 basis points within 2 seconds in the direction the order book already favors, the book hasn't repriced yet, so the bot buys the stale ask before it does. An earlier version tried to forecast the outcome directly; an unbiased backtest showed the order book beat that model at every horizon, so the forecasting layer was retired. The current version doesn't predict, it acts faster than the market updates. Backtested at 195 entries: 77.4% win rate, +$0.35 per dollar risked, profitable on every full trading day tested. All numbers below are paper-traded, no live capital.
Finding one: the exit logic was losing money while the headline stayed green
The bot has a second rule beyond entry: an adverse-jump exit, designed to cut a position early if the market moves against it. Six days live, 12 of these exits fired. Realized exit P&L was −$17.41. Had those same 12 positions been held to expiry instead, they'd have finished at +$18.41. The exit rule cost −$35.82 relative to doing nothing, on a subsystem the backtest had predicted would add about +$5.20.
The aggregate 75.3% win rate never flagged this, because entries were profitable enough to absorb it. Splitting the 12 exits by entry price is what surfaced the actual mechanism:
- Entries at $0.50 or above (n=7): exit-vs-hold +$26.54. The exit genuinely worked here, a low remaining payoff means a jump against the position is worth cutting.
- Entries below $0.50 (n=5): exit-vs-hold −$62.36. The exit destroyed value here, selling a high-payoff ticket into a stale, wide bid on the way out cost more than the adverse move itself.
One rule, two entirely different effects depending on a variable the rule didn't condition on. The fix was mechanical once the split was visible: gate the exit to only fire on entries at $0.50 or above, and let cheap entries ride to expiry. A single aggregate number never would have pointed at that condition. It just would have kept reporting a win rate that happened to still be positive.
Finding two: the edge was decaying inside the same window that looked fine
The six-day P&L was concentrated: the single best trade accounted for 23% of total profit, and the top 10 trades out of 178 accounted for 71%. Fifteen cheap entries (under $0.20) alone produced $487 of the $814 total.
Excluding those cheap tail trades, the remaining EV per dollar dropped to +0.251, and it wasn't flat, it was falling every single day: 0.384 → 0.351 → 0.193 → 0.152 → 0.145 across five trading days. Opportunity rate fell alongside it, from roughly 53 signals a day down to 21–25.
A single six-day aggregate cannot show a trend inside itself. It can only show you the average of a trend, which looks like a stable number right up until it doesn't. The only way to see the decay was to compute the same statistic separately for each day and put them next to each other.
The checklist
- Split any aggregate by time before trusting it. A flat-looking win rate over a window can still be a strictly declining one inside that window. Compute it per day (or per fold, per batch) and look for a trend, not just a level.
- Split any aggregate by subsystem. If a metric blends an entry rule and an exit rule, a model and a filter, a strategy and its risk overlay, decompose it. A positive blend can contain a negative component that's currently being outweighed, not fixed.
- Check what fraction of the result a few outliers carry. If the top few events account for most of the total, the aggregate is describing the tail, not the typical case. Report both numbers: with and without the top decile.
- When a subsystem's effect flips sign, look for the variable it should have conditioned on. The exit logic wasn't uniformly good or bad, it was good above a price threshold and bad below it. The rule was underspecified, not wrong.
- A number that exactly matches its backtest is not automatically validated. It's validated for whatever the backtest actually measured. If the backtest reported one aggregate and skipped the decomposition, the live match only confirms the aggregate, not the parts underneath it.