Writing ·
The 99% Trap: A Field Guide to Label Leakage
A 4-class risk model hit 99.4–99.9% accuracy. That number was the bug, not the result. A mechanistic breakdown of label leakage, the sneaky constructive kind, plus the honest 70.5% model underneath it.
I built a land-dispute risk classifier and it hit 99.375 to 99.875% accuracy across three model families: Random Forest, XGBoost, LightGBM, all agreeing to the third decimal. On a 4-class real-world risk problem with heterogeneous synthetic drivers. That number should never have shipped as a result. It shipped as a bug report instead, once I went looking for it.
This isn't really a post about one mistake in one repo. It's about the mechanism behind it: the specific shape of bug that produces a suspiciously good model, why it's harder to catch than it sounds, and the checklist I now run before I trust an accuracy number. The land-dispute project is just the worked example.
Leakage isn't one thing
"Label leakage" gets used as a catch-all, which is part of why it's easy to miss. It's worth splitting into the actual mechanisms, because each one hides differently.
Direct. The label, or a renamed copy of it, sits in the feature matrix. Easiest to catch: a feature named suspiciously close to the target, or one with a 1.0 correlation to it.
Constructive. The label is built from a set of features, and those same features are left in the training set. The model isn't predicting the outcome, it's reconstructing the formula that defined the outcome. This is the sneaky one, because no single feature name looks wrong on its own.
Temporal. A feature encodes information that wouldn't have existed at prediction time. An outcome-adjacent field, a "last updated" timestamp that only gets touched after the event.
Split-level. Train and test share rows, or near-duplicates, so the model memorizes instead of generalizing.
A five-minute correlation scan catches direct leakage. Constructive leakage survives that scan, because every individual feature looks like a legitimate predictor. It's the combination, plus the fact that the combination is the label, that's the problem. That's what happened here.
How it was built
The synthetic data generator computes a continuous risk score as a weighted sum, then thresholds it into four classes:
risk_score = (
df['dispute_probability_raw'] * 0.4 +
df['dispute_history_score'] * 0.3 +
(df['area_sqm'] > 5000).astype(float) * 0.1 +
(df['ownership_type'] == 'Community').astype(float) * 0.1 +
df['development_index'] * 0.1
)
risk_score += np.random.normal(0, 0.1, self.n_samples)
risk_score = risk_score.clip(0, 1)
df['risk_score'] = risk_score
conditions = [
df['risk_score'] <= 0.2,
df['risk_score'] <= 0.4,
df['risk_score'] <= 0.7,
df['risk_score'] > 0.7
]
choices = [0, 1, 2, 3] # Low, Medium, High, Critical
df['risk_level'] = np.select(conditions, choices)
dispute_probability_raw and dispute_history_score alone carry 70% of the weight that defines
the label. Nothing wrong with that on its own, that's just how you'd simulate a plausible risk
signal.
The bug is one step over, in the preprocessing that builds the training matrix:
self.numerical_features = [
'area_sqm', 'elevation_m', 'slope_deg', 'ownership_duration_years',
'market_value_per_sqm', 'total_market_value', 'price_per_acre',
'distance_urban_km', 'distance_road_km', 'distance_forest_km',
'distance_water_km', 'distance_mineral_km', 'population_density',
'development_index', 'infrastructure_access',
'historical_dispute_count', 'dispute_probability_raw',
'dispute_history_score', 'risk_score', # <- the label's own inputs
]
risk_score, dispute_probability_raw, and dispute_history_score all rode straight into X.
The model wasn't learning what makes a land parcel dispute-prone. It was learning to invert a
threshold function on a column that is a near-deterministic transform of its own label, with 5 to
10% Gaussian noise as the only thing standing between "near-deterministic" and "deterministic."
Any model with enough capacity to fit a threshold (all three of Random Forest, XGBoost, and
LightGBM qualify) closes that gap almost completely. That's why three unrelated algorithms
converged on the same 99.4 to 99.9%. They weren't agreeing about land disputes. They were agreeing
about arithmetic.
What should have tipped me off
The feature importances already had the answer, before I ever opened the generator script.
risk_score and its two components dominated every model's importance ranking, an order of
magnitude ahead of anything domain-plausible like distance_road_km or infrastructure_access.
One feature (or a tight cluster) carrying the bulk of importance, on a task with no obvious single
dominant cause, is the pattern I now distrust before I even look at the accuracy number. The
accuracy is just where that distrust eventually cashes out.
Second tell: three structurally different algorithms, a linear-boundary-averaging ensemble and two different boosting implementations, landing within half a point of each other. Real tasks have texture. Different model families exploit it differently and post different scores. Convergence that tight usually means every model found the same shortcut, not that every model found the truth.
The fix, and the honest number underneath
Pull the three label-derived columns out. Keep historical_dispute_count, it's a Poisson-sampled
downstream draw, one step removed from the label construction rather than an input to it, closer
to a real observable record than a leaked signal:
self.exclude_features = [
'parcel_id', 'latitude', 'longitude', 'risk_label',
'last_dispute_days_ago', 'dispute_start_date',
'dispute_resolved', 'dispute_resolution_date', 'dispute_duration_days',
]
# excluded: dispute_history_score, dispute_probability_raw, risk_score.
# these are the literal weighted inputs used to construct risk_level (the
# target). including them is label leakage, not a feature.
Retrained on the same synthetic set, same three model families:
| Model | Val accuracy | Test accuracy |
|---|---|---|
| Logistic Regression | 69.25% | 70.5% |
| Random Forest | 69.0% | — |
| XGBoost | 67.5% | — |
| LightGBM | 66.75% | — |
A roughly 29-point drop from the leaked numbers, about the size you'd expect once the shortcut is gone. The interesting part isn't the drop, it's that the plain linear model won this round. With the label-derived signal removed, there's no nonlinear structure left worth an ensemble's capacity. A boosted tree just has more room to overfit sampling noise on a five-thousand-row synthetic set. When your simplest model matches or beats your fanciest one, that's not a fluke to shrug off. It's telling you something about how much real structure is actually in the data.
The finding that survived the fix
The honest model has a real weakness, and it's worth reporting instead of smoothing over: the Critical class, 3.8% of the data, collapses.
precision recall f1-score support
Low 0.71 0.91 0.80 223
Medium 0.69 0.61 0.65 348
High 0.73 0.74 0.73 391
Critical 0.17 0.03 0.05 38
One of 38 Critical parcels correctly identified. The confusion matrix shows why: 36 of the 38 get called "High" instead. The model isn't confusing Critical with Low, it's failing to draw the line between adjacent, imbalanced classes at the sparse tail. That's a legitimate, unresolved class-imbalance problem, the kind that shows up constantly in real risk-scoring work: fraud, default, safety incidents, the rare class is usually the one you actually care about. It's a better result than 99.9% accuracy, because it's an actual result. It points at a specific next step (resampling, class weighting, a cost-sensitive threshold) instead of a specific fabrication.
The checklist I use now
- Audit the label-construction code against the feature list before trusting an accuracy number. Not by eye on column names, actually diff the columns that build the target against the columns that feed the model.
- Check feature importance for concentration. One feature or tight cluster carrying most of the importance, with no domain reason for that dominance, is a leak until proven otherwise.
- Distrust convergence. If structurally different model families land within a point of each other, ask what they all had access to that made the task that easy.
- Ablate the top feature. Drop it, retrain. If the score falls off a cliff instead of degrading gracefully, that feature wasn't contributing signal, it was doing the whole job.
- Let the simple model compete. A linear model matching your ensemble isn't an embarrassment for the ensemble, it's a diagnostic. It tells you how much of the "performance" was ever about real structure versus a shortcut both models found equally well.
I'm pushing the corrected code and metrics to the GitHub repo alongside this post.