58% vs 90%: what measuring my own classifier changed
An ablation ladder over 450 of my family's real transactions, and the one product change it justified.
We run a private finance app for our household. One of its jobs is boring and constant: a bank alert email arrives, a small model (Claude Haiku) pulls out the amount, merchant and date, files it into one of our budget categories, and pings me on Slack and my phone to confirm.
I assumed that pipeline was "pretty good". Then I measured it, and the most useful thing it told me was something I could fix without touching the model.
The setup
Every transaction we had ever corrected was already in the database, along with the raw email that produced it. So I didn't need to collect anything. I could replay history: re-run the classifier against stored inputs and score it against what we had actually filed.
I built it as an ablation ladder, adding one piece of the real pipeline per rung:
| Rung | Configuration | Category accuracy |
|---|---|---|
| 0 | Lookup table from past corrections, no model call | 46.7% |
| 1 | Bare model | 31.8% |
| 2 | + our category list in the prompt | 65.8% |
| 3 | + past corrections as few-shot examples | 71.3% |
| 4 | + deterministic override from memory (production) | 76.4% |
450 scored cases, after excluding 42 rows that were typed in by hand and so aren't labels at all. Extraction (amount, date) was 95–100% accurate on every rung. Categorisation was the hard part.
Three things I didn't expect
A lookup table beat the model. With no model call at all, our own history got 46.7%. The bare model got 31.8%. For filing our transactions into our categories, our past decisions were worth more than the model's knowledge of the world. The model earns its place on extraction, not on filing.
Three quarters of the lift was one line of prompt. Going from rung 1 to rung 4 added 44.7 points. 34 of them came from putting the category names in the prompt. The memory machinery I was proud of added about 10.
The confidence score didn't mean what the code assumed. 421 of 450 answers claimed 0.9 or higher confidence; they were right 79.8% of the time. Tightening the threshold from 0.50 to 0.95 still left one in seven accepted answers wrong. The cause was in the prompt: confidence was defined as how much it could extract, which was nearly always everything. We were gating categorisation on an extraction-quality signal. Every line of code involved did exactly what it said, so review would never have caught it. Measurement did.
The split that mattered
The pipeline already records where each decision came from: the memory override, or the model. Splitting accuracy by that field gave a much clearer picture than the confidence score ever did:
| Decided by | Share | Accuracy |
|---|---|---|
| Memory override | 54% | 90.2% |
| Model | 46% | 58.3% |
Half of my notifications were about transactions that were already right nine times in ten, and they got exactly the same interruption as the half that was wrong four times in ten.
So the change was small: stop pinging about items the memory decided. It shipped behind a per-household switch that is off by default, because turning it on means accepting that roughly one in ten quieted items will be miscategorised until someone notices. That is a decision about acceptable risk, and I wanted it to be made on purpose rather than by a default.
The harness had bugs too
Two of them would have changed the conclusions:
- Rung 0 was being scored on fields it never attempted and credited for a merchant it had been handed as ground truth.
- The
--limitflag took the first N cases chronologically, which is exactly the cold-start stretch where memory is empty. A 25-case sample made memory look worthless. It wasn't noise; it was bias. Sampling evenly across the timeline fixed it.
I now treat a surprising eval result as a reason to audit the eval before the system.
Rules I gave the agent
Claude Code wrote most of this harness. Two rules stayed with me rather than the agent:
- The run against real data happens locally, by me, and only the aggregate report leaves the machine. Agent sessions stay off our household's data entirely.
- The daily automated routine may run the free synthetic ladder as often as it likes, but never the real one. That is about 1,800 live model calls on my API key, and it is a human decision taken once a change is ready.
What I'd tell someone starting
- If your data already holds corrections, you already have an eval set. Replay it.
- Add one component per rung. The answer to "is this part worth it?" is the difference between two rows.
- Check what your confidence number is actually measuring before you build a threshold on it.
- Look for a signal you already store before asking the model for a new one. Mine was a column called
decision_source.
The next experiments are already queued: a separate confidence for the category itself, and normalising merchant names, since exact string matching caps the memory at about 78% of cases. Each one gets the ladder re-run before and after. That is what it's for.