The eval harness gap: why agent pilots stall at 78% accuracy
We have seen the same number across dozens of pilots: accuracy plateaus around 78%. The team launches at 65%, climbs quickly to the high seventies, and then the curve goes flat. Everybody blames the model. The team starts evaluating frontier swaps. The new model lands at 79%. Everybody goes home.
The reason it plateaus is not the model. It is the missing eval harness.
What a real eval harness looks like
An eval harness is the closed loop that turns supervisor corrections into structured training signal and validates new model behavior before it goes live. It has four parts:
- Labeled corpus. Every supervisor edit, accept, and reject becomes a labeled example with the input, the agent's output, the supervisor's correction, and the rationale.
- Reference scoring. A way to score new candidate outputs against the labeled corpus, automatically, with deterministic metrics. Not just accuracy, also latency, escalation rate, false-confidence rate.
- Drift detection. Weekly comparisons against a rolling baseline so a 1.4-point regression on a sub-segment is visible before it becomes a 14-point one.
- Replay. The ability to take a real production decision, run it through a candidate model, and diff the output against what the supervisor actually approved.
If you do not have all four, you do not have an eval harness. You have a dashboard.
Why pilots stall without it
Without an eval harness, every model improvement looks like a fight. Engineers ship a prompt change; ops insists accuracy went down; nobody can prove it either way. Confidence thresholds drift. Supervisors lose trust. The team falls back to manual review on more and more cases, and the auto-resolution rate quietly degrades back below 70%.
With one, the loop is clean. The supervisor edits a ticket. The edit becomes a labeled example by the next training cycle. The next model run validates against that example. Accuracy moves from 78% to 81% to 84%, and you can prove it on a curve.
The build vs buy question
Building this in-house is a real project. We know, we did it before we built the platform. What surprised us was not the modeling work; it was the data plumbing. Connecting Zendesk corrections to a typed schema, deduping near-identical examples, versioning the corpus alongside the model, exposing a sane query interface to ops leaders, that is six engineer-months minimum, and it never feels done.
If you are starting fresh and the ops workflow is the goal (not the harness itself), buying it back is almost always cheaper. The harness is the part of the agent stack that has the lowest differentiation and the highest hidden cost.