Three people read about forty outputs between them. Everyone agreed it looked good. That was the evaluation, and it is the one running in most companies shipping AI right now.
It has a name problem. Nobody calls it a method, so nobody audits it. The system ships. A month later someone in a planning meeting asks how it is performing, and the answer is a feeling with no number attached. The feeling might even be right, and that is the trap. A correct guess and a measurement look identical from the inside. You cannot tell improvement from noise, you cannot prove the thing got better after you changed the prompt, and you will not notice a regression until a customer finds it for you.
The fix is smaller than the internet suggests. Most guides on how to measure AI performance hand you a taxonomy of twenty metrics and a platform to track them in. Wrong first move. What you need is a baseline, a set of real examples with agreed-upon answers, two or three numbers that map to how your system actually fails, and a cadence for publishing them. A week of work, not a quarter.
What "it seems to be working" actually is
Spot-checking is a measurement method. It just happens to be one with properties nobody writes down.
The sample is tiny. Forty outputs out of ten thousand is a 0.4% sample, and it was not drawn at random. It was drawn by the people who built the thing, looking at the cases they thought of, on the days they happened to look. Four mechanisms are at work:
- Sample size. With forty examples, a real 5% error rate and a real 15% error rate look the same. You cannot distinguish them, so you cannot detect a change of that size either.
- Selection bias. The person checking chose the inputs. People choose inputs they understand, which are the inputs the system was designed around.
- The demo effect. You test the cases you imagined. The failures live in the cases you did not imagine, which is why they are still there.
- No memory. There is nothing to compare against. Last month's quality was also a feeling, so "better than last month" is not a claim anyone can check.
None of this means the system is bad. It may well be excellent. It means you have no instrument, and without an instrument every decision that follows - ship it wider, swap the model, cut the human review step - gets made blind.
This gets worse as models improve, not better. Stanford HAI's 2026 AI Index reports that frontier models gained thirty percentage points in a single year on Humanity's Last Exam, a benchmark built specifically to be hard for AI, and makes the broader point that capability is now outpacing the benchmarks designed to measure it. If the public benchmarks are struggling to stay meaningful, a private sample of forty is not going to carry your production system.
Before you can fix the instrument, though, there is an earlier number almost nobody has.
The number you skipped
Before any metric means anything, one question has to be answered: what does the process you are replacing actually score?
Almost nobody captures this. The team stands up an AI summarizer, measures it at 87% accuracy, and has no idea whether that is a triumph or a disaster, because nobody ever measured the humans writing those summaries by hand. It is the most common gap in AI measurement, and also the cheapest one to close.
Take 100 items the existing process already handled . Score them with the exact rubric you intend to use on the AI. Write down the number and the date. That is your baseline.
Two things tend to happen when teams do this. The human number usually comes in lower than anyone expected, because people are inconsistent, they skip fields under time pressure, and two reviewers disagree more often than either would predict. Then the baseline turns into the most persuasive artifact the team owns. "The AI is 87% accurate" invites an argument. "The AI is at 87% where our manual process measured 79% on the same 100 items in August" ends one.
Baselines also expose the cases where the honest answer is that you should not automate this at all. If the manual process scores 96% and the AI scores 88%, the interesting conversation is about cost and volume, not quality. Measuring first is how that conversation happens before the build instead of after it.
Get an AI Readiness Snapshot - a free 30-minute call that maps where AI will actually pay off in your business, and what to measure before you start. Book a free 30-min readiness call .
Build the golden set before you argue about metrics
The eval set is the asset. Metrics are functions applied to it, and teams routinely spend three weeks debating which metric to use while owning zero labeled examples.
A golden set is 100 to 300 real items with agreed-upon correct answers. Real is the operative word. Invented test cases test your imagination, which is the thing that already failed.
Composition matters more than size:
- Common cases - the bulk of your traffic, roughly in the proportion it actually arrives.
- Known-hard cases - the ones that already produced a complaint, an escalation, or an awkward Slack thread.
- Edge cases - rare shapes that carry outsized consequences when they go wrong.
- Cases you expect to fail - include them deliberately. A set that only contains winnable items reports a number that flatters you and predicts nothing.
Two people should label a subset independently, maybe 30 items. Where they disagree is where your rubric is underspecified, and finding that out now costs an hour. Finding it out after the whole set is labeled costs the whole set, because a set labeled against a vague rubric produces numbers that drift the moment the labeler changes.
Then freeze it. Version it, date it, and store it somewhere that is not a laptop. A set that changes between runs makes every comparison meaningless, and that is the failure mode that kills the practice inside two months: numbers that move for no discoverable reason, and a team that stops trusting them.
Keeping the set fresh without breaking comparability
Real traffic drifts, so a set frozen in March slowly stops representing June. The way through is additive versioning rather than editing. Keep v1 intact and score it every time, then build v2 by adding new items alongside. Report both for a period. You keep a comparable trend line and gain a current picture at the same time, which is not something you can get by swapping items in and out.
Scoring outputs when there is no single right answer
This is where most teams stall, and it is not a tooling problem. Classification tasks are easy to score, because a label is either right or it is not. Generative outputs have no answer key . Three mechanisms are available, and you will end up using all three at different points.
| Scoring method | Cost | Speed | Reliability | Best for |
|---|---|---|---|---|
| Programmatic checks | Very low | Instant | High, within narrow scope | Format validity, required fields present, schema compliance, forbidden content, numbers that must reconcile |
| Rubric-based human review | High | Slow | Highest, when the rubric is tight | Safety, compliance, anything a customer will escalate, calibrating the other two methods |
| LLM-as-judge | Low | Fast | Moderate, and it drifts | High-volume screening, relative comparisons between two versions, coarse rubrics with clear criteria |
Programmatic checks are underrated. A surprising share of "quality" failures are structural - a missing field, a malformed date, a total that does not match its line items - and those need no judgment at all. Write them first, because they run on every output in production, not just on eval day.
LLM-as-judge is the one that needs care. It works when the rubric is explicit, the criteria are separable, and you are comparing two candidates rather than assigning an absolute score. It fails without announcing itself when the rubric is vague, when the judge shares a blind spot with the model it is grading, or when the judge model changes underneath you during a routine version bump.
Calibrate before you trust it. Have humans score 50 items, have the judge score the same 50, and look at where they diverge. If agreement is poor, the problem is almost always the rubric rather than the judge. Then re-check that agreement whenever the judge model version changes, because a silent upgrade can move your entire quality trend line without anything about your system changing at all.
One rule holds regardless of method: never let the system that produced an output be the only thing that grades it.
The four metric families, and picking two or three
Metrics fall into four families. The point of knowing all four is not to track all four. It is to make sure you have not picked three metrics from the same family and called it coverage.
| Family | What it tells you | Who reads it | Sensible cadence |
|---|---|---|---|
| Quality | Whether the output is correct or useful | The team building it | Every change, and weekly |
| Operational | Whether it responds fast enough and stays up | Engineering and support | Continuous, alerting |
| Cost | What each unit of work costs to produce | The budget owner | Weekly, monthly in aggregate |
| Outcome | Whether the business result moved | The sponsor | Monthly |
Quality varies by task type. Classification gives you accuracy, precision, recall, and F1. Precision answers "when it flagged something, how often was it right." Recall answers "of everything it should have flagged, how much did it catch." Which one you optimize is a business decision, not a technical one: a compliance screen that misses violations is worse than one that over-flags, and a customer-facing auto-reply is the reverse. Generative tasks swap these for factual accuracy, relevance, completeness, and the rate of invented content. Agents add another layer, because an agent can produce a perfect final answer through a sequence of wrong steps , which is why tool-selection correctness and task-completion rate matter separately from output quality.
Operational is latency at p50 and p95, throughput, error rate, and fallback rate. Median latency hides the tail. The tail is what users feel.
Cost is cost per task, not monthly token spend. Include the human minutes. If an AI draft still takes a reviewer four minutes to fix, that is part of the unit cost, and it is often the part that decides whether the system pays for itself.
Outcome is the number the business already tracks - resolution rate, cycle time, escalation rate, revenue per interaction. You do not invent it. You attach to it.
The cut-down rule is three lines long. One quality metric that matches your dominant failure mode. One operational or cost metric that constrains it. One outcome metric the business already watches. That is your scorecard. Everything else stays available as a diagnostic, which you look at when the headline number moves and you need to know why. Keeping KPIs and diagnostics separate is what keeps the practice alive, because a twenty-metric dashboard gets ignored by week three.
Offline evaluation and production signals do different jobs
Picking the metrics still leaves the question of where you measure them, and the answer is both places, for different reasons.
Offline evaluation runs your frozen set against a candidate version. It is repeatable, it is comparable across time, and it catches regressions before users meet them. What it cannot do is tell you about the inputs you never collected.
Production signals come from real traffic. They catch the distribution you did not anticipate, and they are the only place drift becomes visible. Several are cheap enough to instrument this week:
- Edit distance between the AI's draft and what the human actually sent. Most teams already have the data for this one and have never computed it.
- Escalation rate - how often an AI-handled item comes back.
- Silent abandonment - the user started, saw the output, and left without acting. No thumbs-down, no complaint, just a dead end.
- Explicit feedback - useful, but it is a biased sample. Angry users and delighted users rate. The indifferent majority does not.
Teams that only run offline evals get blindsided by input shapes they never imagined. Teams that only watch production metrics see the number move and have no controlled way to find out what caused it. Running both is what gives you detection and diagnosis at the same time.
A worked example, with the numbers kept
All of that stays abstract until you watch a number change and see what it costs. The following case is illustrative, constructed to show the shape of the arithmetic rather than to report a client result.
A support team runs a ticket summarizer. Each closed ticket gets a summary that feeds a searchable knowledge base. Two hundred real tickets are sampled and labeled against a three-part rubric: does the summary capture the customer's actual problem, does it capture the resolution, and does it contain anything the ticket does not say.
The manual baseline, scored on 100 of those tickets written by agents, comes in at 79% fully correct. Most misses are omissions, usually a skipped resolution step.
The first model run scores 84% fully correct. Better than the humans, and the temptation is to stop there and report the number. The error breakdown is where the decision actually lives. Of the 16% that failed, 10 points are omissions, the same failure the humans had, and 6 points are invented facts: a resolution step that never happened, stated with total confidence.
Those two failures are not equivalent. An omission makes the knowledge base thinner. An invention puts a wrong answer in front of the next agent, who has no reason to doubt it. So the rubric gets reweighted, an invention now counts as a hard fail, and the composite number drops to 81% against a human baseline of 79%. The system got worse on paper and more honest in reality, which is what a good metric change usually looks like.
Then the threshold moves. Summaries where the model's confidence falls below a cutoff get routed to a human instead of publishing directly. Invented facts drop from 6% to 2%. The cost is 18% more items in the human review queue. That is the shape of every real evaluation decision: quality bought with throughput, at a rate you can now state in a sentence.
Six weeks later the same 200-item set gets rerun after a model version bump. The composite holds at 82%, but invention creeps from 2% to 4%. Nothing in the prompt changed. The model did. Without the frozen set, that surfaces as a support complaint in November, traced to nothing.
Metrics that lie
An instrument can be worse than no instrument if it reports confidently and wrongly. The recurring ways that happens:
- Leaked eval sets. Items that made it into a prompt, a few-shot example, or a fine-tune. The model has seen the answers. The score is fiction.
- Averages hiding a segment. An overall 91% can be 97% on the common case and 62% on a segment that happens to be your enterprise accounts. Segment before you trust.
- A judge that moved. Your LLM-as-judge got upgraded. The trend line shifted. Nothing about your system changed.
- A stale set. Frozen in March, scored in September, describing traffic that no longer exists.
- Quality with no cost or latency beside it. Optimize quality alone and the system drifts toward more compute, more retries, more review. The number goes up. The economics go down without anyone watching.
- Vanity metrics. Usage, adoption, and number of queries measure enthusiasm, not correctness. They are the easiest numbers to collect, which is exactly why they end up on slides.
The gap between adoption and impact is now measured. DX's engineering research (2026) reports AI tool adoption around 93% across the industry while most organizations see only 5 to 15% gains in pull-request throughput - real, but far below what the tool vendors claim. High usage with unmeasured output is how that gap stays invisible.
What good looks like: the one-page scorecard
Everything above collapses into one page:
- System and use case - one line, in plain language.
- Golden set - version, item count, date frozen.
- Baseline - what the prior process scored, and when it was measured.
- Quality - current number, previous number, direction.
- Operational - p95 latency, error rate.
- Cost - cost per task, including human review minutes.
- Outcome - the business number, with the caveat that attribution is partial.
- Owner - a named person.
- Cadence - reviewed weekly by the team, monthly with the sponsor.
The cadence is the part that gets dropped, and dropping it is what turns measurement back into vibes. Publish the number on schedule whether it moved or not. A boring scorecard that appears every Friday beats an impressive one-off analysis, because the boring one builds a trend line, and a trend line is the only thing that can tell you the model changed under you.
One honest caveat: outcome metrics rarely isolate cleanly. Resolution rate moves for a dozen reasons and the AI is one of them. Say so on the page. Overclaiming attribution is the fastest way to lose the credibility the rest of the scorecard earned.
Key takeaways
- Judging AI by reading a handful of outputs is a measurement method with four known failure modes: tiny sample, biased selection, untested cases, and no memory. It cannot distinguish improvement from noise.
- Measure the process you are replacing first. Without a baseline, an accuracy number is unanchored and unpersuasive.
- The eval set is the asset, not the metric. 100 to 300 real items, labeled by two people, versioned and frozen.
- One quality metric matched to your dominant failure mode, one operational or cost metric, one outcome metric the business already tracks. Everything else is a diagnostic.
- Measurement is a cadence, not a launch gate. A scorecard published every week beats a thorough analysis published once.
Put a real number on your AI
If the honest answer to "how is it performing?" is currently a feeling, the gap is a week of work, not a platform purchase: a baseline, a golden set, three metrics, and an owner.
AI Transformation Discovery - a one-week sprint that produces a concrete AI roadmap, including the measurement system that tells you whether it worked.
