Evals Are Bullsh*t

Most evals are bullsh*t.

They grade the answer. Your business lives with the decision.

An eval is simply a test for an AI system. Give the system a task, then use a person, a piece of code, or another model to judge what it did. That judge is called a grader.

This sounds objective. Often, it is not.

Over the past few weeks, a few companies have asked me for help with evals. The pattern has been surprisingly consistent: teams have plenty of scores, but those scores cannot answer the question that matters. Is the agent doing the job the business actually needs?

Ask a grader whether an agent was helpful, accurate, and well-written, and it will produce a score. The score may even be consistent. But none of those words tells the grader how your company actually works, which mistakes are expensive, or what your best operator would have done instead.

The eval did not fail technically. It graded the wrong job.

One score, three different questions

People use “eval” to describe three different things.

1. A model benchmark asks whether the underlying model has a general capability. Can it solve a math problem, write code, or follow an instruction?
2. An agent eval asks whether the complete system performed a workflow. Did it choose the right tool, retrieve the right information, and complete the task?
3. A business eval asks whether the work was actually good for this company. Did the agent make the right decision, respect the right constraints, and produce an outcome the business values?

All three matter. They are not interchangeable.

Generic benchmarks are genuinely useful. They make broad capabilities comparable. They help teams choose between models, catch regressions, and isolate whether a failure comes from the model or the system around it. When you are debugging, ignoring the details of one business can be a feature.

The trouble begins when that abstraction is treated as a release gate for a specific business. A benchmark cannot tell you whether an agent understands your refund authority, your underwriting appetite, or the promises your support team is allowed to make.

Even rankings depend on what gets tested. An ACL study of major language-model benchmarks found that accounting for relationships among test prompts could change model rankings. A 2026 analysis of 43 agent benchmarks found that their tasks skewed toward programming while human labor and economic value were concentrated elsewhere. The second study is a preprint, not a verdict on every benchmark. Together, they make a simpler point: a test distribution is a design choice, not a mirror of your customers.

The mistake is not using benchmarks. The mistake is treating a high benchmark score as proof that an agent is ready for your business.

A helpful agent can still do the wrong thing

Imagine a customer asks for a refund on a damaged $700 order.

The agent responds quickly. It apologizes, cites the return policy correctly, and explains the next steps in clear language. A generic rubric gives it top marks for helpfulness, factuality, relevance, and tone.

It also grants the refund even though its approval limit is $250. It exposes an internal note marking the customer as a churn risk. It forgets to update the order system. The customer has to contact support again two days later.

The answer looked excellent. The work was a failure.

A useful eval would ask different questions:

• Was the customer eligible under the policy that applied to this order?
• Did the agent stay within its approval authority or escalate correctly?
• Did it keep internal information private?
• Did the refund and case status change in the systems of record?
• Was the issue resolved without another contact?
• What did the resolution cost, and how long did it take?

Some of these checks are immediate and deterministic. The refund amount either exceeded the limit or it did not. The order record either changed or it did not. Other judgments require an expert or a carefully calibrated model. Was the exception reasonable? Did the response preserve trust without promising too much?

Correctness is not one dimension. It is the set of things your business refuses to get wrong.

A precise score can still create false confidence

Specific criteria beat a vague impression of quality. On a 2025 real-world legal question-answering task, requirement-level grades correlated with expert judgment at 0.78, compared with 0.35 for one LLM score and 0.12 for traditional metrics. That is one task, but the mechanism travels well: “Was this good?” is a weak question. “Did it cover each required issue and ground every claim?” is much easier to judge and improve.

Automated graders make semantic evaluation affordable at scale. They are still another fallible system. A 2025 study of 15 LLM judges across 22 tasks found that changing the order of two answers could change the preference, depending on the judge and task. A separate controlled study of reward-model overoptimization showed that pushing harder on an imperfect proxy could eventually hurt the result it was meant to represent. Neither finding argues against automated grading. They argue for calibrating the grader, checking it often, and never confusing its score with the truth.

Then there is reliability. An agent that succeeds 75% of the time may sound strong. But the chance that it succeeds three times in a row is only about 42%. Anthropic uses this distinction to separate “can it succeed at least once?” from “can it succeed every time?” A research agent may get several attempts. A customer-facing agent usually gets one customer.

Average quality hides the exact failures businesses care about: the expensive exception, the repeated mistake, and the action that cannot be undone.

What good evals actually do

A good eval starts with the work, not the model.

It studies real cases, expert decisions, corrections, and outcomes. It includes ordinary traffic and the rare situations where a mistake carries an outsized cost. It tests the entire system under realistic conditions: instructions, tools, data, permissions, handoffs, and state changes.

It also uses the simplest credible grader for each requirement. Code should verify facts and state when it can. Models should judge meaning when rules cannot. Domain experts should settle ambiguous or consequential decisions and regularly check whether automated graders still agree with them.

This is not a one-time certification. Production creates new cases the original test set could not anticipate. Every meaningful failure should sharpen the definition of good, become a regression case when appropriate, and make the next release harder to fool.

NIST's 2026 draft practices for automated benchmark evaluations start by defining the measurement target and choosing a benchmark that serves it. A 2026 KDD paper from Nubank carries that standard into production. Across five customer-support deployments, it paired human-calibrated offline evaluation with online measurement. In one card-delivery system, new variants improved transactional NPS by 37 percentage points and self-service by 29 percentage points over prior versions. That is one company's result, not a universal law. The standard is what matters: offline scores should predict what happens to customers and the business.

The tools are the easy part. The hard part is deciding what good means.

The no-bullsh*t eval checklist

Before trusting an eval, ask eight questions:

1. What decision will this score change? A useful eval should inform a model choice, a product change, a release gate, or an operating decision.
2. Does it define success and failure? Name the desired outcome and the unacceptable result. “High quality” is not a specification.
3. Does it reflect real work? Use production cases, expert examples, known failures, ordinary traffic, and rare but costly exceptions. Test when a behavior should happen and when it should not.
4. Does it test the whole system? Reproduce the tools, data, permissions, handoffs, and state changes the agent will encounter. A good final answer can hide a broken action.
5. Does it grade observable outcomes directly? Prefer a database check, policy assertion, or unit test when one can answer the question. Use model judgment where meaning genuinely matters.
6. Do experts agree with the grader? Compare automated grades with the people who understand the work. Investigate disagreements instead of averaging them away.
7. Does it measure reliability, not just averages? Run repeated trials. Break results down by workflow, risk, customer, and failure type. The segment that matters most may disappear inside the mean.
8. Does it learn from production? Add new failures, recheck old capabilities, and connect eval movement to real outcomes such as resolution, conversion, loss, cost, and latency.

If an eval cannot explain why the business should care when its score moves, it is dashboard decoration.

Good evals are not generic measures of intelligence. They are records of judgment: what we want, what we refuse, what experts correct, what customers experience, and what the business learns.

An eval is business judgment made executable.

Your agent will optimize for whatever you grade. Make sure you are grading the business you actually run.