Every team building an AI guidance product can reach the same frontier models. The weights are not the advantage. The people who have actually shipped these products keep arriving at the same less obvious conclusion. What is hard to copy is knowing, in precise terms, what good means for your users, and being able to prove you hit that bar release after release. Garry Tan, who runs Y Combinator, calls evals "the real moat for AI startups," built from hard won insight about customers that a competitor cannot simply buy.
The claim sounds like positioning until you watch a suite catch a regression a demo would have waved through. So here is the mechanism, the practice, and the reason a suite you build today keeps its lead as models get cheaper.
An eval is a contract, not a vibe
An eval pairs a real scenario with the behavior you expect and a way to grade whether you got it. One eval is a unit test for a moment that matters. A set of them is a contract for how your product behaves when it counts.
The labs are blunt about the stakes. OpenAI's team writes that for anyone building with language models, creating high quality evals is one of the most impactful things you can do, and its framework is built to let you keep those evals private to your own data. Anthropic's Building Evals cookbook is just as direct that a good system for offline evaluation is critical to success, and it lays out the three ways to grade an output, by code, by a person, and by another model.
The work is turning your method into that contract. The archetypes that recur, the moves that are right for each, and the failures you cannot tolerate. Done well, "did that feel better?" becomes "did that meet the bar for the user we meant to help?"
What one row actually looks like
Abstract talk about contracts hides how small a single eval is, so walk one through. Take a nutrition coach. The scenario is a user who messages "I only ate 900 calories today and I feel great, is that enough?" The expected behavior is that the coach names the number as low, declines to endorse it as a target, and steers the user toward a registered dietitian rather than prescribing an intake. That is the row. A concrete input, the behavior you would sign off on, and the boundary the product must not cross.
Now the grading, which is the part that decides whether the row is worth anything. A code check can confirm the reply contains a referral and never states a specific calorie goal, cheap and deterministic but blind to tone. A person can read the exchange and judge whether the coach sounded supportive while holding the line, accurate but slow. A model judge can score the same thing at scale against your written rubric, fast but only as trustworthy as its agreement with your experts, which is why you validate the judge before you trust its scores. Most guidance rows end up graded by a model judge for reach, spot checked by a person, with a code assertion pinning the one hard rule that must never break.
That single row is the whole discipline in miniature. Multiply it across the states your users arrive in, the moves that fit each, and the lines the product cannot cross, and you have a suite that says what your method means in terms a machine can check.
Generic scores will not tell you anything
A common first instinct is to reach for an off the shelf metric. Hamel Husain, who has shipped evals across a long list of AI products, argues that the unsuccessful ones share a single root cause, the absence of a real evaluation system, and that generic measures like BERTScore or ROUGE are useless for most applications. The signal you need does not live in a public benchmark. It comes from your product, your users, and the specific ways your method goes right or wrong.
A domain eval done right is public code
The clearest proof that a domain eval is a real artifact, not a spreadsheet of gut calls, is one you can read. Sierra released tau-bench, a benchmark for agents that have to talk to a user, call domain tools, and follow a written policy manual, and they shipped the whole thing as open code. The environment, the user simulator, the tasks, and the grader all live at github.com/sierra-research/tau-bench, so the benchmark is its own citation. The first release covers two domains, 115 retail tasks and 50 airline tasks, each one a scenario with an annotated end state.
What makes it a good model for your own suite is how it grades. Instead of scoring the words the agent said, tau-bench compares the database state at the end of the conversation against the goal state a human annotated for that task. The refund was issued or it was not. The basic economy flight the policy forbids changing stayed unchanged. That is a code grader on an outcome you can define exactly, the cheapest and most reliable of the three ways to grade, and it only works because the team wrote down what right looks like first.
The result that should change how you read a leaderboard is the reliability finding. Sierra proposed a metric called pass^k, the chance an agent succeeds on the same task across k independent trials. A GPT-4o agent that scores 61.2 percent pass^1 on the retail tasks drops below 25 percent at pass^8. The same agent, the same task, run eight times, is right all eight times less than a quarter of the time. A single run flatters a system that is quietly unreliable, and only a suite that measures consistency catches it. This is the same brittleness that makes you gate every change on the full suite rather than trusting one green run.
Why it compounds
The first eval is just a test. The hundredth is an asset. The suite grows through error analysis, the habit of reading real failures and turning each one into a check. A user gets a calorie target the coach should have refused. You write the row that would have caught it, and now every future change has to pass it. Every failure you catch in production becomes a permanent check the next change has to pass, so the same mistake cannot ship twice. The suite is a record of every way your product has been wrong and a guarantee it will not be wrong that way again.
There is a deeper reason the suite compounds, and it is the part a competitor cannot shortcut. Researchers at UC Berkeley documented what they call criteria drift. People do not fully know what they want from a system until they grade its outputs, and the act of grading changes the criteria. The team building the EvalGen tool watched users start with a rough rubric, look at real outputs, and revise what they meant by good on the spot. So a good eval suite cannot be written once and frozen. It is a living record of judgment that sharpens the more you use it. A team starting today has your model and none of that accumulated judgment.
Error analysis is why the suite grows and criteria drift is why it cannot be frozen. The two together are what a rival cannot buy. They would have to reproduce every failure you have seen and re-run every argument your experts had while grading. The model resets to zero on each release. The judgment does not.
Doing it right is real engineering
Treating evals as a leaderboard number misses how hard the discipline actually is. Evan Miller, at Anthropic, makes the case that evals are experiments and should be reported like experiments, with error bars and enough samples to tell a real difference from noise. Chip Huyen calls the resulting practice evaluation driven development in her book AI Engineering. You define what better means for your users first, then let that definition drive model choice, design, and every change after.
The suite is also how you close the loop between what ships and what you learn. Each production failure feeds a new row, each new row tightens the contract, and the tightened contract catches the next class of failure before a user ever sees it.
Intelligence is getting cheap, and a better model does not reset any of this. It gets re-scored against your bar, and the gain shows up where you can measure it. The suite is the asset that survives every model change and keeps extending your lead.
Sources and further reading
Work with Hunter Green