The most dangerous change is the one that fixes the case in front of you and quietly breaks ten you were not looking at. With language models this is not a rare event. OpenAI's own guidance states it flatly. A prompt that works well one day can produce unexpected and low quality results the next. You edit one instruction to stop a coaching bot from over-apologizing, ship it, and a week later a parent gets told to tough out a fever that should have gone to a pediatrician. The edit did what you asked. It also moved something three archetypes away that you never tested.
Why one change ripples
This brittleness predates language models. It is the oldest lesson in production machine learning. The Google paper on hidden technical debt named the principle a decade ago and gave it an acronym, CACE, changing anything changes everything. A nudge to improve tone in one archetype can erode a safety boundary three archetypes away, because the parts are entangled in ways you cannot see from a single example.
A prompt is not a set of independent rules the model applies one at a time. It is a single conditioning context, and every token in it shifts the distribution over every answer. So a line you add to soften refusals does not stay in the refusal lane. It reweights how the model reads tone, urgency, and when to escalate, everywhere at once. That is why the win you can see and the regression you cannot tend to arrive in the same commit. The eval suite is what makes the invisible half visible before your users find it.
Spot-checks do not scale
Reading a few transcripts after each change feels responsible. It is not enough. On a busy day a hand check skims right past the candidate that diagnoses, or deflects, or drifts. You are sampling maybe fifteen conversations out of thousands, and the fifteen you pull are almost never the ones that moved.
There is a sharper problem underneath. The same prompt does not even give the same answer twice. Researchers at Thinking Machines showed that even at temperature zero, production language models are not deterministic, because your request gets batched with others on the server and the batch shape changes the floating point reduction order, which changes the output. The nondeterminism is not in your prompt. It is in the serving stack, and you do not control it. If you cannot reproduce a single run by hand, you certainly cannot certify a release by hand. Hand-checking assumes a fixed answer to inspect. There is no fixed answer, so the only honest certification runs the same inputs many times and reasons about the distribution, which is a machine's job, not a reviewer's.
Gate every change like a unit test
Run the full eval suite against every candidate, automatically, and block on regressions. Hamel Husain puts the cheapest assertion style checks at level one, meant to run on every code change the way unit tests do. A level-one check is a hard yes or no you can code, the answer never recommends a dosage, the crisis line appears when a user says the word, the response stays under the length the UI can render. These run in milliseconds and cost nothing, so they belong on every candidate.
OpenAI's evals guide says to make evals part of your CI pipeline so you hit the bar before you deploy, not after. Wire it the way you already wire tests. A change to the prompt, a model version bump, or a tweak to the retrieval step opens a pull request, the suite runs against that candidate, and a regression on a case that mattered fails the check and holds the merge. No human decides whether to run it, the same way no human decides whether to run the unit tests. The bar is not "the new version looks good." It is "the new version beats the old one and breaks nothing that mattered."
This is the same discipline evals-are-the-moat is built on, and it only works if the suite is measuring the moments that decide trust rather than generic quality. What belongs in it is the subject of what-to-measure.
A raw score bump can sit inside the noise
Comparing a candidate to a baseline sounds simple until you remember the scores are noisy, and the noise is usually bigger than you think. Say your suite has 400 items and the baseline passes 320, a score of 80 percent. The candidate passes 336, or 84 percent. Four points up. Ship it.
Not yet. Treat each item as an independent coin flip and the standard error on an 80 percent rate over 400 items is about 2 points, so a 4 point gap looks like two standard errors, comfortably real. Evan Miller, at Anthropic, shows why that math is wrong for most evals. Eval items come in related groups, several questions drawn from the same passage, several turns from the same conversation, several archetypes from the same persona, and answers within a group are correlated. When you ignore that clustering, you count correlated observations as if they were independent and undercount the true variance. Miller finds the correct clustered standard error can run more than three times the naive one on real evals. Push our 2 point naive error up past 6, and the 4 point gain is now well inside a single standard error. You have measured nothing.
The fix is to stop scoring the two versions in isolation and compare them paired, item by item. Run the baseline and the candidate on the exact same items, look at where each one flipped, and compute the error bar on the difference with the clustering accounted for. Paired comparison cancels the per-item difficulty, so an easy item that both pass and a hard item that both fail add no noise to the estimate, and the signal you are left with is only the items where the two versions actually disagree. Ship only when the paired difference clears its error bar, then watch production to confirm the win held. The failure you find there becomes a new eval item, and the loop closes a little tighter each time. That closing loop is its own discipline, covered in close-the-loop.
When the gate is not worth it
Gating has a cost, and it is honest to name it. A graded suite that calls a judge model on every item runs slower and costs real money per candidate, and if you run it on every keystroke of prompt tuning you will spend more on grading than on serving. So split it. Keep the level-one assertion checks on every candidate, since they are free, and reserve the expensive graded run for the pull request that actually proposes to ship. Early exploration does not need the full gate. The merge to production does.
The gate is also only as good as the suite behind it. A green check on a suite that never tests the escalation moment is worse than no check, because it grants false confidence in exactly the place trust is lost. Building the suite is the work, and it is not free. Budget for it the way you budget for the tests themselves. The same gate carries over when you change the model underneath the product, which is where choose-and-swap-models picks up the thread.
The rule to carry
No change reaches users until the full suite has run against it and the win clears its error bar. A prompt edit, a model bump, a retrieval tweak, all of them go through the same gate, because all of them can ripple. The suite catches the regression you did not think to look for, and the paired error bar keeps you from shipping noise as if it were a gain. The first version of this gate feels like overhead. By the hundredth change it is the only reason you can ship at all without holding your breath.
Sources and further reading
Work with Hunter Green