What to actually measure in conversational guidance

It’s easy to measure the things that don’t matter: messages sent, sessions started, tokens spent. The hard part, and the only part that counts, is measuring whether the product actually helped the person in front of it.

Process before outcome

Outcomes are slow, noisy, and hard to attribute. Process metrics — did the system follow the method, in this turn — are fast and actionable. If the process is right and the method is sound, the outcomes follow.

Adherence is the number you can move

Turn the method into graded criteria and score every interaction against them. Adherence to the method is a number you can watch, regress on, and improve, release by release. It is the closest thing to a real quality signal that updates daily.

Watch the trust moments

A handful of turns decide whether someone comes back: the boundary request, the moment of doubt, the hard disclosure. Instrument those specifically. The average looks fine right up until the moment that wasn’t.