The experience is made or lost in the handoffs

Once a conversational tool is wired into a client's systems, the experience lives in the seams. A handoff is any place control passes, from the agent to a tool, from a tool to a client system, from sync to async, from a live stream to a stored log, from the agent to a human. Each seam adds latency, can drop state, and can leak data, and each one is usually the part nobody designed. The felt quality of the product is decided there, not in the model. Set the latency budget, stream what you can, make failures safe, and design the human handoff on purpose.

Stream, because waiting is the felt cost

A long answer that arrives all at once feels broken even when it is fast. Streaming fixes the feeling by showing the reply as it is written. OpenAI's own guidance notes that with streaming you can start processing the beginning of the completion before it is finished, and in its example the first token arrived after about a tenth of a second while the full answer took several. On the web this rides on server-sent events, a one-way stream from server to client over plain HTTP. Anthropic's API streams the same way, with a defined sequence of events and an explicit instruction to handle unknown event types gracefully, so a new event never breaks your client.

Voice tightens every budget. A spoken turn has only a few hundred milliseconds before the user expects a reply, which leaves no room for a slow tool call or a model-based check inside the turn. OpenAI recommends WebRTC over WebSockets for its realtime API to keep the connection consistent. On a voice surface, keep the slow work off the turn and let it shape the next one.

Make the action safe to retry

A tool call that takes an action crosses a boundary, and boundaries fail. A network blip after a booking or a payment leaves you unsure whether it happened, and a blind retry books it twice. The fix is an idempotency key. Stripe's API lets a client safely retry a request without performing the same operation twice by attaching a key it remembers, and it holds the key for 24 hours. Carry a key on every action that changes state, and a retry runs once.

Retry the reads, but retry them politely. Exponential backoff with random jitter is the standard, and the jitter is the part teams skip. Without it, clients that fail together retry together and stampede the service, the thundering herd that turns a blip into an outage. OpenAI's cookbook shows the same pattern, backoff plus random jitter so retries do not all hit at once.

Set the bar as a number you can hold

The experience is not a vibe, it is a set of numbers a monitor can watch. Borrow the discipline from site reliability engineering. Google's SRE book defines a service level objective as a target for a measured level of service, and it warns against picking the target from current performance, because that locks you into defending whatever you happen to do today. Set the bar the experience needs, a first-token time and a tool-call success rate over a window, then hold the deployment to it.

Plan the failure path too. The same book describes graceful degradation under load, serving a simpler answer that is cheaper to compute rather than failing outright. A tool that answers a little worse when a connection is slow keeps the user. A tool that hangs loses them.

The human handoff is the one to get right

In a client deployment, the handoff that decides trust is the one to a person. Done wrong, the user repeats their whole story to a support agent and the product feels worse than no product. Done right, the person arrives equipped. The contact-center pattern has a name, a warm transfer, passing context to the receiving person before handing off. For an AI agent the same idea means the transfer preserves the full history and a summary and keeps the user in the same window. Design what state carries across that seam, and what happens when no person is available, before a user finds the gap.

List the seams on purpose

A handoff you did not design is a handoff that will surprise you. Standard tracing helps you see them. The OpenTelemetry semantic conventions for generative AI define spans for each model and tool call and a metric for time to first token, so a slow turn has an address instead of a shrug. List every seam, decide what carries and who owns the failure, and the experience stops being luck. The workshop has a step for exactly this, and it feeds the logging and audit plan the client's security team reads next.