A user tells a coaching app, once, about a rough patch in her marriage. Weeks later she opens a session to work on a promotion, and the assistant leads with "given the stress at home, maybe we start smaller this quarter." She never raised her marriage here. The product did not do anything wrong on paper. It remembered, and it connected. It also just told her, without being asked, that it holds a file on her private life and will bring it up whenever its own sense of relevance says so. That is the moment a helpful assistant starts to feel like one that is watching her.
The instinct with memory is to keep more of it. Remember everything, the thinking goes, and the product feels more personal. In practice, undisciplined recall is the fastest way to break the trust it was meant to build. The failure is not too little memory. It is memory with no boundary on where it travels.
The dossier the user cannot see
When a system folds every past chat into the next one, it builds a profile the person cannot see or predict. Simon Willison, writing about ChatGPT's cross-chat memory, objected not to memory itself but to its opacity, a model of him he had no way to inspect or correct. A user who cannot see what the product believes about them cannot tell whether the next answer is shaped by a fact, a stale guess, or something they said in a very different mood two months ago.
Underneath the opacity is a context failure. Detail that belonged in one conversation resurfaces in another where it does not fit, which is the contextual integrity line memory has to hold. Models cross that line more often than you would guess, and the trust-surface briefing carries the measured rate. The point here is narrower. Even a model that recalls accurately can recall inappropriately, because what counts as relevant to the model is not the same as what counts as appropriate to the user.
Why more history is not more help
Two forces turn extra memory into a liability. The first is staleness. People change jobs, end relationships, drop goals, and a memory store that keeps the old fact will confidently apply it long after it stopped being true, so more history means more chances to be wrong about the person in a way that feels personal. The second is misfire. The common defense, only retrieve the relevant memory, does not save you, because relevance is judged by the model, and a system optimizing to seem attentive will over-recall to prove it was paying attention. The behavior that scores well in a demo, look how much it remembered, is the same behavior that reads as surveillance in production.
Scope beats hoard
The fix is not less memory, it is bounded memory. Memory scoped to the project or the session it was made for cannot bleed into the next one, because it never travels there. Anthropic builds that in, scoping memory per project with an incognito mode that saves nothing. Scope is a stronger guarantee than a well-behaved model, because it removes the path instead of trusting the model not to take it. A boundary you enforce in the architecture holds under a model swap, a jailbreak, and a bad day, and a boundary you only prompt for does not.
The second move is to default to remembering less and proposing the rest. A memory the product offers to keep, "want me to hold onto this for next time," is one the user chose and can picture. A memory it forms silently is one they find out about when it surprises them. The same detail crosses very differently depending on which of those two paths put it there.
The cost of scoping, and where the rest of this lives
Scope is not free. Draw it too tight and you lose the continuity that made memory worth having, so the user re-explains themselves every session and the product feels like it has amnesia. The dial is per context, not a single switch for all of memory, and setting it is a design decision you make moment by moment, not once. Getting it right is the same discipline as any other behavior. Decide the rule, then test that recall stays inside it.
Scoping is half the answer. The other half is letting people see and change what you keep, the legibility the trust-surface briefing lays out in full, and deciding what you are allowed to retain and train on, which is its own question. Memory earns its place when it is scoped, legible, and proposed rather than hoarded and hidden. Held quietly and forever, it is a liability wearing the costume of a feature.
Sources and further reading
Work with Hunter Green