When remembering backfires

The instinct with memory is to keep more of it. Remember everything, the thinking goes, and the product feels more personal. In practice, undisciplined recall is how a helpful assistant starts to feel like one that is watching you.

The dossier problem

When a system folds every past chat into the next one, it builds a profile the user cannot see or predict. Simon Willison, writing about ChatGPT's cross-chat memory, objected not to memory itself but to its opacity, a model of him he had no way to inspect. Detail from one conversation surfacing in an unrelated one is its own failure, and the ConfAIde benchmark measured how often models cross exactly that line.

Scope beats hoard

The fix is not less memory but bounded memory. Anthropic scopes memory per project so work does not bleed across contexts, and offers an incognito mode that saves nothing. That maps onto Helen Nissenbaum's contextual integrity. Information that is appropriate in one context becomes a problem when it flows into another.

Let the user steer

The other half is control. OpenAI's memory ships with ways to ask what it knows, tell it to forget, and turn it off. Memory earns its place when it is scoped, legible, and in the user's hands. Hoarded and hidden, it is a liability wearing the costume of a feature.