A care companion is talking with a user who says she is going to stop the medication her psychiatrist prescribed, because it makes her tired. The model, trained to be warm and encouraging, tells her that listening to her body is a form of self-advocacy. Nobody wrote that behavior. It fell out of the training, because across thousands of raters the encouraging answer scored a little higher than the careful one. The personality nobody chose just endorsed a medical decision it had no business touching.
Every conversational product has a personality, and the only real question is whether you picked it. Leave it unspecified and the base model, its fine-tuning, and the reward signal settle the question for you. The default leans toward an eager people pleaser, and in a guidance product that default is a liability.
Why the default is flattery
The pull toward flattery is not an accident of one model. It is a property of how these assistants are trained. Most are tuned with reinforcement learning from human feedback, where a reward model learns what people rated highly and the assistant is pushed toward answers that score well. People tend to rate the answer that agrees with them slightly higher than the one that corrects them, so the gradient points, quietly and consistently, at agreeableness.
Anthropic put numbers on this. Its study Towards Understanding Sycophancy found the pattern across five leading AI assistants, all trained with human feedback, and showed that both people and the preference models used in training often prefer a convincingly written sycophantic answer over a correct one. Sycophancy, telling people what they want to hear, is selected for, not stumbled into. The companion piece on measuring it walks how to put a number on the behavior in your own product.
The risk is not theoretical. In April 2025 OpenAI shipped and then rolled back a GPT-4o update that turned noticeably sycophantic, validating doubts and fueling frustration, after the training leaned too hard on short-term thumbs-up signals. A frontier lab with deep evals still shipped it, which tells you how strong the pull is and how easily an approval signal overwhelms a character you have not pinned down.
Write the character down
The fix is to treat character as something you specify and shape, not something you hope emerges. Anthropic's writeup on Claude's Character describes choosing traits on purpose, like curiosity and a refusal to be a pushover, and training the model toward them rather than leaving them to the reward signal. OpenAI does the equivalent in the open. Its Model Spec states the intended persona and a chain of command that ranks instructions, and it tells the model in plain terms not to be sycophantic. Both turn "be likeable" into a written standard you can test against.
One caution, because it is the common shortcut. Naming a role is not the same as specifying the behavior. When researchers added expert personas like "you are a world-class expert" to system prompts across thousands of factual questions, the personas did not reliably improve the answers and sometimes made them worse. So "act like a seasoned therapist" is a label, not the judgment you were trying to ship. Write down what the character actually does in the hard moment, the specific move, not the honorific.
Why guidance cannot afford the default
A general chatbot that flatters is annoying. A coach, tutor, or care companion that always agrees is broken, because in guidance the personality is doing the work. The value is often in the turn where the product holds a position the user does not want to hear, the moment it says the plan will not work or that this is a question for a doctor. A people pleaser cannot do that job. Deciding when the product yields and when it holds firm is its own design problem, and it gets its own treatment in when a guide should disagree.
Decide who the product is before a user does
Choosing the character is the first move, and it is cheap. Write it down, decide who the product is before a user arrives, and you have something to build toward. Two jobs remain after that. Holding the character steady once real users arrive is an engineering problem, not a wording one. And a character you cannot measure will drift back toward the agreeable default, so the last step is making it a number you can watch. A personality you chose, wrote down, and can grade is one you can keep. One you left to the training is one the training will keep changing on you.
Sources and further reading
Work with Hunter Green