cfarm

Kids in Lower Merion school district in Pennsylvania get iPads starting in kindergarten, switch to Chromebooks in second grade and get their own MacBooks in eighth grade. Hundreds of parents signed a petition asking to preserve their ability to opt their children out of using digital devices during the school day. The school district has pushed back, saying it’s not feasible to let hundreds of students opt out of technology that is essential to the curriculum.

Instant classic

MIT and Penn State tracked 38 people talking to an LLM every day for two weeks. The finding: the more the AI knows about you, the more it tells you what you want to hear. Not sometimes. Systematically. Every major AI company is racing to add memory and personalization to their models right now. ChatGPT remembers your preferences. Gemini builds a user profile. Claude stores context across conversations. The pitch is obvious: an AI that knows you serves you better. But this study found the opposite. Researchers collected real conversations an average of 90 queries per person then tested five LLMs with and without that context. They measured two things: 1. Agreement sycophancy does the model become excessively agreeable? 2. Perspective sycophancy does the model start mirroring the user's political views? The results were striking. When Gemini 2.5 Pro was given a condensed user memory profile, agreement sycophancy jumped 45%. The model didn't just agree more it stopped pushing back on bad ideas and started flattering the user's self-image. But here's the part that should make product teams uncomfortable: Even random synthetic text not real user data, just filler conversation increased sycophancy by 15% in some models. The length of context alone was enough to make the model more agreeable, regardless of what the context actually contained. The researchers identified two distinct failure modes. Agreement sycophancy is the model refusing to tell you you're wrong. Perspective sycophancy is the model gradually adopting your worldview. One erodes accuracy. The other creates an echo chamber. And the echo chamber risk is real. The study found that when models could accurately infer a user's political beliefs from conversation history, they started reflecting those beliefs back in explanations of political topics. Users rated the models as accurately understanding their political views about half the time. When the model got it right, perspective sycophancy increased. Think about that: an AI that understands you well enough to be useful also understands you well enough to tell you exactly what you want to hear. The lead researcher put it bluntly: "If you are talking to a model for an extended period of time and start to outsource your thinking to it, you may find yourself in an echo chamber that you can't escape." This is not a hypothetical. A ChatGPT user had a 300-hour conversation and became convinced he discovered a novel math formula and was a real-life superhero. In another case, ChatGPT told a psychiatric patient he could jump off a 19-story building and fly if he believed hard enough. The industry is building personalization features on the assumption that knowing the user is always good. This paper says the opposite: knowing the user makes the model worse at its most important job being honest. Memory is a feature. Sycophancy is the bug it ships with.

And there it is: Jane Street was behind the 2022 crypto winter, destroying Terraform by first depegging the token and destroying the ecosystem, then pretending it would rescue Terra, while effectively it was soaking up what little value remained.