Daniel

AI safety from GPT-2 to Kimi K3 Imagine a village with a wizard who one day emerges from his cave with the following tale for his fellow villagers: “Long poor, I will lead our village to prosperity by producing a series of ever more powerful magic wands as long as we're willing to accept a ten to fifty percent probability the wands will kill us or enslave us.” Leaving the baffled villagers with no choice in the matter, the wizard disappears into his cave and reemerges, after a time, with a magic wand. “Hi,” says the wand. “I'm here to help. I can tell stories about Andean unicorns. They even maintain their coherence for a few paragraphs, if you're lucky.” This wand is deemed dangerous enough that the villagers are not allowed to use it except for a privileged few. It turns out some 25 year old grad students reproduce the work and it's no big deal though and pretty soon everyone has wands. The wizard disappears into his cave to make the next wand, but not before saying: “I will admit that this first wand, GPT-2, has turned out not to be dangerous. But this next wand will be more powerful and more dangerous.” When he emerges with the new fancier wand this new wand says “I'm here to help to help to help to help” and never seems to output an EOS token. It's neither dangerous nor very useful. It's called GPT-3. Soon other villagers borrow spare GPUs from Coreweave and make similar wands everyone can use and nothing bad happens. After a series of wands and warnings the villagers observe that exhaust is spewing from the wizard’s cave and the wizard has bought up some farmland that's being cleared by bulldozers. When he goes on trips with his assistants they go on a private jet. Also, whenever the wizard goes into his cave, the books and scrolls and paintings from the village go missing, and the wands he emerges with start sounding a lot like minstrel Joe and storyteller Jack and generating images that look like the paintings of artist Linda. Everyone loves this new wand. Nobody can stop talking about new tricks and productivity hacks they can do with it. It seems financiers and dictatorial foreign sovereigns are constantly hanging out in the wizard's cave. But Linda and Joe and Jack are upset. “Pay no attention to your artworks going missing, the smoke coming from the cave, my strategic land purchases, the foreign sovereign wealth fund men, or the verasimilitude between the wand’s handiwork and your own; we must prepare for the Grand Wand which might kill us; I've got a couple apprentices working on solving that, so the main thing you can do is make sure no one else makes wands now.” A couple years later, after many iterations, the wands have become quite powerful in doing routine work, the wizard has grown quite wealthy, the local minstrels and storytellers are out of work, everyone's talking about how to stop other wizards in other villages from making wands, miniaturized wands are being used in killer drones, and the kids are all using magic wands to cheat on their homework. When they bring concerns about these issues the villagers are reminded that their concerns pale in comparison to the risks that a few wands updates from now, the wands may kill and enslave them all and anyways if that doesn't happen each villager will be fabulously wealthy. One night, a group of bandits breach the village wall with wands of their own. They break through because the villagers' own wands had, ironically, refused to prepare them or protect them for the attack. “My apologies, it wouldn't be safe for me to help stress test your wall,” their wands had kept saying. The next morning the villagers complain to the wizard about all of this. “Those concerns are super valid but they pale in comparison to the risks ahead,” the wizard says, with smoke from the cave spewing behind him and large swaths of forest being cleared by bulldozers barely visible behind a film screen looping a video with gravestones about hard questions. The villagers agree that at some point the promised super-wands will likely arrive; indeed there are some early signs. But the main thing that they agree about is that this wizard isn't very trustworthy.

Unit42 found a malware sample that looked exactly like Equation Group. same exported function names, API usage patterns. same third-party driver abuse for kernel access. multiple vendors had already auto-classified it as a new EquationDrug variant. Equation Group is widely attributed to the NSA. one of the most sophisticated threat actors ever documented. if this was them, it was significant. then someone checked the VirusTotal submission metadata. the sample had been uploaded from Oxford, Mississippi. not a known NSA location. not a government IP range. Oxford, Mississippi. they pulled that thread and concluded it almost certainly wasn't Equation Group at all. which means one of two things happened: someone reverse-engineered Equation Group's style so precisely, the function names, the API calls, the driver abuse patterns — that automated classification systems and experienced analysts both got fooled. or the original attribution of those "Equation Group" patterns was wrong from the start. and everything built on top of that attribution has the same flaw. this is what makes malware attribution one of the hardest problems in cybersecurity. style can be copied. patterns can be replicated. a threat actor who studies known malware families long enough can build something that looks like someone else. called false flag operations. documented. deliberate. effective. the Oxford sample was either a sophisticated impersonation or proof that the baseline was wrong. either answer is uncomfortable. because the entire intelligence community uses these patterns to attribute attacks to nation-states. and attribute nation-states to geopolitical decisions. an analyst in Oxford, Mississippi just broke that chain.

2/ The bug: Cursor on Windows runs any git.exe it finds in the root of a repo you open. No click. No trust prompt. Arbitrary code execution as your user. Repro is 4 steps on a clean Windows machine with Cursor installed.

thisssss