David Pfau

Von Neumann’s grave, Princeton, taken today

the pope and anthropic's co-founder just stood together at the vatican to release "magnifica humanitas," the first ever catholic teaching on AI yes, you read that right. the full ceremony was 2 hours. here's the most interesting things for you to know: 1. this is the biggest religious response to AI in history. popes only put out a handful of these huge official letters in their entire time as pope. the fact that one of them is about AI tells you how seriously the church is taking what's coming. 2. small detail with massive meaning: this pope picked the name "leo XIV" on purpose. the last pope named leo was leo XIII back in 1891, and his most famous act was writing the church's response to the industrial revolution. picking the same name is a deliberate signal. this pope sees AI as the new industrial revolution. 3. the catholic church does this every time a major technology reshapes humanity. they wrote "rerum novarum" in 1891 to respond to the industrial revolution. when nuclear weapons threatened the world in the 1960s, they wrote "pacem in terris." climate change and runaway tech got "laudato si" in 2015. now AI gets "magnifica humanitas." they don't issue these often. 4. the pope's main line: "AI needs to be disarmed." he literally compared AI to nuclear weapons. he said the church spent decades pushing for nuclear disarmament because the technology was too dangerous to leave in the hands of a few. he says AI is now in that same category. 5. anthropic co-founder christopher olah told the pope, on stage at the vatican, that anthropic's own research team keeps finding things inside their AI models that "mirror joy, satisfaction, fear, grief, and unease." 6. olah's reframe of what AI actually is: these things are grown. they're trained on a structure roughly modeled after the human brain and fed everything humans have ever written. in his own words: "they are made from us, from our words." he said even the people building them don't fully understand what's happening inside. 7. olah publicly admitted that every AI lab, including his own, faces pressure that can conflict with doing the right thing. commercial pressure to keep shipping, competitive pressure from other labs, plus the older pressures of pride and ambition. his solution: we desperately need outside critics with no skin in the game who will tell the labs when they're failing. 8. olah says there are 3 giant questions the AI labs cannot answer alone and the world needs religion and philosophy to step in on: > how do we make sure poor countries actually benefit from AI? > what does human flourishing even look like in this new world? > and what are these things we're actually building? 9. one of the sharpest lines in the whole encyclical: "the promise of automatic general prosperity often proves illusory." translation: the idea that AI will just make everyone rich on its own is a fantasy. someone has to actually design the system so the benefits get shared. 10. the pope also pulled out a 100-year-old quote: "contemporary man has not been trained to use power well." said by a theologian back in the 1920s. the whole encyclical is basically a long argument that we need to learn how to use this kind of power before it uses us. 11. the pope kept stressing that he doesn't have the technical answers. but he says the church has thousands of years of wisdom on what it means to be human, and that wisdom is exactly what's missing from how we're building AI right now. his closing line: this technology should serve "human flourishing and human dignity, not control consciences."

Kind of embarrassing watching the pope angling for an MTS gig at Anthropic like this.

academics are unprepared for the coming world where much scientific progress is majorly a function of inference compute. whether OpenAI points the Eye of Stargate at your particular field will decide its acceleration. talent will leach away into the labs. it's already begun

🚨 Google DeepMind CEO Sir Demis Hassabis: “Today’s systems, are nowhere near [AGI]. Doesn’t matter how many Erdős problems you solve… I think it’s far, far from what a true invention or someone like a Ramanujan would have been able to do” it’s over for the Erdős hype

The Lloyd's building at night, London, England, 1989

no one could have foreseen that the butlerian jihad will be catholic