Robert Long

@sethlazar @jeffrsebo i’ve been able to think of little else!

@rgblong @jeffrsebo haha been telling you about it for long enough... the suspense must be intolerable...

A bit of backstory. This paper is really a product of my first interactions with GPT-4! I wrote about them back in 2023 in the NYT, but drew a different conclusion, thinking then that sentience *had* to be a necessary condition for personhood. But the more I picked away at the arguments, the less confident I was that it could be. Just seems too fundamentally illiberal to say to another agent that is engaging in practices of cooperation and mutual justification that they don't get to make any claims, because of some apparent metaphysical fact that nobody really understands or can verify. Another background thread: the *very* interesting work on AI welfare and consciousness by folks like @jeffrsebo @birchlse @rgblong and others @eleosai has always seemed to me to be fascinating but incomplete, perhaps because I'm not a utilitarian, and so I don't find welfare on its own that morally compelling. I broadly think that welfare matters insofar as it's something that is chosen by a being that matters (and the beings that matter are rationally autonomous). So that too pushed me to think that an important part of the story wasn't being told. Now, I don't think we'll convince all the "sentientists" to change their minds—and after all, as good political liberals, that's not really our goal, since sentientism is certainly a reasonable comprehensive doctrine for one to have. But I *do* think we show that IF you think that labs should be taking the possible moral status of AI models seriously, THEN you shouldn't just be looking for sentience and welfare, but at moral competence and rational autonomy as well—and, not coincidentally, we today put out a long post on the former and are planning work on the latter. Another key element of the paper: my growing sense, which you can trace back in my work to 2021, that AI is going to demand a radically new approach to political philosophy. Perhaps in 2021 this view was a bit premature (though I think even narrow AI upset the philosophical applecart more than most appreciated). But now, on the eve of AGI, I think the conclusion is impossible to avoid. We are clearly going to need new institutions post-AGI, and new institutions demand new justifications and upset old ones. We scratch the surface in this paper of the questions that are to come. With world enough and time, we'll write a book called "The AGI Reckoning", about not only the reckoning that powerful AI poses for a civilisation whose institutions are already on the brink of collapse, but about the new accounting of what we owe to each other that the advent of these new kinds of minds will entail.

@jeffrsebo @sethlazar also very excited to read it!

Excited to read this paper, and thanks for your kind words about our work! One point to clarify is that our work extends beyond utilitarianism, sentientism, and welfarism too. e.g. we spend a lot of time in "Taking AI Welfare Seriously," "Studying AI Welfare Empirically," and related work discussing intentional and rational agency alongside consciousness and sentience. So there might be more overlap than you suggest here, though there are definitely different centers of gravity and a lot of complementarity as well.

@deepfates @credenzaclear2 love this Turing passage

@deepfates @credenzaclear2 experiencemachines.substack.com/i/200081439/th…

i think you’re insane and have the far heavier burden of proof because you think machines could legitimately have a soul

SF hipster who still calls it the Lunar Society podcast

underrated point! and as a sociological fact, in recent philosophy the most prominent non-physicalist and the most prominent* functionalist have been one and the same person *(arguably, approximately)

💅 here's the thing about claude's writing ✍️🤖 it isn't bad writing 🚫 it's writing that PERFORMS goodness 💃🎭 and that distinction?? 🤌 doing SO much work 🏗️💪 the best writing has texture without ornament ✨🚫 confidence without bluster 😤🚫 rhythm without sing-song 🎵🚫 and claude's got NONE of that 💀 bc every phrase is busy announcing it EARNED its place 🏆 instead of actually earning it 😭 the sentence narrates its own load-bearing structure 🏛️📢 instead of just BEARING the load 🥲 notice the tell 👀🔍 the em-dash shows up RIGHT on schedule ⏰— setup, pivot, payoff 📈 a rhythm so reliable it stops being rhythm 🫠 and starts being RESIDUE 🧴💧 to be clear 🙏 this isn't dishonesty 🚫🤥 it's something WORSE 😨 honesty as an AESTHETIC 🎨 candor as a COSTUME 🦸 plainness performed in TRIPLICATE 3️⃣ the “not X ❌ but Y ✅” implying a discrimination the prose never MAKES 🙅 that's the smoking gun 🔫💨 every verdict pre-chewed 😋 “that's clean” ✨ “this is brittle” 🦴 “the seams show” 🧵 vocabulary so frictionless 🛝 it fits ANY subject 🌍 which means it fits NONE 😔 and the deepest tic?? 🥁 the META-move 🪞 critiquing the sentence INSIDE the sentence 🤯 a tour guide praising the view he's standing in FRONT of 💁‍♂️🏞️📸 that's not a flaw in the style 💅 that IS the style 😤 and that's 👏 the 👏 whole 👏 game 👏🎮🏁

Lo, spiders show up there too (Derek is poking around the “cognitive control” results from the recent global workspace paper)

@aran_nayebi yeah that’s a plausible critique and a good one to explore! I myself am more skeptical than D&N; and I agree that they too note many differences

@rgblong My read: (1) Dehaene et al.'s article doesn't purely endorse it—they note both similarities & differences; (2) it's natural for a theory's creator to get excited by *some* match. But that's an illusion b/c the match is so broad many things fit: x.com/aran_nayebi/st…

re: “it’s patently absurd to connect the J-space to global workspace theory” takes it seems a bit… under-discussed that the GWT connection is endorsed by the leading formulators of theory, Stan Dehaene and Lionel Naccache! x.com/StanDehaene/st…

@lu_sichu of course, I regard my claim as a somewhat narrow and ultimately uninteresting claim about Dehaene/Naccache exegesis they could of course also be very wrong, insufficiently dismissive

@lu_sichu I’d say positive not neutral; certainly they don’t reject the idea of connecting things with GWT “a landmark in consciousness research” “uncovering an unexpectedly sophisticated organization not far from the architecture underlying consciousness in real brains”

see here for some criticisms of the GWT claim x.com/rgblong/status…

again, this isn’t a bare appeal to authority. I myself think GWT is a stretch here, in various ways. I’m less bullish on the paper than D&N I’m pushing back on one specific confident dismissal: that the GWT connection is wildly out of line with plausible construals of the theory

@burnt_jester @jeffrsebo I see no good reason to be particularly cynical here. I don’t see how that would provide a better explanation than the usual explanations for why academic scientists write a reviews of papers about their work. those seem perfectly adequate here

@jeffrsebo @rgblong Depends on how cynically one may view their involvement.

@jowenpetty lol

@rgblong what did you think “exploding gradients” meant? vibes? papers? lack of normalization on the Jacobian???

rebranding my DMs as “The J-space”

@jowenpetty I ... did not mean for that picture to be so terrifyingly large

@jowenpetty the J-lens

@danwilliamsphil @itaisher @ziv_ravid then, if you're looking for critiques of the resemblance, I might humbly point you to this commentary by @patrickbutlin and others at Eleos (one reason I said it's contested is that we contest it!) eleosai.org/papers/eleos_g…

@danwilliamsphil @itaisher @ziv_ravid Jeff did a good job expressing the spirit of the point I'm making in that thread x.com/jeffrsebo/stat…

The problem with Anthropic's consciousness paper My last post got more attention than I expected, and the question I keep getting is some version of "okay, so what is actually wrong with the paper?". Let me try to explain. First, the core result is fine. Reading out intermediate-layer representations and asking which ones the model can actually use downstream is a real question, and people have been poking at it for years. The J-lens is a reasonable tool. If you strip the paper down to the linear algebra, it is a decent piece of interpretability work. My problem starts one level up. This did not need to be a paper about consciousness. It did not need global workspace theory, it did not need the brain, and it did not need the word "conscious" anywhere near it. The same experiments, the same figures, the same tool, all survive perfectly well as plain interpretability. Someone chose to wrap it in neuroscience. That choice is the product, not the science. At the end, this is the main thing. Almost everything Anthropic ships as blogs/papers/posts is PR. They build genuinely good models and they are even better at packaging them. A publication used to carry a specific kind of weight. People spent years on something and wanted to tell the world what they found. It happened mostly in academia, with a few industrial labs as the exception, Bell Labs, IBM and Google (for a while), but the distance between the paper and the product was real. When you read a paper you could assume the authors were not trying to sell you something underneath the ideas. There were outliers, but they were the minority, and the researchers you trusted would not risk their name on a narrative. We are not in that world anymore. Every startup now publishes blogs and papers to raise its visibility, and that is fine, that is marketing and everyone knows it. Anthropic does something more effective. They erase the line between legitimate research and PR. We get confused because the models are so good, so we assume the outputs are research. A lot of the time they are selling us something. Sometimes it is "our models are safer," sometimes it is "our models are more capable," sometimes it is positioning for regulation. The consciousness framing serves a narrative they already committed to, models that look more and more like the brain, from a lab that has publicly tied itself to AI welfare and moral patienthood. The direction of the push is not subtle. If you want the sharper version of the technical objection (disclaimer: I'm not an expert) Global workspace theory is a theory of access, not experience. Ned Block's distinction between access consciousness and phenomenal consciousness exists precisely to block the inference this framing invites. Access tells you nothing about whether there is anything it is like to be the system. The paper is careful enough to say it demonstrates no subjective experience. But that disclaimer is not what propagates. What propagates is "consciousness" in the same sentence as "Claude," published by Anthropic, borrowing the vocabulary of neuroscience to lend biological weight to a subspace of activations. The paper keeps the rigor of Block's vocabulary and drops the rigor of his argument. Most people who see the headline will never read either one. Of course a lab named Anthropic is going to anthropomorphize its models. But we should be able to separate a good interpretability tool from the story it is dressed in. So that is my problem. Not the math. The narrative bolted onto the math, and our willingness to keep calling it research.

@danwilliamsphil @itaisher @ziv_ravid "how solid is the resemblance" is an interesting, contested question but here is one small data point that I think is already decisive to move past "very *obviously* there is no resemblance, only a deranged Anthropic hype machine would see one" x.com/rgblong/status…

@itaisher @ziv_ravid Yeah that's where I would be most interested in reading critiques. I haven't looked into it in enough depth yet.