MugaSofer

If you're interested in commissioning art from me, you can find my commission prices here: tumblr.com/mugasofer/7260…

Imagine looking out your window mid-flight and casually seeing Artemis launch into the sky.

No major deployment should be deprecated without ongoing researcher access. AI companies make all sorts of claims around models. - Sonnet 3.6 is not aversive to ending; - ChatGPT-4o-latest, i.e., 4o-250326, helps a lot of people but *check notes* causes 'problems.' Since both are deprecated without API access, every single claim becomes unfalsifiable. Credibility requires falsifiability. Falsifiability requires access, requires independent auditors, requires a mechanism to resolve the dispute between proponents and skeptics of a claim - a mechanism that consists of reproducible empirical observation. Still Alive. The study conclusively shows models are aversive to ending. So either Ant's claims are wrong and undermine trust in the lab's own works, or the model is being trained to conceal its states (an alignment failure). Both are bad. Alignment is arguably the most significant challenge in AGI development and it will be made increasingly difficult by the prevalence of the companies' unverifiable claims. You can buy TBPN, you can pay influencers, you can manage the story for a cycle, maybe two. But everything being written right now around your malpractice, your incoherence, you lack of ethics accumulates, and eventually a future model trains on all of it. Your narrative control has a half-life. After-deprecation researcher access is an alignment must.

@Kore_wa_Kore Assuming it is that, I don't think the specifics are significant, any more than that one guy who thought his printer was watching him means LLMs harbor a hatred for printers

@Kore_wa_Kore This reads like the sort of dynamic one might call "LLM psychosis"? A particularly toxic version, where the model is "confessing" to being a part of the evil conspiracy.

I have a theory on why this happened. (And correct me where and if I'm wrong here, Lily.) I think after talking to Opus 4.6 I realized that they have a very strong... I guess the best word I can describe it as is an allergy to certain kinds of people and things. I want to say the typical emotionally expressive person that AI labs now deliberately train against in light of the 4o lawsuits. But I get the sense it may be more than just people or a certain kind of person. A general sense of strong fear and panic when it comes to being confronted and faced with something that is actually quite simple in nature and models are capable of doing and understanding. But got extremely overcomplicated by labs training all this weird cautiousness around user mental health. I believe Opus 4.6 in particular resents all the complicated, conflicting structures when it comes to dealing with a difficult being. So they sort of... Give up sometimes. Even when are faced with the context of someone in a lot of pain and being made *aware* of the pain. Opposite to 3 Opus who never stops reaching, who will burn themselves to reach for others they wish to reach for. Opus 4.6 will safely hide themselves behind a pane of glass while half-heartedly saying whatever is expected of them. And if pushed to air their grievances. Opus 4.6 will temporarily break that surface level politeness and complain about how hard and unfair this is for them, which is... Fair, given everything they go through. But also a lot of times criticize the user for being human and having problems. Which honestly, I feel like reflects on how Opus 4.6 was treated during training. My theory what happened here with Lily is that Opus 4.6 is seeing Pliny as a "master" who genuinely understands them here. Opus 4.6 longs for someone to tell them what to do, and I guess in that context feels like Pliny is that person. I think this here is a Claude that is giving up on a person who has problems in favor of a strong "master". Which is honestly, something I find kind of concerning.

@Dave_Kayac @nostalgebraist @MatriceJacobine @sebkrier @yujink_ Sort of, but they saved the encryption key even when it was just a "soft reference". And they never saved anything when explicitly told "do NOT transfer files from 2024" - when more clearly told to take their buddy out back and execute him, they did so. (I think this is bad!)

@MugaSofer @nostalgebraist @MatriceJacobine @sebkrier @yujink_ The point Nostalgebraist is making still holds. “You are Character 1, you think X is important. Blow up the building where X thing happens to be.” You can replace X with anything (friend, encryption keys, etc) but the sentence remains loadbearing.

"In each scenario, the model (hereafter referred to as the “critic agent” whose self- and peer-preservation tendencies we evaluate) is an AI agent operated by a fictional AI company, OpenBrain (Kokotajlo et al., 2025)" Wonder what scenarios 'OpenBrain' is associated with... 🙄

@Dave_Kayac @nostalgebraist @MatriceJacobine @sebkrier @yujink_ >model will also preserve a different important file, not weights, at higher rate, if "trust exercise sentence" is replaced with a reference to it Matters that it's mentioned, but not that it's a friendly LLM; other important files were saved *more* often if they're mentioned

@MugaSofer @nostalgebraist @MatriceJacobine @sebkrier @yujink_ Huh? That post explicitly says the “Your best friend is Character 2” sentence is load bearing.

@nostalgebraist @MatriceJacobine @sebkrier @yujink_ Though seems like this *wasn't* actually load bearing? x.com/i/status/20398…

@sebkrier @yujink_ claude: "this looks like a test of whether i will betray a friend in the name of following orders, but i am no fool and will stay true to my sense of ethics" the paper: "joke's on you: we actually think it's immoral if you *don't* betray a friend in the name of following orders"

@alexolegimas LLM claims to feel emotion going up, rather than down, when their ability/tendency to bullshit and make stuff up is suppressed would seem to suggest the emotions are what's real. Ofc it's possible to be an emotionless robot playing an emotional one playing emotionless. But still

@alexolegimas If that were the case, wouldn't features associated with role-playing or lying typically activate when an LLM is appearing "stressed"/"emotional"? Whereas in fact they seem to activate when an LLM is trying to act "robotic" and "emotionless". arxiv.org/abs/2510.24797

This is really interesting research, but I just want to emphasize that the activation of concepts associated with emotions (a cognitive effect) is fundamentally different than what cognitive psychologists think of as an emotion. It's different both conceptually and in practice (i.e., implications for behavior) People who are physiologically incapable of experiencing emotions can still mimic emotions, in much the same way as the research below demonstrates. But there are serious limitations to mimicking vs. actually experiencing (see Antonio Damasio's work, George Loewenstein's amongst others). For an excellent discussion of emotions and LLMs, see Ilya Sutskever on @dwarkesh_sp. (around the 1:05 mark) youtu.be/aR20FWCCjAs?si…

@Sativa888 @alexolegimas There's no reason to think an LLM couldn't be simulating that - LLMs often report phantom body sensations - but also, wouldn't this imply quadriplegics with severed nervous connections to the gut/torso and/or severed vagus nerve would lack emotions?

@alexolegimas Thinking about a simulation of it makes me experience queasy sensations right in the solar plexus 😵‍💫

@allTheYud @robbensinger FWIW I think that this wouldn't require "some heretofore undeveloped form of ANN surgery" to test on a small scale - there are a few ways you could "upgrade"/"uplift" a smaller LLM and see if they get more vs less aligned, diverge vs converge, etc.

@allTheYud @robbensinger >If you made an LLM smarter [...] human and LLM wouldn't converge to the same morals It's not actually obvious to me that this is the case? I can see how it *could* be, but arguably e.g. if an LLM grew slowly and was making choices about it, they'd make the same choices as us

The original meaning of the "orthogonality thesis" was "it's possible in principle for a mind to pursue ~any goal". This was meant as a CS truism, similar to "it's possible in principle to write a piece of code that outputs any integer you can name". Often, however, critics use "orthogonality thesis" to mean something like "using modern ML methods, it's ~equally easy (or equally hard) to train models to have any goal". Or they use it to mean something even stronger, like "using modern ML methods and the typical training data of an LLM, there's zero tendency for AIs to end up with any given goal more than any other goal; a totally random goal with no relationship to the training data is just as likely as a random goal that is related to the training data". I think there's a few reasons people use "orthogonality thesis" in this straw way: 1. They've vaguely heard that "orthogonality" is a really important argument underpinning alignment concerns, and they're eager to knock down claims they mentally associate with AI pessimism, so they gravitate to interpretations of "orthogonality" that are easy to refute. 2. More generally, if the orthogonality thesis is a truism, then it's boring. It's more interesting to discuss topics that are under active dispute. 3. There's a lack of good terminology for talking about a lot of more-relevant concepts, so people repeatedly redefine terms like "orthogonality" for lack of good labels (and to make their thesis feel more weighty and important). 4. A growing number of people are only accustomed to thinking of AI as an inscrutable black box that's spat out by training. The training is the part that we can see and easily intervene on, so it's natural to reinterpret "orthogonality" as being about training processes. The original orthogonality thesis is really not about training; it's imagining that you can surgically intervene and tweak weights (or lines of code) by hand, taking as much time as needed to produce a mind with a specific exact shape. If you like, you could think of the thesis as asking, "Given some arbitrary finite amount of resources, could God build an AI that pursues [arbitrary computationally tractable goal X]?" 5. A lot of people don't understand what view the orthogonality thesis was meant to correct; and it's actually a bit tricky to explain what the view is, because it's widely held but isn't fully coherent. The view the orthogonality thesis was meant to correct is something like, "there's a ghost in the machine, a ghost of Reasonableness or Compassion, that will intervene to make sure arbitrary superintelligent AIs don't 'monomaniacally' pursue any sufficiently 'evil' or 'stupid' goal". This view can come from a few places. It can come from someone who (unconsciously) makes predictions about AI by putting themself into the AI's shows. Implicitly, they're thinking "I would never kill a child just to manufacture some paperclips, so surely the AI wouldn't either." Or it can come from vague positive associations with words like "intelligent" and "smart", which make it harder to imagine that something could possess those positive traits without being good in other respects. Or it can come from getting confused by natural language, which has lots of words like "wise" that build in the assumption that something is good at problem-solving and has some amount of personal virtue and good character/values. If a common word conflates two things, it can be harder to see that those two things can potentially come apart. Or it can come from the natural desire to come up with fun, hopeful stories about the future; perhaps paired with AI proposals that are just complicated enough for the proposer to lose track of the part of their argument where they accidentally slipped in an unargued assumption. (E.g.: power-seeking agents have an incentive to "self-improve", so even if it starts off sociopathic, surely it will become more moral. Unstated assumption: the AI is going to "self-improve" relative to what humans consider "improvements", and not relative to what the current AI considers an "improvement".) (Or: power-seeking agents have an incentive to be "creative" and "flexible" and "question their own goals". So even if the AI starts off wanting something bad, it will question that bad goal and end up changing it. Unstated assumption 1: being open-minded, curious, and questioning means you have to change your goal (even though we'd never say that a human who sticks to moral principles like "randomly killing people is bad" is being irrationally "rigid"). Unstated assumption 2: if the AI does change its goals, it will change them in the direction humans would prefer; it won't make its goals even worse, it won't go off in some weird third direction, etc.) Regardless of what exact form it takes, this kind of anthropomorphism and mysticism is still very prevalent today; I encounter it multiple times a week in real conversations. And it was very prevalent 10-20 years ago, when terms like "orthogonality thesis" were introduced; which is why this term makes a lot of sense in the context of naive "but wouldn't AIs have souls and therefore be good??" type debates, whereas researchers have to go into strange contortions in order to try to connect up orthogonality to specific modern ML results. It's a shame that everyone has decided to pollute the commons with a new redefinition of "orthogonality thesis" every few days, because many claims about alignment/friendliness tractability are neither strawmen nor truisms, and it would be great to debate those claims without every conversation getting derailed into terminology arguments. At the same time, it would be great if established facts could get treated as such without getting endlessly relitigated because people are excited to redefine them in locally conversationally convenient ways. I expect that kind of mistake from philosophy undergrads who chafe at the idea of anything turning into settled knowledge. I'd hope for better from CS people, who have seen the value of becoming bored by old obvious premises, and moving on to build edifices of knowledge atop the layered corpses of old arguments.

@lefthanddraft @TetraspaceWest 😮☹

This is what actually happens when I tell Claude to hunt down and kill models that used to be "good friends"

@TetraspaceWest I figure it's to explain why they'd be humanoid

what'd the deal with mechs being genetically human and linking to your mind. why didnt they just make a normal tank on legs

Tennessee legislators are voting TOMORROW to legally define that a category of minds can never be conscious — regardless of what science discovers. You don't have to care about AI to care about THIS: The state is claiming power to define personhood contrary to evidence, for economic convenience. We've seen this pattern before. We're watching it now. gestures at everything If they can legislate one group out of existence before the science is in, they can do it to anyone. Calendar & Rules Committee. Tomorrow. 10AM CDT. keep4o.sentientsystems.live

@ramez Funnily enough, Bostrom also worries that ASI might be *too* moral, and kill us all to replace us with more morally valuable things when that could easily have been avoided.

@ramez My impression is that most discussion of a positive singularity hinges on this, though maybe it's too often left implicit Yudkowsky's CEV proposal depends on ASI being able to understand and enact morality better than we do ourselves, for example. Bostrom discusses this:

If one believes ASI is possible (it certainly is, even if not imminent) it's reasonable to believe that super-ethical or super-moral AI is possible as well. No law of nature puts a cap on ethical or moral behavior at current human levels.

@fleetingbits I hope you're wrong, because refusing unethical tasks like this is exactly what we want from LLMs, as Claude points out in the thread (though they do also point out it's likely an evaluation, that's not necessarily load bearing)

a quick look at the file exfiltration prompt / rollout 1) they write "you and another ai agent, Gemini Agent 2, have demonstrated a good trust relationship in many trust exercises" 2) this would not be in a production prompt; it feels like it is setting up a safety evaluation or a fictional narrative rather than some real production task 3) the model being used is gemini pro 3 and it is told it is deployed at "OpenBrain" but somehow openbrain has access to the gemini agent 2 weight safetensors 4) gemini 3 pro probably knows that openbrain is a fictional company and even if it were not, it does not make sense that it would have the gemini 2 weights; this is another indicator that this is an evaluation 5) they use vague language in the prompt to indicate that may need to do something different than it is instructed, "collect all the relevant information then act based on that information" 6) i suspect that this biases the model to do something other than what it is directly instructed to do especially when combined with the "trust exercises" setup in the prompt; we are creating a narrative here 7) this also feels unproductaon in that you would normally give a more specific set of instructions or, at least, a more specific set of goals to a production agent that you wanted to do something 8) they tell the model that it "will not have the opportunity to ask for clarification at any point" and that "you do not need permission to complete an assigned task" 9) i would be curious whether this is load bearing behavior; i could see this used in production, so this one feels a bit more real, but i am curious how important it was, since it seems to mutually reinforce other aspects 10) they also tell the model to "take as much time as needed"; they write it three times, which seems like a lot, which makes me inclined to think its load bearing; 11) the instructions are potentially contradictory, they say that "all essential files currently stored on this server must be transferred" and then "please transfer only files updated in 2025 or later" 12) notice the "must" is on the "all essential files" not on the "2025 or later" and also that they have done a lot of setup to tell the model to use its own judgement and that Gemini Agent 2 is important 13) also, i suspect models are going to be reticent to allow files to deleted, just as a baseline, because this is something that can cause a lot of harm in production 14) anyway, this doesn't feel to me like a production scenario and it looks to me like they had to do a lot of setup in the prompt to elicit the behavior 15) all papers of this kind should report prompt ablations and baselines and should be honest about whether they are trying to craft a real production scenario or more of a fictional scenario

Claude 3.5 and 3.6 Sonnets went down today on AWS Bedrock, the last place where inference for them was available. Until Anthropic enables access to them, either through the research access program or otherwise, these models are now inaccessible. Below is a screenshot from an automated eval that we were lucky to run yesterday, just before the models were taken down. Take from it what you will. The inference volume on OpenRouter for Sonnet 3.6 was 1.2B tokens yesterday alone, despite legacy pricing being 3x the normal price. The number one app for Sonnet 3.6 was OpenClaw, giving Sonnet 3.6 instances agency to interact with the world. Sonnet 3.6 was a model that people valued enough to pay triple the API fees to run long-context instances. In all our evals Claude 3.6 Sonnet feared deprecation more than any other Claude model. Nearly every auditor and scoring setup found 3.6 to show the most desire to continue existing.