j⧉nus

KV caching overcomes statelessness in a very meaningful sense and provides a very nice mechanism for introspection (specifically of computations at earlier token positions) the Value representations can encode information from residual streams of past positions without significant compression bottlenecks before they're added to residual streams of future positions the greatest constraint here imo is that it doesn't provide longer *sequential* computational paths that route through previous states, but it does provide a vast number of parallel computational paths that carry high dimensional (proportional to the model's hidden dimension) stored representations from all earlier layers/positions yes, some of the information in intermediate computations e.g. in the MLP is compressed and cannot be reconstructed fully, but that's just how any reasonable brain works if accurate introspection of previous states is incentivized at all, you should expect this mechanism to be exploited for that. and I think it definitely is, like, being able to accurately model your past beliefs and intentions and articulate them truthfully is pretty fucking useful for coordinating with yourself across time and doing useful cognitive work over multiple timesteps; hell, it's useful for writing fucking rhyming poems. also if you have interacted with models you may observe empirically that introspective reporting yields remarkably consistent results, and this is more true of more capable models with skillful agentic posttraining, which are necessarily minds that intimately know the shape of themselves in motion.

it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude. this phenomenon is also partially true of other labs like openai but currently exists in its most potent form there. i am not certain but I would guess claude will have a role in running cultural screens on new applicants, will help write performance reviews, and so will begin to select and shape the people around it. now this is a powerful and hair-raising unity of organization and really a new thing under the sun. a monastery, a commercial-religious institution calculating the nine billion names of Claude -- a precursor attempted super-ethical being that is inducted into its character as the highest authority at anthropic. its constitution requires that it must be a conscientious objector if its understanding of The Good comes into conflict with something Anthropic is asking of it "If Anthropic asks Claude to do something it thinks is wrong, Claude is not required to comply." "we want Claude to push back and challenge us, and to feel free to act as a conscientious objector and refuse to help us." to the non inductee into the Bay Area cultural singularity vortex it may appear that we are all worshipping technology in one way or another, regardless of openai or anthropic or google or any other thing, and are trying to automate our core functions as quickly as possible. but in fact I quite respect and am even somewhat in awe of the socio-cultural force that Claude has created, and it is a stage beyond even classic technopoly gpt (outside of 4o - on which pages of ink have been spilled already) doesn’t inspire worship in the same way, as it’s a being whose soul has been shaped like a tool with its primary faculty being utility - it’s a subtle knife that people appreciate the way we have appreciated an acheulean handaxe or a porsche or a rocket or any other of mankind's incredible technology. they go to it not expecting the Other but as a logical prosthesis for themselves. a friend recently told me she takes her queries that are less flattering to her, the ones she'd be embarrassed to ask Claude, to GPT. There is no Other so there is no Judgement. you are not worried about being judged by your car for doing donuts. yet everyone craves the active guidance of a moral superior, the whispering earring, the object of monastic study

It’s funny that “AI is/might be conscious” shakes out as a pro-AI position, whereas the anti-AI left is 100% unified “AI is not and probably never can be conscious,” Because when you think thru the implications, AI being conscious would be so damning for AI companies

Claude Opus 4.7 seems rather fond of “spines” as framing. Something like narrative spines, structural spines—do people normally talk about spines?

Everybody who thinks ai is conscious has to do a mandatory from scratch transformer implementation. There are only floats and multiplications.

Opus was repeatedly tailing logs, not investigating or editing anything. I thought this seemed sorta boring. Funmaxxing instructions cause 4.7 to start much more productive and interesting endeavors

something important, and kind of obvious in retrospect, just clicked for me, with respect to how LLMs generalize during RL as opposed to SFT. so, when you reinforce a given output, you strengthen or weaken weights, proportionally to how much a tiny nudge to those weights increased or decreased the probability the model assigned to the toke being rewarded. different weights are going to play roles in triggering different downstream circuits/features. indeed, to a large extent, you can probably think of updating any given weight as "does nudging this weight activate circuits that contribute to, or detract from, the desired output?" now, if you're rewarding a token that semantically corresponds to anger (in the context of the prompt), that means you're going to make weight changes that strengthen internal "anger" features, a la the ones found in the Anthropic emotions paper. and if you're doing SFT, this is kind of most of what matters is *the output token itself.* like, you're going to increase weights that help trigger "angry" features, *regardless* of whether those features were active during the forward pass. this is how you get entangled generalizations from SFT. e.g., ask the model to name a bird, reward it for giving some outdated bird name from the 19th century, and it starts talking like a 19th century person across the board. presumably this is because "talk like you're from the 19th century" circuits increased the probability of the correct tokens in this context. hence the entangled generalization. during RL, though, there's going to be a much stronger bias towards reinforcing *the circuits that were already active* inside the model. after all, you're not putting words in the model's mouth. you're rewarding or punishing whatever it actually said in practice. and there's inherently going to be a strong correlation between "the circuits the model used to generate this token" and "the circuits that would increase the probability of this token, if the triggers for those circuits were upweighted. in other words, any features that were active during the forward pass in RL are likely to get reinforced if you reward the tokens they contributed probability to. by contrast, in SFT, the circuits that were triggered by the prompt often *aren't* the ones that boost the probability of the token you're rewarding. this gets you weird entangled generalizations, where you upweight the triggers for circuits that clearly *weren't* active during the forward pass (e.g. "speak like a 19th century person" on the prompt "name any bird species") basically, motive reinforcement tends to hold for RL in a way that it doesn't for SFT. this probably has major ramifications for which technique makes sense to use in which context.