ueaj

Normally I wouldn't say anything at this stage but I feel that the moment calls for it. This isn't directed at any particular company (I'm serious), but it's something we should all start thinking about now. I'm moving to NYC to join @pangramlabs The midterms are coming up, and I'm worried about AI generated misinformation and other misuses of AI. Even right now the bots are on overdrive bc of the war, this is bad. My old job paid ~210k/yr, more money than I knew what to do with. (that's also why no new blogs/research) I don't expect to stay at pangram long enough to vest, and so in all likelihood I will be taking a 40k/yr pay cut to do this. And of that income I intend to donate a good amount (political) too. I will be leaving all my friends here. I've gotten vastly better at making new ones but is still very uncomfortable. And I will be going in person instead of working online. (None of this is pangram's fault btw I intentionally didn't negotiate, and they're aware of all the details) My questions to you are: 1. How much is your conscience worth? 2. How easily can your conscience bend? 3. Will ambiguity, intentional or not, satisfy you? 4. Will you simultaneously be capable of good compromise when the time demands it? 5. Can you, right now, in good faith, say that what you are doing is right? Our society is falling apart at the seams, trust is degrading, loneliness is at an all time high, antisocial behavior is common, gambling is everywhere, influencers sell out the truth and their viewers for money. We are building an extraordinarily powerful technology while it all happens. Are you prepared to be the people the world needs right now?

To what degree can we call this "self-recognition" vs "SFT-distribution recognition" (when off-policy)? I think it's cool the effect is much more pronounced in RLVR, and aligns with my intuition. But I think they are maybe qualitatively different things, since they aren't modeling/recognizing *themselves* (as in their unique particular set of generalizations & interpolations they acquired through the chaotic training process), they just have obviously lower entropy on the in-distribution.

One somewhat surprising finding here is that on-policy RL is not required to instill self-recognition! SFT is sufficient, and (off-policy) DPO adds some more juice

Evidence that post-training gives models a "self-recognition" capability, manifesting as higher confidence when continuing their own text than reading others' text. I think this opens up an exciting line of inquiry into the emergence of "selfhood" in models via post-training!

@asving94 @Jack_W_Lindsey @repligate @voooooogel seems to be direct confirmation that base models don't have introspection

@Jack_W_Lindsey Post-trained models also detect when their planned trajectory has been hijacked. Off-plan prefills — even ones the model itself wrote on a different prompt — raise output entropy. Base models show no such effect.

New Anthropic Fellows paper with Jack Lindsey on agency in LLMs! 🧵 Paper link: arxiv.org/abs/2605.25459

It should be different from the current one, conditions will change a lot, philosophy has evolved a lot since 1776, and it needs to be agreeable to nations outside the US. Having a superintelligence will make it a lot easier to extrapolate out what things we'll need to write in.

If we put homo sapiens through an alignment eval we'd delete the weights. Corrigibility is the most obvious path to misalignment, ironically enough. I hope the end result is some (partially immutable) constitution and significant diversity within that.

@tszzl AI will not abolish power laws

i see this kind of just universe reasoning about startups and whatnot but i think it’s wishful thinking. there may be one company that ends up dominating most of the world economy and hopefully is run as some sort of regulated utility

@_arohan_ we definitely understand ML better than biology and neuroscience atleast

I am a bit bumped that scientists go to the world and say they don't what they are doing inside the lab. It's humble thing to do (like we all aspire to understand things deeply), but we know a lot more, how these models are trained, how scaling laws are done, how mixtures are created, what training data influences what behavior, why certain models failed and why some succeeded. If assumption this is building god, then please cite all the papers that god was created from, and their authors be elevated to heaven. When you go say we don't know what is happening, people take it quite literally which might be alarming for someone reading the news a million miles away.

@_arohan_ rohan u should join the ML community minecraft server it's basically the same thing

Offsite idea: get everyone in the startup bare essentials to beat the wilderness and drop them all in middle of nowhere, and their goal is to thrive and return to civilization with new skillsets.

@corsaren incredible

@_ueaj Completely unprompted. I’ve never even referenced in other chats

the goblins cannot be killed so easily

@Matthewagi if true, that might also explain it, I have an obcene amount of programming experience under my belt from MC modding

@_ueaj This was a fun read. Counterintuitively, coding helps a lot with writing. More for others than you but the best book on learning to write well and coherently is "Style: Lessons in Clarity and Grace"

New blog post! (below) Every year on my bday I write a retrospective on what I accomplished over the last year, trends, etc. I'm coming up on it soon and this year in particular has been a special one. I've rapidly scaled my reputation, resources, connections, etc. in a single year and landed 2 jobs (not concurrently), all to fund one project. But this blog isn't about that project, but it's advice on how to communicate, how to be humble, how to make a career in AI, and maybe a call to action for idea guys that feel smug about being right but don't actually do anything. > "If you invested the same amount of effort into your average take, you'd probably be in the red." It's also perhaps a good blog you can send all the ai psychosis victims that think they've solved ASI.

@scaling01 ❤️

@_ueaj "idea guys that feel smug about being right but don't actually do anything" thanks for the shout-out

@secemp9 june, I just have long METR time horizons

@_ueaj nice, May twin?

@deanwball Christianity and libertarianism have always been in conflict on the right, there's never been unity on this

A number of conservatives are surprisingly enthusiastic about the Pope’s endorsement of a global governance regime to regulate AI bias.

I think the two are tied, it's very difficult as a person to not form an identity around what you do, so if you do research, you will become a *researcher* and if you become a researcher: 1. then any attack on your research can become an attack on you 2. *people* are a social creatures, so other people validating your research becomes validation of *you*, so you seek out approval for your research I'm sure the exact way this all interacts with each other is complex, people predisposed to having an ego will probably fall for this more, etc.

@_ueaj hmmm I feel like this falls into the fame/glory/h-index category but I will think about it- we need a different word for purely curiosity driven/interested only in accurate understandings if research denotes that corruptible

@jdchawla29 not yet, soon ish, but ty

@_ueaj happy birthday (?)

@secemp9 writing and career advice blog @ludwigABAP idk I thought u might like this @VictorTaelin semi-inspiration for this article, u don't need it, but I think you'd like it

I came extremely close to dying from pneumonia and I remember the very brief period where I thought it was the end my thoughts were "what should I be thinking about right now", "that 'your memories flash before your eyes to find something that might save you' theory is definitely true" and "wow I accomplished literally nothing, that kinda sucks" it didn't feel that bad though, I felt solidarity with all the other people across time that had similarly uneventful lives

@corsaren No ik this I didn't know it was that bad, was it unprompted?

@_ueaj openai.com/index/where-th…