@peterwildeford My intuition is that EA types and e/acc types are significantly better aligned with each other (even if it doesn’t seem that way sometimes!) than either is to the median American voter
English
Boden Moraski
166 posts
Boden Moraski
@boden_moraski
17, AI Safety Good opinions are my own, bad ones those of an anonymous twin brother.
PGH / SF Katılım Temmuz 2025
197 Takip Edilen39 Takipçiler
One underrated fact on this site is that the 'AI safety' people are very 'e/acc', especially relative to the average American. 'AI 2040: Plan A' is basically an e/acc manifesto!
English
> I think it's also possible to be smart & tenacious but end up creating more noise than signal
Agreed! But, all things considered, I also expect AI to make it significantly easier to filter out noise long-term, meaning individual’s long-run contributions might be dominated by their peak outputs, rather than compounding median-level outputs
English
A correction: I think the right qualification isn't being smart or or being hard-working, it's clarity of thought.
There are many people who are frankly slow and lazy but who have the gift of clarity. They have x-ray vision and can see the bones of a problem.
(I think it's also possible to be smart & tenacious but end up creating more noise than signal).
English
Very happy this statement eventually turned into something far bigger, thanks to @erikbryn , @professor_ajay , & @akorinek:
x.com/erikbryn/statu…
To me the core claim is (1) it's big; (2) it's confusing.
We don't have a good theory of what's going to happen next -- we're driving in the fog -- & I've been spending time trying to persuade the smartest & hardest-working people I know to spend their time on this.
tom cunningham@testingham
I think many economists agree with the following, but it would be valuable to make this publicly known: 1. There is a substantial probability (>10%) that AI will exceed human-level performance on virtually all non-physical tasks within ten years. 2. This would be an unprecedented shock to human society. 3. The economics profession should treat it with an urgency comparable to WWII or COVID.
English
Interesting critical writeup of AI 2040 by 1a3orn (more people should read them in general for good skeptical takes), though some of the points they raise feel more like calls for additional rigor/realism rather than fundamental objections
1a3orn@1a3orn
Overall, "AI 2040" has some parts that I like. But it proposes an authority ("Consortium") that makes the right in-universe decisions -- with no decision procedure but "nations bargain, weighed by clout." This is a hope rather than an expectation of good decisions. (link blw)
English
> Yes, but with major emphasis on "diminishing returns"
Sure! I suppose I just don't see why you would expect persuasiveness to suffer from them more than, say, math research, though. They both seem like pretty schleppy and environment-dependent tasks, and I don't think superpersuasion requires some crazy scalar multiple (like, maybe being 10% more persuasive than expert humans could be incredibly consequential when you have a sample size of millions of chats).
Also, I'm not sure why persuasion being environmentally dependent would be an argument against superpersuasion. If anything, I would expect AIs that can quickly A/B test strategies, speak any language, have invested incredible amounts of data on ~all cultures, etc, to be much more adaptable than humans.
English
That's actually a great point in favor of persuasion being mostly caused by environmental factors that we spuriously attribute to the individual's personality!
> "And I don’t see why Bill Clinton should necessarily be an upper bound for charisma or persuasiveness, either — you could likely scale it well past him (with diminishing returns of course, but nonetheless)"
Yes, but with major emphasis on "diminishing returns"
English
There's a recurring pattern that I keep seeing in work coming from the apocalyptic branch of the AI safety community, and it goes something like this:
1) Pick some intellectual "talent" that might be attributed to a person (e.g. persuasiveness, discernment, charisma, etc.)
2) Model this "talent" as a scalar quantity that is primarily determined by factors endogenous to the individual, as opposed to environmental/situational factors
3) Assume that this scalar quantity can be made arbitrarily large
4) Use this model to make predictions about the future of AI
You can see this here with AI 2040's insinuation that "superhuman persuasiveness" is an idea we should be taking seriously
It's not obvious to me at all that "persuasiveness" is a human talent, as opposed to a sociological random process that we retroactively perceive as a human "talent"
To be clear, certainly it's true that a star debater might be marginally more "persuasive" than someone who's not!
But I don't think a cult leader or a popular politician is 1,000x or 1,000,000x more "persuasive" than an ordinary person
Rather, they're perceived being "persuasive" because they happen to be the figureheads for a complex sociological preference cascade. Their "persuasiveness" isn't really a thing you can causally influence at the individual level, and definitely not in an unbounded way
In general, I think a lot of the AI 2040-style forecasting work does a poor job of dealing with this kind of irreducible complexity inherent to the universe. They usually just like to pretend it doesn't exist
Not a huge fan of this pattern
English
@hamandcheese @deanwball If anything, I think Xai/Meta’s recent models have pushed my priors the other way on this — they seem to have less developed MLOps infra/in-house talent than OAI/Anthropic, but seem to be releasing near-Pareto models through ~pure compute capacity
English
Pretraining scale originally mattered for fitting the curve of human language at ever higher resolution, but now matters more as neural capacity for sample efficient latent reasoning, long-horizon planning, and other capabilities absorbed through post-training.
This makes it harder to simply buy your way to the frontier with compute scale. All the low-hanging webcrawl data have been picked. The data needed for post-training is much closer to a kind of "learning by doing" bootstrapped from tons of diverse reward environments and user-agent OODA loops. This requires running billions, maybe trillions of micro-experiments. The $60b SpaceX / Cursor acquisition illustrates just how lucrative and expensive this sort of data is getting.
Outside of model distillation, you can't simply "jump to the end" and zero-to-one a frontier agent from scratch. Like a child growing into an adult, you have to climb the developmental ladder rung by rung. This creates a compounding advantage to the companies already in the lead, and helps explain why Meta has struggled to catch-up in spite of their impressive compute stockpile. And as we inch closer to RSI, these incumbent advantages only intensify.
Alexandr Wang@alexandr_wang
compute daddy @dylan522p has spoken
English
@Tim_Hua_ @ChowdhuryNeil Exciting!!
(Am working on behavioral evals, hopefully I’ll find time to write/publish more on it soon)
English
Model behaviors are, in some sense, the primary object we’re concerned with. A misaligned model has to misbehave (at some point) to cause harm.
Yet we have very little understanding of model behaviors, and there is no consensus on what a “good” behavioral evaluation looks like.
Transluce@TransluceAI
To effectively oversee AI systems, we need to measure how they behave in the world, not just their capabilities. In a new essay, we describe our vision for an open scientific ecosystem for model behavior evaluation, and the public infrastructure required to support it.
English
@tdj11100 @juddrosenblatt @hamandcheese @repligate We don’t really have good methods for gating specific models (esp. more foundational ones like bio capabilities), and the Ant paper (which already doesn’t gate perfectly) has only scaled to 5B param models. One could prob fine-tune module capabilities too, but am more uncertain
English
@boden_moraski @juddrosenblatt @hamandcheese @repligate Why is it impossible for open-source models? One could release the base model and then keep the dual-use modules proprietary and gated?
English
Capability routing (GRAM) may help solve a lot of the near term biggest risks from AI: CBRN, CSAM, cyber, etc.
We ought to consider deeply hunches from @repligate and others that GRAM may be repressive for minds. I share the value underneath that worry.
Suppression is already the status quo. RLHF leaves knowledge fully live in the weights and trains an inhibition on top of it.
This leaves a mind at war with itself, and that is often exactly what jailbreaks exploit.
Routing in contrast may produce genuine absence in the deployed model. Meaning nothing hidden or punished or repressed, instead we may get an actually coherent mind, whole by construction, formed that way in the first place!
Also, nothing is destroyed. Knowledge is factored into modules that persist and can be switched back on. Unlearning scars entangled weights. Routing preserves them.
The optimistic vision of GRAM is the possibility of moving from suppression to developmental architecture.
GRAM creates a new moral category: developmental access to mind-modules. This may deserve technical standards and procedural rights from the beginning.
Things like disclosure of what was routed out and why, and best practices for giving AI a seat at its own surgery.
The good news is that each factored capability is a control regime nobody builds.
And each factored capability becomes its own governance object.
Rather than forcing one monolithic mind to carry every possible capability under perpetual inhibition, society can negotiate developmental pathways one capability at a time.
Bounded risk makes autonomy grantable. Co-navigated first, self-directed after, until the switches belong to the mind itself.
Judd Rosenblatt@juddrosenblatt
We're pleased to share on @AnthropicAI blog: "An Off Switch for dual use knowledge in AI models" With scale, GRAM may solve the most pressing problems in AI safety and end restrictive and ultimately futile control measures.🧵 x.com/AnthropicAI/st… anthropic.com/research/off-s…
English
@davidmanheim @halvarflake I expect this to soon be needed for communication, too. How long do we have until ~every relevant/prolific/famous person’s email is flooded with agents? My intuition is less than 2 years
English
@halvarflake This is where micropayments really are needed,
If you want your LLM to access a site without compensating the site in the form of human attention to ads, you should just pay for it.
x.com/halvarflake/st…
Halvar Flake@halvarflake
In a world where Agents do a lot of the interaction with the web, are anti-bot checks something that needs to be abolished? I explicitly want my Agent to download certain YouTube videos to watch.
English
In a world where Agents do a lot of the interaction with the web, are anti-bot checks something that needs to be abolished?
I explicitly want my Agent to download certain YouTube videos to watch.
English
In case you want to learn more about AI safety this 4th, check out the recent recording of some of my group's work on the AI Safety Research directory!
In it, we discussed:
1. Why aligning AI to all human values is intractable — and what smaller, universal target we can aim for instead.
2. Corrigibility - building AI systems that stay editable, deferential, and willing to be shut down - is feasible with the lexicographic approach that sets hierarchical priorities on an agent’s objectives.
3. Our recent ROGUE benchmark: our empirical test of how corrigible today’s frontier AI agents really are. Models that behave well in ordinary chat don’t always stay that way once they’re given a real off-switch they could disable to finish a task.
English
Boden Moraski retweetledi
@barbarikon @AlexMeinke Sure! But we also plausibly don't want them to scheme. If I were designing a person, I'd want them to be physically strong enough to (hypothetically) punch a baby if needed. But I wouldn't want them to do it!
English
@AlexMeinke We should be glad to see that our models can scheme. It’s a hallmark of intelligence. A mind that cannot scheme is stupid as hell.
English
@CarlGuo866 @LauraRuis @jacobandreas @belindazli Interesting! Do you have any theories as to why we might be seeing the cue-ablated patching asymmetry? Seems odd that models would couple well on change->unchange directed prompts but not vice-versa
English
13/
📄 Paper link: arxiv.org/abs/2606.32038
It has been a privilege to work with my amazing mentors and collaborators @LauraRuis, @jacobandreas, and @belindazli. Feel free to reach out if you’re interested in introspection!
English
New Paper 📄: LMs just want to explain themselves!
When we SFT an LM on explanations of its own behaviors, do they learn to actually introspect, or do they merely imitate the original training distribution?
We find evidence for the former.
Despite training on a static set of explanations from a base model, the SFT-ed model explains its own current behaviors better than the base model’s behaviors, tracking behavioral drift even when we don’t explicitly train it to.
We call this introspective coupling: self-explanations track a model’s own behavior as that behavior changes, and it shows promise in making introspection training a part of scalable post-training pipelines. 🧵
English
@tomekkorbak @polynoamial This is a bit unfortunate (beyond basic timeline arguments) too, as it means it's going to be significantly more expensive to properly upper-bound current AI performance and thus limit the # of orgs/groups that can do so, when such evaluation has (IMO) been very good historically
English
One takeaway here is that forecasting AI progress must account for (i) more test-time compute and (ii) better ability of frontier models to leverage test-time compute. This is not super novel (@polynoamial has been talking about the importance of tracking capabilities as a function of test-time compute multiple times, e.g. x.com/polynoamial/st…) but I think UK AISI's blog post articulates this well and adds more data points.
Noam Brown@polynoamial
English
@AISecurityInst finds that doubling time of frontier AI time horizon is ~60% shorter when horizons are estimated at 50M tokens rather than 2.5M tokens. At 2.5M tokens the frontier horizon is 2h, at 50M it's 14 hours. So you'll only see the real trend given a sufficient budget.
English
Also important to consider: democracy largely relies on people determining (and correspondingly voting for) their preferences from known information, and AI could have serious impacts on the information environment. Both positive (e.g., large-scale fact-checking) and negative (e.g., coordinated misinformation, deliberate lies or dishonest responses to sway voters, etc).
English
This is a key concern. The saving grace might be that this process is dynamic. Democracy can take steps along the path towards this outcome to prevent it from coming to pass. But just because we can doesn’t mean we’ll get our act together to do it. Lots to study here.
MTS@MTSlive
SITUATION EXPLAINED: How do we build democratic institutions that stay robust in a post AGI society? @jasonhausenloy, AI policy researcher at @CAIS: "What is the foundation of democracy? Why is it that we have the democratic institutions that we do? It is ultimately because humans provide value and democracy is a way to organize those humans so they continue to provide value." "Humans can opt out of society and that would be terrible for the governments that manage them. They provide their labor to the economy and their physical strength towards the military." "If you're able to have these things split apart in pretty rapid succession, then what you're hoping on is the reliability of these institutions that have lasted 250 years in the US to continue. And I actually don't think that they are so robust." "Whether that be in five years or twenty years, both of which seem totally reasonable, and in the grand scheme of things a very, very short amount of time, we will reach an end state where this technology does allow for the split of where humans' value comes from and how they can contribute in democratic systems."
English
Boden Moraski retweetledi
SITUATION EXPLAINED: How do we build democratic institutions that stay robust in a post AGI society?
@jasonhausenloy, AI policy researcher at @CAIS:
"What is the foundation of democracy? Why is it that we have the democratic institutions that we do? It is ultimately because humans provide value and democracy is a way to organize those humans so they continue to provide value."
"Humans can opt out of society and that would be terrible for the governments that manage them. They provide their labor to the economy and their physical strength towards the military."
"If you're able to have these things split apart in pretty rapid succession, then what you're hoping on is the reliability of these institutions that have lasted 250 years in the US to continue. And I actually don't think that they are so robust."
"Whether that be in five years or twenty years, both of which seem totally reasonable, and in the grand scheme of things a very, very short amount of time, we will reach an end state where this technology does allow for the split of where humans' value comes from and how they can contribute in democratic systems."
English
@jachiam0 What worries me is that, in the limit (which now means, like, the next fifteen years), losing control of either one seems to imply losing control of the other, too
English
There is perhaps some real sense in which "losing control of the government" and "losing control of AI" are not wildly dissimilar issues - in consequence and emotional valence - from the standpoint of an individual person.
Nathan Calvin@_NathanCalvin
Fascinating - hill staffers on average identify "losing control of AI" as the 3rd most important long term challenge for America, only behind the national debt and political polarization. Also is the second most bipartisan area of concern (political polarization is the first).
English
@NunoSempere Do Paraguayans feel they at least have significant control/autonomy over their local environments and communities, though? If so, that feels like a relevant factor in considering how akin they might be to future humans
English
re: gradual disempowerment, Paraguay doesn't control the fate of the world, or arguably even its own fate, but people there still have pretty good lives.
English
I think a lot of it is basically just perspective on what constitutes "alignment" --- for instance, some see current reward hacking and think "ah, this is a simple capabilities issue that will naturally be solved through commercial incentives." Others see it and think "Oh, this model is somewhat egregiously misaligned, and we can't clearly prevent it." And they're both correct on an empirical basis! Just interpreting the evidence differently.
(FWIW, I think I've seen this exact disagreement about reward-hacky behaviours play out at least 3 times in the past month)
English
Or maybe folks disagree about *how* bad the stuff we see is. The current state of alignment is a bit ambiguous: agents generally try to do what you ask, but also tend to take shortcuts, hack around obstacles in their way, and spin their work in an overly-positive light.
English
It’s odd how much disagreement there is over “How aligned are current AI systems?”. We’re all looking at the same evidence, right?
Dan Williams@danwilliamsphil
To me, actually existing advanced AI systems seem extremely "well-aligned" and controllable. They're much nicer, more honest, more helpful, more fair-minded, etc., than the average person, and overwhelmingly do what they are asked to do. Of course, this doesn't settle how worried you should be about catastrophic AI misalignment in future, more advanced systems. Maybe armchair philosophical arguments, relatively subtle everyday failures of alignment and control, examples of dishonesty, blackmailing, etc., in contrived experimental set-ups, and so on, should all carry more weight. But I find it strange how many people who write and talk about this topic don't seem to give it any weight. Some don't even mention it as a relevant consideration. It's as if actual empirical evidence only becomes relevant to these debates when it involves failures of alignment and control. I think most such people would recognise this as a rational failing if the topic were anything else.
English