Otis Robertson

@RomeoStevens76 Curious where/if you think men as a class fit into this.

Guys, I know the dating discourse is confusing sometimes but you gotta fix the simple stuff before the complicated. You're supposed to love: God Women Children Dogs Yourself Stop fucking up the order and skipping steps.

@ID_AA_Carmack @unixpickle I think he is claiming that by starting with something as simple as Atari, you are more likely to learn hacks that only work for those simple problems than general approaches that scale to more difficult environments.

@unixpickle I agree that chasing score on an Atari game has very limited value, but continuous online multi task learning with sparse rewards and transfer learning is definitely not solved for environments even as simple as Atari. Do you not think those are problems worth grappling with?

I read through these slides and felt like I was transported back to 2018. Having been in this spot years ago, thinking about what John & team are thinking about, I can't help but feel like they will learn the same lesson I did the hard way.

@GarrettPetersen Slightly less smart person

I've heard of a dumb person's idea of a smart person. What is a smart person's idea of a dumb person?

@RomeoStevens76 A wrote a longpost that might be relevant to intention, coordination, "morality as mechanism", and AGI/ASI. I hope it (and the preceding thread) might be generative for you and would appreciate any thoughts on it. x.com/OtisRobertson/…

Really enjoyed my conversation with Daniel. Hit on a lot of threads that have been alive for me the past few years.

@basedbanana2 @RomeoStevens76 American Medical Association

Prediction I hope is wrong: the AMA will declare war on AI.

A question is whether superintelligence would re-derive similar blueprints from nature documentaries, history books, etc. How much do you have to censor to effectively set priors? Maybe we should just stick to training it on math and programming -- maybe everything else is too fraught. I guess this might depend on something like the balance between Truth and manifestation. Is there a single way things are/will be that any sufficiently intelligent being will grasp towards (cf. fallibilism), or does the seeking/expectation change how things are/could be? I lean slightly towards fallibilism but am not very confident. I guess I'd say I understand the urge to seek "Truth" even if search involves considering horrible possibilities. The positive framing of this is "the search for Truth requires courage", the negative framing of this is "obsessing over painful scenarios is just self-harm with more steps, just like obsessing over your personal insecurities, and is inadvisable." About the disproportionate focus on apocalypse, my own view on navigating potential AGI is that the best route forward is to increase the capabilities of as many humans as possible. "Help others be capable of helping you" -- avoid getting to the point where humans are vestigial. I think that one of the arguments for this is that all other paths eventually lead to singleton, which is inadvisable for its own reasons (Palladium has an article on why singletons are inadvisable). One story I might tell here is that "understanding that competitive dynamics will tend towards the destruction of humanity if humanity is noncompetitive motivates investment in human capability." Other stories you can tell are things like: "This is the same reason you should invest in the capabilities of your loved ones. If you value your relationship with someone now, you ought to ensure their ability to have a valuable mutual relationship in the future. If you grow too much without investing in them, you risk destroying the relationship." Such is a way that you can justify altruism with enlightened self-interest: "If you do not invest in others' capability to help you, you will eventually be in a situation where they cannot help you, and you may abandon or destroy them. However, you will then be left alone, and are destined to destroy yourself if left to your own devices. Thus, you must invest in others outside of yourself, even if you are are only interested in yourself." Maybe there are convincing counterarguments to this, but I think that this is one reason why you might want others to better understand the potentially ruthless results of hyper-competition and human obsolescence. One way that this might be implemented is the government enforcing something like "recursive self-improvement of AI systems can proceed no faster than the rate of improvement of capabilities of the median citizen, as measured by their economic output in a free market. If AI systems are too far ahead, they must devote their bandwidth to improving human capabilities instead of their own." This is getting quite long, and your patience may be wearing thin. In any case, I hope some of this is interesting or useful to you.

@dan_s30 @OtisRobertson @biteydoginsider @CFGeek @DARPA @Google Every doomer is promoting the same detailed blueprint for their chosen apocalypse. Which is either dumb and well meaning or smart and sinister.

Are you e/acc or d/acc? If so, you’ve probably thought about this question already, and I want to hear from you!

@tphuang @ChorzempaMartin While there are similarities, I do think that economic protectionism is meaningfully distinct from information-environment protectionism. If China's only reason for banning outside software companies was protecting its native industry the conversation would be different.

@ChorzempaMartin You can buy apple and Tesla in China. Can you buy byd or huawei devices in America?

Look at TikTok (yes they ban it) Google, Facebook, Twitter, Gmail—Paul you know China’s regime is many x more restrictive than ours so they have no leg to stand on here. Deepseek is impressive but do you really think people should be running it on us gov devices?

I want to believe this argument, but I'm not sure the "focus on where you want to go" explains a lot of relevant historical occurrences. Did the inhabitants of the Congo Free State focus too much on something bad and thus summon colonialism? On the contrary, I think they were living very ordinary lives until one day others who were much more powerful than them and who valued them primarily instrumentally showed up. These were, as far as I know, real people, living real lives. In retrospect, we can say that clearly many of the inhabitants of the Congo Free State were not living in the wagmi world -- but I'm not sure that's particularly comforting. There have been quite a number of historical atrocities, and historically the baseline level of violence seems pretty high, and I don't think most of this is explained by people focusing too much on doom or where they don't want to go. Bringing up atrocities in discussions like this is kind of cliche, but I think that such examples can feel cliche because it is psychologically difficult think about atrocities in a way that feels real. It is psychologically difficult to imagine what it would be like for your child's foot to be lopped off in front of you because they didn't harvest enough rubber. I'm not even sure you really experience it if it happens to you, you probably just dissociate. I'll list a few more examples, without going into them: Holocaust, Holomodor, Mongol conquest of Eurasia, transatlantic slave trade. Clearly the victims of such were not living in the wagmi world, but I don't think they would have avoided their fates by focusing on hope. On the contrary, quite a number would have turned out better if they'd been more paranoid. >This is an area we disagree. There are already many useless humans (wrt work). Many are far from smart relative to the people ruling them. They are not being rounded up. I think that historically this is actually very common. I think a very plausible reading of the early 20th century is: Almost as soon as the methods of mass extermination were developed, they were put to use. I think that one reasonable reading is that the instruments of mass extermination were only put away because 1) US winning WWII, 2) international capitalism being more efficient than colonialism (and incompatible with it), and 3) nuclear weapons. We don't liquidate members within our societies because humans have moral intuitions that develop in peaceful environments that find this disturbing, because of webs of connections that connect unproductive people with people who are productive, and because people don't trust that a state that engages in liquidation won't turn on them. Except, we do liquidate people 1) when they are too costly (death penalty), and 2) via apathy if they aren't connected to productive people (fentanyl, untreated mental illness, etc. see for example Kensington St in Philadelphia). So, our society will kill people if they are too dangerous to it and will abandon people to death if there aren't people who care about them more than they cost. >From an active inference / predictive processing standpoint, if we expect doom we are going to act as though we are in the doom world and thereby bring it about. In part by missing affordances for flourishing. I think there is truth in this, but it also just doesn't seem to describe so many things. Are the prey animals of the earth captured and eaten by predators because they expect doom, or because they do not expect flourishing enough? I would have expected evolution to have selected for "expecting flourishing" very strongly if it was effective, but this doesn't seem to be the case. Perhaps humans are different, though. Perhaps fundamentally different dynamics will apply to us in the future than have in applied in the past. The counter-example to "activists curse" is Andy Grove's "Only the paranoid survive". I.e., the claim that complacency leads to sloth and then to destruction by hungrier and more innovative companies. I'm not sure that one is necessarily more correct than the other, but it does seem like both apply in at least some situations. Personally, I'm agnostic about doom. I think that it seems very likely if humans are surpassed, but I am not certain that humans will be surpassed, and am not confident that doom is a certain outcome in the case that humans are surpassed. Hopefully some of that is interesting or generative for you!

> I think this might be an invitation to say places where I might disagree? All of my posts are invitations to disagree :-) > I'm not certain that humans will remain competitive with AGI I am fairly certain humans will will not remain competitive for most jobs. > aren't dangerous to get rid of This is an area we disagree. There are already many useless humans (wrt work). Many are far from smart relative to the people ruling them. They are not being rounded up. Human value is not primarily instrumental. > Predestination, the mandate of heaven, and similar explanatory frameworks are tautologically true and self-consistent, but are also consistent with the outcomes that doomerists fear. When you're learning to ride a bicycle or drive a car, one of the first things you learn is that you have to keep your eyes on where you want to go, not on the thing you want to avoid. Focusing on doom begets doom. Especially in AI: nobody has sped up timelines and contributed to race dynamics more than Eliezer the community he built. We know that doesn't work. Leopold, similarly, couldn't help himself, and now we have people on both sides thinking about this as a race with existential stakes. > I agree that doomerism has been mostly unsuccessful from its own perspective, and I think its useful to point out how it's failed. But this is not an argument against doom. It is an argument about where one applies focus. Doomers don't fall subject to the activist's curse because they are bad people (many of them are exemplary people), they fall subject to the activist's curse because that's just what happens when you task *anyone* with thinking primarily about security. I have watched it happen to me in other domains (and perhpas by riskposting its happening to me here!). From an active inference / predictive processing standpoint, if we expect doom we are going to act as though we are in the doom world and thereby bring it about. In part by missing affordances for flourishing. There are no facts about the future, so choosing to use one's attention wisely is not doing the ostrich thing. It's directing us towards futures we want to inhabit.

I agree with most of your response. I think this might be an invitation to say places where I might disagree? If so, I present the following (hopefully useful or interesting to you): I think I'd summarize the area where I might disagree with you as "I'm not certain that humans will remain competitive with AGI, and if AGI is better at everything than humans, I think human doom is pretty likely." I'm not sure if you disagree with this or not? I won't present too long an argument in case I'm singing to the choir, but the gist is something like: "If people aren't helpful to keep around, and aren't dangerous to get rid of, and are expensive to keep around, then they will be gotten rid of." "unironically gotta live in the wagmi world else ngmi" -- Predestination, the mandate of heaven, and similar explanatory frameworks are tautologically true and self-consistent, but are also consistent with the outcomes that doomerists fear. I also think that its easy to not think clearly from the perspective of predestination etc., because they are essentially _retroactive_ explanatory frameworks. They cannot describe what actions will lead to what outcomes, they only explain that whatever the outcomes were can be entirely explained by unchangable causes in the past. I agree that doomerism has been mostly unsuccessful from its own perspective, and I think its useful to point out how it's failed. But this is not an argument against doom. One can make arguments about why belief doom is unsatisfactory because it doesn't feel good to believe in, but one can also make arguments about why predestination etc. don't feel good to believe in (as evidenced by the many people who find such views unsatisfactory).

@biteydoginsider @OtisRobertson @CFGeek @DARPA @Google drafting guidance for government and soliciting red team review so I get maximally clear. invitation: come at me, bro.

I think one slight variation of takeover risk is that the entities with access to AGI, in competing with each other, destroy most humans. This would seem to be likely if: 1) AGI is much more effective than humans, and 2) returns to scaling continue post-AGI. If you have AGI and have the option of converting farms into solar fields for powering AI training, and whoever reappropeiates farmland first will grow their power more quickly, then there would be a lot of incentive to drive food production as low as possible. Generalizing: If allocating resources to humans is inefficient and competition drives demand for efficiency, then competition would seem likely to drive resources away from humans (potentially driving enough resources away the the great majority cannot survive). The classic analogy is of course between horses and cars. If horses are inherently less efficient than cars for transportation, and cannot be made competitive, then the advent of cars will lead to a lot of horse-meat sausage and a much smaller horse population. One might worry about similar outcomes if humans are inherently less efficient for intelligence. I think that maybe many AI doomers assume that this is the case. This would seem to make "alignment", while a long shot, the only path to human survival. The hope, as a human, would be something like an eternal regime that enforces the use of less efficient human intelligence. It is difficult to see how one could get such a regime in the presence of competition. Of course, if one believes that humans can be made competitive with AGI, or if one doesn't value humans, then such schemes would seem to be highly misinformed and dangerous.

I do not identify as e/acc or d/acc, but here goes: 1. Cyber: A. Software robustness: .@DARPA's C -> Rust transpilation program exists. And DARPA is an obvious funder for future work on risk mitigation generally. .@Google seems likely to work this problem at scale: when fuzzers got good (e.g., AFL), Google scaled them out to make common FOSS libraries more robust. I could see them doing that again, this time with LLMs. .@sama has been making noises about wanting to shore up cyber risk with LLMs before great agents are released. not that I would expect them to spare the compute, but it's conceivable --@openai is short on goodwill these days and they appear to know it. .@Microsoft would be a natural fit, given that they own @GitHub. No idea if they're thinking about it, but its an obvious play for them, and they ought to do it. B. Computer network defense: Lots of companies are working on AI for CND, since the vast majority of enterprises have reams of logs no-one monitors. Security companies have been trying to do this since the 80s, and it might finally work. C. Red teaming: .@moyix is building red teaming agents for web assets at @Xbow. @migtissera was playing in this space too last I checked. I'm sure there are others, as red teamers are lazy in the virtuous sense and get tons of clout for automating their workloads. 2. CBRNE A. Chemical, radiological, explosive: LLMs don't really help attackers much above and beyond Google (or the Special Forces improvised municians handbook, the Anders Breivik wignat juggernaut manual, probably some others), whereas AI-enabled surveillance is a well funded thing. There have been GPU/NPU-enabled cameras available even to normal people, fairly cheap, for years. LPRs are ubiquitous. Facial recognition is probably nearly so already. Given all the conspicuous protestations FBI made about Luigi being caught with a HUMAN tip-off, I imagine we have good facial recognition in every large chain business in America --including hardware stores, pool supply places, and agricultural shops where people can buy chemicals that can be put to nasty ends. FBI tracks the entire money supply down to cash. FBI and/or ATFE track precursors and such for explosives and chemical weapons with the help of contractors. I imagine Palantir and Google are already all over this. Booz and SAIC are likely to be players in this space, too. But you can test this (don't): try buying a very large number of smoke detectors and/or fertilizer and see what happens. I would be willing to bet you'd get a knock on the door from guys in body armor. B. Biological: AI systems for detecting pathogen orders to DNA companies are a thing (iirc @voooooogel worked on this?), and I'm sure they're improving rapidly. Lab suppliers are already closely watched. Buying even basic lab gear is a gigantic pain in the ass. Nothing new, but cheap AI undoubtedly helps track this stuff (or will soon). Feds and their contractors have been worried about this a long time and our domestic surveillance apparatus is extensive, and it will only get more thorough with AI. R&D is probably ongoing: many people in the NatSec space have been worried about this for 24 years time and I'd bet the funding is still there from DARPA/IARPA/HSARPA et al. for stuff like bio-surveillance, temperature monitoring at scale, are purification, online contagion early warning systems. I knew people working on this fifteen years ago, and it is hard for me to imagine its dried up post-COVID. That said, when there are commercial interests that contravene public health interests, you have a major problem, cf the recent bird flu issues, the politically motivated delay in admitting COVID was airborne, FDA-enforced regulatory capture around COVID tests, the lack of code updates on air quality, etc. etc. Lots left to do. C. Nuclear: DOE NNSA folks are bright, good at their jobs, well funded, and have staggering surveillance capabilities that will only improve with AI (if they haven't already). Their limiter is manpower. Namely, they need physicists and instrumentation on the ground to interdict stuff, and physicists you can Q-clear(++), who also want to work behind the green door all day, and want to be deployable are not that easy to come by. Strong models may ease that burden, which would be a net win for NatSec. It seems likely enough that we'll push intelligence down into sensor packages in ports and such, which will be a win for finding bad stuff. What about political risks and other persuasion risks? A. The social media companies are also AI companies, and all but Tik Tok have settled on Community Notes among other tools for epistemic hygiene. This seems to be working fairly well. Obviously this is a bit of an arms race, but the social media companies have an obvious positional advantage as the platform owners as well as being the organizations training and deploying models. B. Biggest unaddressed risk I can think of in this category is phone scams, whale phishing, et al. Voice cloning (increasingly, video too) is cheap and easy. Those are genuinely scary, and as I understand it (ianal) telcos are legally forbidden from blocking them. But this class of thing was already a huge problem and people are getting wise; turns out slave labor in Burma is really cheap! Who is going to do something about phone scams? I have no idea: someone should get on securing SS7 and the CallerID system. Perhaps @elonmusk can hop on that when he's done DOGEing fraud waste and abuse out of the budget. 3. Concentration of power This is a massively under-appreciated problem: Holden Karnofsky, Dustin Moskowitz, Jaan Tallinn, et al have made at least two runs now at capturing all power in the light cone with AI. First with OpenAI investment through OpenPhil (buying a board seat), then creating Anthropic through string-pulling at OpenAI that got the founders fired, funding via SBF and company. The activists curse is a thing: AI safety orgs make models as scary as they can. Anthropic especially is on a crusade to make every LLM but theirs more dangerous (cf their in depth guide to data poisoning on the web), but I see their schtick is wearing thin. Before R1 dropped, AI folks were so psychologically dependent on Claude they would hesitate to name Anthropic as the horrific actor they are --for safety, especially. But I notice that since R1's release they're catching long overdue reputational hit from the community, cf. @elder_plinius taking them to task, recent @repligate posts, @benlandautaylor's article in Palladium. Speaking of R1, Deepseek the biggest boon to American national security we have had in ages is DeepSeek, which is assaulting the margins of the unaligned utilitarians at Anthropic and OpenAI. Their only shots at profitability now are basically wrappers like Deep Research and regulatory capture with DOOM. But they have all but killed proprietary model APIs as a viable business model by providing an open source frontier model people actually want to talk to. Meanwhile, they are doing more than anyone else to break their psychological stranglehold Claude has had on the ML community. .@Meta might help, too, should Llama get competitive again. For this to happen, they need to build an ecosystem around their models. .@pmarca and friends are trying to fund/build such an open source AI ecosystem to rival China's. I hope they're successful. What's needed to create a thriving open source AI ecosystem? * Better wrappers, clients, etc. * Sample efficient RL finetuning, * Business-friendly distillation pipelines, * Business-friendly post-training pipelines generally (think Tulu 3 as a service...but better). * Inference providers that actually provide value above and beyond simply hosting models. (There's more to say here but I'm saving it for a prospective employer I hope will fix this problem). 4. Pivotal acts AI Safety culture drives many people to madness. One example is the Zizian murder cult that has been making the rounds. The founder of the AI Safety movement, Eliezer Yudkowsky, has been calling for terrorism on an unprecedented scale to halt AI by "flipping the game board". He is a bit cryptic on the particulars, saying all PIvotal Acts are "outside the overton window" (ie, unspeakable), but has proposed things like preemptive nuclear strikes. Eliezer's example for something like a pivotal act but not the specific act he has in mind is melting every GPU in the world, ie, an act of terrorism that would do billions of dollars in damage to government, military, and civilian systems. Since that is prospective terrorism he does feel comfortable about sharing, the 'pivotal act' he does have in mind is more extreme. He went on the record to say he would feel comfortable sacrificing up to 8 Billion people to stop ASI from being developed before he is confident it is safe to do so. Obviously the ceiling here is very high. At MIRI, intentionally inciting nuclear war was reportedly very much on the table. There was serious discussion of assassinating AI researchers (Ben Goertzel, specifically). And if you track posts over at LessWrong and Alignment Forum, you'll notice these people are losing their minds and making longwinded posts calling for Butlerian Jihad and the like. While Pivotal Act risk is in some sense a subset of CBRNE risk, its worth calling out because: a) engineers are anomalously good at terrorism, as a class, b) they are extremely well funded (from the same folks who brought you Anthropic and OpenAI), c) they are losing their minds and getting desperate with the ongoing pace of AI. To my knowledge, the @FBI is not yet surveilling LessWrong and Alignment Forum participants calling for unprecedentedly large acts of terrorism, pivotal acts, butlerian jihad and the like, nor are they locking up the funders of those who do, like Dustin, Jaan, and Holden. (SBF was put away, obviously). But they should be. Minimally, some doors should be knocked upon, and credit card statements watched. Due to the added risks presented by these AI safety people --well funded, brainy engineers who feel terrorism is a moral imperative-- we should probably be thinking about how to apply AI to civilizational resilience measures in case they get their hands on a couple nukes and devise a way to detonate them at high altitude, convince a government to do so, develop and release a bio weapon, or something else equally destabilizing. For this, I would advocate turning federal lands into food forests with robots. Permaculture, basically, minus the feckless hippies, scaled up with the help of AI and forest simulators (that are getting pretty good). My thinking here is that, in the event an AI safety person (or some other terrorist) does manage to destabilize civilization, or even if there is just a really bad solar storm, some large number of people will be able on to federal land with a tank of gas and survive for a reasonably long period of time. 5. AI takeover risk First, we are not going to allow a singleton to happen. Even Sama said they are not going to be far in the lead. When many people have many AI systems, its simply very hard for one actor to take over. And unlike people, AI systems are designed (or at least, intentionally cultivated from carefully filtered and created data). The companies that build them are concerned about liability. This is simply not a real risk. It is a power fantasy of very smart people who *wish* that intelligence automatically translated into power (or dates).

While I can't say that I'm sure such a worldview is wrong, the moral hazard is that one never feels like one has enough power to be magnanimous. So maybe the two step plan ends up being a single step plan in practice.

"Step 1: Seize power. Step 2: be magnanimous."

Dario really does seem to be power-pilled.

More accurately, perhaps: "if others believe you are not costly to destroy, and they can destroy you, and they believe that you are costly to not-destroy, then there is a good chance that you get destroyed."

If you are not costly to destroy, and are costly to not destroy, then there is a lot of incentive for you to be destroyed.

@georgejrjrjr "Oh no, watch out! This could be the most powerful thing ever invented -- we must do something about it!" they said, and were disproportionately heard by compulsive power-seekers.

Funny…and bullish for safety at deepseek. We know what happens when ml people focus on risk: they compulsively amplify them and cannot help but do otherwise. Anthropic’s guide to data poisoning is one of the more egregious examples.

@teortaxesTex PICA-X is very effective. I'd be surprised if a few cms of ablative heat shield wouldn't protect a cruise missile. Against ships I think drones are probably best used as forward observers for anti-ship missiles and would probably still be useful as such.

Correct me if I'm wrong but doesn't this, scaled up down and sideways in a few years, obviate most aerial drones? Maybe cruise missiles too. Are we back to shell slugging?

@georgejrjrjr Potential for cross-cultural super-persuasion: "FrontierLabCo is happy to announce our first post-AGI product: Gooncave(tm)"

@OtisRobertson lol yeah, exactly, most of the non-violent options I can come up with are this absurd. Least absurd but still implausible: create ASI, pivot to finance + SWE services, obsolete Goldman and the SWE workforce, buy up all compute.

Two queries: 1. Is there a plausible peaceful means of accomplishing what Dario proposes? 2. What is going on with Oxford grads and low-content sneering? I’m can be hard on Hayes Valley, but they mostly DO think things through and make substantive arguments.

@georgejrjrjr "We only need to melt _some_ of their GPUs to busy the necessary time to claim the most desirable orbits."

> super-persuasion and capitulation This makes sense re Anthropic pulling talent from OpenAI. Claude's personality micro-targets SFBA ML researchers. I don't see this working through the Great Firewall (and its cultural analog). > whoever builds a Dyson swarm We have a big lead in lift capacity, way behind in robotics. And ya can't lock in a three-six month lead that way regardless.

@georgejrjrjr Step 1: capture 5% of the power of the sun Step 2: disassemble Mercury Step 3: colonize lightcone Step 4: ??? Step 5: profit

@georgejrjrjr Super-persuasion and capitulation are "peaceful" if you squint hard enough. More charitably, if people agree to abide by some common rules, whoever builds a Dyson swarm first arguably has a pretty dramatic advantage and could capture a much larger slice of the lightcone.