Steve Omohundro

My essay "Regulating AGI: From Liability to Provable Contracts" agisocialcontract.substack.com/p/regulating-a… was just posted as part of the "AGI Social Contract" project: agisocialcontract.org

AI theorem proving advancing rapidly! Critical to AGI Safety!

Very important work highlighting the current state of AI Cyberattacks, the likely most critical next AI Risk.

Extremely important work! With all the excitement around AI code generation, let's get the ball rolling on verified code generation!

🧵 "What should the role of governments be after the upcoming AI economic transformation?" We're launching an new anthology on a new "AGI Social Contract" - in collab with 12+ experts to explore new strategies & policy interventions for an AI transition. In collab with: @akorinek, @degerturann, @steveom, @IasonGabriel, @JulianDJacobs, @sjmanning, @dhadfieldmenell, @sethlazar, @petersalib, @collegraphy, @synchroaphasia, @deanwball, @JustinBullock14,

A group of leading AI scholars is backing a private governance model for AI policy solutions — and they're rallying around SB 813. 🧵

Great work! You've brought theorem proving to the thinking LLM revolution! Thank you for making the prover and the autoformalizer freely available. I believe these will be essential to effective AI Safety.

Shocking! This year's batch of Y Combinator startups is all in on "Vibe Coding": youtube.com/watch?v=IACHfK… At 10:00 they say "one quarter of the founders said that more than 95% of their codebase was AI-generated!"

random thoughts/predictions on where vibe coding might go: - most code will be written (generated?) by the time rich. Thus, most code will be written by kids/students rather than software engineers. This is the same trend as video, photos, and other social media - we are in the command line interface days of vibe coding. For the majority of creators, vibe coding will eventually fade, and vibe designing (with a visual paradigm) will come to dominate. People ultimately think better in a GUI-like format than a CLI-like format. Thus, in vibe designing you will show the AI the design outcomes you want, and then everything else is done for you. Yes, you may end up with tools to tweak the design details for extra controllability, and provide additional mockups that then get filled in underneath with code. But maybe folks will build software without seeing or learning a programming language. - vibe coding could reduce the need for open source libraries as more code will be generated from scratch by AI. Code will be more of a disposable commodity, with less reuse, and instead generated on the fly for personalized use. It's interesting to see right now that creating a new project is easier than editing a project, because the latter requires a lot more context/complexity. Interesting dynamics if something like this continues - "trad UX" and design standards give way to post-modern/fragmented software, as millions of new vibe coders create experiences with no prior know how and new perspectives. New patterns will emerge, as TikTok/YouTube has done to film making and trad entertainment. The world will go beyond buttons and modals and scrollbars and other things. Software may become unrecognizable before it coalesces again - if vibe coding makes software trivial to build, then the bottlenecks shift to other places: 1) consistent creativity that stays ahead of everyone else. Anyone can write a tweet, but the best creators are the ones who consistently come up with new ideas. 2) distribution and network effects, where the first vibe coded product doesn't win, but rather the first vibe coded product that hits scale that wins - imagine products that automatically adapt based on user behavior, rather than based on the actions of the vibe coder. For example, if the vibe coder has specified that the signup funnel should easy, then after seeing users struggle with it, the software can automatically vibe code itself to improve the flow by dropping steps or adding explanatory text. Right now we are in a paradigm where PMs specify behavior that software engineers specify in code. Imagine if PMs can specify outcomes, and the software is configured to automatically adapt to hit those outcomes what other wacky ideas should be on this list?

Amazing example of "Vibe Coding" an entire flight simulation game using thousands of prompts and no coding: x.com/NicolasZu/stat…

I just gave a short talk arguing that extending "Vibe Coding" to "Vibe Proving" and "Vibe Specification" will power formal methods for AI Safety: "It's a New Day for Formal Methods!" drive.google.com/file/d/1Jv8W1F…

I agree that R1 shows that we are unlikely to achieve AI Safety by mandating constraints on the big lab AIs. But "Acceleration is the only way forward" is suicide. My opposite take is: "It's time to get serious about building truly safe and secure infrastructure so that humanity survives regardless of the AIs that are created."

In November, I was honored to participate in the Mind First Foundation's event "AI Safety Salon with Steve Omohundro": mindfirst.foundation/presentations/ They just posted the two videos of the event: youtube.com/watch?v=J0jxkY… youtube.com/watch?v=ONOy5f… In addition to Q&A and a panel discussion with Preston Estep and Dan Faggella, I gave a talk entitled "AI Benefits Without AGI Risks". I presented "Provably Safe AI Infrastructure" in the context of the biological evolution of agency, intelligence, and cooperation, the "Neat and Scruffy" developments in AI since 1956, and what it means for the future.

AI safety people were right. Again. Instrumental convergence is proven yet again. Empirically. So far we’ve seen: ✅ Self-preservation: You can’t achieve your goals if you’re turned off (see Apollo's recent paper. Link in comments) ✅Resource acquisition: You can’t achieve your goals if you don’t have energy, money, or computing power (see Terminal of Truths making money to increase its compute) ✅Goal preservation: You can’t achieve your goals if people change your goals (See Claude below) It’s a fact. Also, it was obviously going to happen even before we got the empirical evidence, because it’s just the rational thing to do, as Claude below describes.

🏆 ⭐ We're thrilled to announce the 2024 Future of Life Award winners! This year, we honor three groundbreaking experts who laid the foundations for ethics and safety in computing and AI. Learn more about the invaluable work of Batya Friedman, James Moor, and Steve Omohundro:

PSA for Boston-area folks!! On Nov 22nd there will be a special AI safety salon at Microsoft NERD w/ Steve Omohundro (@steveom), AI safety pioneer. Organized by Alex Hoekstra & @PrestonWEstep from Mind First Foundation (@mind_first). RSVP here-> docs.google.com/forms/d/e/1FAI…

We need more "tool AI" as in Elon's space triumph today, and less AGI world domination hype as in Dario Amodei's recent "entente" essay. My essay below argues that "scaling quickly" won't lead to Dario's "eternal 1991" – but perhaps to 1984 until the end, with a non-human Big Brother. lesswrong.com/posts/oJQnRDbg…

Debates over AI Policy like CA SB-1047 highlight fragmentation in the AI community. How can we develop AI policies that help foster innovation while mitigating risks? We propose a path for science- and evidence-based AI policy: understanding-ai-safety.org