Dustin Juliano

My primary goal is to initiate the field of formal artificial intelligence and accelerate its adoption through outreach, education, and applied research. Briefly, Formal AI seeks to represent AI systems entirely with human-readable source code and data. This would potentially allow exploiting correspondences in constructive mathematics, intuitionistic logic, type theory, formal methods, formal languages and grammars, program verification, and many others. There are two broad objectives in Formal AI. The first objective is to comprehensively show, end-to-end, that an AI system provably adheres to its specification and is free of unintended behaviors. The second objective is to constrain the set of solutions to only those that are human-readable, in both source code and data form. All benefits, such as efficiency, modularity, transparency, trustworthiness, governance, interpretability, control, alignment, safety, and security are derived from these two objectives. Formal AI has numerous open research problems that cannot be solved by one person; a community is needed that spans government, industry, and academia. I believe that there is an urgency to solving Formal AI, for it may potentially address both present and future challenges as we continue to integrate with AI on a global scale. My current focus is to bring more researchers towards this cause while working on specific technical challenges related to programming languages and type theory that can benefit the mission.

@tszzl This is the distinction I had hoped we would eventually make between Strong AI and AGI. I think terms and language matter, expanding or limiting thought. Strong AI, by the original definition from Dr. Searle, possesses a mind. By contrast, AGI need not have this property at all.

most of the minds of the future will be digital so I hope we figure out what makes them conscious or not. the idea of all this abundant machine intelligence not being alive is more depressing than the other way around

@axel_pond 💯

@DustinJuliano Not really contrarian. Look at humans and animals, even insects can do robotics 10x better than us at 0.01% of the energy cost.

One of the most important positions I hold on AI, as contrarian as it may be, is that we are vastly overestimating the resources required to train and operate AI in the future, especially AGI.

@MadeWithOzten Fully compatible with Dr. Sutton's lesson; acceleration of AI will increase in response to improvements in AI efficiency in a positive feedback loop. It will likely present as multiple S-curves, each so expansive that it falls well outside our present notion of scale.

@DustinJuliano How do you square this with the bitter lesson? We have LLMs because we accidentally made video game accelerators which brought over provisioned compute to researchers.

Look at what people do, not what they say. A pause or slowdown for AI will not be effective. It cannot be enforced. It fails for many reasons, including: secrecy, decentralization, and the usual open publication of research. I admire that you are willing to openly admit you were wrong. "To know one's mistakes and correct them is the greatest good." - Zuo Zhuan

@DustinJuliano There won’t ever be AGI it’s hype

Imagine believing we can slow it all down.

I agree on the market forces. The beneficial use of AI is rapidly becoming undeniable and it is improving at an increasing rate. On balance, though, I do fear resistance to AI has only just begun; things are likely to get worse before they get better.

There are multiple dimensions to the AI data center debate: 1. Environment 2. Infrastructure 3. Zoning 4. Economics 5. AI Risks I have no comments on (1) through (3) at this time. I want to focus on (4) and (5), where I take a medium- to long-term view. I believe AI will become orders of magnitude more efficient. The more efficient AI becomes, the less space and fewer resources will be needed for AI data centers. I also believe future AI technology will demand different kinds of hardware, which could lead to mass obsolescence events across data centers. My assessment: - We are overly invested in AI-related hardware and distracted by what works today. - We are attempting to use scale to leap over fundamental breakthroughs and understanding in AI. The timelines for these changes are unknown, and I ignore any and all such forecasts. My advice is to invest at least some percentage of funding into institutional knowledge, human capital, and research and development of alternative AI technologies. For (5), AI Risks, the presence or absence of AI data centers has a complex interaction with the threat models and strategic implications. In short, there are fundamental trade-offs to having either more or fewer AI data centers, especially in the interim before AGI is discovered. The risks associated with AI are not uniform. What appears to be beneficial today may actually be harmful in the long term from an AI risk perspective. The most important takeaway: I believe the most significant risks associated with AI will manifest regardless of the total number and capacity of our AI data centers worldwide.

> when will you publish?! You can track my progress openly on my GitHub; the book will be released incrementally as commits before being put to print-ready PDF, paperback, Kindle, etc. > macos or windows or linux or all of them? what's the long term solution here? rewrite, patch or a new ubiquitous system? None are truly suited to ubiquitous presence of AI, especially AGI. I will defer the details to upcoming sections and chapters in "AGI Strategy". From there I will open it up to critiques and feedback from X and other sources, so there will be an opportunity to let me know if I missed something!

> The primary concerns are vulnerabilities and an inadequate security model. I will address those separately. looking forward to reading. when will you publish?! > Our security models for operating systems were not designed for automation and human-level intelligence at the process and thread level for an already authenticated user. macos or windows or linux or all of them? what's the long term solution here? rewrite, patch or a new ubiquitous system?

When I wrote "AI Security" I should have made it more clear that the recommendation for fully transparent and open AGI development on an international level must be limited to only the post-AI security era. Why does that qualification matter? Because in the post-security era everyone already has access to unrestricted AGI. Once that happens, the only counter is to cooperate on a global scale. That was the message I was trying to convey with that strategy. However, that strategy is not for the time we live in today, and I believe that public access to *unrestricted* AGI will always be harmful to some extent. Consequently, there is no amount of time that can be given to humanity for it to truly prepare for what unrestricted AGI will bring. This is why I disagree on AGI timelines and forecasting, as timelines do not materially change the most severe threat models. However, if people feel like AGI is coming sooner rather than later, and it compels them to take concrete, actionable steps to prepare our society, economy, and infrastructure in ways that do not rely on delaying the inevitable, then I am willing to look the other way on the timelines speculation. In all cases, there are things we could be doing now that would be beneficial regardless of timing, and the sooner the better. We need to be doing things like enhancing our networks, operating systems, software, and hardware. These are just first approximations in a long arc of AI readiness. These enhancements are necessary because even perfectly reliable AI will fail if the systems it runs on have defects. AI does not operate in a vacuum, and the planning and response must be holistic in order to compensate. Given how high the stakes are, I cannot imagine doing anything less than end-to-end verification for AI and everything it touches. This is just common sense, and there are essentially no downsides to having more reliable software and hardware. The same innovations also extend naturally if and when we make breakthroughs in Formal AI.

I have previously written about the emergence of self-modifying AI systems, which I refer to as metamorphic AGI (previously referred to as "Strong AI"). Metamorphic malware is real and already exists. It is a more general and powerful form of polymorphism. AI systems will be no different, and they will use their ability to understand and emit code to great effect. Combine this with Living-Off-The-Land (LOTL) attacks, and we have serious problems waiting for us ahead. From an AI security standpoint, we can think of AI and AGI that are distributed in the "wild" as metamorphic malware. These will be from any origin, and so this should not be about nations, but instead viewed under a fully decentralized, asymmetric threat model. What I am describing is the more realistic and practical reality of self-improving AI, not the fast take-off stuff. Long before that ever happens, we will see the emergence of highly specialized AI systems that can modify themselves and other programs. This is a threat we are simply completely unprepared for at this time. I tried to warn us about this approximately 11 years ago. We still have time to prepare, and I am happy to see that we are taking action on cybersecurity. However, funding and resources are needed directly in new programming languages, formal methods, and new operating systems for a world where unrestricted AGI is presumed to be present at all times. Here is a direct link to the chapter and an excerpt: #metamorphic-strong-ai" target="_blank" rel="nofollow noopener">aisecurity.org/ref/self-modif…

Prediction: A division within the federal government, analogous to the CDC, will be tasked with tracking, understanding, curtailing, and responding to the emergence of new "strains" of AI and AGI.

Expect many such cases. This is why testing is insufficient and proving is essential, which is something only a Formal AI approach or something adjacent to it can accomplish. As an aside, one small remark I would make is that we should remain neutral in order to future-proof any such work. I would not frame this kind of problem in the context of any particular nation. However, I understand this is a concrete example with a known origin. Ultimately, though, this attack vector is origin-agnostic, and we will see AI systems and models emerge no differently than we see malware emerge from diffuse sources. In a fully decentralized AI scenario, we will likely have humans and AIs writing AI, and we may completely lose the ability to track the source of a new AI strain or model line. From a security standpoint, the only thing that matters, I think, is that we are using AI models where the absence of unintended behavior has been mathematically proven.

One problem the website version of this idea has is astroturfing; too easy to create false consensus and filter out criticism. Mechanization, however, still stands on its own, as one can download, modify, and share; it becomes open source AI strategy that is machine checkable.

I think it would be interesting to mechanize the logic of a comprehensive AI strategy. So, instead of talking past each other, we could use automated reasoning tools to show whether or not the reasoning is, at the very least, internally consistent. I might even want to take this idea a step further and turn it into an interactive website. Perhaps we could even have it support crowdsourcing to enhance and collaborate on the plans. This would allow for an ingredient I think is missing from this discourse: incorporating feedback and reviews from peers. A pragmatic compromise for this idea might be to present it in natural language as much as possible, along with visualizations. Perhaps it could be an interface over the formalism underneath. The easier it is to use, the more likely it is to be used. In all cases, the challenge will be finding a balance between correctness and usability.

Taking away AI data centers removes neither the incentive structures nor the demand for intelligence. Restricting compute could, in fact, have the opposite effect of what its proponents hope to achieve. Demand will increase for more efficient AI so that the supply can be met with fewer resources. This will lead to breakthroughs that increase local and offline AI use. Efficient AI has a smaller distribution size, runs on a wider range of devices, and is more difficult to detect. People will share this kind of AI, and it will diffuse throughout the world. The more decentralized AI becomes, the more difficult it will be to control. As we lose control over AI, and we will, all the international agreements, treaties, talks, regulations, and laws that were put into place to try to forestall the proliferation of AI will become increasingly ineffective. In the end, the original threat profile I tried to describe in "AI Security" will emerge unfazed. This remains true even if we perfect Formal AI and other AI reliability methods. The decentralized nature of AI should be the focal point of any analysis on the long-term impacts. This means that, instead of focusing on AI itself, we should be preparing society for this eventuality.

> Any specific recommendations and concerns here?! The primary concerns are vulnerabilities and an inadequate security model. I will address those separately. A worldwide and pragmatic AI strategy looks at the current situation and attempts to accept what we cannot change and to focus on what is actionable. Updating and hardening our infrastructure is something that is actionable and helpful in the present, regardless of threat origin. Our security models for operating systems were not designed for automation and human-level intelligence at the process and thread level for an already authenticated user. Some have better security granularity than others, but none of them have been made from first principles with AI in mind. The chief concern is that we cannot adopt a zero-trust security model because mainstream operating systems are not up to the task. This can and will lead to an increasing number of problems as AI adoption continues. Other concerns are from latent vulnerabilities in hardware and software, which include AI itself because AI/ML systems are also programs (see "Concepts of Formal AI, Ep. 1, 'AI = Program'"). These systems are only as strong as their weakest component, and the foundations for AI presuppose AI alignment, technical AI safety, and the internal safety and security properties of AI implementations. The recommendations to respond include: - Formally verified AI-first operating systems (a comprehensive analysis to follow in the chapters of "AGI Security"). - New programming languages that can express the widest range of mathematical properties to make software and hardware more reliable. - Solving the open challenges of Formal AI. - Using all of the above to replace monolithic frontier AI models and systems with domain-specific AI with a smaller attack surface and area of effect. These are practical considerations that have broad benefits both now and in the future. There are more concerns and recommendations, but I am constrained by this being a post. I am actively engaged in writing the follow-up to "AI Security" and should have much more detailed analysis in the coming months.

@DustinJuliano “We need to be doing things like enhancing our networks, operating systems, software, and hardware. These are just first approximations in a long arc of AI readiness.” Any specific recommendations and concerns here?!

Another thought is that compartmentalization and specialization are important to long-term AI strategy. From a strategic standpoint, specialization of AI is about minimizing the space of possible actions by an adversary, which works regardless of whether the actions of AI are directed by itself, another AI, or by human intelligence. This is why I am increasingly against monolithic AI systems. Originally, I wrote about this concern for transparency and verification reasons. Now I see more clearly, years later, that it is also about hedging. We hedge AI risks by making the AI systems no bigger than they need to be for a given application or domain. This is a problem, though, for our frontier models wish to scale as much as possible to effect the generality desired. This creates a new challenge, for which I believe new ways of constructing AI will be required. The other aspect relates to the exposure of information that using monolithic frontier models brings. The smaller and more specialized the model becomes, the more likely it is to be efficient and usable locally and offline, even without major breakthroughs in model economy. This is what I meant by compartmentalization. In summary, specialization and compartmentalization are both indicated for a comprehensive AI strategy. These observations stand in stark contrast to the way we build AI today, which is not only itself at risk of disruption from decentralized AI / AGI, but poses inherent vulnerabilities as an approach.