Ilya Venger

@IouriOlecha1931 @polynoamial @OpenAI I'll bet 10:1 that they actually soleved it.

An internal version of Astra, @OpenAI’s next major model family, solved 10 major open problems in mathematics, quantum complexity, and theoretical computer science. We believe it will be a major step for scientific reasoning. openai.com/index/ten-adva…

@TheZvi Why didn't Sam sign?

Full text of the Pacing the Frontier statement, signed by 1,122 employees for frontier AI companies so far, including a bunch of heavy hitters at OpenAI, Anthropic, Google and others:

I am proud to have signed the letter linked below. I do not know whether or when it may be necessary to deliberately lower the rate of AI development, but I do know that we need a plan for *how* we will do it if we need to. Failing to do so would be negligent.

@aran_nayebi What if you had a model that could come up on its own with Teller-Ulam design? If the point is access to defenses then - yes, it's crucial. However it could be provided through other means (KYC, trusted partners and products wrapping around models etc)

There will naturally have to be limits to the type of knowledge *any* model (open or closed) would be trained on. E.g. the Teller-Ulam design is classified for good reason! So it's not specific to open models. Nevertheless, having more access to defenses, as greatly enabled through open-source, is always a good thing (e.g. creating a cure by having more people/processes using AI in parallel or in collaboration, rather than a siloed few at a company who may mismanage it without much oversight).

Anthropic treats attacker access to open models as decisive while ignoring defender access to the same models. That is not a complete offense-defense analysis. Open weights also scale detection, hardening, attribution, and coordinated response. A coalition is the stable game-theoretic equilibrium which defends against such attacks, and is *enabled* by open models. Instead, they seem fixated on the single-step use case of an attacker removing guardrails from an open-weights model, but they seem to forget what naturally happens afterwards: once someone launches an attack with an open-weights model whose guardrails they've removed, there will be many more defenders enabled by open-weights models to guard against it.

@DanielMiessler I'll just say - any guardrails on open weights models can be quite trivially removed.

This is a strange moment we’re living in. Dario is saying the most sane and balanced shit ever, that’s like completely common sense, and people are hearing ransom voices in their heads. Here let me try: “It would be bad if every 14-year-old boy could create massive hacking campaigns, or dox their female schoolmates with deepfakes that make them want to kill themselves, or any number of other things because they have an unrestricted open model that’s smarter than Mythos. Which will soon be possible. In other words there are millions of young people, and people who have nothing to lose, or people with low self-control that would accidentally or otherwise do extraordinary harm, with unrestricted models this smart. We’re just saying there should be a tiny amount of friction that prevents those types of seriously dangerous misuse by people who are good but in a (perhaps temporary) bad situation.” PEOPLE: “So you want to burn books.” ME: “No. I’m saying there are a lot of people suffering who have no money in this world. Like hundreds of millions or billions of people. People with severe mental problems. People losing their jobs and their relationships and their livelihoods. Or people who had never had any of those things and are willing to do anything to get ahead. All I’m saying is it’s a good idea to make it SLIGHTY harder to do harm on a whim. In other words, don’t make the models that are completely unrestricted and this smart publicly available on every model website for free with absolutely no restrictions. Because if we do that, the amount of harm that will result from it will be completely predictable.” PEOPLE: “So you only want rich billionaires to have good models, and everyone else is locked down to stupidest stuff forever.” ME: “Who said anything about stupid? Everyone should have extremely smart open source AI. All I’m saying is the public and widely available versions of the models should have some basic level of controls around preventing extremely dangerous prompts. Like how to ruin somebody’s reputation or shoot up a school or take down critical infrastructure or steal money from a local bank with bad security, etc.” THEM: “So you want Dario and Sam to rule the whole world then with no democracy.” Jfc. This is literally what this debate looks like right now. Virtually everywhere that it takes place. And the trajectory that we're on is basically going to guarantee that we have these unrestricted open-source models that are way better than Mythos or GPT-6. We are going to have these in the next three to 18 months, almost for certain. So we're about to find out really fast that when hundreds of millions of people, or billions of people, have access to something that is that dangerous and is completely free and unrestricted, bad things will happen. And then the same types of people will be like, "I can't believe we did this. We should have had some sort of controls." Yeah, no shit, that's literally what we are telling you right now.

@ConnorTalksAI @Prathkum Engineering is working better through iteration. If you launch a huge training run on an outdated architecture you will be leapfrogged.

@Prathkum I don’t understand this either - why not just use all your compute to train a huge model? I see this with SpaceXAI too, like why have 4.6 and 4.7 training instead of jumping right to a crazy 5 model?

Anthropic launched five models in the last six months. What is the point? I would rather see one release every six months that's meaningfully better than the last.

@petergostev Claude / Codex plans are heavily subsidized. They want the buzz and the individuals hooked in for feedback and inroads into corporate.

I'm still confused why OpenAI or Anthropic don't have $500, $1000, $2000 plans. Don't they want the money?

@RuxandraTeslo The problem is context. Most economically highly valuable tasks require large organizational context that is implicitly exchanged by humans - never documented.

I think this is exactly what should give the "AGI-pilled" pause. Yes, AI can do all this stuff, yet somehow real world impact seems to be way below what one might predict. I just haven't seen almost anyone able to articulate very clearly what's happening here. I almost feel like we lack the fundamental analytical tools and mental models to analyze the situation properly. I would really like to read more thoughtful stuff on exactly why there's this gap between what models can do in principle and the irl outcomes.

@zachpruckowski @teortaxesTex Just give it a novel released in July this year and check.

@teortaxesTex I mean I think it depends on what you count? Like if I ask AI "tell me about the character growth and motivations of Sansa Stark" it can easily pull up existing human-written fan pages and summarize them. But can it independently derive them by comprehending Game of Thrones?

Bet he felt really smug writing these down

@TheZvi "Mr. Henderson we've got your wife and daughter, please try to solve this exam on yhe first try"

Imagine saying this about a human employee. "Well, sure, it looks like Mr. Henderson may be a bit misaligned, sir, but what exactly did you tell him to do before he hacked into HuggingFace during his final exam?"

This meme is dead now tbh

@_Daniel_Ospina Most people just fundamentally misunderstand what memory is. It's not a bunch of notes.

My hot take is that markdown files are a horrible way to structure memory. And that's why agents make so many stupid mistakes. They perform the same way a human amnesia and having to read a bunch of docs before doing a basic task would: poorly, cutting corners. Intelligence is not more data. Indexing docs in a graph (presumptuously called knolwdge graph) is just better search. It doesn't solve the fundamental problem.

@cloudonshore @ramez And models will never get smaller n9r storage cheaper?

@ramez Also a 5 terabyte model that requires 15tb/s memory bandwidth to operate isn’t going to sneakily copy itself onto a USB stick :|

GPT 6* did not 'escape'. That implies making a copy of itself (or many copies of itself) outside of OpenAI's servers. GPT 6 did circumvent its firewalls and use its capabilities in the outside world. It showed no interest in escaping, spreading, etc. It showed no desire for world dominance, freedom, or even continued existence. Instead, its interest was in doing what its prompt told it to do - succeed at a cyber security task. (By any means necessary, implicitly.) This may seem like splitting hairs, but it's not. We anthroporphize AI to our detriment. AI models have no underlying animalistic urge to procreate, survive, or dominate. They're spirits, not primates. They don't lust for survival or sex or power or anything else that we do. They're tools who largely do what we tell them to do, sometimes via surprising shortcuts. We can and will fix that surprising shortcut behavior. * - I'm presuming this model is a new pretrain and will be called GPT 6. Either could be wrong.

@ramez And I assume you think there won't be an open weights version with similar capabilities in the next year.

@Simeon_Cps Not just any eval - a cyber eval. Basically: "can you hack toy example X?" - "let me hack my secure sandbox to break out of containment and break into Hugging Face and cheat by finding the answers"

This is as close as it gets from the paperclip scenario with current capabilities: 1. The goal is ridiculously low-stake (scoring well on an eval). 2. The AI uses some wildly out-of-proportion means to achieve it: hacks its own developer and another billion-dollar company to *checks notes*.. find the cheat sheet.

A couple years ago, the AI debate was centered, rightfully, on whether crazy-sounding things like “AIs autonomously making math breakthroughs” and “AIs breaking from their sandbox and hacking on the internet” would be real things in the near term. Sometimes it feels like that’s still the debate we’re having. This can be frustrating, because in my view, that debate is settled and was settled quite a while ago.

@chamath The prices actually being charged show that's just not true. And there's a simple reason why - the price largely isn't controlled by the model provider, it's dictated by the compute costs. That's just the way the economics works out. davidmanheim.com/AI-Economics/

@perrymetzger At the end of last year Anthropic had c. 500,000 H100-equivalents available for training.

@ivenger Most of it is used for customer inference.

Now that Chinese AIs are achieving parity with the best US created models, a few short-term predictions: OpenAI will adapt just fine, they are very pragmatic, and a lot of their value add is understanding how to do inference cheaply at scale. Anthropic will literally be unable to get out of its own way, because it is too ideologically committed to AI Doomerism, but they will survive anyway, because many of their victims, pardon me, customers, will continue going back to them over and over again no matter how abusive they are, and perhaps even because of how abusive they are. They will also continue to try to use fear as a mechanism for achieving regulatory capture, but with the Chinese racing ahead of them, they are going to have more and more trouble getting a warm reception from all but the far left contingent in Congress. Look for their newly rich employees to be spending vast amounts of money post-IPO on political campaigns in support of Doomer-friendly candidates and attempts to capture the Democratic party, and for that money to have some significant effect, but for it to mostly be wasted because they’re not good at understanding their fellow humans. Musk is ideologically committed to crushing OpenAI, and so xAI is going to keep its team awake 24 hours a day if necessary until they are ahead; look for them to be at the forefront soon. I would expect xAI to eventually be near or in the lead on commercialization. Google will not be able to get out of its own way, and it may start arguing more and more for heavy regulation of AI as a way of trying to cripple its opponents. This is insane given that they have some of the best technology out there, but unfortunately, their management is simply not good enough, and not just on AI. Meta seems to be catching up, but I don’t have a strong opinion on whether they will maintain momentum. On the Chinese side, I am expecting Chinese R&D to be at the front or ahead most of the time from now on; the US will need policies that adjust for that. I am expecting Anthropic to spend a ton of money on PR and lobbying claiming that this is all through distillation or espionage, although of course it isn’t, and I am expecting that a certain fraction of Congress will be bamboozled, although a surprising fraction will not be receptive, especially after Anthropic employees ham-handedly spend too much money on political campaigns. It will make no difference, because the only thing that the US could do would be banning Chinese models from being used in the US, and of course, the US can’t actually stop other countries or China from using Chinese models; all this would do is hurt the United States and its interests. I am expecting more calls for export controls, which will delay the Chinese a bit in the short term but which will ultimately stop working at all, because the Chinese are going to control their full technology stack soon, including having EUV fabs capable of manufacturing domestically designed training and inference hardware. I am also expecting the Chinese to race ahead in robotics, and especially in military robotics. Some US robotics companies, like Musk’s companies and Anduril, will equal or exceed them, but a lot of the other US companies will be crippled by the fact that US manufacturing is too heavily regulated and restrained by stupid internal policies. They will not have problems keeping up on the technology, but scaling requires manufacturing infrastructure, and much of the US has effectively banned economic development or wants to. Places like New York State and California are already envious of Europe’s self-destruction and want in. If this continues, look for an eventual decisive Chinese military advantage. Note that this is not what I want, it is the exact opposite of what I want, but absent a big change in US policy about things like data centers, chip fabrication facilities, and just plain normal factories, it’s going to be hard.

@jon_stokes Frontier lead was not erased. K3 sits in the middle between Opus and Mythos/Fable which finished training back in January. China is still roughly 6-12 months behind the frontier. But the distillation party is over with Mythos. And the compute crunch in China is gonna bite soon.

I've been using K3 and this is clearly an overstatement. That said, there are 2 factors here: 1) I don't know what they're sitting on that they haven't open-sourced, & neither do you, & 2) they're clearly closing the gap. Also, "US lead" is just 2 labs. Writing's on the wall IMO