benthamite🔸

I like that OAI has been discussing things in public and feel more positively towards them than if they had just made their original post

@csvoss @robbensinger @BenKorpan @IsaacKing314 @tszzl I appreciate you and other OAI staff pushing for changes and having public discussion, and I hope that you continue doing so!

@benthamite_ @robbensinger @BenKorpan @IsaacKing314 @tszzl slight correction to both: technically I said “through contract and more,” and by doing so I wasn’t necessarily claiming that the contract alone is *sufficient* to protect the redlines; just that it’s one part contributing to a *system* that does Evidence of the system working:

Is this a fair recap of what's known about Anthropic v. DoW? 1. It's a mystery how any of this started. Some sources claim that an Anthropic official was talking to a Palantir executive, and expressed "disapproval" for using Claude to do things like capture Maduro; and then the Palantir executive reported this to DoW (techpolicy.press/a-timeline-of-…). Anthropic seemingly disputes whether this happens, saying, "We have never raised objections to particular military operations nor attempted to limit use of our technology in an ad hoc manner" (anthropic.com/news/statement…). But perhaps Anthropic means that they never formally raised objections...? Separately, a defense official told the Washington Post that Dario was asked "If an intercontinental ballistic missile was launched at the United States, could the military use Anthropic's Claude AI system to help shoot it down?", and that Dario gave a response along the lines of, "You could call us and we'd work it out." But, per the Washington Post: "An Anthropic spokesperson denied Anthropic gave that response, calling the account 'patently false,' and saying the company has agreed to allow Claude to be used for missile defense." A lot of people have floated the idea that this may have been a scheme by Palantir and/or OpenAI to kneecap Anthropic. But I don't think there are any smoking guns there, aside from the obvious fact that OpenAI's President is the world's single largest Trump donor, and somehow the direct consequence of all of this was an OpenAI competitor suffering a massive blow while OpenAI scores a major DoD contract. 2. It's suspicious that all of this happened on the same week that (a) Anthropic rolled back all of its core safety commitments (x.com/twitter/status…) and (b) the US invaded Iran. But as far as I know, there's no public evidence that any of these events are connected? 3. Based on all the public info so far, it seems that OpenAI fully capitulated to DoW's substantive requests and then @sama lied and deceived the public about this (with some help from @tszzl and other OpenAI staff)? Peeling past the layers of rhetoric, the substance of OpenAI's deal seems to be "DoW does whatever it wants" (x.com/TheZvi/status/… + x.com/ARozenshtein/s… + x.com/ShakeelHashim/… + x.com/jeffrsebo/stat…). Presumably the point of Altman's rhetoric was just to add some confusion and noise to the room in the hope of preventing a staff exodus; by the time people realize what's happened, if they ever do, the news cycle will have moved on and the urgency will have diffused from the room. (One similarly has to imagine that Anthropic walking back all of its core safety commitments might have prompted a staff exodus at Anthropic, if the timing hadn't worked out so perfectly for the org. But maybe that's an over-optimistic take on Anthropic staff? It's an incredibly well-paying, cushy job, and now it's a job where you get to feel like a hero to many people even as you actively work to reduce the lifespan of your kids (x.com/robbensinger/s…). The halo effect is a doozy, and I have to imagine there's a lot of pressure to find some excuse to focus on the happier side of things.) 4. It's a mystery what DoW's goal was here (if it wasn't just 'find some excuse to shift money and power from Anthropic to OpenAI'). Possibly it was just a series of misunderstandings that escalated via clashing egos. Some have suggested that DoW's objection to Anthropic's terms was just "a matter of principle" (x.com/deanwball/stat…), and OpenAI claims that DoW has no plans to do mass domestic surveillance (x.com/ShakeelHashim/…). (Not that OpenAI's claims about the physical world carry that much weight nowadays...?) But either way, it's certainly suspicious that DoW has focused the discussion on "legal uses", when Anthropic claims that the big issue is that US law hasn't caught up to some mass surveillance loopholes that SotA AI now enables (anthropic.com/news/statement…). Similar questions exist re how actually-important Anthropic's bright line on near-term fully autonomous killbots was, from the DoW's perspective. Given that OpenAI's new DoW deal seems to genuinely give vastly more leeway to DoW to do killbots and mass domestic surveillance, this seems like some evidence for "Anthropic's red lines were actually meaningfully constraining things DoW wanted to do, and that's what this was all really about", and some evidence against a mere personality clash or a ploy to shift power to OpenAI. 5. Nearly all of this discussion is ignoring the real elephant in the room, which is that AI companies are locked in a race to build a technology that most leaders in the field agree poses a serious chance of killing us all. Anthropic and OpenAI seem to be aware of what the real name of the game is, even if they dramatically understate the risks and frequently misunderstand their nature. In contrast, I haven't seen recent evidence that DoW or the White House are tracking this at all. (And many of their actions don't seem to make sense if you view AI in those terms.)

@robbensinger @csvoss @BenKorpan @IsaacKing314 @tszzl Fair. I may be misunderstanding them

@benthamite_ @csvoss @BenKorpan @IsaacKing314 @tszzl Chelsea says above that OpenAI has three clear redlines that "are protected *in depth*, through contract". This contradicts what you're saying, which is that the contract doesn't protect against anything, it just takes note of which things are already protected anyway (by law).

I understand the OAI justification to be: 1) bad things are illegal, so you don't need to ban them in contracts, 2) to the extent bad things aren't illegal you should make them illegal instead of banning them in contracts. I'd be interested to hear from @csvoss and @tszzl if that's correct and if so which one is doing the work for them personally

Ah, in that case, sure? I'm not quite sure I see why specifying "legal" changes anything; it's not as though the DoD is supposed to be permitted to break the law when contracts don't specify this.

@robbensinger @csvoss @BenKorpan @IsaacKing314 @tszzl I think @csvoss is objecting to your phrase "can do whatever it wants" since dow is restricted to doing only legal things. IIUC it would be more accurate to say that dow "can do all of the things ant was worried about"

@csvoss @BenKorpan @IsaacKing314 @tszzl Seems relevant?: - x.com/jeremyphoward/… - x.com/jeremyphoward/…

@tszzl What suggestions do you have for people without operational insight?

most of the so called ai watchdog groups on here are barely concealed partisans. they operate almost entirely on personal trust and immediately enter the frame control of whichever side they’re on when they have no operational insight

@ARozenshtein I think he was trying to point out that it's good that the usw is saying this, not make an independent claim that the statement is true

I think I understand what Boaz is trying to say, but given that the National Security Agency is part of the military and given the amount of incidental collection of domestic communications that (legally) occurs under FISA and 12,333, this statement is simply not true.

#1 post in /r/OpenAI is titled "The end of GPT". 7/10 top posts are about cancelling accounts/anti-OAI. 10/10 /r/anthropic posts are pro-A\, 3/10 about people switching from oai

"I wish we had more empirical evidence re: Holden's claim 'there are companies not terribly far behind the frontier that would see any unilateral pause or slowdown as an opportunity rather than a warning'"

I stand with Sec. Hegseth. I too would rather invoke the DPA than do a project without claude

@_ueaj What are you requesting they do?

Call you reps! It's the first time I've done it, and it turns out it's very easy and it takes like 5 minutes. Every AI researcher in America should be doing this right now. Your work is going to be used for surveillance and automated weaponry

@TetraspaceWest Yes, addressed by Paul here: alignmentforum.org/posts/fRSj2W4F…

is METR at all worried that they seem to be an external marketing department for AI corporations

@xlr8harder Toby ord found that the mechanism you propose fits the metr data reasonably well: forum.effectivealtruism.org/posts/qz3xyqCe…

I think people are overinterpreting these time horizon evals. They are very impressive! But when error rates are near zero, and tasks require many successful steps in order to complete, small absolute improvements in error rate have a multiplicative effect. Consider a task with 1000 steps. A model that makes errors on 1% of steps will succeed about 37% of the time. A model that makes errors on 0.5% of steps will succeed nearly 61% of the time. This is a dramatic apparent improvement in reliability and could come from only eliminating, for example, one single kind of reasonably uncommon error. That's not to downplay the accomplishment here, but I think people are reading more into these numbers than they should.

@MichaelTrazzi Why is 80% better? Naively I would have thought that 50% is better because it's less affected by outlier data points

For people freaking out: 14.5 hours is the 50% horizon with a limited task suite The 80% horizon (which matters more for the automated coder in AI 2027 type scenarios) isn't on a faster trend

Yes, Zvi's were on risk [1-4] I'm confused about what you think they should have done here then. Should they have just cancelled the contest because people didn't give submissions pointing in the right direction? Said "We will give Eliezer the prize even though he didn't submit anything?" 1. thezvi.substack.com/p/the-crux-list 2. thezvi.substack.com/p/to-predict-w… 3. thezvi.substack.com/p/stages-of-su… 4. thezvi.substack.com/p/types-and-de…

@benthamite_ @MatthewJBar I think Zvi was (wrongly) more optimistic than I was about OP epistemics and submitted something, but maybe it was about risk numbers instead of timelines. (The risk essays also argued in the opposite direction of later OP updates.)

You ran a $50,000 "Change Our Views" essay contest in 2022 about your AI timeline of 2050, and gave the prize to an essay arguing for even longer timelines, as I had predicted would be the case, even as I also knew your timelines would later update shorter.

@allTheYud @MatthewJBar (though of course it's not so long ago that 47% by 2043 was considered a "short" timeline!)

Were there any submissions arguing for shorter timelines? At least of the ones made public on the EA Forum[1], the closest I can find is @MatthewJBar arguing 47% by 2043.[2] So I think maybe the long timelines people won by default? 1. forum.effectivealtruism.org/topics/open-ph… 2. forum.effectivealtruism.org/posts/fsaogRok…

@doesnotplaydice @ben_j_todd Then I would encourage him to reference those facts in future communications

@benthamite_ @ben_j_todd he has the facts on his side

Jensen Huang when asked by his biographer whether the world is prepared for AI risk: "This cannot be a ridiculous sci-fi story,” he said. He gestured to his frozen PR reps at the end of the table. “Do you guys understand? I didn’t grow up on a bunch of sci-fi stories, and this is not a sci-fi movie. These are serious people doing serious work!” he said. “This is not a freaking joke! This is not a repeat of Arthur C. Clarke. I didn’t read his fucking books. I don’t care about those books! It’s not– we’re not a sci-fi repeat! This company is not a manifestation of Star Trek! We are not doing those things! We are serious people, doing serious work. And – it’s just a serious company, and I’m a serious person, just doing serious work.”

When you run claude --dangerously-skip-permissions and it does something dangerous without asking permission