Drake Thomas

Explaining things you do is often costly and draws down on various resources, like "time of your comms team" and "political capital of pro-weirdly-high-transparency factions" and "ability to do future decisionmaking efficiently without having to track and ensure that details of your process will be logged and publicized" and "share of total external attention paid to everything else you say". These costs are super worth paying in a bunch of cases but I don't think it's a free action and often these are the same resources I want to save for stuff like "doing good transparent comms about the risks of Anthropic's models". (Epistemic status: I will glomarize about specific instances, but in general I tend to have ~no interesting private info about things like this and would leave similar commentary about a corresponding thread on another AI company; almost all my tracking-ant-decisionmaking hours go into arenas I think are much more important to monitor.)

@bshlgrs @claudeai Does that make sense?

We're extending Claude Fable 5 access on all paid plans, as well as keeping Claude Code’s weekly rate limits 50% higher, through July 19.

@carl_feynman Paper reads as pretty strongly AI-written to me; Pangram thinks it's at least moderately composed of AI text. Idea could still be worthwhile but I'm a lot more skeptical of any choices about emphasis or claims of evidentiary strength than I would be from a human author.

Here's a marvelous paper. It proposes that genetically programmed human language learning doesn't require any genetically specified language mechanism, just a tendency to pay attention to certain stimuli, and to try really hard to predict them. Then generic learning mechanisms can do the hard work. The stimuli to pay attention to are rapid ostensive rhythmical streams of actions. This covers language, and sign language, and also music. "Noticing language: What echolocation tells us about language emergence" by Brett Reynolds. (Epistemic status: I had this idea a couple of years ago. But I didn't think hard about it or write a paper, so I deserve no credit.) lingbuzz.net/lingbuzz/00930…

I guess I'm imagining that in full research transparency world, there's like 20x as much total info to sift through, but the number of things that can be salient enough to count as non-obscure is still bottlenecked by human attention, and it will still take paying a bunch of attention to implications of obscure stuff to reach correct takes about what's going on in a way that requires a bunch of hours of effort, such that in practice the pool of people willing to put in that effort will still be in large part the people working at AI companies (rather than the discourse opening up to some 10x larger pool of people who would have had really good takes about model training decisions if only there were full research transparency). Above is mostly specifically about research and fiddly details of what the AI cos are doing to train models, I definitely think things like "what capability level is there internally" and "what is the current level of risk posed by models" and so on would be usefully salient to a bunch of people and clearly have large transparency benefits.

@MaskedTorah @DKokotajlo I think this is a non-sequiter: currently if you want to understand various details of what's going on, you need to pay a bunch of attention to implications of obscure stuff; it would be way less work to know that stuff if important points were just stated directly!

As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it's very important for preventing concentration of power. (Explanation below)

@etirabys seems like there’s disagreement in replies, poll time! what happens if you go to eti’s profile and scroll through the 5 most recent posts, looking for the original blog post thread (not this QT of it)?

This post and subsequent posts are hidden from me, on my own profile. I logged onto an alt and visited my profile and saw the thread for half a second before it blinked out of view. That's annoying. I already accounted for being deranked by the algo but this seems a bit much.

Hm, I don’t feel like I see many of these benefits showing up in the most transparent pieces of the pipeline now? Eg Anthropic gives a lot of pretty specific detail on the architecture and training pipeline for their production bio classifiers; I can’t think of any ways this has propagated to regulator decisionmaking or public commentary from other companies (though maybe it has privately influenced other cos?). So I’m skeptical by default that we’d see a lot of these benefits if everything were like that. But maybe very low interest and commentary is what you’d expect at this point in the timeline when most of the public isn’t freaking out, so absence of evidence isn’t much signal?

@MaskedTorah @DKokotajlo A lot is: - Practical+working regulation depends a lot on having a legible public scientific understanding. - It helps a lot of companies can comment on practices of other companies and public can comment on this (once other companies have info, may as well go public)

@DKokotajlo TBC, I do think the conversation would get significantly better in several ways with total research transparency, but I don’t know that its SIZE would - not sure there are more than like a thousand people waiting in the wings to start having good takes.

@DKokotajlo I frequently see people having technical conversations, who are literally in the top few hundred people globally for degree of interest and attention on this stuff, get things wrong because they haven’t paid attention to obscure stuff AI labs have published.

@BronsonSchoen Sorry, I’m not saying the RR itself substitutes for AST, I’m saying the external reviewers’ public assessments OF the risk report do.

I’m not very clear on how the risk reports relate to the AST reports. It seems like the AST reports are understandably a lot of objections to how things are framed elsewhere, so not surfacing them is going to create an overly optimistic view. It doesn’t seem like an unredacted risk reports fills the same role as a stress test? If so, I would expect there to no longer *be* a separate AST report. Is there external review of the unredacted AST report?

RSP v3.4 is out! Not a huge change, but slightly more diffs than some previous incremental updates, and I helped out with workshopping a bunch of the changes here and feel pretty good about them. Happy to offer takes where people have questions. anthropic.com/responsible-sc…

Yeah, I'm a bit torn on this - I think it's pretty useful and good for public transparency, but also introduces moderate pressure on AST's communications if they know they're grappling with public comms as well (redactions make it more tractable but don't solve this, and also add their own friction that consumes AST time which could otherwise point at object level work). I'd feel a lot more draw to do this if we weren't already getting public external reviewers to weigh in on unredacted risk reports (not yet required by the RSP at this capability level, but being regularly piloted in practice), which I think fulfils a lot of the same goals, and does many of them better given the reduced COI of the reviewer. I currently lean towards probably AST review sharing being good in some form, but I haven't really talked it through with folks much yet and I could easily imagine changing my mind.

Are there any plans to start publishing a (presumably redacted) form of the Alignment Stress Testing Team Report? I found the system cards to be a more systematically optimistic / positively skewed presentation of evidence in the one case we do have (for example, see the major disagreements the AST team’s report had in the one publicly released version we do have), so it seems like it’d be significantly useful for some version of the Alignment Stress Testing Team’s Report to be public.

@BronsonSchoen Some reasoning is discussed in the changelog and I gave some more thoughts in this thread, can you say more about what you want to know here? x.com/MaskedTorah/st…

@MaskedTorah Why was this criteria added?

@dylanbowmanSF a recent tweet of yours and something on google, I forget which?

@MaskedTorah dylanbowmansf.com/blog.html I just switched domains yesterday, sorry about that. Was the dbow.me one linked anywhere?

you can now subscribe to my blog if you feel like your email inbox is lacking in truth nukes

@dylanbowmanSF ah i think maybe something's down?

@dylanbowmanSF do you have a url? I spent like 30 seconds googling and didn't manage to find it, it doesn't show up in the first page of search results for the quoted phrase "superhuman articulacy"

@TheZvi And if you find yourself in a world where the god of straight lines will keep on offsetting your AI uplift with diminishing returns absent such uplift, you wouldn’t obviously be in that state. slatestarcodex.com/2019/03/13/doe…

@TheZvi (Which is the intent of the threshold, to capture the point at which you’re like “ah shit might be some scarily fast RSI coming up we’d better have our shit together to handle way more powerful models and soon”.)

@TheZvi these have always been there, they’re linked on the main RSP page on the website! Rarely get discussed though.

@MaskedTorah omg thank you thank you actual redlines this so good in terms of procedure, also ty for the notice.

@allTheYud But again, that's just my off the cuff take, I'm not claiming 2021!Drake or 2021!anybody had consensus about it, I just don't think they had consensus about your thing either and it's bad form to imply they did in a context where a reader might assume you literally believe that.

As to where the threshold should be: in one sense, the current state of affairs ought to motivate very aggressive slowdown action already, with our existing state of evidence, and so nothing more is needed. In a more pragmatic sense, what would make me think "oh this result could make a real dent in pause tractability"? Eh I'd have to think about it, credible evidence of a coherent malign secret goal in a current frontier model which was good enough to make surprising behavioral predictions about contexts in which the model would take otherwise-inexplicable actions to further that goal seems like it'd probably be enough to at least get the vast majority of Serious Technical People into a state of extreme concern in public? (And would update a bunch of people down on the tractability of alignment, probably, such that they'd be more into longshot pause advocacy instead of trying to muddle through the race.) Don't really expect most policymakers to be able to distinguish great science with spooky results from bad science with spooky results, so I think the impacts route mostly through influencing legible experts. Not very confident.

Way back when, interpretability work justified its existence by postulating that if the below result was found, further AI work would be immediately halted by companies and governments. That was, supposedly, the whole point of interpretability.