Olcan retweetledi
The answer (~44:40) to Noam's question on @NoPriorsPod
---
@karpathy: Well, I was there for a while, right? And I did re-enter. So to some extent I agree. And I think that there are many ways to slice this question. It's a very loaded question a little bit. Um, I will say that... I feel very good about what people can contribute and their impact outside of the frontier labs, obviously. Not in the industry, but also in like more, like ecosystem-level roles. So your role, for example, is more ecosystem level. My role currently is also kind of more on an ecosystem level, and I feel very good about the impact that people can have in those kinds of roles. I think conversely there's... there are definite problems in my mind for, um, for basically aligning yourself way too much with the frontier labs too. So fundamentally I mean you're, you have a huge financial incentive to, uh, with these frontier labs. And by your own admission, the uh, the AIs are going to like really change humanity and society in very dramatic ways, and here you are basically like building that technology and benefiting from it like and being like very allied to it through financial means. Like this was a conundrum that was in, um... at the heart of, you know, how OpenAI was started in the beginning, like this was the conundrum that we were trying to solve. Um, and so you know, that—so it's kind of...
@saranormous: It's still not resolved.
Andrej Karpathy: The conundrum is still not like fully resolved. So that's number one. You're not a completely free agent and you can't actually like be part of that conversation in a fully autonomous, um, free way. Like if you're inside one of the frontier labs. Like there are certain things that you can't say, uh, and conversely there are certain things that the organization wants you to say. And you know, they're not going to twist your arm, but you feel the pressure of like what you should be saying, you know? Cause like, obviously. Otherwise it's like really awkward conversations, strange side-eyes, like what are you doing, you know? So you can't like really be an independent agent, and I feel like a bit more aligned with humanity in a certain sense outside of a frontier lab, because uh, I don't, I'm not subject to those pressures almost, right? And I can say whatever I want.
So those are like some sources of misalignment I think, to some extent. I will say that like, in one way I do agree a lot with that sentiment that, um, I do feel like the labs, for better or worse, they're opaque and a lot of work is there, and they're kind of like at the edge of capability and what's possible, and they're working on what's coming down the line. And I think if you're outside of that frontier lab, uh, your, your judgment fundamentally will start to drift, because you're not part of the, you know, what's coming down the line. And so I feel like my judgment will inevitably start to drift as well. And uh, I won't actually have an understanding of how these systems actually work under the hood. That's an opaque system. Uh, I won't have a good understanding of how it's going to develop and etc. And so I do think that in that sense I agree and it's something I'm nervous about. I think it's worth basically being in touch with what's actually happening and actually being in a frontier lab. And if some of the frontier labs would have me come for, you know, some amount of time and do really good work for them and then maybe come in and out—
Sarah Guo: Guys, he's looking for a job, this is super exciting!
Andrej Karpathy: (Laughs) Then I think that's maybe a good setup. Because I kind of feel like it kind of, um... you know, um, maybe that's like one way uh to, to actually be connected to what's actually happening but also not feel like you're necessarily fully controlled by those entities. So I think honestly in my mind like, uh, Noam can probably do extremely good work at OpenAI, but also I think his most, um, impactful work could very well be outside of OpenAI.
Sarah Guo: Noam, that's a call to be an independent researcher, if you got auto-research.
Andrej Karpathy: Yeah, there's many things to do on the outside and it's a... and I think ultimately I think the ideal solution maybe is like yeah, going back and forth, uh, or um, yeah, and I think fundamentally you can have really amazing impact in both places. So very complicated, I don't know, it's a very loaded question a little bit, but um, I mean I joined the frontier lab and I'm outside, and then maybe in the future I'll want to join again, and I think um, uh, that's kind of like how I look at it.
Noam Brown@polynoamial
@saranormous @karpathy @NoPriorsPod Why is he not at a frontier AI lab at the most pivotal time in human history since at least the industrial revolution?
English
