Jacob Hilton

136 posts

Jacob Hilton

Jacob Hilton

@JacobHHilton

At the Alignment Research Center, formerly at OpenAI

Katılım Kasım 2012
56 Takip Edilen3.3K Takipçiler
Jacob Hilton retweetledi
Jacob Hilton
Jacob Hilton@JacobHHilton·
@bzogrammer @davidad Although note that we only ever unroll the RNN for 10 steps (and there is a fresh input changes at each step), so no fixed point is ever actually reached!
English
0
0
3
948
Charles Rosenbauer
Charles Rosenbauer@bzogrammer·
I'm not a machine learning expert, but a recurrent network? Just a single nonlinear transform iterated until a fixed point? 1. An iterated nonlinear function is a fantastic recipe for a fractal! Expect to find many properties of fractals here. The decision boundary is likely horrifyingly complex. Not to mention, it would be an extremely high-dimensional fractal! 2. Computing fixed points of iterated functions is NP-complete. Not just NP, but complete, as in expressive enough to encode any possible problem in NP. What you have is in some sense the neural equivalent of a SAT solver with a hard-coded instance. Something that's likely cycling through a state space with embedded inference rules to decide when to make a big jump and when to make a small jump, and in higher dimensions the directions may be analogous to SAT solver unit propagation. The fixed points (no jump) are the solutions.
English
2
0
4
1.6K
Jacob Hilton
Jacob Hilton@JacobHHilton·
A challenge to the mechanistic interpretability community: fully interpret our 432-parameter RNN. (Thread)
English
15
37
560
64K
Jacob Hilton
Jacob Hilton@JacobHHilton·
@mrsirrisrm Good work so far! This roughly matches our understanding of neurons 1, 2, 4, 6 and 7 as explained in the post.
English
1
0
1
73
deliciousSandwich
deliciousSandwich@mrsirrisrm·
@JacobHHilton Those same neurons also encode the position of 2nd argmax, but with smaller amplitude and opposite phase! (Ignoring position 9 as a special case). So the difference in position between argmax and 2nd argmax is encoded
English
1
0
0
55
Jacob Hilton
Jacob Hilton@JacobHHilton·
@nickinpractice Analysis by hand might not; good scalable approaches should probably transfer to some extent. Note the model zoo includes both RNNs and transformers.
English
0
0
8
2.7K
Nikola Georgiev
Nikola Georgiev@nickinpractice·
@JacobHHilton Do you expect mech interp techniques to transfer from RNNs to Transformers? Assuming your RNN doesn't use attention.
English
1
0
5
3K
Jacob Hilton
Jacob Hilton@JacobHHilton·
Thanks to Zihao Chen, George Robinson, David Matolcsi, Jacob Stavrianos, Jiawei Li and Michael Sklar for work on this and other 2nd argmax models.
English
0
0
34
4.1K
Jacob Hilton
Jacob Hilton@JacobHHilton·
Overall, I'm grateful to the Attorneys General of both California and Delaware, as well as their staff, for taking this issue seriously. There still seems to be the potential for the nonprofit to play a meaningful oversight role, although it is not doing so yet.
English
0
0
3
423
Jacob Hilton
Jacob Hilton@JacobHHilton·
The full Memorandum of Understanding between OpenAI and the California Attorney General has more details about these and other governance measures, including some degree of continued accountability to the Attorney General: oag.ca.gov/system/files/a…
English
1
0
2
469
Jacob Hilton
Jacob Hilton@JacobHHilton·
Although this doesn't fully preserve OpenAI's obligation to its nonprofit mission, I'm pleased to see a number of governance measures buried in the details here. These could act as meaningful guardrails, but only if further steps are taken. (Thread)
OpenAI@OpenAI

We completed our recapitalization. The non-profit, the OpenAI Foundation, is now one of the best resourced philanthropies ever, with equity valued at ~$130B. It continues to control the OpenAI for-profit, which is now a public benefit corporation. openai.com/index/built-to…

English
1
1
11
1.3K
Jacob Hilton retweetledi
page
page@michaelhpage·
Silver linings from DE AG: 1. PBC directors can consider only the mission in safety / security decisions. 2. Safety and Security Committee will be run by the nonprofit and have the power to require mitigation measures, including halting deployments.
OpenAI@OpenAI

We completed our recapitalization. The non-profit, the OpenAI Foundation, is now one of the best resourced philanthropies ever, with equity valued at ~$130B. It continues to control the OpenAI for-profit, which is now a public benefit corporation. openai.com/index/built-to…

English
2
4
37
6.5K
Jacob Hilton
Jacob Hilton@JacobHHilton·
@tobyordoxford One issue could be limited data. When data is bottlenecked, compute scaling is worse, but eventually scaling data is more cost-efficient than scaling compute. Also, the information-density argument implies a different coefficient (shift on log plot) but the same exponent (slope).
English
0
0
1
143
Toby Ord
Toby Ord@tobyordoxford·
And while RL let models learn to perform better, inference-scaling is just giving them more time to think. That improves performance, but not through increasing their intelligence. Yet it looks like the only scaling paradigm we have left… Full post: tobyord.com/writing/how-we…
English
4
4
80
7.3K
Toby Ord
Toby Ord@tobyordoxford·
New post on RL scaling: Careful analysis of OpenAI’s public benchmarks reveals RL scales far worse than inference: to match each 10x scale-up of inference compute, you need 100x the RL-training compute. The only reason it has been cost-effective is starting from a tiny base. 🧵
Toby Ord tweet media
English
27
53
500
191.7K
Jacob Hilton
Jacob Hilton@JacobHHilton·
@Jess_Riedel @peterwildeford I don't think I would personally feel comfortable approving something less than around 50%, although my overall comfort level may depend more control and governance issues (which remain not fully addressed).
English
1
0
1
89
Jacob Hilton
Jacob Hilton@JacobHHilton·
@Jess_Riedel @peterwildeford Furthermore, I have little confidence that final decisions about the restructuring will be unconflicted, regardless of whether arguments can be made in its favor.
English
1
0
2
83