Jacob Hilton

Seems like a great opportunity for technical talent to come into government and help the USG make sound, technically informed decisions on AI

@bzogrammer @davidad Although note that we only ever unroll the RNN for 10 steps (and there is a fresh input changes at each step), so no fixed point is ever actually reached!

I'm not a machine learning expert, but a recurrent network? Just a single nonlinear transform iterated until a fixed point? 1. An iterated nonlinear function is a fantastic recipe for a fractal! Expect to find many properties of fractals here. The decision boundary is likely horrifyingly complex. Not to mention, it would be an extremely high-dimensional fractal! 2. Computing fixed points of iterated functions is NP-complete. Not just NP, but complete, as in expressive enough to encode any possible problem in NP. What you have is in some sense the neural equivalent of a SAT solver with a hard-coded instance. Something that's likely cycling through a state space with embedded inference rules to decide when to make a big jump and when to make a small jump, and in higher dimensions the directions may be analogous to SAT solver unit propagation. The fixed points (no jump) are the solutions.

A challenge to the mechanistic interpretability community: fully interpret our 432-parameter RNN. (Thread)

@mrsirrisrm Good work so far! This roughly matches our understanding of neurons 1, 2, 4, 6 and 7 as explained in the post.

@JacobHHilton Those same neurons also encode the position of 2nd argmax, but with smaller amplitude and opposite phase! (Ignoring position 9 as a special case). So the difference in position between argmax and 2nd argmax is encoded

@nickinpractice Analysis by hand might not; good scalable approaches should probably transfer to some extent. Note the model zoo includes both RNNs and transformers.

@JacobHHilton Do you expect mech interp techniques to transfer from RNNs to Transformers? Assuming your RNN doesn't use attention.

Thanks to Zihao Chen, George Robinson, David Matolcsi, Jacob Stavrianos, Jiawei Li and Michael Sklar for work on this and other 2nd argmax models.

Read more on ARC's blog: alignment.org/blog/algzoo-un… Play around with the models using the AlgZoo GitHub repo: github.com/alignment-rese… Cross-postings for comments at LW/AF

Overall, I'm grateful to the Attorneys General of both California and Delaware, as well as their staff, for taking this issue seriously. There still seems to be the potential for the nonprofit to play a meaningful oversight role, although it is not doing so yet.

The full Memorandum of Understanding between OpenAI and the California Attorney General has more details about these and other governance measures, including some degree of continued accountability to the Attorney General: oag.ca.gov/system/files/a…

Although this doesn't fully preserve OpenAI's obligation to its nonprofit mission, I'm pleased to see a number of governance measures buried in the details here. These could act as meaningful guardrails, but only if further steps are taken. (Thread)

Silver linings from DE AG: 1. PBC directors can consider only the mission in safety / security decisions. 2. Safety and Security Committee will be run by the nonprofit and have the power to require mitigation measures, including halting deployments.

@tobyordoxford One issue could be limited data. When data is bottlenecked, compute scaling is worse, but eventually scaling data is more cost-efficient than scaling compute. Also, the information-density argument implies a different coefficient (shift on log plot) but the same exponent (slope).

And while RL let models learn to perform better, inference-scaling is just giving them more time to think. That improves performance, but not through increasing their intelligence. Yet it looks like the only scaling paradigm we have left… Full post: tobyord.com/writing/how-we…

New post on RL scaling: Careful analysis of OpenAI’s public benchmarks reveals RL scales far worse than inference: to match each 10x scale-up of inference compute, you need 100x the RL-training compute. The only reason it has been cost-effective is starting from a tiny base. 🧵

@Jess_Riedel @peterwildeford I don't think I would personally feel comfortable approving something less than around 50%, although my overall comfort level may depend more control and governance issues (which remain not fully addressed).

@Jess_Riedel @peterwildeford Furthermore, I have little confidence that final decisions about the restructuring will be unconflicted, regardless of whether arguments can be made in its favor.

The nonprofit currently owns 100% of the residual value after capped returns. Not sure how being reduced to 20-30% is a fair assessment, especially when the nominal control of the non-profit over operations also decreases significantly.