Robert

In a reward model I found you could turn off 90% the activations in the second half of the NN with no loss in performance. Just need to keep the final N position activations and pass through the residual for all positions greater than N.

How it felt to be a software developer before ai.

By forcing myself to only launch and review training runs in 9-11 and 5-7 windows. It means I’m forced to automate, notice where I wasting time before. Cuts burnout through constant experiment context switching and reviewing experiments which finished late.

The performance of an LLM as a judge is a function of compute just like many other AI task. Thinking longer makes a better judge. Best of N. Larger models. Judging one example at a time rather than a batch.

Going for a long jog can turn a bad day into a good one.

At a first guess I would say AI has 4x my coding output but only 1.2x my actually output (business impact). Mainly because there are so many other bottlenecks which limit output.

When was the last time you saw a corporate space of any kind that blew you away Time to take chances again

@PaulSkallas It’s kind of like grip strength. It’s the first thing to go.

They stopped making good movies. Can't believe it. imagine telling someone this 20 years ago. "they don't make good movies in the future anymore" what?

I viewed my AI work as being a “scientist” which meant an emphasis on understanding. It’s going very poorly this year. I think a better model is “esports”. Emphasis is on clicks per minute, volume of experiments, experiment strategy, psychology, racing to the goal.

A lot of my RL failures have come from thinking “as long as KL is in this range it’s ok.” But this is wrong, the optimal range depends on the context, the base model, the target, the algorithm. It’s just not a good way to measure overfitting.

Always get my best run times in the Stanford quad. Knocks a couple mins off my 5km.

The best part of Japanese high streets is adding a third dimension. In Europe/US all the interesting stuff is on the ground floor. A 2D world, limited. In Japan you add verticality. Random buildings with 12 floors of niche bars, bordels, and karaokes.

Update: it's pretty clear now that *the* thing to do is 1) figure out how to make your research as verifiable as possible 2) spin up some agents

(I still have the bigger cousin running on prod nanochat, working a bigger model and on 8XH100, which looks like this now. I'll just leave this running for a while...)

Trying to get more retention on a product been working on for years. Feels like late stage California gold rush. The surface gold is gone, the valley is overcrowded.

An LLM should be watching my work at all times to immediately help with debugging with all the context.