HanRong YE @ Nvidia

nanochat now trains GPT-2 capability model in just 2 hours on a single 8XH100 node (down from ~3 hours 1 month ago). Getting a lot closer to ~interactive! A bunch of tuning and features (fp8) went in but the biggest difference was a switch of the dataset from FineWeb-edu to NVIDIA ClimbMix (nice work NVIDIA!). I had tried Olmo, FineWeb, DCLM which all led to regressions, ClimbMix worked really well out of the box (to the point that I am slightly suspicious about about goodharting, though reading the paper it seems ~ok). In other news, after trying a few approaches for how to set things up, I now have AI Agents iterating on nanochat automatically, so I'll just leave this running for a while, go relax a bit and enjoy the feeling of post-agi :). Visualized here as an example: 110 changes made over the last ~12 hours, bringing the validation loss so far from 0.862415 down to 0.858039 for a d12 model, at no cost to wall clock time. The agent works on a feature branch, tries out ideas, merges them when they work and iterates. Amusingly, over the last ~2 weeks I almost feel like I've iterated more on the "meta-setup" where I optimize and tune the agent flows even more than the nanochat repo directly.

This decision is deeply concerning and unfair to authors who invested significant effort into the rebuttal process. For example, our students did an amazing job turning a paper with negative reviews into an 8/6/6/6. This policy feels like it just resets all that hard work to zero.

Yep

Tempted to look up reviewer names? Don’t. As an author, you simply cannot emotionally detach your work from the person behind the review. This results in grudges that might last decades. I assure you, if someone disliked your paper, you do *not* want to know who it was. Unfortunately, this makes this leak a major issue as the research community relies on the double-blind review process in order to provide objective and fair evaluations.