tokenbender

@Yuchenj_UW > I can summon my autoresearch army to win it… if I have time. the point of autoresearch is that it won't really need much of "your time".

OpenAI just dropped a training challenge: Train a <16MB language model in 10 minutes on 8×H100s and minimize held-out loss on a fixed FineWeb dataset. Basically NanoGPT Speedrun. They’re sponsoring $1M in compute. I can summon my autoresearch army to win it… if I have time.

@tpottery_ extended thinking is a luxury. we are in the cup noodle research era. but that bowl looks good.

@tokenbender What about extended thinking ramen?

i cook my instant ramen and sit with my instant coffee to find some instant relief in a nice research drop that might be out there and trigger my instant research. i need instant insight. instant clarity. instant expertise in a field i learned about 14 minutes ago. i need the abstract to hit my bloodstream like ultra burnt caffeine and the conclusion to knead my nausea. i need to scroll past one chart and become a different person. i need to misunderstand something at a historic pace.

Blog post - Transformers as Constrained Optimization Rewriting pre-norm decoder-only transformers as solutions to regularized objectives. Changing regularization to hard constraint gives a canonical temperature, generalizing to KL-divergence, ideas of cross-layer interaction.

@exploding_grad sounds great, hope to see it.

@tokenbender I'm doing something similar. GPU poor - so I'm building 2 toy models (std vs attn res stream). Planning to run some basic mech interp experiments on them - so check if they are more interpretable. If it works, then i'll try scaling up the model arch and test further.

To further my understanding and also reproduce the results of the paper - I have setup with repository where we can compare and understand the baseline vs mHC vs attention-residual-full vs attention-residual-block algos properly. Sharing more as runs complete later. link - github.com/tokenbender/na…

@clbswrs we have come from some 40% bs rate from 3.7 to 10-15% bs rate if the task isn’t too complex. but this last 10% is where we improve the slowest.

@tokenbender Tried doing this with sonnet 3.7 last spring/summer and it pranked me probably 75% of the time. Maybe more. Definitely took more time to figure out how it had reward hacked than running the campaign myself at that point. Can u believe sonnet 3.7 was only a year ago

true automation of research has been achieved. we just need to get good at the trivial job of finding the 10-20% cases of fraud in 2 hours of looping and 1k LOC PRs.

@SarahLacard how are we going to revert without knowing something is wrong? how would we write tests for logical issues or general policy violations without checking? survivorship bias code can be shipped but reliability is still necessary if you want to own the code you ship.

@tokenbender yeah exactly - for me it's a hard revert and not bother to care if it hid something somewhere, you're not going to be able to exhaustively check everything - revert and move on with better prompting during round 2

@SarahLacard it already took a short-cut and was caught later. this is my confirmation of if clean-up is done or not. even that may need some quick manual checks.

@tokenbender that is setting it up to lie to you

@LukaIvanic73477 yep, similar observation.

@tokenbender Straight up responds to my previous message requests again, instead of the last one. For me it's around 400k. But non-chat ability, it can hold its own until 600k, even on messy kernel code.

450-500k context seems to be the mark for gpt 5.4 where it stops understanding what it is replying to and reaches its confused state.

@tokenbender doing my bit simcore.netlify.app/explainers/moo…

@cto_junior wow the quality has improved so much.

@himanshustwts mostly because my notes and perspective on mHC and residuals is clean already so i can fit things in that frame easily.

@tokenbender awesome G. this was fasttt!

I wrote something on Moonshot's latest research release - Attention Residuals. Intuition, notes and how you can understand standard residuals vs mHC vs attention residuals.

since Jan ‘26 itself, the number of such projects has increased to 4. these are all almost concluded but i face dopamine drought as soon as i tell myself to start writing a semi-academic article. if i could just throw my artifacts into a directory and finish and end with - “ohh this looked cool so i did that next”, everything would be so much more fun.

every now and then i relapse towards my habit of experimenting with really important things but never releasing in public because it is way more effort for me to make things clickworthy and presentable than sate my curiosity and maximise tinkering with the next challenge.

@muzzdotdev not sure if you use opencode but if you do, DCP used to be one of my must-have plugins there.

@tokenbender what is it

how uninstalling opencode-dcp feels like after it has lost its way