will depue

first tranche of @runpod credits should be rolling out

We’re sending out the first tranche of compute grants today, so make sure to sign up before then.

@shawnbuilds @OpenAI it’s really not that hard! try grabbing the train gpt starter script and have it explain to you how it works + suggest an experiment to run. the MLX script should run on your mac and you can configure it to run in about a minute

@OpenAI genuine question - how does someone with zero knowledge in this space get started attempting a problem like this? i'd imagine you'd need some serious ai fundamentals

Are you up for a challenge? openai.com/parameter-golf

@evilmathkid hope to see you on the leaderboard mithil!

Reduce kolmogorov complexity in a ~turing machine defined by 8xH100s, PTX, a CPU and 10min while also optimising the hell out of the code execution Love it

@industriaalist i’d be surprised if the search space in the limit of L(N) wasn’t equally rich as L(D)? why do you say so

few thoughts on openai's parameter golf: - first, you'd be surprised how many researchers at big labs (not just openai) are interested in our slowrun - i'd expect openai to be already automating parameter golf *entirely* with agents. and i'd also expect agents to be better than humans at this already. - for slowrun, we've deliberately kept it less gamified. the search space over learning algorithms for data efficiency is much larger than for compute/parameter efficiency. so slowrun is less of a competition and more of an open research effort toward interesting, new learning algorithms

@artificialguybr i would be surprised if it wasnt at least somewhat of an AI <> human collaboration!

It would be funny if the winner of this were a user with Codex/Claude.

@itsandrewgao ok will see if we can get more. if you run the 1xh100 baseline just make a pr with the log and the submission and i'll add it to the non record submissions for iteration

nooo how am i supposed to parameter golf when there are no 8xh100s help @willdepue

@Laz4rz 🫡🫡🫡🫡🫡

This is actually extremely exciting and something companies should be doing more

@vitransformer that sounds awesome! please do try it

i mean diffusion language models?

@bilaltwovec @typedfemale we gotta do #3 honestly, been pushing for a while

@typedfemale who remembers openai requests for research 1 and 2

we are all filipino mercor workers now

@adamyathegreat of course! troop 223

@willdepue Are you an Eagle Scout?

Remembering Mike Lanning, who passed today, leader of the greatest Boy Scout troop in America: Troop 233. Mike was an incredible person, leader & mentor to many. He maintains the record for most Eagle Scouts from one Scoutmaster: 1000+ with Troop 223. He will be deeply missed.

@cocohearts lmao

proud to announce we're open sourcing a new model GPT1.1-nano

@Yuchenj_UW thanks for sharing! i expect the best submissions to look pretty different than the nanogpt speedrun models, given parameter constraints

OpenAI just dropped a training challenge: Train a <16MB language model in 10 minutes on 8×H100s and minimize held-out loss on a fixed FineWeb dataset. Basically NanoGPT Speedrun. They’re sponsoring $1M in compute. I can summon my autoresearch army to win it… if I have time.

@test_tm7873 @Yuchenj_UW feel free to train and test on whatever! we just require final leaderboard submissions to be on h100s you can always share a github repo if you’re not submitting to the leaderboard (see non record submissions), just make sure it still follows the rules of fixed dataset and eval

@Yuchenj_UW man. its epic, i love small language models. but all my experience i have with em is on tpus. 😭and they want H100s. nnnnnooooooo.

@karpathy so cool

The signature is alluding to NVIDIA GTC 2015, where Jensen excitedly told an audience of, at the time, mostly gamers and scientific computing professionals that Deep Learning is The Next Big Thing, citing among other examples my PhD thesis (one of the first image captioning systems that coupled image recognition ConvNet to an autoregressive RNN language model, trained end to end). This was back when most people were still unaware and somewhat skeptical but of course - Jensen was 1000% correct, highly prescient and locked in very early.

Thank you Jensen and NVIDIA! She’s a real beauty! I was told I’d be getting a secret gift, with a hint that it requires 20 amps. (So I knew it had to be good). She’ll make for a beautiful, spacious home for my Dobby the House Elf claw, among lots of other tinkering, thank you!!

@onetwoval @kellerjordan0 thanks val! you should compete 👀👀

@willdepue @kellerjordan0 this is lit

we're launching a challenge, inspired by @kellerjordan0's NanoGPT Speedrun, to train the best model under extreme parameter limitations anytime i get asked how to break into ML, i point to challenges like this: just start training models! also, we're covering $1M of compute

some random photos from troop 223, was an incredible time. send all your kids to bsa!