George A. Tilesch

It is with immense happiness that I announce the arrival of our first book: BetweenBrains. Co-authored with my friend and NASA leader Omar Hatamleh, the book is our attempt to prepare responsible readers worldwide for the #AI Age.…lnkd.in/gb5QRbJ betweenbrains.ai

Excited to release new repo: nanochat! (it's among the most unhinged I've written). Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI. It weighs ~8,000 lines of imo quite clean code to: - Train the tokenizer using a new Rust implementation - Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics - Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use. - SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval) - RL the model optionally on GSM8K with "GRPO" - Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI. - Write a single markdown report card, summarizing and gamifying the whole thing. Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc. My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved. Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.

Grok estimates with 75%-85% certainty that Trump is a Putin compromised asset. @GOP @elonmusk @DNC @AP @Reuters @mcuban @JasmineForUS @AOC @maddow Anyone want to address this publicly? x.com/i/grok/share/W…

My take on o3: the global economy is cooked, we need a new economic and societal framework.

Last week, the chairs of the EU Code of Practice for general-purpose AI models presented to the #AI Board what they identified as the tension points and trade-offs in the first draft, just as they completed the second one. mlex.com/mlex/articles/…

Introducing Genie 2: our AI model that can create an endless variety of playable 3D worlds - all from a single image. 🖼️ These types of large-scale foundation world models could enable future agents to be trained and evaluated in an endless number of virtual environments. → goo.gle/4f48y2e

RIP language teachers and interpreters. Turn on volume. Goodbye old world. New GPT Advanced Voice. Thoughts?

We’ve trained a model, CriticGPT, to catch bugs in GPT-4’s code. We’re starting to integrate such models into our RLHF alignment pipeline to help humans supervise AI on difficult tasks: openai.com/index/finding-…

Excited to share that I have accepted the Ambassadorship for the AI Summit happening in Budapest this September! With 2000 participants and top international speakers, this event is set to be the largest gathering dedicated to AI in the region.

NEW: The EU's #AI Act will be signed off on Thursday, but it will only be published in July due to a significant backlog, @mlexclusive has learned. That means the entry into force has been further delayed. Full story 👇 content.mlex.com/#/content/1569…

🚨 SCOOP 🚨 Google's chief privacy officer is out after 13 years at the company. Google is also retiring the role. Matthew Bye, Google’s head of competition law, is also gone after 15 years. Huge exec departures during turbulent times. forbes.com/sites/thomasbr…

OpenAI Board forms Safety and Security Committee, responsible for making recommendations on critical safety and security decisions for all OpenAI projects. openai.com/index/openai-b…

Meanwhile, over in Google Search. Andrew Johnson has been killin it, I never knew.

The final version of the approved EU AI Act even specifically recognizes model cards and datasheets: The work that me and my former Google Ethical AI team co-lead @timnitGebru spearheaded.

BREAKING: EU ministers unanimously adopted the #AI Act. The law will be published in the official journal of the European Union in the coming days.

I just resigned from @OpenAI. I was the guy responsible for communicating to the public that the voice feature that they have in the ChatGPT app isn’t the one they just demoed

As I predicted, Microsoft is training its own LLM It's called MAI-1, a 500B param model, and may be previewed at their Build conference. When this model becomes available, it will only be natural for MSFT to push this instead of the GPT line. As predicted, OAI and MSFT are becoming more and more competitive. Any which way, LLMs have become a commodity (link in alt)

Not going to build a bigger supercomputer than these chaps. But you don’t need to to build aligned AGI if you decentralize Swarm, collective intelligence will beat centralised collected intelligence

Sora for artists and filmmakers: