Art of the Problem

@karpathy When it comes to formance what's the key difference between an LLM "knowing" in its weights vs context?

LLM Knowledge Bases Something I'm finding very useful recently: using LLMs to build personal knowledge bases for various topics of research interest. In this way, a large fraction of my recent token throughput is going less into manipulating code, and more into manipulating knowledge (stored as markdown and images). The latest LLMs are quite good at it. So: Data ingest: I index source documents (articles, papers, repos, datasets, images, etc.) into a raw/ directory, then I use an LLM to incrementally "compile" a wiki, which is just a collection of .md files in a directory structure. The wiki includes summaries of all the data in raw/, backlinks, and then it categorizes data into concepts, writes articles for them, and links them all. To convert web articles into .md files I like to use the Obsidian Web Clipper extension, and then I also use a hotkey to download all the related images to local so that my LLM can easily reference them. IDE: I use Obsidian as the IDE "frontend" where I can view the raw data, the the compiled wiki, and the derived visualizations. Important to note that the LLM writes and maintains all of the data of the wiki, I rarely touch it directly. I've played with a few Obsidian plugins to render and view data in other ways (e.g. Marp for slides). Q&A: Where things get interesting is that once your wiki is big enough (e.g. mine on some recent research is ~100 articles and ~400K words), you can ask your LLM agent all kinds of complex questions against the wiki, and it will go off, research the answers, etc. I thought I had to reach for fancy RAG, but the LLM has been pretty good about auto-maintaining index files and brief summaries of all the documents and it reads all the important related data fairly easily at this ~small scale. Output: Instead of getting answers in text/terminal, I like to have it render markdown files for me, or slide shows (Marp format), or matplotlib images, all of which I then view again in Obsidian. You can imagine many other visual output formats depending on the query. Often, I end up "filing" the outputs back into the wiki to enhance it for further queries. So my own explorations and queries always "add up" in the knowledge base. Linting: I've run some LLM "health checks" over the wiki to e.g. find inconsistent data, impute missing data (with web searchers), find interesting connections for new article candidates, etc., to incrementally clean up the wiki and enhance its overall data integrity. The LLMs are quite good at suggesting further questions to ask and look into. Extra tools: I find myself developing additional tools to process the data, e.g. I vibe coded a small and naive search engine over the wiki, which I both use directly (in a web ui), but more often I want to hand it off to an LLM via CLI as a tool for larger queries. Further explorations: As the repo grows, the natural desire is to also think about synthetic data generation + finetuning to have your LLM "know" the data in its weights instead of just context windows. TLDR: raw data from a given number of sources is collected, then compiled by an LLM into a .md wiki, then operated on by various CLIs by the LLM to do Q&A and to incrementally enhance the wiki, and all of it viewable in Obsidian. You rarely ever write or edit the wiki manually, it's the domain of the LLM. I think there is room here for an incredible new product instead of a hacky collection of scripts.

@yacineMTB made this youtu.be/cbD4NsZQKYw

I can't believe radios work. Like wifi. Or satellite coms. Or LTE. How is that even possible

@aaronistan @bySyncere what's your approach to fusing LLM and real-time policy?

About six months ago, I left Stanford to build a robotics company for the home. At the time, humanoid hype felt like it was getting ahead of its promises but no one had really shown an alternative. So we’ve been building one. Excited to share more soon.

@yacineMTB if you anchor it to physical project the progress is easier to realize and build on

it is insane what i am now capable doing with a computer

i'm working on simple bipedal robot which general capability. with onboard real-time policy networks for locomotiin and LLM control in outer loop for adaptation. two questions 1. would your product help me make this sim data? 2. how have you thought about passing sensor data between to LLM in a flexible way

2026 is a terrible time to build a robotics company. Imagine building a software company without AWS, Stripe, or GitHub. Everything from scratch. That's robotics today. Most robotics teams waste their first 6–12 months rebuilding the same data infrastructure, telemetry, internal tools, pipelines. None of it is core IP. I started @AlloyRobotics to SPEEDRUN all of it. A modern stack so robotics teams can iterate 10x faster than their competition. If you're building in this space, lets talk.

@yacineMTB the key thing i've noticed is RL of "real-time" behaviour (learning policy networks in sim) and RL of "long term" behaviour (which LLMs solve). of you accept that you go a long way.

reinforcement learning algorithms are so incredibly immature, it is a miracle that it is working at all. there is *so* much room for improvement on algorithms, parallelization and optimization for speed of learning. so, so, so much

@robofearth thank you "baby steps"..

@Artoftheproblem nice work Brit!

My robot learning to walk in one minute with neural network only !

@karpathy @doodlestein @Qivshi1 I was about to try a “clean up agent” whos only job is to keep memory updated in a way that makes sense (ie obviously delete old things we replaced etc)

If I had to guess it's less decay and more that memories have naive RAG-like implementations, so you're at the mercy of whatever happens to retrieve in the top k via embeddings. They don't process you in aggregate and over time (probably compute constraints) so they struggle to identify what's fleeting (?). Anyway just guesses, but it's cringe :D

One common issue with personalization in all LLMs is how distracting memory seems to be for the models. A single question from 2 months ago about some topic can keep coming up as some kind of a deep interest of mine with undue mentions in perpetuity. Some kind of trying too hard.

@Squigglee_lines growbot is working title, and these are just early tests soon much more will be possible - the walking policy is a small feed forward network with memory of previous few states

@Artoftheproblem This is so cool, so proud of buddy. Does your robot have a name?

@yacineMTB i'm flying now

the state of machine learning research: "youd be faster trying it than trying to work out from papers if it would work"

@jsuarez can i train robot locomotion policies with this?

This is currently the layout of puffer 4

better late than never to the party

@3DPrintAficio :) what have you been messing with?

@Artoftheproblem OH YES. Love the LEGO integration! Rapid ML convergence on microcontrollers is exactly why I tinker.

well had a lot of surprises this week. I finally got a low level instinct policy running trained with RL which allows it to run (so cool!) and while messing with that i realized i'd want an outer loop too for more expressive interactive behaviour, and so tried letting a second slower LLM loop control the system with memory. and while I see lots of people doing "voice in voice out" with robots i'm trying to do physical in physical out (but with LLM in background). and that was amazing, for example I can have it "mimic whatever it feels" so if I shake it a certain way, it can go into a self improving loop and mimic it withint 4-6 tries. head is spinning. it's like an clawdbot but for real

@Artoftheproblem This is so cool! I thought growbot would just be using RL for basic controls. You’re incorporating chain of thought for higher level planning?

My insect robot's brain trace

@TheJamieWooster thank you :)

^ @Artoftheproblem Respect your amazing work.

If you still genuinely think LLM's have emotions and can generate novel ideas, please watch a few of these youtube.com/watch?v=NmCRQP… videos from the Art of the Problem youtube.com/watch?v=fKf6Kl… he simplifies and makes understandable the core concepts & limitations of LLM's

@deepfieldnorth Thanks David

@Artoftheproblem Best work yet!

The Alien Signal That Looked Like Language full video 👇

@juliarturc great idea

Diffusion models clicked for me when I started seeing them through the lens of particle motion. I built this interactive playground where you too can clickety-clack to understand how drift, noise, and other hyperparams control diffusion. I hereby submit this as penance for the sin of YouTube edu-tainment 😇 Link in the first comment.