Sabitlenmiş Tweet
"The Cursive Transformer" is a project I finished about a year ago but forgot to post about.
We trained a Transformer to write cursive!
greydanus.github.io/2025/03/30/cur…
English
Sam Greydanus
848 posts
Sam Greydanus
@samgreydanus
Physics, AI, comparative history. Dartmouth, CERN, Microsoft, Google, open source. Put your outdoor work in order, after that, build your house.
Katılım Eylül 2017
51 Takip Edilen2.4K Takipçiler
More on this soon. Pretty much works but want to build more projects on top of it to work out the bugs
Sam Greydanus@samgreydanus
@karpathy Yes! I built a humble attempt at solving this issue: - minimalist dev server for building with Claude Code - API for deployment, DNS setup, etc - Stripe + email + database + storage + etc - Self-documenting API (curl tidepool.sh/api) See tidepool.sh/quickstart
English
Zooming in on eg "The Terminator" helps you discover similar but *much less popular* movies.
Helps with discovery of niche films on the long tail of content
English
This screenshots show interesting islands - a WWE island, a southeast asian cinema continent (bollywood, etc), Scooby-Doo, Peter Pan (+Hook etc)
The individual movies have different genres, cast, etc and yet since they share plot, characters & themes the UMAP puts them together
English
@karpathy Yes! I built a humble attempt at solving this issue:
- minimalist dev server for building with Claude Code
- API for deployment, DNS setup, etc
- Stripe + email + database + storage + etc
- Self-documenting API (curl tidepool.sh/api)
See tidepool.sh/quickstart
English
When I built menugen ~1 year ago, I observed that the hardest part by far was not the code itself, it was the plethora of services you have to assemble like IKEA furniture to make it real, the DevOps: services, payments, auth, database, security, domain names, etc...
I am really looking forward to a day where I could simply tell my agent: "build menugen" (referencing the post) and it would just work. The whole thing up to the deployed web page. The agent would have to browse a number of services, read the docs, get all the api keys, make everything work, debug it in dev, and deploy to prod. This is the actually hard part, not the code itself. Or rather, the better way to think about it is that the entire DevOps lifecycle has to become code, in addition to the necessary sensors/actuators of the CLIs/APIs with agent-native ergonomics. And there should be no need to visit web pages, click buttons, or anything like that for the human.
It's easy to state, it's now just barely technically possible and expected to work maybe, but it definitely requires from-scratch re-design, work and thought. Very exciting direction!
Patrick Collison@patrickc
When @karpathy built MenuGen (karpathy.bearblog.dev/vibe-coding-me…), he said: "Vibe coding menugen was exhilarating and fun escapade as a local demo, but a bit of a painful slog as a deployed, real app. Building a modern app is a bit like assembling IKEA future. There are all these services, docs, API keys, configurations, dev/prod deployments, team and security features, rate limits, pricing tiers." We've all run into this issue when building with agents: you have to scurry off to establish accounts, clicking things in the browser as though it's the antediluvian days of 2023, in order to unblock its superintelligent progress. So we decided to build Stripe Projects to help agents instantly provision services from the CLI. For example, simply run: $ stripe projects add posthog/analytics And it'll create a PostHog account, get an API key, and (as needed) set up billing. Projects is launching today as a developer preview. You can register for access (we'll make it available to everyone soon) at projects.dev. We're also rolling out support for many new providers over the coming weeks. (Get in touch if you'd like to make your service available.) projects.dev
English
@j0hnparkhill Yes sir, that paper was a big inspiration for this work and we cited it profusely in our paper
English
@samgreydanus Hopefully y'all saw the Graves paper from Pre-transformer! arxiv.org/abs/1308.0850
English
"The Cursive Transformer" is a project I finished about a year ago but forgot to post about.
We trained a Transformer to write cursive!
greydanus.github.io/2025/03/30/cur…
English
@jwuphysics It's the kind of question I'd love to spend some time answering with a toy example (MNIST-1d? :) greydanus.github.io/2020/12/01/sca…) but for now all I can give are vague intuitions
English
@samgreydanus > By contrast, when one bins continuous data and tokenizes the bin indices, one is able to train with a cross entropy loss, which generally works much better than an RMSE loss
Aside from skewed distributions, do you have good intuition why this is? Why not histogram norm?
English
@jwuphysics The loss surface is generally better when using cross entropy than RMSE. I know that's a cop out answer, and I'd have to look at some analysis of why the gradients are more well-behaved. Part of it is that NNs are much better at having on/off output neurons than scalar
English
@theodorus5 Haha! I had my papers marked down for a year if I didn't write cursive. But I'm grateful that as an adult I can use it for letters. Now AI can write back in proper form.
English
@samgreydanus Well if we had to suffer learning it at school it should too right 😉
English
You can run it in a Colab and also get the dataset via links on the blog post.
This was a fun project wherein I got to build and train Transformers from scratch, and also was able to get them to do something that I haven't seen them do very well yet.
English
With a few modifications to the sampling code last night, we were able to start doing custom generations. Model supports uppercase, lowercase, digits, and some punctuation. It's not perfect, but it's improving rapidly, and we expect to be writing entire paragraphs soon!
English
The model can now generate complex character sequences with near perfect fidelity
English
# Cursive Transformer Update: Sept. 4
One change in the augmentation code -- randomizing downsampling rates a little bit (so that each sample contained a slightly different # of points/character) -- led to massive improvements.
English
For a dataset with just 500 words (used combinatorially to synthesize 250k examples) and 5000 gradient steps (7 minutes walltime) we get reasonable if imperfect generation.
Suggests results will be pretty good in the 1k-2k word range, which is what Zach & I are pushing for now.
English
In a step towards a full-featured cursive transformer, we are now building and debugging on a dataset of lowercase, uppercase, and punctuation characters, eg:
English
Better downsampling method is leading to additional quality improvements.
English