Jan Tempus

Interestingly, Craig W. Schmidt has a second paper using LPs for tokenisation hitting arXiv today as well! Check it out: arxiv.org/abs/2605.22705

In our new paper, we reinterpret tokenisation as a problem in high-dimensional geometry (100M dims to be precise!), which we can solve efficiently to get a globally near-optimal tokeniser! Our method consistently improves language models over BPE. See 🧵for details.

As a bonus, by solving a relaxed linear program, our method allows you to upperbound how far from optimal any tokeniser is. w/ @tpimentelms Paper: arxiv.org/pdf/2605.22821 Code: github.com/JanTempus/toke…