kirk

@che_shr_cat Didn't we already knew that transformers fail at algorithmic tasks? They mostly solve these kind of tasks by using spurious correlations youtube.com/watch?v=aBGXgW…

11/ I also made a comic version for this one — sometimes a picture is worth a thousand tokens. #MachineLearning #AI

1/ We know Transformers fail at length extrapolation. But new research shows a deeper flaw: they fail at IN-DISTRIBUTION state tracking. They don't learn algorithmic rules, they just memorize isolated circuits per length. 🧵

@JFPuget Welcome to the era of "we'll delete everything and you'll be happy"

You can't make this up. And this happened to a someone working on safety and alignment at META Superintelligence Lab. I don't get how people can let AI agents go on unsupervised like this. I'll stick to my supervised use of AI for now.

@DimitrisPapail Nice write up! I lf I'm not mistaken I would say the pair encoding trick of tokens resembles a lot the RLE trick where u compress repeating values/letters in a condensed format representation. Has been used a lot in vision & I bet there's a lot of codebases on web with it.

oh snap! we now have a 980 (Claude Code) and 970 (Codex) parameter transformesr that can add 10-digit numbers!!!! After I pushed both to "just try harder to get it below 1k params, and be creative" they managed to do it!

@ZimingLiu11 @naturecomputes @SuryaGanguli @AToliasLab Nice thread! Any thoughts on how do continuous but non autoregressive models like fno and pinns compare to transformer based? Based on your findings they should be avoiding both issues present in transformers?

@naturecomputes @SuryaGanguli @AToliasLab 15/N To close this thread, let me remind you of the "no free lunch theorem", and that everything boils down to inductive biases, which is the central theme of this paper.

🚨Transformers don't learn Newton's laws? They learn Kepler's laws! Like us, transformers don't predict a flying ball via a differential equation, but by fitting a curve. Moreover, reducing context length steers a transformer from Keplerian to Newtonian. Compression in play.

@giffmana @crude2refined Indeed, the optimizers we use are quite sensitive and as such lr can help when stuck in a local min to get unstuck, all other hyperparams are there to smooth the optimizer trajectory.

@crude2refined Yes, they are too, but (usually) lr is the highest bit.

As per my recurring rants:

@JFPuget @jm_alexia 🤔 isn't that already happening? The majority of academics use overleaf and given that overleaf has integrated 3rd party AI models, those could potentially be sending data to underlying companies, no?

@jm_alexia That was my reaction too. I was amused to see academics happy to share their unpublished research with OpenAI. I didn't post myself as writing latex paper is a low priority for me , I do it once every other year or so now.

Don't use a tool made to steal your IP

@branerico @adrian1977 @burkov I think it already has lost its value if u consider that on average a PhD makes what a 20 year old earns with vocational training working as an electrician in datacenters.

There’s competition for academic jobs not because of quality PhDs graduating. It’s because there are more than needed PhDs ( unnecessary numbers) every year while the jobs have not increased, yet research funds have been limited. Everyone is doing a PhD , it would loose its value soon!

It's an easiest time to make a master's or a PhD: 1. You pick a bunch of recent papers from scientists in your research domain, submit them to a chatbot, and ask it to analyze the "future works" sections and propose experiments to try. 2. You ask the chatbot to write code for these experiments. 3. You run this code and get some incremental results not published anywhere else. 4. You ask the chatbot to write a paper about these incremental results. 5. You submit this paper to a third-tier conference in your domain where the acceptance rate is above 0.5. Three papers like that and you are ready to write a thesis. I guess you already know how to get the thesis written fast.

c.f. science.org/doi/10.1126/sc…

History has a funny way of repeating itself...

@connordavis_ai Didn't LLM's already knew how to answer what-if questions, even at a superficial level. What was the baseline here?

Holy shit… this paper might be the most important shift in how we use LLMs this entire year. “Large Causal Models from Large Language Models.” It shows you can grow full causal models directly out of an LLM not approximations, not vibes actual causal graphs, counterfactuals, interventions, and constraint-checked structures. And the way they do it is wild: Instead of training a specialized causal model, they interrogate the LLM like a scientist: → extract a candidate causal graph from text → ask the model to check conditional independencies → detect contradictions → revise the structure → test counterfactuals and interventional predictions → iterate until the causal model stabilizes The result is something we’ve never had before: a causal system built inside the LLM using its own latent world knowledge. Across benchmarks synthetic, real-world, messy domains these LCMs beat classical causal discovery methods because they pull from the LLM’s massive prior knowledge instead of just local correlations. And the counterfactual reasoning? Shockingly strong. The model can answer “what if” questions that standard algorithms completely fail on, simply because it already “knows” things about the world those algorithms can’t infer from data alone. This paper hints at a future where LLMs aren’t just pattern machines. They become causal engines systems that form, test, and refine structural explanations of reality. If this scales, every field that relies on causal inference economics, medicine, policy, science is about to get rewritten. LLMs won’t just tell you what happens. They’ll tell you why.

@elonmusk The default should be reject all by definition. There problem solved.

The EU Commission has destroyed countless life-seconds with their idiotic “accept cookies” pop-up!

@elonmusk The problem is purely financial/economic. People were given the promise that if u work & study hard u’ll be better of than ur parents, that’s a lie of course, not in this economy & with this inflation. Also, 1% is hoarding all the money. Why don’t u give some to boost families?

An immediate increase in the birth rate is needed

@Dorialexander Both

based current vibes, not sure if neurips 2025 is a conference or a job fair.

@TomGur @fortnow Shite just hit the fun

Times are changing. From the acknowledgments of @fortnow's new paper on search vs decision for S_2^P. (arxiv.org/abs/2512.02808)

@dejanseo @SchmidhuberAI @kimmonismus @WhatsApp Still not bad at all

@68kirk @SchmidhuberAI @kimmonismus @WhatsApp most likely just rule based sampling of pre-recorded audio files

Yann LeCun’s 1989 convolutional neural network demo, the foundation for the CNNs we still use today. It's amazing how far we've come since then!

@francoisfleuret U need a model to decode the pattern in order to appreciate the hidden beauty...

@SchmidhuberAI @kimmonismus It has a voice answer in '86, impressive if u think that todays apps like @WhatsApp fail to transcribe even a basic voice message of couple of minutes.

@kimmonismus Fukushima's 1986 video shows a CNN that recognises handwritten digits, three years before LeCun's video: x.com/SchmidhuberAI/…

@ben_eysenbach @ZiarkoAlicja @WiMLworkshop Are these recorded?

To what extent can combinatorial reasoning problems be solved with learned representation and no search? Check out @ZiarkoAlicja's talk at @WiMLworkshop today at 2:45pm PT!

@burkov Haven't read the paper yet but from ur description sounds like there are some resemblances with diffusion approaches.

This paper really is groundbreaking. It solves a long-standing embarrassment in machine learning: despite all the hype around deep learning, traditional tree-based methods (XGBoost, CatBoost, random forests, etc) have dominated tabular data—the most common data format in real-world applications—for two decades. Deep learning conquered images, text, and games, but spreadsheets remained stubbornly resistant. This paper's (published in Nature by the way) main contribution is a foundation model that finally beats tree-based methods convincingly on small-to-medium datasets, and does so very fast. TabPFN in 2.8 seconds outperforms CatBoost tuned for 4 hours—a 5,000× speedup. That's not incremental; it's a different regime entirely. The training approach is also fundamentally different. GPT trains on internet text; CLIP trains on image-caption pairs. TabPFN trains on entirely synthetic data—over 100 million artificial datasets generated from causal graphs. TabPFN generates training data by randomly constructing directed acyclic graphs where each edge applies a random transformation (using neural networks, decision trees, discretization, or noise), then pushes random noise through the root nodes and lets it propagate through the graph—the intermediate values at various nodes become features, one becomes the target, and post-processing adds realistic messiness like missing values and outliers. By training on millions of these synthetic datasets with very different structures, the model learns general prediction strategies without ever seeing real data. The inference mechanism is also unusual. Rather than finetuning or prompting, TabPFN performs both "training" and prediction in a single forward pass. You feed it your labeled training data and unlabeled test points together, and it outputs predictions immediately. There's no gradient descent at inference time—the model has learned how to learn from examples during pretraining. The architecture respects tabular structure with two-way attention (across features within a row, then across samples within a column), unlike standard transformers that treat everything as a flat sequence. So, the transformer has basically learned to do supervised learning. Talk to the paper on ChapterPal: chapterpal.com/s/a1899430/acc… Download the PDF: nature.com/articles/s4158…

@michaelzluo x.com/andrew_n_carr/…

Transformers without positional embeddings are functionally the same as dLLM and has better scaling law than transformers with positional embeddings. It is also much easier and stable to train transformer than dLLM due to much prior work. I would position dLLM as a "cost arbitrage" over LLMs, i.e. faster generation with much higher token throughput.