James Gin Pollock
223 posts

James Gin Pollock
@gin_james
CTO Orbital Materials, prev. Pluto Data Analytics, @datasine (acquired) - @datakind Data Ambassador





Introducing DroPE: Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings pub.sakana.ai/DroPE/ We are releasing a new method called DroPE to extend the context length of pretrained LLMs without the massive compute costs usually associated with long-context fine-tuning. The core insight of this work challenges a fundamental assumption in Transformer architecture. We discovered that explicit positional embeddings like RoPE are critical for training convergence but eventually become the primary bottleneck preventing models from generalizing to longer sequences. Our solution is radically simple: We treat positional embeddings as a temporary training scaffold rather than a permanent architectural necessity. Real-world workflows like reviewing massive code diffs or analyzing legal contracts require context windows that break standard pretrained models. While models without positional embeddings (NoPE) generalize better to these unseen lengths, they are notoriously unstable to train from scratch. Here, we achieve the best of both worlds by using embeddings to ensure stability during pretraining and then dropping them to unlock length extrapolation during inference. Our approach unlocks seamless zero-shot context extension without any expensive long-context training. We demonstrated this on a range of off-the-shelf open-source LLMs. In our tests, recalibrating any model with DroPE requires less than 1% of the original pretraining budget, yet it significantly outperforms established methods on challenging benchmarks like LongBench and RULER. We have released the code and the full paper to encourage the community to rethink the role of positional encodings in modern LLMs. Paper: arxiv.org/abs/2512.12167 Code: github.com/SakanaAI/DroPE





One point I made that didn’t come across: - Scaling the current thing will keep leading to improvements. In particular, it won’t stall. - But something important will continue to be missing.





Some politicians love to talk the UK down, but UK AI & Tech is moving at 100mph! Time to shift the narrative - so last night I started building a scrappy real-time dashboard to track the UK AI story: vibeshift.uk The code’s on my GitHub if you want to help build it out :) (Shot on my iPhone: Zero polish but maximum energy!)


You have heard of AI slop in the context of short video creation. But the same principle applies when it comes to improving drug discovery: we absolutely do not need a deluge of new hypotheses; we need better predictive validity (as per @JackScannell13). writingruxandrabio.com/p/what-will-it…


The UK is a great country with an extraordinary history. Our stagnation is real, but it's fixable and worth fixing. Enjoyed giving this talk at @lfg_uk last week and so encouraged by the optimistic responses I've had from people who are building a brilliant future for Britain 🚀











Matbench Discovery is out in Nature Machine Intelligence @ Paper: rdcu.be/esRFB Leaderboard: …tbench-discovery.materialsproject.org No better time to thank all my co-authors @RhysGoodall, @PhilippBenner2, Yuan Chiang @cyrusyc_tw, @Bowen_D_, Mark Asta, Gerbrand Ceder @cedergroup,








