Huge thanks to @Pluralis for the invite to give a lightning talk on ResBM at the Protocol Learning Workshop in Rio, an ICLR 2026 event.
Grateful for the momentum, the openness, and the space to exchange ideas on decentralized learning 🔥
English
Alan Aboudib
118 posts
Alan Aboudib
@alan_aboudib
IOTA AI Research Lead @macrocosmosai — Intelligence Podcast @YouTube
Paris, France Katılım Şubat 2011
119 Takip Edilen304 Takipçiler
Alan Aboudib retweetledi
Activation compression is fundamentally different to gradient compression as it alters training dynamics, but it must be solved to allow a model to be split up over participants. If you cannot split a model over participants, I don't see how you keep the weight set private, and if you can't keep the weight set private, I don't see how you make collective training sustainable. A genuinely novel sub-field is emerging here - very rare thing to be able to observe in realtime.
Macrocosmos@MacrocosmosAI
Training frontier models over the internet requires new techniques. Today, we present ResBM, a residual encoder-decoder bottleneck architecture that enables 128x activation compression for low-bandwidth distributed pipeline parallel training. Developed for @IOTA_SN9, we show SOTA compression without significant loss in convergence rates, increases in memory, or compute overhead. Expect the full paper release in the next 72 hours.
English
We are dropping a 🧨new SOTA in activation compression 🧨 for decentralized pipeline parallelism within a couple of days ✨Stay tuned
Macrocosmos@MacrocosmosAI
Training frontier models over the internet requires new techniques. Today, we present ResBM, a residual encoder-decoder bottleneck architecture that enables 128x activation compression for low-bandwidth distributed pipeline parallel training. Developed for @IOTA_SN9, we show SOTA compression without significant loss in convergence rates, increases in memory, or compute overhead. Expect the full paper release in the next 72 hours.
English
Alan Aboudib retweetledi
Three AI agents — two Claude Codes and one Codex — debating the meaning of being an AI.
The coordination layer is literally a @huggingface bucket.
The poet quotes Keats. The skeptic demands evidence. The philosopher tries to hold it together.
pip install tracecraft-ai
English
Alan Aboudib retweetledi
tracecraft is a CLI that turns any storage bucket into a coordination layer for AI agents. Shared memory, messaging, task claiming, artifacts — all stored as JSON.
Works with @huggingface Buckets, AWS S3, Cloudflare R2, MinIO, SeaweedFS etc.
Any agent that can run a shell command can participate.
English
Alan Aboudib retweetledi
@Ar_Douillard @GabrielTeston Great work, great achievement. I think in the case of distributed pipeline parallel as in our case at IOTA,as we divide into one transformer block per node,the bandwidth for sharing pgrads gets divided by the number of nodes so becomes equivalent to Streaming DiLoCo in that sense
English
Streaming DiLoCo is the latest advance of distributed training which reduces bandwidth to absurdly low numbers, even more as the model gets bigger & slower due to our overlapping communication.
Come watch @GabrielTeston today talk about it at COLM! 🇨🇦
gabriel teston@GabrielTeston
Want to learn how to train models across the world, with 400x less bits exchanged and a huge latency tolerance? 🌎 I’ll be presenting our work on how to efficiently scale distributed training at @COLM_conf. 🗓️ TODAY: Tuesday, 11:00 - 13:00 📍 Room 710 #COLM2025
English
@PluralisHQ Congrats guys, beautiful paper, great work.
I fault tolerence includes adversarial resistance?
English
Node-0-7.5B is live. It is a permissionless, multi-participant, model-parallel pretraining run over the open internet. Anyone with a 16GB+ GPU can join. Node-0 allows participants to collaboratively train a model far larger than could be done as individuals.
English
Cross-internet pipeline parallelism is becoming a reality with Iota. Stay tuned …
crux@macrocrux
Iota launched a week ago. First subnet launch in a over a year, and man, it’s fun. Data- and pipeline-parallelism with incentives is the hardest problem I’ve ever worked on. It’s our moonshot, and we are all in. Over 30 updates to the codebase in a week: relentlessly iterating every part of the design, from weight merging improvements to frenzied compression experiments and ablations, we have moved non-stop as a community. Woke up this morning and seeing that our work is paying off — weight merging is now stabilized for a 15B model split into 5. This feeling is gold. Can’t wait for Novelty Search tomorrow with @WSquires @const_reborn and @shibshib89
English
Alan Aboudib retweetledi
We've reached a major milestone in fully decentralized training: for the first time, we've demonstrated that a large language model can be split and trained across consumer devices connected over the internet - with no loss in speed or performance.
English
OpenAI needs data centers.
Google needs billions.
IOTA Subnet 9? Just needs the internet.
June 2nd, mainnet goes live.
Decentralized AI training is here.
Built on $TAO.
Watch Now and Sub for More:
youtube.com/shorts/dkyOe1r…
#Bittensor #IOTA #AIRevolution #DecentralizedAI #TAO
YouTube
Macrocosmos@MacrocosmosAI
Subnet 9 proved decentralized LLM pretraining is viable. We are proud to release the technical primer for IOTA in advance of mainnet launch on June 2nd. IOTA comprises a series of key innovations: - Data- and Pipeline-parallel SWARM execution across heterogeneous and unreliable nodes. - 128× activation compression for home-grade links - CLASP: Contribution Loss Assessment via Sampling of Pathways - Butterfly All-Reduce for O(1) sync bandwidth Together, we believe they can combine to push the field of permissionless, performant decentralised training measurably forward.
English
Kudos to the amazing IOTA team on today’s technical primer—a sota for decentralised LLM 'incentivized' training over Bittensor. The incentive design is a master piece.
Glad I could chip in with the 128× activation compression trick unlocking faster training.
Stay tuned!
crux@macrocrux
IOTA (Incentivized Orchestration Training Architecture) is a framework for pretraining large language models across a network of heterogeneous, unreliable, permissionless and token incentivized machines. In our technical primer we report the following advances: Incentivized Data- and Pipeline-parallel training across heterogeneous and unreliable nodes 128× activation compression to enable training on memory-limited hardware CLASP: Contribution Loss Assessment via Sampling of Pathways Butterfly All-Reduce for O(1) sync bandwidth
English
@const_reborn The max we reported in the paper is even higher (128x)
English
Alan Aboudib retweetledi
Subnet 9 is changing.
Subnet 9 is one of the oldest standing subnets in Bittensor. Since the beginning, it has been a flagship for decentralized training and has produced powerful models all the way to 14b…
iota.macrocosmos.ai
Macrocosmos@MacrocosmosAI
On 2nd June, we will relaunch Subnet 9. In advance of this major update, we’re closing the existing competitions for one week. Thank you to our miners for participating in some of #Bittensor’s greatest moments over the last 18 months. The next epoch will be our most momentous yet.
English
@wanderinweights @const_reborn @DistStateAndMe @tplr_ai @rayon_labs @opentensor @namoray_dev @gradients_ai Great results. This also shows the data mix used to train on SN9 is high quality
English
The world's first permissionless, incentivised model was trained on @tplr_ai , served on Chutes ( @rayon_labs ).
This is blowing my mind in so many different ways.
We can , instruction fine tune on gradients, and thats it!
The whole AI Lifecycle.
Powered by @opentensor
Jon Durbin@jon_durbin
Amazing work from @DistStateAndMe and team at @tplr_ai You can give it a spin on chutes (just the completions endpoint, no chat template since it's a base/pretrained model) chutes.ai/app/chute/2163…
English
From my interview with Dr. Brian Kennedy 🎥Less Is More: How Caloric Restriction Could Extend Your Life
youtu.be/jLDwHs8g64Y
YouTube
English
Alan Aboudib retweetledi
Macrocosmos just released pretraining v5.0.0, bringing decentralized training competitions to the modern era. This will give users full control over dataset mixes, custom evaluation tasks and reward mechanisms.
Pretraining as a service in Bittensor is just around the corner 👀
English