Jonghyun Choe

We're releasing a technical report describing how Composer 2 was trained.

Inference compute is on track to be a massive computational workload by the end of this decade. I think it will be much bigger than training (especially if you consider RL rollouts / inference needs for training). And it's still an open playing field in terms of the hardware, the platforms, and the models. It's also increasingly clear that people are willing to pay a premium for reduced latency. On the hardware side there are several interesting directions to keep an eye on: - SRAM style setups seem promising (GPT Spark on Cerebras, Groq acquisition by Nvidia) - Disaggregated systems (prefill on one machine / processor, generation on a different one) probably make a lot of sense. The computational characteristics of prefill vs decode are so different, specializing at the hardware level will yield efficiency gains - I also wouldn't discount more exotic technology like the Taalas chip / near memory computing / etc. While they are still pretty far out from large scale deployment, the economic pressure for efficiency gains could be a catalyst On the algorithm / architecture side: - Pretty much every major open-weights model has at least one optimization which makes it faster for inference. Whether it be MoE, SSM (or other hybrid variety), or sliding window or sparse attention. There are more differences here than there were a year ago. And it will be interesting to see where we converge. - Will diffusion models unify the prefill / decode split? - Still believe there are big gains to be had in further co-design of model to hardware and workload I also don't think we will have a one-size fits all solution in the future: - Cloud-based models may look very different than edge-optimized models - Models may be more and more co-designed for the hardware they are deployed on - There will be at least one knob which trades-off latency and power efficiency / cost.