Holden

@mattmireles PowerInfer 1, the open-sourced code, works on Apple Silicon with CPU only. We are progressing on GPU support and adopting the method in this work where applicable. Stay tuned!

@hodlenx Does it work on Apple Silicon?

🚀 Excited to introduce PowerInfer-2: A game-changing LLM inference engine for mobile devices by the #PowerInfer team. It smoothly runs a 47B model with a staggering 29x speedup on smartphones! Watch our demo to see it in action! 🎥 Technical details at: powerinfer.ai/v2/

@078sky Some initial tests have shown approximately 40% savings on energy per token🔋 We would love to share more findings after comprehensive tests.

@hodlenx Cool! would be interesting to see power consumption metrics

@canav4r @vimpunk We expect this method to work well on other mobile platforms, but the current testing machines best fit our requirements for large RAM, high-speed flash, and NPU. It’s only a demonstrative implementation and we have no preference on the hardware.

@hodlenx @vimpunk What about mediatek or apple SOCs?

@IlyasHairline We’ve included the predictors into transformers model weights and you can train them end-to-end now. We found it more efficient to co-train instead of training the predictors offline.

@hodlenx Did you release training code for the predictor?

@IlyasHairline @wey_gu If the foundation models use these activation functions and are pretrained from scratch, it would be ideal. We have demonstrated their negligible loss/perplexity compared to SwiGLU, but the trained model exhibited very sparse FFNs in TurboSparse paper and arxiv.org/abs/2402.03804

@hodlenx @wey_gu What if you could change the activation function?

👐 PowerInfer-2 will be open-sourced based on the PowerInfer repo. We’re refining it to untangle from our testing platform and making it accessible on PCs for the community. Open-sourcing will happen in stages starting soon. Stay tuned for updates at github.com/SJTU-IPADS/Pow…

🔓 The power of cloud-scale models and local privacy isn't mutually exclusive. We're pioneering to bring LLM's incredible capabilities directly to your device without compromising privacy. Explore how we're making AI accessible to everyone, everywhere: powerinfer.ai

@wey_gu The method we proposed in TurboSparse enables us to sparsify a mainstream foundation model within 150B token (5% of pretraining). The continuous training of Mistral and Mixtral costs us less than $0.1M. We hope other researchers found it a nice trade😎

@wey_gu Those ground-breaking speedups are all based on intrinsic sparsity and depends on ReLU more or less. We have confirmed some alternatives, like ReLU^2, and dReLU proposed in TurboSparse. They are very promising but not adopted by mainstream LLMs yet. Retraining is still essential

@vimpunk It is implemented on Qualcomm Snapdragon 8 Gen 3 and utilises CPU & NPU. The test machine in the video is OnePlus 12.

@hodlenx very impressive work. what backends is this implemented on?

@tetrachino 需要NPU算力相对CPU强得较多，我们主要的测试是在Snapdragon 8 Gen 3做的。这张图是论文中的算力实测结果，NPU在FP16下可以跑到~5TFLOPS

@tetrachino 是的，我们在prefill阶段用了NPU。这时推理的batch size比较大，很适合NPU计算

@tetrachino Yep. It utilises NPU at the prefill phase

@lin72h MoE is the highlight, but it also applies to other LLMs. For a Mistral-7B level model, it can save nearly 40% of memory while achieving the faster inference speed than SOTA

@hodlenx Very impressive, does it only support MoE models?

Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameters - Proposes a novel dReLU function, which is designed to improve LLM activation sparsity - 2-5× decoding speedup model: huggingface.co/PowerInfer abs: arxiv.org/abs/2406.05955

Thrilled to unveil Bamboo-v0.1: A groundbreaking 7B LLM by the #PowerInfer team, matching Mistral's performance with 85% activation sparsity. Built on Mistral's weights, supercharged with dReLU for up to 4.38x hybrid computing speedups. Discover github.com/SJTU-IPADS/Bam….

🌟PowerInfer boosts LLM serving speeds by up to 11x on consumer-grade GPUs! Inspired by our Deja Vu paper (ICML'23 arxiv.org/abs/2310.17157), it serves ReLUified LLMs, keeping heavy hitter (hot) neurons in GPU and offloading sparsely fired ones on CPUs. Proud of my undergraduate/master alma mater SJTU (Shanghai Jiao Tong University) for this work! github.com/SJTU-IPADS/Pow…