RunLocal

🔥 VLMs on mobile devices with world-facing cameras key for proactive, intelligent computing. Local/on-device inference key for real-time, private experiences. Great to see an emphasis on smaller VLMs. Excited to see where @huggingface, @moondreamai, etc. take things 🚀

Snap's announcement about their on-device text-to-image model seems to have slipped under the radar… Apparently, it generates 1024x1024 images with quality that's comparable to cloud-oriented models like Stable Diffusion XL. But it can do that locally on an iPhone 16 Pro Max in <1.5 seconds! 🤯 Snap are planning to ship it soon to their ~450m daily active users, and I wouldn't be surprised if it's free. I wonder how all these subscription-driven, cloud-based image generation apps will respond… Announcement: newsroom.snap.com/ai-text-to-ima… Paper: arxiv.org/abs/2412.09619

Short but sweet talk about the WebNN API: youtube.com/watch?v=FoYBWz… Def worth checking out the YouTube playlist from @jason_mayes WebAI Summit last year. It's packed with great talks! Looking forward to the next summit!

Awesome work from @soldni and team at @allen_ai! If you're interested in shipping on-device AI language features in your app, I highly recommend checking out their demo app to get a sense of what's possible these days on an iPhone: apps.apple.com/us/app/ai2-olm…

creativestrategies.com/research/white… Stable Diffusion power consumption between M3 and X Elite: "M3 MacBook Air, 8-core CPU 10-core GPU with 16GB RAM spec, we see averages of 87.63 Joules used per image generated. On the Snapdragon X Elite system, we used a prototype Surface Laptop 15-inch with 16GB RAM, with the X1E78100 SKU of Snapdragon X Elite. We see averages of 41.23 Joules used per image generated"

“An old Asian man” Stable Diffusion 1.5 with ControlNet. Used 6.94 bit mixed-bit palletization to get the model size down to <1gb. Results are decent (around 1 it/s)

Benchmarked @bria_ai_'s RMBG CoreML model across almost every iPhone with an NPU yday. Latest iPhones 5x quicker for this model than XS/XR (from 2018).

Bitesize Benchmark (iPhone 15 Pro Max) WhisperKit (v0.6) vs Whisper.cpp (v1.6): - Accuracy: Both perfect for this short clip - Speed: WhisperKit at 45 w/s. Whisper.cpp at 30 w/s. (average over 3 runs) - Peak RAM (note: physical footprint only): WhisperKit at 180Mb, Whisper.cpp at 680Mb

Always benchmark your on-device models! Tried to run whisper-tiny on MacOS, using computeUnits.all in CoreML. Produced worse inference/load times than only using ANE, which is surprising because you'd expect CoreML to pick the optimal config. Why CoreML did this 👇

Just converted @bria_ai_’s awesome background removal model to CoreML for inference on iOS/MacOS. There's a 3x inference speedup when using the NPU (ANE) vs CPU/GPU on an iPhone 15 Pro Max. Shout out to @yairadato and team for their work (HF repo: briaai/RMBG-1.4).