Mohammad Rastegari
107 posts
@morastegari
Distinguished AI Scientist at Meta. Affiliate Assistant Professor at University of Washington.
Apple presents CatLIP CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data Contrastive learning has emerged as a transformative method for learning effective visual representations through the alignment of image and text
Apple presents OpenELM An Efficient Language Model Family with Open-source Training and Inference Framework The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and
Apple presents Speculative Streaming Fast LLM Inference without Auxiliary Models Speculative decoding is a prominent technique to speed up the inference of a large target language model based on predictions of an auxiliary draft model. While effective, in application-specific settings, it often involves fine-tuning both draft and target models to achieve high acceptance rates. As the number of downstream tasks grows, these draft models add significant complexity to inference systems. We propose Speculative Streaming, a single-model speculative decoding method that fuses drafting into the target model by changing the fine-tuning objective from next token prediction to future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 - 3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and Meaning Representation, without sacrificing generation quality. Additionally, Speculative Streaming is parameter-efficient. It achieves on-par/higher speed-ups than Medusa-style architectures while using ~10000X fewer extra parameters, making it well-suited for resource-constrained devices.
LLM in a Flash by Apple is the most upvoted paper on Hugging Face huggingface.co/spaces/hysts/d…
Apple announces LLM in a flash: Efficient Large Language Model Inference with Limited Memory paper page: huggingface.co/papers/2312.11… Large language models (LLMs) are central to modern natural language processing, delivering exceptional performance in various tasks. However, their intensive computational and memory requirements present challenges, especially for devices with limited DRAM capacity. This paper tackles the challenge of efficiently running LLMs that exceed the available DRAM capacity by storing the model parameters on flash memory but bringing them on demand to DRAM. Our method involves constructing an inference cost model that harmonizes with the flash memory behavior, guiding us to optimize in two critical areas: reducing the volume of data transferred from flash and reading data in larger, more contiguous chunks. Within this flash memory-informed framework, we introduce two principal techniques. First, "windowing'" strategically reduces data transfer by reusing previously activated neurons, and second, "row-column bundling", tailored to the sequential data access strengths of flash memory, increases the size of data chunks read from flash memory. These methods collectively enable running models up to twice the size of the available DRAM, with a 4-5x and 20-25x increase in inference speed compared to naive loading approaches in CPU and GPU, respectively. Our integration of sparsity awareness, context-adaptive loading, and a hardware-oriented design paves the way for effective inference of LLMs on devices with limited memory.
Excited that our "Dataset Reinforcement" paper introducing "ImageNet+ dataset" is accepted to #ICCV2023! Up to ~7x FASTER training! Paper: arxiv.org/abs/2303.08983 ImageNet+/Code: Coming Soon w/ @HPouransari @sacmehtauw @MFarajtabar @morastegari @OncelTuzel Ali Farhadi 1/7
Introducing the fastest neural network on mobile so far - MobileOne, an improved one millisecond mobile backbone. Latency range: 0.79 ms - 1.86 ms. ImageNet top1 accuracy range: 71.4 % - 79.4 %. arxiv: arxiv.org/abs/2206.04040 code and models: github.com/apple/ml-mobil…
(1/N) New paper! Separable self-attention for mobile vision transformers. Improves the speed and accuracy of transformer-based mobile vision network, MobileViT, significantly across tasks and input sizes. Paper: arxiv.org/abs/2206.02680
1/2 @CVPR so if somebody posts wrong facts about a paper I co-authored, I may not correct them in an answer in any way? i.e. you prefer the spread of falsehoods? What if someone hates me and creates anon account, acting like me and answering threads about my submission?
1/N The return of patch-based self-supervision! It never worked well and you had to bend over backwards with ResNets (I tried). Now with ViT, very simple patch-based self-supervised pre-training rocks! First BeIT, now Masked AutoEncoders i1k=87.8% arxiv.org/pdf/2111.06377… 🧶
Excited to release the source code and pre-trained models of MobileViT. github.com/apple/ml-cvnets
Excited to share our new method, "Learning Compressible Subspaces (LCS)", for adaptive model compression in real-time: (arxiv.org/abs/2110.04252).
