Daniel Soudry

Accelerate your transformer model with the new Block-Sparse-Flash-Attention! github.com/Danielohayon/B… This training-free, drop-in replacement extends FlashAttention-2 with minimal code changes (CUDA Kernels Included). Paper: arxiv.org/abs/2512.07011

Scaling FP8 training to trillion-token LLMs From Intel. Trained a 7B model using FP8 precision on 256 Gaudi2 accelerators. Matched BF16 with 34% throughput improvement while using 30% less memory. Introduces Smooth-SwiGLU as solution to outlier amplification. Links below

I'm back in Vienna to present our paper at @icmlconf with Itamar Harel on Wednesday at 1:30 pm, at poster #907. Looking forward to meeting and chatting! #ICML2024

Selected as Spotlight for #ICML2024 ! 🥳

Q: You sample random neural networks until you find one with perfect training accuracy. What will be the generalization error? A: Typically good — We prove that when a “simple explanation” exists, such sampled NNs (MLP/CNNs) generalize well! arxiv.org/abs/2402.06323