Indico Data Labs

There are some really compelling results in this paper (some intuitive, some not so much). The causality analysis shows some non-linearity worth further investigation and further analysis of the effect of parameter counts may be warranted, assuming the true dynamic is sigmoidal.

Counterfactual analysis suggesting a causal relationship between removal of large numbers of relevant documents and QA performance.

Large Language Models Struggle to Learn Long-Tail Knowledge by: Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel arxiv.org/abs/2211.08411

Overall, this work provides thorough and encouraging results for distilling pre-trained language models into recursive transformers. The idea of adding per-layer adaptors whilst re-using the MLP and Attention particularly interesting.

The authors find that distilling the model with adaptors, that are different for each iteration of the recursive block, improves performance across all tasks. The adaptors seem to help the layers better mimic the behaviour of the separate layers from the teacher.

MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers by Nouriborji et at. Proposes a method for distilling Bert-Style transformers into Albert-style recursive transformers. arxiv.org/abs/2210.06425

Overall this was a refreshing work from OpenAI that shines light on often underappreciated aspects of ML -- dataset curation and generalization behavior! Models and code are openly available at: github.com/openai/whisper

Finally, since a portion of the training examples were non-English audio transcriptions or non-English audio translated to English, the model can be used in these settings as well. Scaling trends show clear improvements from model scale, especially in multilingual settings.

This week we're highlighting the open-source Whisper speech recognition model outlined in "Robust Speech Recognition via Large-Scale Weak Supervision" by former Indico founder @AlecRad, @_jongwook_kim, @txhf, @gdb, @mcleavey and @ilyasut. openai.com/blog/whisper/