Announcing a holiday gift: 🎅SantaCoder - a 1.1B multilingual LM for code that outperforms much larger open-source models on both left-to-right generation and infilling! Demo: hf.co/spaces/bigcode… Paper: hf.co/datasets/bigco… Attribution: hf.co/spaces/bigcode… A🧵:

SantaCoder is trained on Python, Java, and JavaScript and outperforms other large multilingual models such as InCoder (6.7B) or CodeGen-multi (2.7B) considerably! A lot of pieces from a lot of collaborators came together to get to that result:

@BigCodeProject I think the hugging face demo is erroring out?

@BigCodeProject @huggingface @grandorganics

@BigCodeProject @RecuerdameBot Recuérdamelo en 3 horas

@BigCodeProject @memdotai mem it

@BigCodeProject Congrats on this achievement. Tried with some basic prompts, ex: “group all elements of an array by the value of key property”. Could not get it to generate meaningful code, in fact generated code looks incomplete. Im doin sth wrong for sure.

@BigCodeProject Hey there, Is the code for this project available? (I can't find it). I'd like to see the details on how you implemented both Multi-Query-Attention (MQA) and Fill-in-the-middle (FIM). thanks! cc: @huggingface

@BigCodeProject @memdotai mem it

@BigCodeProject Doesn't seem to work that well but maybe that's because it's a small model and it's not fine-tuned.