Ali Al-Kaswan 🍉

Option to present your work at FSE and optionally publish a short paper. Explore the details and join us. Together we can build transparent, reliable benchmarks for code LLMs. poisonedchalice.github.io

Key information: Inclusive submission criteria: anyone can submit and attendance is not mandatory. We accept existing techniques or work under review at other venues Inclusion in proceedings is optional Submission deadline: 17 March 2026 Competition date: 6 July 2026 Montreal.

📣 Exciting competition for the AI & Software Engineering communities! 🎯 Large language models (LLMs) for code are powerful but they often memorize what they’ve seen. When evaluation datasets overlap with training data, reported performance can be misleading

6/6 🎉 We're looking forward to discussing our work at #FSE2025! This work highlights the importance of targeted alignment strategies tailored to the unique challenges of software engineering tasks 💡.

5/6 📢 We're excited to share our findings & resources with the community! Explore our results and resources: 📄 Preprint: arxiv.org/abs/2504.01850 💻 Repo: github.com/AISE-TUDelft/C… 🤗 HF Space: huggingface.co/spaces/an0nymo…

1/6 🎉 Thrilled to announce that our collaborative paper, "Code Red! 🚨 On the Harmfulness of Applying Off-the-shelf Large Language Models to Programming Tasks," has been accepted to @FSEconf 💻 @avandeursen @MalihehIzadi @serg_delft @AISE_TUDelft

🗣 Call for Papers! 🗣 Are you working on #NLP methods that address #SoftwareEngineering challenges & would like to submit your work to a specialized workshop? We invite you to submit your work to the 4th edition of the NLBSE workshop co-located w/ the @ICSEconf '25, in Canada!

@pr0me @nearcyan I think decoder-only models scale better; more data are available and they're more efficient. However, for some tasks, encoder-decoder models are still preferred.

@nearcyan i missed the memo on why we stopped scaling encoder-decoder models for code. codeT5 was pretty nice but then they just... stopped around 1B params?

BigCodeBench LLM Leaderboard

Cool ML4SE papers presented at @ICSEconf by our colleagues at @serg_delft, @aalkaswan1 and @MalihehIzadi. 1- Traces of Memorisation in Large Language Models for Code 2- Language Models for Code Completion: A Practical Evaluation #ICSE2024

@aalkaswan1 presenting our recent @ICSEconf work with @avandeursen on “Traces of Memorisation in Large Language Models for Code” in the final ICSE session on language models and code generation! @AISE_TUDelft @serg_delft #llm #llm4code #ICSE2024

At @AISE_TUDelft, we're happy to wrap up the year with 2 papers accepted at @ICSEconf (ResearchTrack)🎉 -Language Models for Code Completion: A Practical Evaluation -Traces of Memorisation in #LLMs for Code @aalkaswan1, @katzy_jonathan, @timvandamdev, Marc, Razvan & @avandeursen

A little less than 2 weeks (due Saturday, Dec 9) to submit your @NLBSE_workshop tool competition paper! Looking forward to your submissions for both issue and code comment classification tasks. More info: nlbse2024.github.io/tools/

Looking forward to your @NLBSE_workshop tool competition entries on code comment classification, due **Dec 9** We give a larger dataset, a new scoring formula, and a base Colab notebook to train/test your solutions. See: nlbse2024.github.io/tools/ Questions? Contact us!

Heartfelt thanks to Prem @devanbu for spending an enlightening week w @serg_delft & delivering a thought-provoking lecture on quality of code generated by #LLMs in our course (ML4SE'23)! I am sure your expertise added a rich dimension to the learning experience of our students.

Why do #LLMs perform better on some programming tasks than others? In our #IEEESCAM2023 paper, we show early evidence for an inherent difference in representations of tokens between languages. arxiv.org/abs/2308.13354 @MalihehIzadi @avandeursen @ieeescam #ML #LLM #ML4SE