Devjeet Roy retweetledi
1/ In the AI era, specifications are becoming the primary interaction mechanism between humans and coding agents. Inferring, validating, and maintaining them is foundational to trustworthy software engineering.
English
Devjeet Roy
38 posts
Devjeet Roy
@devjeetrr
PhD student at Washington State University
Katılım Temmuz 2013
316 Takip Edilen47 Takipçiler
Devjeet Roy retweetledi
Measuring learning, not just capability, is fundamental to evaluating continual learning methods. Continual Learning Bench is built around exactly that!
Was a pleasure to contribute to this, check out the full write up:
Parth Asawa@pgasawa
Today, we’re releasing Continual Learning Bench 1.0: the first, realistic benchmark for measuring how AI systems can improve in online settings. Benchmarks today assume models are stateless. Each example is independent, and once a system finishes a task, it moves on as if nothing happened. But deployed AI systems should learn from experience. We tested 10+ frontier systems against novel, expert-validated tasks and find there’s still plenty of headroom for learning. (1/n)
English
Devjeet Roy retweetledi
Very excited to announce the v1.0 of SlopCodeBench release:
- Doubling the size of the dataset
- @harborframework support
- scb-check: a CLI that flags slop anti-patterns
- Way more model results
scbench.ai
github.com/SprocketLab/sl…
🧵
English
Devjeet Roy retweetledi
Our MLSys 2026 paper is live on arXiv: “Learning from Less: Measuring the Effectiveness of RLVR in Low Data and Compute Regimes.”
@realjustinbauer @Walshe_tech @pham_derek @harit_v @ArminPCM @fredsala and @paroma_varma present a comprehensive empirical study of open-source SLMs after RLVR in low-data regimes, revealing that dataset composition matters more than dataset size for scaling performance across number counting, graph, and spatial reasoning tasks.
Read the paper: arxiv.org/abs/2604.18381
English
Devjeet Roy retweetledi
Our #MLSys2026 paper is live on arXiv 📄
We ran a systematic study of RLVR in low-data regimes across 3 procedurally generated benchmarks (counting, graph, spatial reasoning).
Key finding: dataset composition matters more than dataset size.
arxiv.org/abs/2604.18381
English
Devjeet Roy retweetledi
We found that agents generate progressively worse code with each iteration. Real developers do not.
SlopCodeBench is the only eval that faithfully measures quality degradation on iterative, long-horizon coding tasks.
arxiv.org/abs/2603.24755
scbench.ai
🧵
English
Devjeet Roy retweetledi
AI coding assistants generate a lot of slop. Now there's a way to measure it: SlopCodeBench!
Gabe Orlanski@GOrlanski
Often, a bug I have had to fix has been tied directly to overly verbose or defensive code from an AI agent that has eroded my project as the specs changed. Why isn't there an eval for this? There is now. Very excited to announce SlopCodeBench scbench.ai
English
Devjeet Roy retweetledi
If you want to add your agent to the harness/run the evals/or anything related, please reach out. I would be more than glad to help 😀
English
Devjeet Roy retweetledi
Often, a bug I have had to fix has been tied directly to overly verbose or defensive code from an AI agent that has eroded my project as the specs changed. Why isn't there an eval for this?
There is now. Very excited to announce SlopCodeBench scbench.ai
English
Devjeet Roy retweetledi
What's the future of AI for Software Engineering?
🤖 Join Alex Gu (@minimario1729) (MIT; StarCoder, CRUXEval, LeanDojo contributor) tomorrow at the #DL4C workshop! He'll cover current challenges in AI for SE and promising directions for what lies ahead. #ICLR2025 #ICLR
English
Devjeet Roy retweetledi
We're excited to welcome our fourth speaker @taoyds, Assistant Prof at @HKUniversity and director of @XLangNLP. His groundbreaking work in grounding language into code and actions spans digital and physical environments, aiming to democratize data science and enhance human-computer interaction.
Join us at the 3rd Deep Learning for Code workshop at #ICLR2025, this Monday in Singapore! 🇸🇬
#DL4Code #NLP #AI
English
Devjeet Roy retweetledi
Go beyond code completion! Baptiste Rozière (@b_roziere) presents his work at @MistralAI automating code tasks (IDE tools, agents) for better developer focus. Catch him this Sunday at #DL4C! #ICLR2025 #iclr
English
Devjeet Roy retweetledi
Just 6 days until #DL4C! 🗓️ Daniel Fried (CMU / Meta AI) @dan_fried @AIatMeta will be sharing insights on how inducing functions from code makes LLM agents smarter and more efficient. Don't miss it! See you Sunday! #ICLR2025 #iclr
English
Devjeet Roy retweetledi
🚀 ICLR week is upon us! Join us at the #DL4C Workshop to hear Xingyao Wang (@xingyaow_) discuss LLMs evolving into SE agents, covering the CodeAct framework (code exec as action), the OpenHands platform (dev-like generalist agents), & SWE-Gym (real-world task training). @iclr_conf
English
Devjeet Roy retweetledi
Lots of great papers in – thank you all!
Review will start from tmrw. Last chance to sign up to be a reviewer for #DL4C forms.gle/gPosJujrdAQqUv…
@iclr_conf #iclr #iclr2025 #iclr25
English
Devjeet Roy retweetledi
🚀Excited to share the 3rd Deep Learning for Code workshop is back at @iclr_conf'25! This year we’ll focus on emergent challenges in the field, e.g., agents, post-training, developer productivity, open science, and benchmarking for code Submit by Feb 3 dl4c.github.io🧵⬇️
English
@sameercassim007 @vfsglobalcare I had the same issue. This is a DNS problem, as you can see here #A/schedule.vfsglobal.co.uk" target="_blank" rel="nofollow noopener">dnschecker.org/#A/schedule.vf….
For me, what worked is to use my phone. But you can also change your DNS server in your network settings.
English
@vfsglobalcare
Good Day, pls advise if the problem with UK Visa appointments has been resolved.
Applications captured and paid , but link does not take you to the appointment page... link not available
Johannesburg, South Africa 🇿🇦 English
Devjeet Roy retweetledi
What does it mean for the LLMs of Code to be open and responsible? Our fourth invited talk by Harm de Vries (@harm_devries) from @ServiceNowRSRCH and Leandro von Werra(@lvwerra) from @huggingface is about BigCode (bigcode-project.org)! #ICML2023
English
Devjeet Roy retweetledi
Thank you to all authors who submitted their works to the DL4C workshop @ ICLR 2023!
Instructions and Upload Form for Authors of Accepted Papers: forms.gle/LiTdK393NTz4VJ…
Deadline: May 1st
English
Devjeet Roy retweetledi
📢We are excited to announce our incredible lineup of speakers for the Second DL4C Workshop @ ICLR 2023!
Link: dl4c.github.io/speakers/
The speakers are:
English