Steven Xia

We’re happy to release 𝐒𝐄𝐂-𝐛𝐞𝐧𝐜𝐡 𝐏𝐫𝐨: a benchmark for measuring the bug-hunting capabilities of AI agents in critical software systems such as Chromium V8, Firefox SpiderMonkey, and more. Explore the details here: sec-bench.github.io

@slimshetty_ @LingmingZhang Also feel free to check out the paper for more detailed analysis. We have a whole bunch of examples and results looking at the different tools agents generate!

@slimshetty_ @LingmingZhang Cool question! For Gemini-3-Pro, it mainly creates tools that are well-suited for analyzing the repo (e.g., read/search/write files). There are also task-specific ones like creating edit tools that apply targted patches that are quite difficult to do via bash commands only.

🤯🤯🤯 Gemini 3 Pro + Live-SWE-agent hits 77.4% on SWE-bench Verified, beating ALL existing models, including Claude 4.5!! 🤖 Live-SWE-agent is the first live software agent that autonomously self-evolves on the fly — and it even outperforms the manually engineered scaffold used by the Gemini 3 Pro team (76.2%)

🚀 Introducing Live-SWE-agent: 🤖 An autonomous coding agent that self-evolves on the fly while solving real-world issues. ✨ Simplistic design: no offline training, no heavy workflows. 👇 Surprising gains: 📌 75.4% on SWE-bench Verified 📌 45.8% on SWE-Bench Pro (new SOTA!)

WE'RE GOING TO THE WORLD SERIES!!!!! #WANTITALL

We're releasing a new iteration of SWE-bench, in collaboration with the original authors, to more reliably evaluate AI models on their ability to solve real-world software issues. openai.com/index/introduc…

I'm on the way to USENIX Security'24. We will present one paper about privacy-preserving app authentication. I'm also happy to discuss TEE-based AI security and other AI-related security topics. If you're interested, please get in touch with me! #usenix #USESEC2024

Introducing OpenAutoCoder-Agentless😺: A simple agentless solution solves 27.3% GitHub issues on SWE-bench Lite with ~$0.34 each, outperforming all open-source AI SW agents! It's fully open-source, try it out: 🧑‍💻github.com/OpenAutoCoder/… 📝huggingface.co/papers/2407.01…

Magicoder: Source Code Is All You Need paper page: huggingface.co/papers/2312.02… introduce Magicoder, a series of fully open-source (code, weights, and data) Large Language Models (LLMs) for code that significantly closes the gap with top code models while having no more than 7B parameters. Magicoder models are trained on 75K synthetic instruction data using OSS-Instruct, a novel approach to enlightening LLMs with open-source code snippets to generate high-quality instruction data for code. Our main motivation is to mitigate the inherent bias of the synthetic data generated by LLMs by empowering them with a wealth of open-source references for the production of more diverse, realistic, and controllable data. The orthogonality of OSS-Instruct and other data generation methods like Evol-Instruct further enables us to build an enhanced MagicoderS. Both Magicoder and MagicoderS substantially outperform state-of-the-art code models with similar or even larger sizes on a wide range of coding benchmarks, including Python text-to-code generation, multilingual coding, and data-science program completion. Notably, MagicoderS-CL-7B based on CodeLlama even surpasses the prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1). Overall, OSS-Instruct opens a new direction for low-bias and high-quality instruction tuning using abundant open-source references.

In the past 6-mon release of HumanEval+ we have been improving its toolchain usability and dataset quality from v0.1.0 to v0.1.7 releases. 🔥 Now we release MBPP+, a new benchmark in EvalPlus v0.2.0: tinyurl.com/4pw82wb8 🧵

@nenrightjerger make it 3 years in a row😅

As Purdue claims the top spot in this week's poll, I'm wondering which scrappy team from NJ is going to be our undoing this year?

Introducing the EvalPlus leaderboard! evalplus.github.io/leaderboard.ht… 🔥28 models have been evaluated on coding HumanEval & HumanEval+ 🔥7B CodeLlama outperforms ~16B models e.g. StarCoder&CodeGen 🔥Phind-CodeLlama-34B-v2 and WizardCoder-Python-34B-V1 as open models both beat ChatGPT 🧵

@fried_rice @DomSteinhoefel Definitely! We are working on that and hoping to add additional targets

@DomSteinhoefel Will it be opensourced?

OK now we have @steven_xia_ on his work with @YuxiangWei9 and @LingmingZhang, all of @plfmse. Talking about program repair using large pre-trained language models

@saikatdutta2012 @cs_cornell @CornellCIS So happy for you! Updating the ICSE slide to include this rn!

Super excited to announce that I will be joining Cornell as an Assistant Professor of CS (@cs_cornell, @CornellCIS) starting Fall 2024. I am truly grateful to my advisor, mentors, friends, and family, without whom this journey would not be possible!

We welcome everyone to try out 📚𝐇𝐮𝐦𝐚𝐧𝐄𝐯𝐚𝐥+! A dataset to reflect the "real" correctness of LLM-generated code. Using📚𝐇𝐮𝐦𝐚𝐧𝐄𝐯𝐚𝐥+ is the same as HumanEval. You can easily pip install it and evaluate in our prepared sandbox (optional). github.com/evalplus/evalp…

🚨 Evaluating LLM-generated code on datasets with just "3 test-cases" is NOT enough! 🚨 We built ✨HumanEval+✨: improving HumanEval with up to thousands of new tests to fully evaluate functional correctness of LLM generated code! @JiaweiLiu_ @YuyaoStarling @LingmingZhang

We have released our dataset as well as our code at: github.com/evalplus/evalp… Furthermore, we also included all studied LLM-generated code at: github.com/evalplus/evalp… ✨ x 14 models (10 model types) ✨ x 5 temperature settings ✨ x 200 code samples

Compared to the original HumanEval, our results on our new dataset show that on average, the pass@K performance of current popular state-of-the-art LLMs drops by 📉15% including both ChatGPT and GPT-4.