Egor Bogomolov

RL training and coding-agent experiments not scaling locally? IdeGYM fixes that – and it's now open source. jb.gg/rsrch-idegym

A pleasure to see our work posted by AK 😊 📉Issue: GRPO hurts generation diversity and pass@k/max@k do not grow as well as pass@1 📈Solution: we estimate gradients for both on- and off-policy cases to optimize max@k directly, and it shows better yields in code generation tasks (measured as the ratio of passed tests)

Interesting @NeurIPSConf experience: a weird mix of likely the most positive review I’ve received and an overriding PC reject on top of it 🤔

The 3rd IDE Workshop @ICSEconf 2026 is scheduled for Saturday, April 18th! Please submit your short papers and extended abstracts on anything IDE-related: plugins, studies, refactorings, environments, AI in IDE, etc.! All information here: ide-workshop.github.io 🏝️🏝️🏝️

JetBrains Junie is live — for everyone! single AI subscription. pro tier included with all products pack and dotultimate.

@InceptionAILabs Is there an API to access the model? I would be happy to run it on some coding benchmarks, but I have not find any points to the API yet.

We are excited to introduce Mercury, the first commercial-grade diffusion large language model (dLLM)! dLLMs push the frontier of intelligence and speed with parallel, coarse-to-fine text generation.

@john_lam @headinthebox Hi, author here. For code generation task we give model an instruction in natural language and access to the library. The reference code is only used for evaluation to compute metrics

I don't disagree with your assertion, but I find the code generation examples quite suspect. They have a file containing some "reference code" and from that they are expecting the model to generate working code based on the reference code. The problem then becomes one of "what is the right reference code to put into the model's context"? How would that happen in the real world vs. a benchmark? I find that in my own experience using Sonnet 8 hours a day every day that the context is critical. Perhaps the most important context is having a shared understanding between me and Sonnet about what we want to achieve before Sonnet goes about writing the code. If we screw up defining the shared context (you could think about this as a "spec via chat", bad results and wasted time follow. If we are aligned on the shared context, I find 80% of the time I get acceptable results.

blog.jetbrains.com/ai/2024/08/lon… " ... The results show that even the best model – GPT-3.5 in this case – can only fix 17% of failing builds, indicating just how much work yet needs to be done for AI to be practically useful in this context. ..." Understatement of the year! Who would have expected 3 years ago that plain stupid next token prediction can solve 1 in 6 of failing builds automatically. That is a huge productivity improvement. Developers, you are fucked as a profession. Completely and utterly fucked.

Interested in impacting millions of developers with your ML research? We at JetBrains Research are hiring! jetbrains.com/careers/jobs/s…

🤝 Great thanks to @saridormi, Timur Galimzyanov, Evgeniy Glukhov, Anton Shapkin, @tigina_maria, @areyde, Alexander Kovrigin, @avandeursen, @MalihehIzadi, and @timofeybryksin for a fantastic work! Please submit to our leaderboard, and we will see you in Long Code Arena! 🏟️

🗂️ Module summarization — based on the module’s or project’s source code and a short description of the desired documentation, the model should generate it, testing its abilities in large comprehensive natural language texts. This benchmark includes custom LLM-based evaluation.

Models' contexts are getting so big that they can (and often should!) include an entire repository, while we are still evaluating them on methods and files. That's why we created Long Code Arena. Pre-print: arxiv.org/abs/2406.11612 Datasets: huggingface.co/spaces/JetBrai… Details in🧵!

💜Exciting news from JetBrains Research! 💜 We're publishing the Kotlin ML Pack — a set of datasets, multiple fine-tuning checkpoints, and a handwritten evaluation benchmark. Check out our HuggingFace page jb.gg/kotlin-ml-pack & the details below! & the details below (1/5) :