Yuning Mao

Training on issue-solving only does NOT guarantee transfer to other tasks. 🎨Introducing Hybrid-Gym - synthetic training tasks for generalization (hybrid-gym.github.io) +25.4% on SWE-Bench / +7.9% on SWT-Bench / +5.1% on Commit-0 with NO issue-solving / test-gen/... training

I was laid off by Meta today. As a Research Scientist, my work was just cited by the legendary @johnschulman2 and Nicholas Carlini yesterday. I’m actively looking for new opportunities — please reach out if you have any openings!

What if LLMs knew when to stop? 🚧 HALT finetuning teaches LLMs to only generate content they’re confident is correct. 🔍 Insight: Post-training must be adjusted to the model’s capabilities. ⚖️ Tunable trade-off: Higher correctness 🔒 vs. More completeness 📝 with @AIatMeta 🧵

How to construct repo-level coding environments in a scalable way? Checkout RepoST: an automated framework to construct repo-level environments using Sandbox Testing (repost-code-gen.github.io) Models trained with RepoST data can generalize well to other datasets (e.g., RepoEval)

📢My New Paper: Diversity-driven Data Selection for Language Model Tuning through Sparse Autoencoder TLDR: We proposed to use features from SAEs as a measure for data diversity&complexity and proved it's effectiveness on data selection for LLM tuning. arxiv.org/pdf/2502.14050

Among the most impressive aspect of the Llama 3.1 release is the accompanying research paper! Close to 100 pages of deep knowledge-sharing on LLMs like we havn't seen very often recently What a treat! It covers everything, pretrainining data, filtering, annealing, synthetic data, scaling laws, infrastructures, parallelism, training recipees, post-training adaptation, tool-use, benchmarking, inference strategies, quantization, vision, speech, videos... Mind-blown! Maybe the single paper you can read today to join the field of LLM from zero right to the frontier Read it here and feel the open-science ai.meta.com/research/publi…

Go team🦙

Introducing Meta Llama 3: the most capable openly available LLM to date. Today we’re releasing 8B & 70B models that deliver on new capabilities such as improved reasoning and set a new state-of-the-art for models of their sizes. Today's release includes the first two Llama 3 models — in the coming months we expect to introduce new capabilities, longer context windows, additional model sizes and enhanced performance + the Llama 3 research paper for the community to learn from our work. More details ➡️ go.fb.me/i2y41n Download Llama 3 ➡️ go.fb.me/ct2xko

Introducing 🌈 Rainbow Teaming, a new method for generating diverse adversarial prompts for LLMs via LLMs It's a versatile tool 🛠️ for diagnosing model vulnerabilities across domains and creating data to enhance robustness & safety 🦺 Co-lead w/ @sharathraparthy & @_andreilupu

Announcing Purple Llama — A new project to help level the playing field for building safe & responsible generative AI experiences. Purple Llama includes permissively licensed tools, evals & models to enable both research & commercial use. More details ➡️ bit.ly/3ReRNHI

Happy to share our new work on advancing LLM safety with auto red-teaming

Please remove the default system prompt if you find Llama 2-Chat too safe. Many of the false refusal examples people shared online can be addressed by this simple change

Honored to be a core contributor of Llama 2 release. Give it a try and I hope you'll be pleasantly surprised. Feel free to DM me for any feedback (especially on safety).

LIMA : LLaMA 65B + 1000 supervised samples = {GPT4, Bard} level performance. From @MetaAI arxiv.org/abs/2305.11206

🔥Excited to present Progressive Prompts - a method for continual learning in language models: no data replay, trains 0.1% parameters, 20% improvement over previous SOTA on T5. #ICLR2023 🤖👀 arxiv.org/abs/2301.12314 GitHub: github.com/arazd/Progress… Details in🧵1/N

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models Joint work w/ @hila_gonen, @yuning_pro, @AuspiciousRay6, @NamanGoyal21, @gh_marjan, @LukeZettlemoyer, @MadianKhabsa Idea: increase vocabulary size (intelligently), & fix over-tokenization.🧵[1/5]

UniPELT often surpasses the upper bound when taking the best performance of all its submodules used individually on each task, indicating that a mixture of multiple PELT methods may be inherently more effective than single methods. 🧵[4/4]

On the GLUE benchmark, UniPELT consistently achieves 1~3pt gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups, exhibiting superior model effectiveness and robustness. 🧵[3/4]

🚨 Working on parameter-efficient language models? Check out our 🆕 preprint "UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning" with @scottyih, @MadianKhabsa et al. at @facebookai, which achieves 1~3% gains on GLUE. arxiv.org/abs/2110.07577 🧵[1/4]