Feng Wang

So happy to see our latest work ViT-5 getting attention and sparking discussions on social media! 🚀 We’ve already seen many new stars on our repo over the past week — really appreciate the support! In ViT-5, we modernize the original ViT architecture by systematically upgrading normalization, positional embeddings, introducing register tokens, and more. The result: SOTA performance among plain ViTs on both understanding and generation tasks. paper: arxiv.org/abs/2602.08071 code: github.com/wangf3014/ViT-5 I also noticed a few misunderstandings, so let me clarify: 1️⃣ ViT-5 does not introduce any new architectural components. Instead, we conduct a systematic study of which existing modules truly benefit ViT and whether they are compatible with each other. Our goal is to build a strong backbone using components that have already been rigorously validated through long-term empirical use for effectiveness and robustness. 2️⃣ Some pointed out missing references (e.g., EVA series). We fully agree these are excellent models that pushed structural improvements in ViTs. We will add missing references in the updated version. Over the past five years, there have been so many incremental architectural refinements to ViT that it’s hard to cover every single one — but please feel free to share interesting related work with me! 3️⃣ Scaling up is on the way. 👀

If you’re still on a ViT-style visual backbone… consider switching to ViT-5 🚀 (Also, QK-Norm is my favorite piece 🫶 — fixes a lot of training instability headaches 🤯)

Why do most LLM agents hit a wall? They don’t accumulate skills. Introducing SkillRL📚 — recursive skill-augmented reinforcement learning that lets agents learn skills from failure and evolve over time. 🔥A 7B model: • +41% over GPT-4o • ~20% fewer training tokens • 33% faster convergence SkillRL bridges raw experience → policy improvement by distilling trajectories into structured, co-evolving skills during RL. Most agents forget. SkillRL evolves. 🔄 📄 Paper: arxiv.org/abs/2602.08234 💻 Code: github.com/aiming-lab/Ski… Great work @richardxp888, Jianwen Chen, Hanyang Wang, @JiaqiLiu835914, @lillianwei423, @AiYiyangZ, and nice collab. w/ @__YuWang__, @XujiangZhao, Haifeng Chen, Zeyu Zheng, @cihangxie.

ViT-5: Vision Transformers for The Mid-2020s "a systematic investigation into modernizing Vision Transformer backbones by leveraging architectural advancements from the past five years" * LayerScale * RMSNorm * original MLP design with GeLU activation * both APE and 2D RoPE jointly * registers with a separate 2D RoPE * QK-Norm * remove bias terms in the QKV projection layers 84.2% top1 accuracy on ImageNet-1k, 1.84 FID on ImageNet-256

@ducha_aiki Another correction is that our main model in this scaling work is a Mamba-based architecture, so the scaling cost is not that "expernsive"

@ducha_aiki Thanks a lot for sharing our recent work! An important finding of this paper is that reducing the patch size can serve as an effective scaling dimension for vision models. Compared to increasing the parameter count, it offers greater scaling potential and is easier to implement.

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie tl;dr: we trained 1px patch size ViT so you don't have to. It improves results, but costly. arxiv.org/abs/2502.03738

@ducha_aiki So rather than taking our work as "you don't have to do patch size scaling," we believe it actually delivers a promising signal that "currently image data is under-utilized, and everyone should do patch size scaling to get stronger model."