Yuren Cong

1/🚀 Excited to announce Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation! We built an omni model utilizing direct patch embedding layers for raw image inputs and achieves SOTA in multimodal understanding AND generation. Paper: huggingface.co/papers/2604.24… Code: github.com/facebookresear… Thanks to all the co-authors! @__Johanan, @wmren993, @xiaoke_shawn_h, @ShoufaChen, @TianhongLi6, Mengzhao Chen, Yatai Ji, Sen He, Jonas Schult, Belinda Zeng, Tao Xiang, @WenhuChen, Ping Luo, @LukeZettlemoyer!

@ducha_aiki I think it’s worth to try but I wouldn’t like to give a rude guess🤔 What is sure is that you need more patience (data)

@CongYuren I am more asking if I replace DINO in VGGT with this model, would it be better at point cloud and camera prediction. Or using it inside Roma v2 for image matching:)

1/🚀 Excited to announce Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation! We built an omni model utilizing direct patch embedding layers for raw image inputs and achieves SOTA in multimodal understanding AND generation. Paper: huggingface.co/papers/2604.24… Code: github.com/facebookresear… Thanks to all the co-authors! @__Johanan, @wmren993, @xiaoke_shawn_h, @ShoufaChen, @TianhongLi6, Mengzhao Chen, Yatai Ji, Sen He, Jonas Schult, Belinda Zeng, Tao Xiang, @WenhuChen, Ping Luo, @LukeZettlemoyer!

@KlemenKotar do you mean the patchily size?

@CongYuren This is really cool! Did you experiment with the size of the flow matching pixel prediction head? Is there a noticeable improvement in prediction quality as you scale it ?

@_Suresh2 These simple layers are initialized and trained from scratch.

@CongYuren are the patch embeddings trained from scratch, or bootstrapped from a pretrained tokenizer?

3/🚀 Why does this matter: Pretrained vision encoders may be limiting your multimodal model. By using patch embedding layers, Tuna-2 enables fully end-to-end optimization from raw pixels and can achieve higher upper limit! Just need a bit of patience..

2/🚀 How: We first derive Tuna-R, a pixel-space model that relies solely on a representation encoder. Furthermore, Tuna-2 streamlines the design by bypassing the representation encoder entirely, utilizing direct patch embedding layers for raw image inputs.

@ducha_aiki If you were asking whether using the patch embedding works than using Dino, then Yes:)

@CongYuren Is it better than DINOv3?

3/🚀 Why does this matter: Pretrained vision encoders may be limiting your multimodal model. By using patch embedding layers, Tuna-2 enables fully end-to-end optimization from raw pixels and can achieve higher upper limit! Just need a bit of patience..

2/🚀 How: We first derive Tuna-R, a pixel-space model that relies solely on a representation encoder. Furthermore, Tuna-2 streamlines the design by bypassing the representation encoder entirely, utilizing direct patch embedding layers for raw image inputs.

@liuziwei7 Great work!👍👍👍How large is the model?

🚫 No Vision Encoder (VE) 🚫 No Variational Autoencoder (VAE) ✅ Just one end-to-end model directly engages with native signals, pixels and words, for both understanding and generation. 💊NEO-unify💊 is the first step toward **truly end-to-end unified models**, learning directly from near-lossless inputs via a representation space shaped by the model itself.

(3/3) Scaling Zero-Shot Reference-to-Video Generation arxiv.org/abs/2512.06905

(2/3) Mixture of States: Routing Token-Level Dynamics for Multimodal Generation arxiv.org/abs/2511.12207

Glad to share we have 3 papers accepted to #CVPR2026🥳🥳🥳: (1/3) TUNA: Taming Unified Visual Representations for Native Unified Multimodal Models arxiv.org/abs/2512.02014

Excited to share TUNA is accepted to #CVPR 2026, which is among the first to scale native UMMs to video understanding+generation🚀🚀🚀

@giffmana @cloneofsimo Great insight! 🧐If we extend this to multimodal learning, do you think initializing the modality encoders randomly will also bring any benefits?

Not saying that's what they see, but if you have data and compute, in the long run, from scratch always wins. Here is one striking example from our distillation pape; blue is random init, yellow is init with sota model *on the same target task*. And I've seen more variants of this over the years.

SAM3 was pretrained from scratch? I wonder why they didnt init from dinov3? I have a hypothesis that SSL pretrained models are actively hurtful when you have large compute budget, i wonder if thats what they saw

@ziqi_huang_ amazing work! Is it too late to get TUNA on the table?

Unified multimodal models can jointly enable visual understanding and generation, but how and when do these abilities truly reinforce each other? Meet the 🔍𝗨𝗻𝗶-𝗠𝗠𝗠𝗨 benchmark to know more. 🌐 vchitect.github.io/Uni-MMMU-Proje… 📄 arxiv.org/abs/2510.13759 💻 github.com/vchitect/Uni-M…

Excited to announce that @ManusAI has joined Meta to help us build amazing AI products! The Manus team in Singapore are world class at exploring the capability overhang of today’s models to scaffold powerful agents. Looking forward to working with you, @Red_Xiao_!

6/📷Attribute-Centric T2I: attribute-aware compositional T2I generation. Better control, better disentanglement. Also done during my PhD📌IJCV accepted. link.springer.com/article/10.100…

5/🌉SPAN: scene graphs × images, aligned with Transformers. Done during my PhD📌TPAMI accepted, a long-running🏃 ieeexplore.ieee.org/document/11231… #scene_graph #scene_understanding

✨My 2025 Research Wrap: pushing boundaries in multimodal research, thanks for every collaboration 1/🐟TUNA: unified representation for generation/understanding (image + video). No encoder mismatch! see more at tuna-ai.org! #GenAI #multimodal #LLMs