Phillip Rust

Happy to share our paper on language modelling with pixels has been accepted to ICLR‘23 (notable-top-5% / oral) 🎉. Big thanks and congrats to Team-PIXEL @jonasflotz @ebugliarello @esalesk @mdlhx @delliott and looking forward to presenting in Kigali! 🌍 #ICLR2023

Tough week! I also got impacted less than 3 months after joining. Ironically, I just landed some new RL infra features the day before. Life moves on. My past work spans RL, PEFT, Quantization, and Multimodal LLMs. If your team is working on these areas, I’d love to connect.

Humans see text — but LLMs don’t. I wrote a short blog post exploring how models can perceive text visually rather than tokenize it: 🔗 csu-jpg.github.io/Blog/people_se… From PIXEL, CLIPPO, VisInContext, VIST to DeepSeek-OCR, this is a quick story of how vision-centric modeling is changing how machines read, and a reflection on some of our own small efforts in the past two years.

I will be presenting this work in-person at ACL🇹🇭 this week. Drop by if you'd like to chat! Oral: Today (Monday) 16:30 Poster: Tuesday (Tomorrow) 10:30 - 12:00

This project is a collaboration with my amazing peers and mentors during my internship @AIatMeta: Bowen Shi, @skylrwang, @ncihancamgoz @j_maillard. ⭐ 🧵(9/9)

For more experiments and all the details, check out our arXiv preprint linked above. We are working on releasing our code and data, so stay tuned! 👨‍💻 🧵(8/9)

Introducing “Towards Privacy-Aware Sign Language Translation at Scale” We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data. 📄: arxiv.org/abs/2402.09611 🧵(1/9)

New preprint "Improving Language Understanding from Screenshots" w/ @zwcolin @AdithyaNLP @danqi_chen. We improve language understanding abilities of screenshot LMs, an emerging family of models that processes everything (including text) via visual inputs arxiv.org/abs/2402.14073

In PHD: Pixel-Based Language Modeling of Historical Documents with @NadavBorenstein @rust_phillip and @IAugenstein, we apply pixel language models to processing historical document and to more standard NLP classification tasks too. See it in Poster Session 6 on Sunday 10th.

In Text Rendering Strategies for Pixel Language Models with @jonasflotz @rust_phillip and @esalesk, we design new text renderers for visual language processing to improve performance or to squeeze the model down to just 22M parameters. See it in Poster Session 2 on Friday 8th.

anon policy survey is out: tinyurl.com/aclarxivpolicy

Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks across speech-to-text, speech-to-speech, text-to-text translation & speech recognition for up to 100 languages depending on the task. Details ⬇️

📢 I am hiring a postdoc to join our project on pixel-based natural language processing. The position is based in Copenhagen 🇩🇰 for 18 months. Applications are due by March 29 employment.ku.dk/faculty/?show=…. Informal inquiries are welcome.

Thrilled to receive a grant from @VILLUMFONDEN to carry out blue-skies research on tokenization-free NLP veluxfoundations.dk/en/about/proje… I will hire Ph.Ds and Postdocs to build up the group so feel free to reach out. We're starting off with a paper at #ICLR2023 openreview.net/forum?id=FkSp8…