Timo Schick

🎉 New paper 🎉 Introducing the Toolformer, a language model that teaches itself to use various tools in a self-supervised way. This significantly improves zero-shot performance and enables it to outperform much larger models. 🧰 🔗 Link: arxiv.org/abs/2302.04761

Looking forward to my first non-virtual conference in almost 2 years. If you’re attending #ICML2024 and want to chat, drop me a message 😊

🤖 Want an agent that can learn new tasks from only a handful of demonstrations and no weight updates? 🚀 Check out our new work on In-Context Learning for Sequential Decision-Making, where we show how we can use transformers to few-shot learn new Procgen and MiniHack tasks. 👋 If you want to learn more about it, come chat with us at the FMDM workshop @NeurIPSConf on Friday, December 15. 🙌 Kudos to @sharathraparthy who did an outstanding job leading this work, designing and running lots of experiments, and digging deep trying to understand the model’s behavior. 🧵👇

Excited to be giving an oral presentation at @NeurIPSConf on Toolformer: Language Models Can Teach Themselves to Use Tools [arxiv.org/abs/2302.04761]! When: Wednesday at 10:15am Where: Ballroom A-C (level 2) neurips.cc/virtual/2023/o…

Inflection AI just announced Inflection-2, a HUGE new 175 billion parameter language model. Capabilities exceed Google and Meta's top models and “is very close” to catching GPT-4. The CEO also said the company’s next model will be 10x larger in six months.

Thrilled to announce that Inflection-2 is now the 2nd best LLM in the world! 💚✨🎉 It will be powering Pi.ai very soon. And available to select API partners in time. Tech report linked... Come run with us! inflection.ai/inflection-2

It has been nothing short of incredible to be a part of this team and celebrate every accomplishment! And we’re still *just* getting started 🏃🏽‍♀️🏃🏽‍♀️🏃🏽‍♀️

🎉 Introducing Inflection-2, the 2nd best LLM in the world! Get ready to experience the future of AI with us. bit.ly/3TaUpcD

Utterly insane weekend. So sad. Wishing everyone involved the very best. In the meantime, we finished training Inflection-2 last night! ✨ It's now the 2nd best LLM in the world... & we're scaling MUCH further. Details v soon. Come run with us!

In just over 100 days since launching Pi, we’ve just hit one billion messages exchanged. A huge milestone 🤯 Any predictions on how long it will take us to get to 2 billion?!

🚨New Paper 🚨 Self-Alignment with Instruction Backtranslation - New method auto-labels web text with instructions & curates high quality ones for FTing - Our model Humpback 🐋 outperforms LIMA, Claude, Guanaco, davinci-003 & Falcon-Inst arxiv.org/abs/2308.06259 (1/4)🧵

Lost in the Middle: How Language Models Use Long Contexts arxiv.org/abs/2307.03172 Exciting work exploring the effectiveness of long context, led by @nelsonfliu and with Kevin Lin, Ashwin Paranajape, John Hewitt, @percyliang @Fabio_Petroni @MicheleBevila20

Excited to announce that we’ve raised $1.3B to build one of the largest clusters in the world and turbocharge the creation of Pi, your personal AI. forbes.com/sites/alexkonr…

We’re proud to announce Inflection-1, the best-in-class LLM developed at Inflection! Inflection-1, which powers Pi.ai, outperforms GPT-3.5, Chinchilla, and LLaMA on a number of academic benchmarks. More details in our technical memo: inflection.ai/inflection-1

One of our key sources of human data is no longer fully “human"! We estimate that 33-46% of crowd workers on MTurk used large language models (LLMs) in a text production task - which may increase as ChatGPT and the like become more popular and powerful. arxiv.org/abs/2306.07899

We present LongForm📜, optimizing instruction tuning for long text generation with corpus extraction. LongForm models outperform baselines such as FLAN-T5 and Alpaca. 📄Paper (w/@timo_schick @annalkorhonen @HinrichSchuetze): arxiv.org/abs/2304.08460 💾Repo: github.com/akoksal/LongFo…

// Deep Papers #3: Toolformer // LLMs like Bing and ChatGPT use external tools like calculators and web search to answer questions. How do you teach LLMs to *use* these external tools? Toolformer shows how! We interviewed the authors :) Spotify: open.spotify.com/episode/6uXohG…

For this week's @MilaNLProc reading group, @peppeatta presented "Toolformer: Language Models Can Teach Themselves to Use Tools" by @timo_schick et al. Paper: arxiv.org/abs/2302.04761 #NLProc #ReadingGroup

// Toolformer Podcast: Preview // Today I'm interviewing the Toolformer authors! LLMs like Bing (and soon, ChatGPT) can use external tools like calculators or internet search to answer questions. But how do language models *learn to use* these tools? 1/5

We are reproducing Flamingo, a vision and language model developed by Deepmind (arxiv.org/abs/2204.14198). We spent a good amount of time fighting training divergences (aka "instabilities"). Surprisingly, even at the ~2-3B scale. Some learnings from overcoming these 🧵:

Overcoming current LLMs limitations by augmenting them with better reasoning and tools is an exciting research direction. Check out our survey on this topic!