Ishaan Gulrajani
1K posts
Ishaan Gulrajani
@__ishaan
Hi! I’m a machine learning researcher @openai. Previously @stanford @facebook @google @mila_quebec
San Francisco Katılım Kasım 2010
512 Takip Edilen4.2K Takipçiler
Absolutely crazy to me there is a whole generation of people who know what a transformer is but might not know what an RNN is (not meant to be a comment on Helen). Is this what getting old is like?
Helen Toner@hlntnr
Never forget @karpathy training a recurrent neural net (precursor to transformers) to imitate @paulg in 2015—a thing of syntactic and semantic beauty:
English
@azmythalauris and one day you’ll say your last goodbye without knowing it at the time, and one day you’ll long for them, and one day …
English
I often get giddy thinking about the fact that there are people out there I don’t know, who I have yet to meet and will become close to and love. I don’t know who they are but eventually our paths will cross and I’m SO EXCITED ABOUT IT.
English
@curiousceros my life thinks I am objectively attractive but is not attracted to me
English
OK Apple Mail I know I’m not on top of my inbox but you don’t have to call me out like this
English
English
@sirbayes @ylecun @srush_nlp This is a non-issue but I've never understood why scientists prefer to minimize/descent. Maximization/ascent feels so much better. Much better vibes IMO 😎
English
Glad this one is also out! It was really fun working on this with Blake and Cengiz. I hope people enjoy :) Happy to chat if anyone has questions or comments
Kempner Institute at Harvard University@KempnerInst
NEW! Part two of a #KempnerInstitute blog series: @blake__bordelon, @ABAtanasov & @CPehlevan propose a simple and solvable model where many of the aspects of #LLMs are already present. Read more: bit.ly/3RqYMhX #neuralnetworks #AI
English
from a glance this is an excellent sequel to @lucastheis ' 2015 paper, which remains the most important thing i've ever read on evaluating generative models
Lucas Theis@lucastheis
What does it mean for an image, video, or text to be 𝑟𝑒𝑎𝑙𝑖𝑠𝑡𝑖𝑐? Despite how far we've come in 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑖𝑛𝑔 realistic data, 𝑞𝑢𝑎𝑛𝑡𝑖𝑓𝑦𝑖𝑛𝑔 realism is still a poorly understood problem. I've shared my thoughts on how to correctly quantify realism here: arxiv.org/abs/2403.04493 #icml2024 #genai #compression
English
I realize that one of the hindrances to my making fast progress on the PhD is that I don’t have family here, and I would really appreciate it if my friends could sign up to attend my defense if I ever have one.
English
English
I’m excited I’ll be joining MIT next fall, for a shared interdisciplinary faculty position between Music (@MIT_MTA @MIT_SHASS) & EECS.
I’m recruiting PhD students @MITEECS for Fall 2024 (apply by Dec 15), and also Postdocs. See czhuang.github.io for details. Come join us!
English
@chenlin_meng @judyhshen Congratulations, looks like really great work!
English
Super excited for the launch of Pika 1.0! I am extremely grateful to be working with such an amazing and talented team on this journey! ❤️
I am also very thankful for the support from our incredible investors, advisors, friends, and community members. We couldn't have achieved this without you! :)
Pika@pika_labs
Introducing Pika 1.0, the idea-to-video platform that brings your creativity to life. Create and edit your videos with AI. Rolling out to new users on web and discord, starting today. Sign up at pika.art
English
@karpathy @DBahdanau @kchonyc Attention in deep learning goes back at least a little farther! See Alex Graves' "soft-window convolutions" in arxiv.org/abs/1308.0850. (This of course does not diminish @DBahdanau et al's excellent work in any way.)
English
Thanks for highlighting; The paper that introduced Attention (by @DBahdanau, @kchonyc, Bengio) gets ~1000X _less_ attention than the paper "Attention is All You Need". And it is historically amusing that both are very general but happened to be developed for machine translation.
Jim Fan@DrJimFan
Today 6 years ago, "Attention is All You Need" went on Arxiv! Happy birthday Transformer! 🎂 Fun facts: - Transformer did not invent attention, but pushed it to the extreme. The first attention paper was published 3 years prior (2014) and had an unassuming title: "Neural Machine Translation by Jointly Learning to Align and Translate", from Yoshua Bengio's lab. It is a combination of RNN + "context vectors" (i.e. attention). Many of you likely haven't heard about this paper, but it's one of the greatest milestones in NLP and has been cited 29K times (compared to Transformer's 77K). - Neither Transformer nor the original attention paper talked about the general-purpose sequence computer. Instead, both were conceived as solutions to one narrow & specific problem: machine translation. It's remarkable that AGI (some day soon) can trace its origin to the humble Google Translate. 😅 - Transformer was published at NeurIPS 2017, one of the top AI conferences worldwide. Yet it didn't even get an Oral presentation, let alone awards. There were 3 best papers at NeurIPS that year. Combined, they have 529 citations as of today.
English
@jwthickstun @LucaAmb @sedielem @vlastelicap By a change-of-basis, Gaussian diffusion on learned embeddings is exactly equivalent to Gaussian diffusion on one-hots with a learned noise covariance. I really wanted one-hots to win, but reality had other plans.
English
@LucaAmb @sedielem @vlastelicap Ishaan has run experiments on one-hots (e.g., in the Diffusion-LM paper last year: Appendix F). I'm not sure whether a direct ablation of one-hots vs. embeddings has made it into any published papers. Maybe @__ishaan can comment.
English
Making diffusion language models work as well as autoregressive ones will be a challenge (see my earlier blog post: sander.ai/2023/01/09/dif…).
This paper quantifies this and finds a 64x efficiency disadvantage across all scales 👀 a big gap, but at least it's a constant factor!
Ishaan Gulrajani@__ishaan
New paper with @tatsu_hashimoto! Likelihood-Based Diffusion Language Models: arxiv.org/abs/2305.18619 Likelihood-based training is a key ingredient of current LLMs. Despite this, diffusion LMs haven't shown any nontrivial likelihoods on standard LM benchmarks. We fix this!🧵
English
@sedielem Fun fact that didn’t make it into the paper: when we started the project, the gap was 10,000x 😳. It took stacking many 10% improvements to make it this far.
English