Yaroslav Bulatov

Gradients are energy-inefficient due to long-range dependencies, yet we lack a viable alternative. What if we crowdsource the discovery of what's next? Starting a weekly in-person reading group in SF. WebGPU, online learning, Hinton. Email yaro.slavvb@gmail.com.

Was happy to host @stephen_wolfram at South Park Commons. One discussion point, why do we use real numbers for learning? Computers are discrete ... Stephen brought up a potential culprit, 1943 McCulloch/Pitts paper link.springer.com/article/10.100… youtube.com/watch?v=4kae7Y…

@FrancoisChauba1 Also I wonder if "order" is a false dichotomy. For instance take Richardson iteration. It is an "adjoint-free" method, no derivative is needed. But you could also derive it as form of gradient descent. So logically it's first order, but it has cost of 0th order

what's wrong with backprop

@tkanarsky This has the explanation -- medium.com/tensorflow/fit… . You need all L layer activations in order to compute a gradient. As L grows, you exhaust cheap/fast SRAM and get forced to use expensive HBM

@yaroslavvb Could you explain what you mean by "L layers simultaneously"? isn't the whole point of chain rule that it allows gradient of an op further up thr graph to be defined only in terms of the most proximate gradient down the graph

@FrancoisChauba1 I thought Hinton's Forward-Forward is a pretty cool idea. It addresses the energy issue. It doesn't work sufficiently well from learning standpoint, but I wonder if it can be fixed

Spending Sunday morning reading about Recurrent Laryngeal Nerve

Perhaps the most important skill in the age of agents becomes management skill. Famous example from organizational theory, Boeing 777 got things unstuck by switching from functional partitioning to product partitioning

Finally got around to using researcher agents and they work much better than I expected, thanks to @YadKonrad for getting things going at github.com/cybertronai/Su…

@cosminnegruseri but yeah, I think there are not enough funding strategies which allow founders to be honest, hence a lot of "fake it till you make it"

@cosminnegruseri non-profit like OAI ? :)

Have been thinking today about the practice of starting company before you know what it's for, seems backward to me. Zuck talked about same issue in an interview in 2016 gemini.google.com/share/63600785…

@srush_nlp @AryaTschand I don't see much codesign happening. Alex Smola gives a tutorial on evolution of attention at icml-2019. Something like 7 steps by researchers, trying things out on CPUs and GPUs. TPUs were pretty clunky in the early days, not a good fit for research

@AryaTschand Oh lol. Looks like you worked on TPUs so you probably know way more about this than me.

Hypothetically, Tensor Cores could have predated most general purpose computing, right? Feels like there should be people creating worlds where GPT-5 exists, but MS Word is wildly impossible.

Chaotic Solitons in Gray-Scott reaction

@FeinbergVlad ok, the AI generating the slide was a little too self-congratulatory here

@yaroslavvb How does (gen) AI help with matchmaking?

Coasen Singularity - firms exist to minimize transaction costs. If AI agents reduce these transaction costs to near-zero, economic need for large firms disappears, potentially leading to a "singular" economy of individuals or "one-person unicorns."

Happy to host @dmsobol at @southpkcommons next Tue. She leads Mixture of Experts training at @cerebras and will tell us everything she knows about MoEs luma.com/moe-daria

@sopharicks the listening group is at 8:30am tomorrow

@yaroslavvb When is the reading?

For this week's ML reading group in Precita Park Cafe, I created a song summary, so that people can get an overview of the paper on the way to the park science-radio.discourse.group/t/auto-regress…