Kyle retweetledi
CSHL’s Kyle Daruwalla discovered a novel approach for AI algorithms to process data more efficiently, resembling the ways in which the human brain learns. 🧠 cshl.edu/can-ai-learn-l…
English
Kyle
2.7K posts
Kyle
@darsnack
NeuroAI Scholar @CSHL | EE PhD @UWMadison | Comp. Eng. / math @RoseHulman | CS + neuro + AI
Katılım Eylül 2009
459 Takip Edilen187 Takipçiler
Kyle retweetledi
We present Cheese3D, a system to sensitively track 3D movement of the entire face in lab mice.
First #preprint from the lab!
biorxiv.org/content/10.110… 1/n
English
@DimitrisPapail If it’s for linear algebra, then I would say Julia hits the sweet spot of intuitive syntax. Even better than Matlab. But if this course goes all the way up to DL pipelines…hard to beat PyTorch.
English
what language should an intro to linear algebra for ML course use?
English
@DimitrisPapail @agi_watcher At best you have the exact gradient and at worst a really noisy one. Reducing how animals learn to computing an approximate forward gradient is unlikely to result in a “better” model.
English
@agi_watcher i don't know. "because neurological plausibility" doesn't cut it for me as an answer & there is no significant evidence that the algorithm is really comparable to backprop. So I'm not convinced that it's worth considering more carefully.
English
Perhaps the only interesting practical implication of only using forward passes during training is significantly less memory use, i.e., no need to store activations. Is there another one? Eg, no need to build a computation graph?
Neural plausibility doesn't cut it btw
Santiago@svpino
I just read The Forward-Forward Algorithm paper by Geoffrey Hinton. In summary: It's an algorithm to train neural networks with two forward passes instead of using backpropagation. Very early, but it definitely shows that this approach deserves more research.
English
@DimitrisPapail Data movement is the energy hog in large, distributed training right? Less memory = less to move. Plus any forward algo is presumably converging in the presence of gradient noise. Those tricks might also mean resistance to noise from asynchronous, distributed updates.
English
@DimitrisPapail Those seem like important implications as you scale up the model size. Or if you have recurrence in play (I’m thinking scaling up arxiv.org/abs/2102.11011). The best answer for gradient propagation through time has been “make time into space” (i.e. transformers).
English
Kyle retweetledi
The capabilities of LLMs are now causing a resurgence in folks saying that humans are "statistical parrots" mindlessly predicting the next word based on what they've heard before.
5 minutes of observing a young child still learning to speak should cure you of this notion.
François Chollet@fchollet
In general I've been sensing a new current deep learning maximalists recently, going from "our models can definitely reason" to "well our models can't reason, but neither can humans!"
English
Kyle retweetledi
Did you know, that you can build a virtual machine inside ChatGPT? And that you can use this machine to create files, program and even browse the internet? engraved.blog/building-a-vir…
English
@DimitrisPapail Aren’t they both evidence for the prediction? Second shows that shortcuts are limited the # of steps in the training data unless you change the problem to force it to learn the recurrence relation. Adding recurrence to the model is a more flexible solution.
English
some interesting papers to read
- evidence for the prediction arxiv.org/pdf/1807.03819…
- evidence against the prediction arxiv.org/pdf/2210.10749…
English
Prediction for the future of LLMs:
- GPT-X will be a *recurrent* Transformer, for X=4 or 5.
I can't see how TFs can become general purpose without simulating complex for-loops/GOTOs, without recurrence. Unless depth is linear or width becomes exponential in depth of recursion.
English
Kyle retweetledi
A dramatic example of this is how we engineer randomness into computers. Our amazing pseudo random number generators are reliable, predictable and rely on deterministism all the way down to transistors whereas biology is random at all levels and becomes deterministic when needed
François Chollet@fchollet
Humans as a collective can develop some incredibly sophisticated systems (your computer and the software running on it is proof of that), but those still remain orders of magnitude short of the superhuman sophistication of biological systems. For now, at least.
English
Kyle retweetledi
#NeuroAI: Could principles of embodied sensorimotor neuroscience unify and improve the various Self-Supervised Learning (SSL) methods? How could the brain self-supervise itself?
We are happy to share our #NeurIPS2022 paper arxiv.org/abs/2210.09224 with @franz_scherr and Q. Guo🧵:
English
@neurograce For stuff that’s mostly complete but you decided isn’t worth pursuing further. Somehow it seems wrong that these ideas are never disseminated. Anything that could be a preprint but still worth pursuing is something that I assume you’ll hang onto for next year.
English
Kyle retweetledi
White paper —Rallying cry for NeuroAI to work toward Embodied Turing Test !
Let’s overcome Moravec’s paradox: Tasks “uniquely” human like chess and even language/reasoning are much easier for machines than “easy” interaction with the world which all animals all perform.
Kording Lab 🦖@KordingLab
Massive whitepaper just dropped on why neuroscience progress should continue to drive AI progress: arxiv.org/pdf/2210.08340… Argues for an embodied turing test. Needed: real interdisciplinary people, shared platform(s), fundamental research
English
Sometimes you come across numbers that are so mind boggling that you have to share them. Some people in America pay $100 / Mbps that others pay $0.25 / Mbps (in the same city)! We desperately need a scientific corps in Congress.
themarkup.org/still-loading/…
English
Kyle retweetledi
who would win? millions of TPU-hours of RL or two cheeky austrians 😜 arxiv.org/abs/2210.04045
English
Kyle retweetledi
Scholars in all disciplines & countries, we invite you to sign this open letter to @WHOSTP calling for equitable access & participation in research for all
Why we think you should sign
ostp-letter.github.io/why.html
Or go straight to the letter
ostp-letter.github.io
#OSTPLetter
English
@lerlerimed @gta_rollergirl @UWMadison Wow. I didn’t even know that rule for international students. That’s messed up.
English