Ethan

12.6K posts

Ethan banner
Ethan

Ethan

@torchcompiled

trying to feel the magic. global research lead at @canva | prev: cofounder at @leonardoai

sydney - florida - SF Katılım Nisan 2022
897 Takip Edilen9.8K Takipçiler
Sabitlenmiş Tweet
Ethan
Ethan@torchcompiled·
personally I feel like the inflection point was early 2022. The sweet spot where clip-guided diffusion was just taking off, forcing unconditional models to be conditional through strange patchwork of CLIP evaluating slices of the canvas at a time. It was like improv, always trying to riff of mistakes and sitting right at the fine line between interesting and incoherent.
Ethan tweet mediaEthan tweet mediaEthan tweet mediaEthan tweet media
EPROM@eprombeats

Image synthesis used to look so good. These are from 2021. I feel like this was an inflection point, and the space has metastasized into something abhorrent today (Grok, etc). Even with no legible representational forms, there was so much possibility in these images.

English
29
40
704
244.9K
Ethan
Ethan@torchcompiled·
There’s a few typos and bits that need to be fixed, which will resolve once arxiv updates with the v2 version, and will have code up soon, paper here! arxiv.org/pdf/2607.09967
Ethan tweet media
English
1
0
0
14
Ethan
Ethan@torchcompiled·
I think this is one of the most fascinating projects I’ve tinkered on yet. Parameterizing our weights as a learned weighted blend of symexp and regular linear weights gives up to ~1.42x training speedup wallclock, and can be fused back into standard weights for deployment🧵
Ethan tweet media
English
1
0
1
73
Ethan
Ethan@torchcompiled·
@alexUnder_sky I recall a distil pub on something like this, cellular automata with a learned rule set, is this the extension to 3d case?
English
1
0
1
128
sacha🥝
sacha🥝@alexUnder_sky·
Saw this paper long time ago, so many details about lego self-organisation. So cool that such interesting and unconventional work had made in to such prestigious venue. Dr. Risi is the goat, I guess
Sakana AI@SakanaAILabs

We are pleased to share our latest research, now published in Nature Communications: “Smart Cellular Bricks: Physical Modules That Recognize Their Own Shape and Repair Themselves.” Blog: sakana.ai/smart-cellular… Paper: nature.com/articles/s4146… A long-running theme in our work is collective intelligence: the idea that sophisticated, robust behavior can emerge from many simple parts following local rules, with no central controller, as it does in a colony, a tissue, or a brain. We had mostly studied this in software and simulation. So this time we asked a simple question. Do the same decentralized principles hold up in the physical world, where communication is noisy and modules fail? To find out, we built a collection of simple cubic bricks. Each brick runs the same small neural network and talks only to the bricks it is physically connected to. No brick is told its position, or which shape it is part of. Yet from these purely local exchanges, the collective converges on the correct global shape, locates where modules are missing or damaged, and can even guide its own repair, inspired by how living tissue self-organizes and regenerates after injury. For us, this is a first step in a broader direction: taking the principles of collective intelligence we have studied in software and letting them emerge, decentralized and robust, in the physical world. In the future, we imagine smart materials that let structures sense and report damage on their own, and LEGO-like systems that recognize their own configuration and adapt in real time, pointing toward environments that are more robust, adaptive, and regenerative. This work is a collaboration between Sakana AI, IT University of Copenhagen and Autodesk.

English
1
0
4
811
Ethan
Ethan@torchcompiled·
@wolftivy a random init itself is arbitrary information. And we know from NTK/overparmeterized analysis that you get the lazy-training/barely moves while still reaching an optimal solution. ultimately why im saying, it's not so clear, but I suspect with growing width, it matters less
English
1
0
2
266
Wolf Tivy
Wolf Tivy@wolftivy·
@torchcompiled The point is about space, not optimality of the resulting function. If you can casually pack a megabyte of arbitrary information into two megabytes of weights, you’re not anywhere near optimal. And yes wouldn’t we all love a constructive proof of a superior method.
English
2
0
28
1.8K
Wolf Tivy
Wolf Tivy@wolftivy·
In other words, neural networks are shockingly inefficient and the algorithmic ceiling on intelligence is way higher. (An efficient representation would use more of the variance for itself)
will depue@willdepue

kinda funny you can draw a smiley face in your neural net before training and it’ll be there afterwards you can also use photos. i trained a MNIST classifier initialized to my face and you can still see me at the end works across inits, weight decay, LR, optimizers, see below

English
22
20
585
86.3K
Ethan
Ethan@torchcompiled·
@wolftivy If anything makes me wonder if scheduled noise injection is a missing point in optimization.
English
0
0
0
135
Ethan
Ethan@torchcompiled·
I’m inclined to agree, but might need a proof? FWIW these do underperform a bit IIRC, just as we’re deviating from a symmetric random init. But to the point of convergence point being very determined by your initialization (tensor programs and infinite width studies cover this a lot) seems like a hindrance towards an optimal solution but we also have many equivalent local minima that I believe get arbitrarily close to what the global optima yields?
English
2
0
8
2.2K
EMILY SUNDBERG
EMILY SUNDBERG@Emily_Sundberg·
Who is the most fascinating person in San Francisco
English
61
3
90
65.4K
Ethan
Ethan@torchcompiled·
@paularambles Fellas landing in the psych ward just to be building ChatWithYourPDF
English
0
0
6
423
“paula”
“paula”@paularambles·
"Claude Fable" is insane because it appears to be one of the most legitimately dangerous drugs with the potential to gigafry your brain but is exclusively taken by literal turbonormies who unironically want to like "make a personal website" and basically get oneshotted by it.
“paula” tweet media
English
18
9
340
15.9K
Ethan
Ethan@torchcompiled·
@willdepue > preventing attention gradients into past tokens The transformer with no transformer
English
1
0
39
927
Nicole
Nicole@elocinationn·
Prediction: Anthropic likely to suffer a major fall from grace when the people discover they are no better than OpenAI in terms of actually caring for the public good, and probably worse (i.e. following in the footsteps of whatever drove the left to self destruct over the past decade). Likely to be in relation to some major environmental damage with their aggressive data center roll out and how it contradicts their public narrative. I will never understand why corporations about to IPO bother with virtue signalling when they’re about to face a market that will just dump their stock if they do anything other than the most financially attractive option. It’s like asking to be burned at the stake. You face the same incentives as everyone else, so you’d be better to direct the public against your true enemy (bad incentives) vs your short term competitor—long game players know this.
English
21
3
153
15.4K