Anthony Barker

2.1K posts

Anthony Barker banner
Anthony Barker

Anthony Barker

@FiveIsGreen

Physicist, martial artist, AI enthusiast

Hamburg, Germany Beigetreten Mart 2009
420 Folgt155 Follower
Anthony Barker
Anthony Barker@FiveIsGreen·
@SebastienBubeck Notice that if you have just a bit of memory, you can improve by counting the number or requests and switch from the “Move to the front” policy to the Transportation rule after the lines cross around request 250.
English
0
0
0
547
Sebastien Bubeck
Sebastien Bubeck@SebastienBubeck·
My good friend Christian Coester solved a 50 years old open problem in self-organizing lists (was it really open? did people even care? yes and yes), and, you guessed it, ChatGPT-Pro made the key step in the proof*! (*a Christian Coester on the other side of the screen was needed to see through the full proof!) The problem is extremely simple: you are organizing (online) a list of items, and when item in position i is requested you pay i. Requests arrive i.i.d., and so the best thing to do is to organize the list in order of decreasing probability of appearance of items. But now what happens if the probabilities are not known, and you want a memoryless algorithm (note that with memory one could just estimate frequencies and approximate the optimal ordering)? 40 years ago it was shown that the most stupid memoryless rule of moving the requested item to the front achieves a competitive ratio of pi/2 (and this is tight). But empirically there seems to be a better idea, which is to be a bit less aggressive and simply move up the requested item by one position (the "Transposition Rule"), see Codex's experiment below. In the very first paper on this topic 50 years ago, Rivest conjectured that indeed this should be a very good rule. What did Christian prove? He showed that Transposition Rule's average cost is at the most the optimal fixed list plus ONE. Just a regret of one. Beautiful. In fact the proof shows something more delicate, and this is where ChatGPT comes in: Denote p_i for the probability of the i^th item (in the optimal ordering). Now under the stationary distribution of the Transposition Rule, denote s_i for the "excess cost" that item i incurs from wrong placement, namely: s_i = sum_{j < i} (p_j - p_i)*Prob(i appears before j) The regret of Transposition Rule is exactly the sum of those slack terms s_i. Now here is the magic: ChatGPT suggested that s_i might be smaller than p_i. If true that concludes the proof, since then we have sum s_i <= 1. Note that it is quite magical, I don't see a reason a priori why s_i would be smaller than p_i ... ChatGPT proposed a an argument to prove this for small n, and Christian found an actual proof for all n.
Sebastien Bubeck tweet media
English
6
30
258
23.8K
Anthony Barker
Anthony Barker@FiveIsGreen·
Airdropping Starlink is an obvious move to let Iranians coordinate taking back their country.
NetBlocks@netblocks

⚠️ Update: #Iran's internet blackout is now entering its 18th day after 408 hours without international connectivity for the general public. Chosen users are granted privileged access, while the remainder are left with a limited domestic intranet under increasingly tight control.

English
0
0
1
22
Anthony Barker
Anthony Barker@FiveIsGreen·
Why does anyone use the term “humanoid robot” when we already have the much nicer term “Android”? Is it just because of the phone OS?
English
0
0
2
36
VraserX e/acc
VraserX e/acc@VraserX·
I’m genuinely curious. If AI and robotics eliminate most human labor and scarcity… what does daily life actually look like? What do people do with their time? What gives life meaning? How do you imagine a post-labor world?
English
410
14
240
42.3K
Anthony Barker
Anthony Barker@FiveIsGreen·
To open Hormuz and save the petroleum dollar, the US gov needs to go into the business of shipping insurance.
English
0
0
0
13
WW3finalboss
WW3finalboss@WW3finalboss·
🇪🇺🇪🇺🇪🇺🇪🇺 Should the European Union become a country?
WW3finalboss tweet media
English
950
75
1.4K
181.4K
Anthony Barker
Anthony Barker@FiveIsGreen·
@VictorTaelin Make a compiler, and make the optimization flag require a micropayment. AI wash the optimization flag to make it seem like you have to do it on some giant server while magically also keeping code private.
English
0
0
0
97
Taelin
Taelin@VictorTaelin·
I think a lot of people will be using Bend2 soon. Been wondering about the business model though. How to sell a programming language? Since it is meant to be used by AI's, I think we could make it free for all, except AI labs must pay a license to use it on their agents 🤔
English
67
3
302
23.5K
Anthony Barker
Anthony Barker@FiveIsGreen·
This is a clear symptom of the Singularity: AI researching new and better AI faster and more thoroughly than human performance.
Andrej Karpathy@karpathy

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project. This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.: - It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work. - It found that the Value Embeddings really like regularization and I wasn't applying any (oops). - It found that my banded attention was too conservative (i forgot to tune it). - It found that AdamW betas were all messed up. - It tuned the weight decay schedule. - It tuned the network initialization. This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism. github.com/karpathy/nanoc… All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges. And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

English
0
0
0
11
Wholesome Side of 𝕏
Wholesome Side of 𝕏@itsme_urstruly·
You wake up and it's 1990. No wifi. No mobile phones. No social media. No doom scrolling. What do you do first?
English
3.4K
149
3.4K
582.2K
Anthony Barker
Anthony Barker@FiveIsGreen·
@clashreport Gosh that sounds like a German version of DOGE and the Argentinian chainsaw
English
0
0
1
6
Clash Report
Clash Report@clashreport·
Germany's Merz: We must deregulate every sector. I call for a “regulatory clean slate.” Minor corrections to laws are not sufficient. We need to systematically review the whole set of existing EU legislation.
English
555
976
7.7K
1.6M
Kurus
Kurus@kuruscam·
@DJSnM Second?? When was the first?
English
3
0
0
3.6K
Science girl
Science girl@sciencegirl·
What is stopping humanity from living peacefully together?
English
29.6K
2.8K
22.1K
6.5M
Anthony Barker
Anthony Barker@FiveIsGreen·
I reckon from the snap invasion that Iran got its spicy metal out of Fordow before Midnight Hammer.
English
0
0
0
32
Anthony Barker
Anthony Barker@FiveIsGreen·
A light-femtoDay is 1.02 inches.
English
0
0
0
4