The_DarrenS

20.4K posts

The_DarrenS banner
The_DarrenS

The_DarrenS

@The_DarrenS

♿Autistic. I am who I am and nothing more. Beware: I say it as I see it.

UK Katılım Haziran 2010
103 Takip Edilen456 Takipçiler
The_DarrenS
The_DarrenS@The_DarrenS·
@karpathy Hi there, I have a simple question on this from and old guy just leaning about AI the autoreacher does using it take up alot of power at home? any help from anyone on this is appreciated
English
0
0
0
23
Andrej Karpathy
Andrej Karpathy@karpathy·
Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project. This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.: - It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work. - It found that the Value Embeddings really like regularization and I wasn't applying any (oops). - It found that my banded attention was too conservative (i forgot to tune it). - It found that AdamW betas were all messed up. - It tuned the weight decay schedule. - It tuned the network initialization. This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism. github.com/karpathy/nanoc… All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges. And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.
Andrej Karpathy tweet media
English
974
2.1K
19.4K
3.6M
BBC Breaking News
BBC Breaking News@BBCBreaking·
Person dies after being shot by federal immigration officers in Minneapolis, US, officials confirm bbc.in/4bAxb91
English
462
256
712
304.4K
Volodymyr Zelenskyy / Володимир Зеленський
Together, we met in Dublin with @PresidentIRL Catherine Connolly. We are here on an official visit today. Among the main issues, we discussed the return of Ukrainian children abducted by Russia, two Irish universities joining the Global Coalition of Ukrainian Studies, Ukraine’s accession to the EU, and the steps and decisions to help accelerate this process. We also discussed cooperation on ensuring global food security, the rehabilitation and social reintegration of Ukrainian warriors, and the construction of shelters in our schools. I thank Ireland for its support, and for the financial and humanitarian assistance it has provided since the start of Russia’s full-scale war. For all types of support to our people. This means a lot to Ukraine.
English
187
590
4K
112.9K
The_DarrenS
The_DarrenS@The_DarrenS·
I only found #comphyui yesterday and want to do one thing. Convert some art i make into black and white depthmap for lasering any ideas #AIart
English
0
0
0
30
The_DarrenS
The_DarrenS@The_DarrenS·
@Kijaidesign Hi, I wonder if I may pick your mind on the depthmapanything in comphy. I only found comphy yesterday and want to do one thing. Convert some art i make into black and white depthmap for lasering' Your name keeps coming up so ask you i thought the best
English
0
0
0
10
The_DarrenS
The_DarrenS@The_DarrenS·
@falkirkcouncil Hi folks. Where is it on the site can one request bags for clothing to be disposed off or recycled?
English
0
0
0
3
BBC Breaking News
BBC Breaking News@BBCBreaking·
US sanctions major Russian oil firms, accusing President Vladimir Putin of refusing to end "senseless war" in Ukraine bbc.in/47mvbxl
English
137
221
779
260.1K
The_DarrenS
The_DarrenS@The_DarrenS·
@RolandDGEurope could you tell me if the versa art re 640 works with versaworks 6 or 7 and if not what version would it work with
English
1
0
0
8
The_DarrenS
The_DarrenS@The_DarrenS·
#Question for any #artist out in X. I am going to start learning to draw, #humans So where is the place to start when learning to #draw them? In my 50s so I hope this old dog can learn new tricks. 🙂
English
0
0
1
31
The_DarrenS retweetledi
The_DarrenS
The_DarrenS@The_DarrenS·
Redoing my room. Q: To LED or not to LED?
English
0
0
0
8
The_DarrenS retweetledi
Social Security Scotland
Social Security Scotland@SocSecScot·
Carer Support Payment has replaced Carer's Allowance for people in Scotland. Help us spread the word about Carer Support Payment amongst eligible people with shareable social media content, flyers, posters and factsheets available 🔽 bit.ly/CarerSupportPa…
English
2
14
12
1.4K
The_DarrenS
The_DarrenS@The_DarrenS·
Sad news. Just lost my father to multiple brain tumors Monday. I will miss him very much.
English
1
0
0
11