Dry

Using an hologram fan to visualize 360° ancient relics. twitter.com/i/status/17806…

Remember Devin? Apparently demo's fake. Paul is sad. 😥

A few new CUDA hacker friends joined the effort and now llm.c is only 2X slower than PyTorch (fp32, forward pass) compared to 4 days ago, when it was at 4.2X slower 📈 The biggest improvements were: - turn on TF32 (NVIDIA TensorFLoat-32) instead of FP32 for matmuls. This is a new mathmode in GPUs starting with Ampere+. This is a very nice, ~free optimization that sacrifices a little bit of precision for a large increase in performance, by running the matmuls on tensor cores, while chopping off the mantissa to only 10 bits (the least significant 19 bits of the float get lost). So the inputs, outputs and internal accumulates remain in fp32, but the multiplies are lower precision. Equivalent to PyTorch `torch.set_float32_matmul_precision('high')` - call cuBLASLt API instead of cuBLAS for the sGEMM (fp32 matrix multiply), as this allows you to also fuse the bias into the matmul and deletes the need for a separate add_bias kernel, which caused a silly round trip to global memory for one addition. - a more efficient attention kernel that uses 1) cooperative_groups reductions that look much cleaner and I only just learned about (they are not covered by the CUDA PMP book...), 2) the online softmax algorithm used in flash attention, 3) fused attention scaling factor multiply, 4) "built in" autoregressive mask bounds. (big thanks to ademeure, ngc92, lancerts on GitHub for writing / helping with these kernels!) Finally, ChatGPT created this amazing chart to illustrate our progress. 4 days ago we were 4.6X slower, today we are 2X slower. So we are going to beat PyTorch imminently 😂 Now (personally) going to focus on the backward pass, so we have the full training loop in CUDA.

Seems like a busy game of hungry hungry hippo happened this weekend

Human evolution process

Le 8 Mai 1902, la Ville de Saint-Pierre, ancienne capitale de la Martinique, disparaissait suite à une éruption Volcanique 🌋🙏🏾

#Aérien : @CorsairFr s’est associé au chef guadeloupéen Jimmy Bibrac pour ses repas en business au départ des #Antilles 🍽️🛫Une façon de mettre en avant "la #gastronomie créole" et le "terroir antillais" ➡️bit.ly/3AJ5neO

JUST IN: #Bitcoin whitepaper is hidden on every Apple MacBook computer running recent versions of macOS software.

Old joke about agnostic technologists building artificial super intelligence to find out if there’s a God. They finally finish & ask the question. AI replies: “There is now, mfs!!”

There are two rules in life: 1) Never give out all the information.

L'exercice international de préparation aux tsunamis #CaribeWave2023 se déroule aujourd'hui dans toute la Caraïbe à partir de 10h, avec comme scénario, l'effondrement d'un pan de la Montagne Pelée causant une vague destructrice. Mais savez-vous quoi faire en cas de tsunami ?

🇫🇷 FLASH - "Si les Français étaient vraiment en colère, je n'aurais pas été réélu il y a un an", affirme Emmanuel #Macron. (France 2) #Macron13h #reformedesretraites

Are you?

“I used to be in crypto, but now I got interested in AI"

There was no VAR check for that penalty call. Wow

Never seen them give one man so many penalties in one competition, this shit is rigged

poktcomet.com:49 c0d3r.org:49 benvan.com:48 zp-labs.net:44 2jx.com:43 blockspaces.io:42 nodeinfra.io:33 liquify.info:32 nachoracks.com:19 trgc-pokt.com:10

Top nodes runners 24h blackholestake.io:79 poktscan.net:74 sendnodes.org:74 🚀 nodefleet.org:72 easy2stake.com:66 2jx2jx.com:52 benvanman.com:50 benvannode.com:50 thunderstake.io:49👇

👇 benvannode.com:99 2jx2jx.com:97 poktcomet.com:91 zp-labs.net:90 2jx.com:88 blockspaces.io:76 nodeinfra.io:65 liquify.info:59 nachoracks.com:38 trgc-pokt.com:17

Top nodes runners 48h sendnodes.org:157 🚀 poktscan.net:148 blackholestake.io:148 nodefleet.org:146 easy2stake.com:141 benvanman.com:102 benvan.com:102 c0d3r.org:101 thunderstake.io:100