Michael G

@Quicken your quicken license verification server is down. I would expect that adds part of paying the license fee, I can actually use the software

@elonmusk theverge.com/2023/2/14/2360…

Please stay tuned while we make adjustments to the uh .… “algorithm”

Wow. Just wow. theverge.com/2023/2/14/2360…

@karpathy @benjamin_bolte XFormer and Flash custom kernels come standard with PT2, so you can use them off the shelf. Check out PyTorch nightlies! PT2 has both separate functions for each custom kernel, and generic _scaled_dot_product_attention to pick the best implementation for specific parameters

@benjamin_bolte yep great repo

having fun optimizing minGPT today - base: 495ms - zero_grad(set_to_none=True): 492 - torch.jit.script gelu: 463 - OMP_PROC_BIND=CLOSE: 453 - torch.backends.cuda.matmul.allow_tf32: 143 - torch.autocast(torch.bfloat16): 121 - FlashAttention: 102 now: more fused kernels more better

@soumithchintala @karpathy Using the new scaled dot product attention custom kernels that come with PT2 (your choice of vanilla, xFormer, Flash) is as easy as using a context manager: with torch.backends.cuss.sdp_kernel(…) See Google Colab example here => colab.research.google.com/drive/1L9xbagV…

@karpathy Once you publish the "optimized minGPT" repo, we'll probably send some more patches in. 1. with the latest nightlies, the xformers/flash-attention kernels are in PyTorch core now. 2. we have a matmul autotuner that is about to land that gives significant boost in perf

@JimiDevine Amazing progress for civilization. Dismantle slums and build student housing.

Listening to the horrible sound of chainsaws in People's Park. Watching trees drop. Sad day for Berkeley.

@POTUS @VP @PeteButtigieg greetings from California where TESLA, an American car maker you may not have heard about, has created an alternative to internal combustion engines. Check this out: teslarati.com/tesla-crushes-…