David Page

Ever wanted to train CIFAR10 to 94% in 26 SECONDS on a single-GPU?! In the final post of our ResNet series, we open a bag of tricks and drive training time ever closer to zero... Colab: colab.research.google.com/github/davidcp… Blog: myrtle.ai/how-to-train-y…

@badlogicgames My git repo 🙈 github.com/davidcpage/ski…

Anytime someone posts "skill issues"/"ngmi", ask them to show you what they build, preferably with a link to a git repo.

@jeremyphoward @hi_tysam @kellerjordan0 @karpathy Impressive, well done! Hi Jeremy! Still lurk here occasionally and hopefully start blogging again soon..

@hi_tysam @kellerjordan0 @dcpage3 @karpathy This is awesome. Love it :D I hope David sees it too, although I haven't seen him on Twitter for a long time.

New training speed record for CIFAR-10: 94% accuracy in 3.29 seconds on a single GPU Paper: arxiv.org/abs/2404.00498 Code: github.com/KellerJordan/c…

@samgd @jeremyphoward @nanopore Yep not me this time! Looks like fascinating work from Giuseppe though

I'm a bit late to the party, but I just noticed that @dcpage3 has been writing an awesome series on Vision Transformers over the last few years: myrtle.ai/learn/leo-1-lo…

@JoaquinAlori Thank you!

@dcpage3 Links seems to be broken, here is a working one myrtle.ai/learn/how-to-t… Thanks for the great thread by the way!

The paper that introduced Batch Norm arxiv.org/abs/1502.03167 combines clear intuition with compelling experiments (14x speedup on ImageNet!!) So why has 'internal covariate shift' remained controversial to this day? Thread 👇

Excited to announce our new work, a unified theory towards explaining 3 black magics in deep learning: (1) ensemble, (2) knowledge distillation, and (3) self-distillation. An accessible blog post is below.

@bozavlado @iiSeymour CTC_CRF extends flipflop to output scores for multiple (six) consecutive bases not just two. Output layer is mostly orthogonal to choice of RNN/CNN encoder so CNN improvements are very welcome! More details coming soon..

@iiSeymour Oh the pair decoding! BTW, is there any description what CTC_CRF does? Is it flipflop? Or something even completelly different?

Well, it seems that our bag of improvement for bonito is sort of useless, since they switched back from CNNs to RNNs :)

Big accuracy update coming in the next version of Bonito 🚀 v0.3.0 combines everything we have learned with structured and unstructured approaches - @dcpage3, Tim and myself are working hard on the finished touches this week - watch this space 👀

@TMVector @nanopore Thanks Jonny, that’s very kind!

@dcpage3 @nanopore Congratulations and all the best in your new role! I hope they know how fantastic a hire they've made 😎

First day of new job @nanopore where I get to apply ML to a bunch of fun science and engineering problems. Pretty excited!

@achacond @nanopore Thanks, looking forward to it!

@dcpage3 @nanopore Congrats for the position! Enjoy the new Journey!

@Sisseljuul @nanopore Thank you!

@dcpage3 @nanopore Welcome!

Bonito v0.2.2 - SAM output - Sequence and alignment tsv summaries - Grab bag of training improvements from @dcpage3 github.com/nanoporetech/b…

Preparing a short course on neural nets can be fun. Below is one of the fast Resnets by @dcpage3 on CIFAR10. Would have been nice to track a UMAP-like representation of some internal layer, but have not found a reasonably fast/stable way to do so. Any idea? @NikolayOskolkov

Undertraining a large model is a good way to speed things up on toy problems myrtle.ai/how-to-train-y… but it was far from clear this should extend to large scale.

Great study of training efficiency at large scale + nice results on compression for inference!

Simple setup + attention to details -> sota self-supervised reps! LARS -> large batches -> no need for memory bank of -ve examples Random crops + color aug (to prevent hist cheating) -> no need for special arch Projn head for contrastive loss -> hidden reps preserve info

Blitzing fast CTC decoding github.com/nanoporetech/f…

@Buntworthy It's fixed now. Thanks for letting me know!

@dcpage3 FYI one of the images in Part 2 of "how to train your resnet" is broken. myrtle.ai/how-to-train-y…

@ylecun @viglovikov @timetravellertt @kaggle The problem though with "you can always add those tricks to get the numbers up" is that *very* often I see papers that don't do data aug, or don't tune hyper-params, etc, then claim their new idea helps. But then I find it's actually just a poor proxy for the things they skipped

@ID_AA_Carmack If you're interested in training small accurate nets efficiently, then the best in the world is @dcpage3, and he told all his secrets in this amazing series: myrtle.ai/how-to-train-y…