Guillaume Leclerc

Last week @Trevornoah asked @OpenAI @miramurati: How can we safeguard against AI-powered photo editing for misinformation? MIT students hacked a way to "immunize" photos against edits: gradientscience.org/photoguard/ @aleks_madry

Last week on @TheDailyShow, @Trevornoah asked @OpenAI @miramurati a (v. important) Q: how can we safeguard against AI-powered photo editing for misinformation? youtu.be/Ba_C-C6UwlI?t=… My @MIT students hacked a way to "immunize" photos against edits: gradientscience.org/photoguard/ (1/8)

Can we cast ML predictions as simple functions of individual training inputs? Yes! w/ @andrew_ilyas @smsampark @logan_engstrom @gpoleclerc, we introduce datamodels (arxiv.org/abs/2202.00622), a framework to study how data + algs -> predictions. Blog: gradientscience.org/datamodels-1/ (1/6)

@PreetumNakkiran @dustinvtran This was our goal when we open-sourced github.com/libffcv/ffcv-i…. It is the same length ans the pytorch imagenet example but up to 10x faster. We would love to hear feedback from the community to know if we can improve in any way.

@dustinvtran This was exactly my complaint -- I just want the pytorch imagenet example but faster. And bare minimal hacks -- no adversarial-mix-up-erasing-dropsmooth with 10 hyperparameters...

misc question but are there good benchmarks for training "hackable" models quickly? for research, "ImageNet in 1 hr" is useless to me if changing 2 lines makes it "NaN in 5 hrs"

@jefrankle @PreetumNakkiran (3) because FFCV does GPU augmentation and data movement in parallel it might still give you a little boost in other cases. It's always worth a try. Feel free to ask technical questions on our slack!

@jefrankle @PreetumNakkiran @jefrankle. From our experience, yes. (1) First case is when you are IO bottlenecked, with the appropriate parameters FFCV will dramatically improve the throughput you get from your storage. (2) FFCV makes it easy to move augmentation from/to the CPU to maximize speed. 1/2

@ArashVahdat - Allows declaring arguments where they need to be - Allows capturing the arguments where they are needed - Supports the definition of arguments through a combination of both config files (easy to checkout on git), and CLI for env dependent args github.com/GuillaumeLecle…

Machine learning Twitter: how do you pass a large list of arguments to your python training scripts? If you are happy with any other library please comment below.

@odbol @aleks_madry Just added experimental support for tensorflow. It's available here: github.com/libffcv/tf-ffcv

@aleks_madry Neat! Could this work with tensorflow as well?

ImageNet is the new CIFAR! My students made FFCV (ffcv.io), a drop-in data loading library for training models *fast* (e.g., ImageNet in half an hour on 1 GPU, CIFAR in half a minute). FFCV speeds up ~any existing training code (no training tricks needed) (1/3)

@crude2refined @aleks_madry Colab only runs python 3.7 which doesn't include `multiprocessing.shared_memory`. Therefore the earliest compatible version is 3.8 :/ As soon as Colab updates python we will have an example notebook!

@aleks_madry Colab link or it didn’t happen?

@odbol @aleks_madry Yes, it should be pretty straightforward. We are experimenting with that as we speak.

@yanndubs @aleks_madry @williamfalcon @PyTorchLightnin We have been using PTL with FFCV in our lab with success for quite a bit now. They are definitely complementary. The only caveat is that one has to override a few things from PTL. We will release a demo soon but feel free to join our slack it has been discussed there.

@aleks_madry Pytorch lightning is 10x slower 😳 @williamfalcon @PyTorchLightnin Anyways to get those speed ups in lightning? 😇

@code_star @giffmana We have been using FFCV internally on shared clusters with many different GPUs including V100s, 2080ti, 1080ti and it really helped a lot, especially since most of these clusters use network attached storage and don't have fast local storage and you share CPU with other users.

@giffmana I’m not sure how much it would speed up training on most department servers anyways (Without A100s). Assuming you have 2080tis, 3090s, or even v100s I don’t know that you get the specific speed up benefits they demonstrate.

And I'm sure we will *still* see "we cannot afford an ImageNet experiment" in papers for years to come. The 3200$ publication fee and 2000$ conference travel are no problemo though.

@Anshumali_ @aleks_madry We did try it (it was definitely better but still much slower than FFCV), but due to lack of good interop with PyTorch and the fact that webdataset is meant to fulfill the same function, we decided to stick with the latter for our thorough benchmarking.

@aleks_madry Did we try tfrecords? medium.com/mostly-ai/tens… They are much much faster than pytorch for dense datasets.

@code_star @michalwols @kevin_zakka @chriswolfvision @aleks_madry @soumithchintala Most likely not as much but from our experience it always gives significant benefits (+ will leave you with more CPU for other jobs and reduce power consumption!)

@gpoleclerc @michalwols @kevin_zakka @chriswolfvision @aleks_madry @soumithchintala So would we expect less speed up on older cards?

@jacobgorm @aleks_madry @schrep JPEG and RAW are just two example data types that FFCV can work with. It's really easy to add other Field Types. You can either keep it for yourself or submit a pull request! We would love to have WEBP support.

@RafailFridman @aleks_madry @ml_norms If it is sampled only once (i.e., getitem returns the same thing for the same index), FFCV can be used out of the box! Otherwise, you can (1) have getitem return the parameters of the distribution/do any needed pre-processing (2) use FFCV's fast data pipeline to do the sampling.

@aleks_madry @ml_norms Can you use it if getitem in the dataset samples from some distribution?

@PhongStormVN @aleks_madry While we haven't personally experienced with this one, FFCV was designed to accommodate virtually any dataset. In the case of COCO (segmentation map), one can easily store the segmentation map in an additional field. Feel free to join our slack if you need help getting started!

@aleks_madry Hi, does this library works on different dataset for different tasks. For example, coco for object detection?

@michalwols @kevin_zakka @chriswolfvision @aleks_madry @soumithchintala Indeed none of the ready-to-go PyTorch code we found was able to saturate 8A100s (that's part of why we wrote FFCV!). We also compared against the fastest speed/accuracy numbers we could find for PyTorch in the ResNet-50 8xA100 accuracy scatterplot.

@kevin_zakka @chriswolfvision @aleks_madry @soumithchintala They preprocess the dataset to a smaller size, cache all of it in ram, use progressive resizing, test time augmentation and tuned cyclical learning rate. I'm pretty sure the baselines they compare against are not optimized to saturate 8 A100s.