I’m excited to finally showcase our work on developing a generalizable deep learning system for cardiac MRI. This is the culmination of over 3 years of research spanning my time as a postdoc in @HiesingerLab and now as a cardiac surgery resident at @pennsurgery, and I couldn’t be happier with the finished product. Weights to be released for academic use (coming soon!). All built with @pytorch and @pytorch_lightning. Lots of really cool stuff to unpack, but here are few main results: Contrastive learning with cine-sequence CMR studies from multiple different views (short axis, 2-chamber, 3-chamber, and 4-chamber views) with long-form CMR free text reports is really hard. A good sign for us was how the video transformer embeddings evolved with each epoch. We train for about 600 epochs taking about 2 weeks. In a zero shot fashion, the model can then do some really interesting things: separate out different diseases (ACDC dataset), and different genders, age groups, and ejection fractions (UK BioBank) with no explicit supervised instruction. No such behaviour with kinetics-600 initialized weights (see preprint)! Finetuning over our system yields results superior to baseline approaches on downstream tasks of interest, with 10x and sometimes 100x less data. Here’s a quick figure showing how we achieve superior results for UK BioBank LVEF estimation with just 1% of the available data. Better yet, that same model when tested on a Kaggle dataset does exceptionally well (dashed line = kinetics-600 performance on left; lower is better!). Focusing on some disease diagnosis tasks, we label a separate dataset of some 6000 individual patients. This is real-world representative unenriched dataset. Our methods show massive improvements in AUC compared to baseline methods. Validation set shown here, internal test set (Stanford / Medstar / UCSF), and external test set (UPenn) figures are in the paper.
