Bibb Allen

Tools for evaluating and comparing AI models will be part of the ACR AILAB ailab.acr.org #ACRDSI #ACRInformatics @RadiologyACR #RadAIchat#RadAIchat

T5 - not only need agreement with radiologists but also exam metadata (equipment, protocol, patient info) so we can understand where models fail. #RadAIchat

FDA recognizes need for adaptive AI, but challenged to regulate so models stay locked. #RadAIchat

@US_FDA Roadmap for AI regulation focuses on: 1) Good machine learning practices (structured use cases) 2) Premarket assurance of safety and effectiveness (validation) 3) Real-world performance monitoring (registry reporting) #RadAIchat #ACRDSI #ACRInformatics #RadAIchat

We can centralize data sets to be used for validation similar to a multicenter clinical trial but lots of challenges – HIPPA, data use agreements, etc. Or, how cool would it be if we could send the models to multiple sites and let the patient data stay on premises? #RadAIchat

ACR DSI is also working toward centralized data sets with NIH, RSNA, AAPM and the University of Chicago through the Medical Imaging and Data Resource Center (MIDRIC) project. midrc.org #RadAIchat

Curt would say AI + humans = best result. #RadAIchat

Of course, AI does not have to be perfect to be useful. See article in Lancet Digital Health. thelancet.com/journals/landi… Generalizability is important but even models that fail occasionally can assist radiologists provide better care for our patients. #RadAIchat

Check out Casey Ross's Stat+ article @caseymross: “The result is a dearth of information on whether AI products will ...trigger unintended consequences, such as an increase in incorrect diagnoses, unnecessary treatment, or an exacerbation of racial disparities.” #RadAIchat

Lay press also noticed concerns our safety and bias in FDA cleared algorithms. #RadAIchat

Woojin also said --- You wouldn't buy a pair of shows without trying them on first, would you? You should try AI models at your institution before purchasing to make sure they work in you setting #RadAIchat

At last year's ACR DSI Data Science Summit, Woojin Kim (@woorinrad) advocated for using data sets enriched with abnormal and challenging cases when evaluating AI models – especially for disease processes that have low prevalence #RadAIchat #RadAIchat

In radiology, brittleness may be caused by variations in equipment, imaging protocols & heterogeneous patient populations. Models may understand output from some but not all devices. Need multisite validation prior to deployment in clinical workflow. #RadAIchat #ACRInformatics

AI models are brittle which means they may not perform as expected outside of the environment where they were trained. This is true for all industries – not just healthcare. Algorithms may work beautifully and then fail at unexpected moments. #RadAIchat #ACRDSI #ACRInformatics

A1 AI models are brittle which means they may not perform as expected outside of the environment where they were trained. This is true for all industries – not just healthcare. Algorithms may work beautifully and then fail at unexpected moments. #RadAIchat #ACRDSI #ACRInformatics

Hello Everyone. I am Bibb Allen. I am a community practice radiologist in Birmingham, Alabama at Grandview Medical Center. I am the Chief Medical Officer for the @RadiologyACR Data Science Institute. #RadAIchat #ACRDSI #ACRInformatics #RadAIchat

Do@m

@RADPAC congrats to Ted Burnes our whole ACR RADPAC GR team and RADPAC chair Jim Brink for another great year

@BubbAllenMD so how bout following @alabamarad

Glad to be following our state chapter @AlabamaRad