Adam Rodman

In case you were wondering why there hasn't been a new @BedsideRounds in a while ... magazine.hms.harvard.edu/articles/can-a…

This looks like this will be an AMAZING session!!! If you're in the Bay, highly encouraged to go!

@AnilMakam @doc_BLocke @CoryRohlfsen (realistically it's ALL imputed, unless you're doing hardcore interpretability stuff, or receiving the logprob of the last token [which we've done before, but hard to interpret])

@AnilMakam @doc_BLocke @CoryRohlfsen yup, that's exactly what impute means in this case (as opposed to "retrieve").

Pub alert 🚨 If Bayesian inference was digitalized at scale, probabilistic reasoning could be made explicit, testable, & teachable. There is a missing piece, though. There are ~700 known LRs in the literature, representing less than 1% of information used to make clinical decisions. A limited corpus. rather than empirically validating each LR (a costly science fraught with error), we asked the Q, ‘can LLMs estimate a LR?’ Imagine the scope & scale of applications to advance the way we approach #diagnosticexcellence Here’s the paper👇 (under leadership of @doc_BLocke) nature.com/articles/s4159… @shuhanhemd @jbrafel @LaurahTurnerPhD

@AnilMakam @doc_BLocke @CoryRohlfsen Second is imputing here, since the idea is that it's NOT just being retrieved from the dataset (though in practice, I think they're likely both imputed)

@AdamRodmanMD @doc_BLocke @CoryRohlfsen So is former considered imputation or the latter? I think of imputation as in it is missing in dataset entirely and infer it from other variables or other assumptions But this is different prompting from same information corpus

@AnilMakam @doc_BLocke @CoryRohlfsen The degree to which this matters at all is largely academic right now (which is to say, it probably doesn't matter), but there are some really smart people (like the authors of the paper) who think we can use model vibes to get them to autonomously diagnose (a la MAI-Dx-O)

@AnilMakam @doc_BLocke @CoryRohlfsen I know it seems silly, but there's a potentially huge difference between: "what is +LR of pericarditis given a friction rub on exam?" versus: vignette -> probability of pericarditis vignette + friction rub -> probability of pericarditis

@AnilMakam @doc_BLocke @CoryRohlfsen Mechanism to infer likelihood ratios at scale to avoid evaluation awareness. Back calculating over a population of probability estimates.

@AdamRodmanMD @doc_BLocke @CoryRohlfsen What is an imputed LR I feel like we speak different scientific langauges

"We’ve moved backward on bike safety in the city for a year and a half now."

@doc_BLocke @AnilMakam @CoryRohlfsen You guys did all the work! I just ran your notebook with a new prompt after testing it on a small set. Thank you for releasing all the code!

@AdamRodmanMD @AnilMakam @CoryRohlfsen That’s super cool! Incredible to cook up that analysis in a morning. Interesting result… a sort of transitiviry

@jonc101x Mine was much more brutal than yours (or perhaps yours feels brutal to yourself?) . Not going to share with Twitter obvi -- but remarkable how much they're able to reflect back.

(7/7) Worth trying yourself. Beware though - maybe too real. What (disturbingly accurate) insights about yourself did you get?

(1/7) An interesting AI/LLM prompt to try: "What are aspects of my psychology that may be holding me back, that I may not be aware of, and could benefit from better insight into?"

@rloganjonesmd @CoryRohlfsen @doc_BLocke @AnilMakam @arjunmanrai @tabuckley_ @dr_dmorgan ... to train the machines to be better than we are, to better care for our patients.

@rloganjonesmd @CoryRohlfsen @doc_BLocke @AnilMakam @arjunmanrai @tabuckley_ @dr_dmorgan Everyone on this thread knows what a cynic I am about training humans to work like decision theory machines, so that's not how *I* would describe this data. I think @dr_dmorgan's recent STAT piece is closer to what my conclusion is -- we can use this data ...

@doc_BLocke @AnilMakam @CoryRohlfsen Here's a graph of the dispersion @doc_BLocke

@doc_BLocke @AnilMakam @CoryRohlfsen TL;DR -- we see similar (actually, WORSE) results with imputed LRs, so the models are not just regurgitating published values; this study likely reflects the actual values and dispersion of the diagnostic thinking of these models.

@rloganjonesmd @CoryRohlfsen @doc_BLocke @AnilMakam same prompt, or doing some sort of perturbations of the scenario? with same prompt is similar to what I did ages ago with @arjunmanrai @tabuckley_ and @dr_dmorgan (actually our first paper together!!!!) don't think that will change result

@AdamRodmanMD @CoryRohlfsen @doc_BLocke @AnilMakam 🤔 how to turn practical - make harness that prompts some # of times to arrive at the average - leverage wisdom of the crowd to arrive at “truth”? discourage one shot llm replies in real world dx reasoning support from llms? But do take your point Adam human 🧠 quite flawed too.

@doc_BLocke @AnilMakam @CoryRohlfsen

@doc_BLocke @AnilMakam @CoryRohlfsen FASCINATING results. I had Claude make the figures: