OpenEvidence

Rigorous evaluation of medical AI is good for everyone, and we welcome it. Counter to a half-dozen independent studies from institutions such as the Mayo Clinic that were highly positive on OpenEvidence—a lone paper now purports to show that generalized AI beats specialized clinical AI (@UpToDate, @EvidenceOpen). The paper has a massive undisclosed conflict of interest and irredeemable methodological flaws. Behind the scenes: The study authors run a competing in-house medical AI at their hospital, and asked OpenEvidence for an API to power it — including rights to build a "competing product" with OpenEvidence's own API. OpenEvidence declined. Then, this paper coincidentally appeared. Point-by-point, looking closely at the datasets used in the study, the disingenuous and fatal flaws become immediately apparent 🧵.

Last summer an ER doctor at Mount Sinai watched two med students and a resident pull up OpenEvidence mid-shift on a hard case. He assumed they were unusual. Then the health system found that a third of its 9,000 physicians were already using it. That's how OpenEvidence spread. Doctors found it, tried it on real cases, and told the doctors next to them. The hospitals are now formalizing what their clinicians already do. @SteveLohr piece in today’s New York Times follows that thread out to a small hospital in Alaska, where Dr. Barbara Creighton uses it on complex cases she’d otherwise send to a paid specialist consult. She calls it “like having a bunch of specialists in your pocket.” The hype around medical AI usually skips the part doctors care about most, which is what the tool is for. Daniel Nadler summed it up in one line: “It’s not an oracle, it’s a tool. Knowledge and knowledge workers still matter.”