@DrDiGiorgio @EvidenceOpen This sort of AI is built into Epic at our institution now. It makes hospital courses, daily insights, chart reviews, give specific citations in the chart and links them to specific notes.
English
Austin Meyer
3K posts
Austin Meyer
@austingmeyer
w/ @b_ritt23 • MD/PhD/MS/MPH/MS • data scientist • virus modeler • IM/Peds Hospitalist @bswhealth • Assistant Prof @bcmhouston • I 💛 infections, jazz, data, ☕️
Austin, TX Katılım Mart 2009
1K Takip Edilen580 Takipçiler
This could actually be great.
Right now we pay people to comb through a very fragmented medical record and piece together a cohesive picture. They often fail.
This could change all that. I’m cautiously optimistic.
OpenEvidence@EvidenceOpen
Until now, physicians using AI in clinic had to assemble the patient’s context themselves. Allergies, comorbidities, medications, prior procedures, copy-pasted in from the chart. Today we’re announcing a partnership with @CedarsSinai. OpenEvidence now works directly inside Epic, drawing on the patient’s full record and interpreting the medical literature through the lens of that specific patient. Cedars-Sinai is the first academic health system to deploy patient-aware clinical intelligence at enterprise scale. The clinician asks a complex question in natural language. The answer reflects both the best available evidence and the patient in front of them. Patient data is never stored after the clinical session or used for any other purpose.
English
Technology to verify citations is easy and numerous tools already exist. They just have to be implemented. It would be trivial to have one or several AI models do initial screening for citations (both meta-data and appropriateness) and whatever else was high yield. Not to mention, we could just require doi’s for every citation and have an editorial assistant click on them. That would take 2 minutes.
English
I don't have a clean answer. Science will be AI-touched at every level. AI-assisted and AI-generated are not the same thing. What does the first actually look like in practice, and who should be building it? thelancet.com/journals/lance…
English
One of my NIH grant reviews came back last year with comments clearly LLM-generated. The PMIDs cited against the proposal were hallucinated. The papers did not exist. The score still counted toward the funding decision.
English
See the pre-print of our new article. Quantifying the Protection Gap: RSV Activity Outside the Recommended RSV Prophylaxis Administration Window academic.oup.com/jpids/article-…
English
One additional thing to keep in mind, is the RLHF that general models undergo for alignment is likely to make real time unintentional biasing/steering a significant problem for non-crystallized case presentations. There would probably need to be some specific alignment tuning to this use case.
English
This is a nice thread. I feel like a lot of non-physicians do not understand the caveats here. I might go farther and argue that I'm not sure it is actually ready for prospective trials except perhaps in a very limited sense.
Ultimately, the training data includes lots of heavily curated case presentations from the published literature and essentially no real world presentations with confusing and disorganized thoughts/data.
The biggest sticking point is likely to be that very large models should be able to memorize or at least easily pattern match test data that was very similar to the curated content in the training data, but whether they are useful at all on a stream of consciousness presentation (whether presented by physician or a patient) is unclear. I'm not sure it is even clear that a model can consistently take in the information from a patient and present it as a completely hallucination free curated case.
English
🧵1/ Our new study on AI and physician reasoning just came out in @ScienceMagazine. As co-senior author, I'm excited about our findings, and I do think AI will reshape medicine. But after seeing some of the discussions, I'm also worried about how our findings may be misinterpreted.
English
Gemini 3.1 gets citations wrong frequently. Even if it doesn’t hallucinate the title of the article, it hallucinates the other metadata of the citation or inserts the citation for something that the citation doesn’t support. The GPT and Claude models are, admittedly, much less likely to hallucinate. Given the highly variable expertise of authors, it might make sense to still have them note what models were used.
I agree that AI review should be at least allowed (many journals strictly prohibit it). It would be particularly useful in medicine where most authors are subject matter experts and relatively poor with methods. Though it is worth mentioning that AI reviews often bring up lots of relatively meaningless pedantic (non-)issues. So reviewers with subject expertise are still needed to strictly prioritize or eliminate concerns raised by AI. Models also tend to lack nuanced subject matter intuition. Since most academic papers are being written by subject matter experts, the main value of AI reviews is probably in methods verification and replication rather than discussion of implications.
English
There is also too much focus on two issues in discussions of paper reviewing: hallucinations & privacy.
Hallucinations are not gone, but the latest models rarely hallucinate sources (& it is relatively easy to make the human responsible)
And you can get IP compliance easily now
English
I think that academia has not absorbed the fact that AI agents are now good enough to independently reconstruct complex papers without access to code or the papers themselves; just the methods & data.
They aren’t perfect but the errors are often in the human paper, not the AI.
English
I agree that it is definitely a good thing for things to be cheaper. It’s just that the thread was about saving lives. I worry about the narrative that extensive imaging saves lives because it can be surprisingly insidious and dangerous. Especially coming from important and influential people.
English
@austingmeyer @Dan_Jeffries1 I wasn't advocating the indiscriminate use of full-body MRIs, but the fact that the technology that makes this possible reduces the time and cost of an MRI exam is intrinsically a Good Thing.
English
More people will die from suppressing AI than from the imaginary AI apocalypse.
They'll die from restricting safe self-driving cars that are 90% better drivers than people who kill 1.5 million people and injure 50 million more every year.
They'll die from the vaccines and cures that never get created.
They'll die from all the myriad of helpful inventions that never get created by geniuses in a datacenter.
They'll die from preventable diseases that they could have asked their chat bots about so they were better informed when they went to see their doctors but who couldn't ask because short-sighted legislators made it so the chat bots had to refuse to answer.
They'll die from the slower economy that stifles robot driven factories over wildly overblown jobs apocalypse fears which will mean we never get a vast array of new and more affordable goods.
They'll die from the cheaper solar panels and batteries that would get made by those automated factories which would slow climate damage and provide cheap energy to undeserved areas.
They'll die from the super smart tele-AI doctors that never get deployed to remote areas.
And they'll die as fanatics from the stop AI movement radicalize their followers to shoot people or throw firebombs.
Max Tegmark@tegmark
Senator @BernieSanders has invited me and three other AI researchers to a public panel on AI existential risk & international cooperation at the U.S. Capitol 7pm Wednesday April 29th. RSVP here to join us for this important conversation: forms.office.com/Pages/Response…
English
Isn’t this a bit too strong? GRADE clearly has issues, but saying that only prospective controlled trials should count as evidence seems like a kind of epistemic fundamentalism. There has to be a path for accepting other types of evidence when prospective controlled studies are infeasible, unethical, or not yet complete. There are lots of decisions physicians have to make with limited evidence and there needs to be a pathway for physicians practicing away from major centers to know what most of their colleagues would do in a certain situation. Do we just have a different outlet for that information?
English
@BoussageonR @LGHemkens @ABsteward That’s not the point at all. Don’t over complicate. This is very simple. If something is proven true with reproducible prospective controlled studies, make a recommendation. If not, don’t. It’s really that simple.
English
Appreciated the thoughtful response from IDSA. It reflects shared concern that strong or highly directive statements should be used cautiously when evidence is weak, and a convergence toward aligning guidance with evidence strength and improving transparency in uncertainty, while continuing GRADE alongside structured clinical reviews. WikiGuidelines is acknowledged as having complementary strengths in this space. Continued dialogue and future collaboration remain open, with a shared focus on patient and clinical needs. #IDXposts
Antibiotic Steward Bassam Ghanem 🅱️C🆔🅿️🌟@ABsteward
@BradSpellberg @WikiGuidelines @DrToddLee @AnilMakam @medrants 💥Update: Our LTE has prompted a response from IDSA. "Rethinking How We Provide Guidance When Evidence is Limited" Looking forward to where this discussion goes next #IDXposts academic.oup.com/cid/advance-ar…
English
To be clear, there is no consistent evidence that asymptomatic/average risk people have improved outcomes from full body MRI, regardless of cost or time. It seems intuitive, but there are a lot of relatively risky invasive tests that come after an incidental finding. In many cases, it would be better not to find them while randomly imaging asymptomatic people.
English
Actually, AI already saves lives.
In several countries, mammograms are examined by AI and radiologists. Reliability is improved.
In the EU, every car sold must be equipped with Automatic Emergency Braking Systems. That's AI. They reduce frontal collisions by 40%.
Modern MRI machines are equipped with AI technology that reduces the time of imaging by 4x or more. You can now get a full-body MRI in 40 minutes for about $1000. Reduced time -> reduced cost -> more/earlier detection.
And that's not counting the progress in medicine enabled by modern AI, including Nobel Prize-winning protein structure prediction.
English
Does it really explain a 60% drop though? It just seems quite unlikely to have such an obvious alternative explanation coincidentally occur, and honestly looking at fairly imperfect plots feels like reading tea leaves. There is also a national decline of something like 10% from peak, but one would expect programs near Silicon Valley to be more sensitive to this.
English
@austingmeyer @ylecun What happened is the university itself made an intentional decision to shrink the CS major, starting with the entering freshmen in Fall 2023. See x.com/minilek/status…
Jelani Nelson@minilek
English
Many folks seem to be confused, and think the collapse of the CS major graduation numbers at Berkeley could be linked to the "AI is taking SWE jobs" hysteria narrative. Here's the easiest way to see that this is false: the timeline doesn't fit.
The graduating class in 2027 (first small CS cohort graduating) has students who arrived on campus as freshmen in Fall 2023, with freshman admission targets set (i.e. shrunk) by the university in Fall 2022. So, the hysteria narrative obviously doesn't match the timeline; ChatGPT didn't even come out until November 2022.
Now consider the plot below; orange curve is what % of bachelor's degrees are CS degrees each year at Berkeley, and blue curve is what % of applicants applied to be a part of that graduating year, intending to be a CS major in their application (combining both junior transfer and freshman applicants). In other words:
* orange measures CS graduate production
* blue measures CS demand (via % of all applications to the university)
What do you notice? The collapse in orange (CS grads) isn't because of a collapse in blue (demand). In fact it's the opposite: orange collapsed at a time when blue was going up. 1/
English
It may also be that a portion of eventual CS grads were transfers in and those have complete stopped? Or that there was some incentive to complete dual majors with another science or engineering and those have stopped? It just seems like the other models to explain this massive drop are considerably less powerful.
English
I don’t understand why this can’t be the AI narrative. Most students have the flexibility to switch majors through the first two years of their program. This plots suggests exactly that. Per the chart, the incoming students could be naive and making decisions based on parent preferences. Once they get to campus, they have two years to switch majors before they get locked in. They hear about job market concerns and switch before year three. That would be a perfect timeline given the release of GPT-4 in May 2023.
English
Our Method/ology paper. An Introduction to Machine Learning for the Pediatric Hospitalist publications.aap.org/hospitalpediat…
English
Most importantly and probably the biggest bottleneck, is they need to stop testing against benchmarks that consist of cases that have been well curated and summarized by medical professionals. If people want to see how well these models perform, they need to allow random patients to input their own queries and allow the models to ask their own questions in response. The cognitive disorganization of real patients without a doctor filtering it for the models is probably a significant limitation in training.
English
We only have spotty information about this very important topic. It suggests AI can be good at diagnosis, but the real world doesn't always match the experiments. x.com/emollick/statu…
Ethan Mollick@emollick
Across most medical benchmarks, including when real cases & human doctors are involved, there is a clear trend of AI models improving over time (and many where today's AI beats human doctors) But we do not have many studies measuring real-world performance of AI in medicine, yet
English
This paper shows people are asking a lot of medical questions of AI already, but we have little evidence of how good or bad this is.
Most of the published research uses old models & compares to doctors. How do new models compare to the info people would have gotten without AI?
English
@mcuban The level of hubris in the tech community is remarkable.
English
You aren’t wrong Patrick. But the challenge is that few supplements have actually been tested to assure the ingredients on the label match what is actually in the supplements you ingest.
Be careful
Patrick Collison@patrickc
I'm lucky enough to have a great doctor and access to excellent Bay Area medical care. I've taken lots of standard screening tests over the years and have tried lots of "health tech" devices and tools. With all this said, by far the most useful preventative medical advice that I've ever received has come from unleashing coding agents on my genome, having them investigate my specific mutations, and having them recommend specific follow-on tests and treatments. Population averages are population averages, but we ourselves are not averages. For example, it turns out that I probably have a 30x(!) higher-than-average predisposition to melanoma. Fortunately, there are both specific supplements that help counteract the particular mutations I have, and of course I can significantly dial up my screening frequency. So, this is very useful to know. I don't know exactly how much the analysis cost, but probably less than $100. Sequencing my genome cost a few hundred dollars. (One often sees papers and articles claiming that models aren't very good at medical reasoning. These analyses are usually based on employing several-year-old models, which is a kind of ludicrous malpractice. It is true that you still have to carefully monitor the agents' reasoning, and they do on occasion jump to conclusions or skip steps, requiring some nudging and re-steering. But, overall, they are almost literally infinitely better for this kind of work than what one can otherwise obtain today.) There are still lots of questions about how this will diffuse and get adopted, but it seems very clear that medical practice is about to improve enormously. Exciting times!
English
We submitted a paper to a prominent Pediatrics journal that found, "... our analyses suggest that a fixed window would need to be extended to be both a month earlier and a month later (September–April) to more definitively capture the entire RSV season." It was rejected 6 weeks ago (and is under review at a different journal) because the reviewers felt like an additional season of data was required to be sure. Here is the additional season from @AAPNews. publications.aap.org/aapnews/news/3…
English
@BradSpellberg Not to mention there are so many health systems pushing to remove procal from laboratory availability, which seems crazy to me.
English
I'm not sure one has to require a virus be present for non-severe, right? Just that bacteria not be present. There are so many fully negative panels and most standard sputum arrays that I've seen don't have any typical bacteria on the array. To be safe, one could define a set of trials sequentially reducing duration with a small set of antibiotic options. Then, test non-inferior reduction from 5 days to 3 days. Then, 3 days to 1 day. Then, 1 day to 0 zero days.
English
Several important points here.
First, historical data on abx efficacy for PNA is not relevant to Q you are asking bc when the historical data were generated, there were no vaccines for pneumococcus or H flu. So they accounted for the vast majority of PNA (particularly S. pneumo).
Austin Meyer@austingmeyer
@BradSpellberg @ABsteward @DrToddLee Has there ever been a randomized placebo controlled (with no antibiotic in the control group) trial for outpatient pneumonia treatment in a developed country?
English
Yes. It just seems like all of the evidence in inapplicable to the most common CAP prescribing context. As you noted, I can't find any actual evidence that we should be giving empiric antibiotics in the outpatient setting for microbiologically typical CAP coverage based on clinical or radiographic evidence for CAP in developed countries (those with high pneumococcal vaccination rates)? As far as I have been able to find, virtually every trial is a non-inferiority trial based on prior standard dating back to penicillin use in case series of severe pna in the inpatient setting prior to pneumoccocal vaccination and not in mild or moderate disease. Almost all non-inferiority trials show that ever increasingly short courses of antibiotics are non-inferior. It seems like we should really go back and study this in the right (modern) context.
English
The article I posted has detailed review of historical data of abx effectiveness. There were alternation studies of abx vs. no, studies of serum therapy, cohort studies of abx vs. no, etc, all of which demonstrated a MASSIVE reduction in death d/t Abx. The references are there.
English
Thanks. Are there any for typical pneumonia also? I believe all of those were atypicals. I'm just having trouble finding any for typical CAP in the outpatient setting (where there likely wouldn't be microbiologic confirmation). I'm mostly wondering because most of these radiographic pneumonias are viruses or atypicals.
English
@austingmeyer @ABsteward @DrToddLee Apologies I gave you the link to the wrong article before. Please disregard the 2010 position paper. I was doing it on my phone. The 2008 workshop summary is the correct paper here (refs 16-20):
pmc.ncbi.nlm.nih.gov/articles/PMC28…
English
Time to revisit empiric atypical coverage in non-severe CAP?
🆕⚡🔴Target trial emulation using a propensity-weighted cohort from 68 hospitals in Michigan >66,000 patients:
• No difference in time to clinical stability
• No difference in ICU transfer or antibiotic duration
• No difference in 30-day mortality
• But ↓ composite 30-day mortality/rehospitalization with azithromycin
Signal… or bias?” academic.oup.com/cid/article-ab…
English