Sabitlenmiş Tweet
Sam Finlayson
10K posts
Sam Finlayson
@IAmSamFin
Clinical Assist. Prof of Pediatric Genetics @seattlechildren SVP Medical AI @EvidenceOpen Past: MD-PhD @harvardmed, @MIT_CSAIL, @Stanford
Seattle, WA Katılım Ekim 2014
705 Takip Edilen5.4K Takipçiler
always interesting and jarring to see normal people’s idea of what a dumb question is. “why would the earth form into a sphere?” and “what prevents penguins from flying, and why would that be an evolutionary advantage?” are actually high level. can the average adult answer these?
Rock Solid@ShitpostRock
What do you do in this scenario?
English
A minor technical point: EvidenceGrade scores the evidence retrieved and used to answer the query. This is related to — but is still distinct from — the task of grading the full body of evidence that exists and is relevant to the answer. So there’s a subtle difference here from the scope and therefore claims of, say, a full Cochrane review.
With that in mind, happy to discuss the methods in more detail if questions come up.
(And to the other commenter’s point: no, there is absolutely no role of pharma in influencing the grades or the retriever or answer writer, nor is there thumb on the scale for specific journals).
English
Unsurprised to see Endocrinology up there with Pharmacology in evidence grade. Surprised to see Transplantation all the way down at the bottom. Have not gone through the methodological details @EvidenceOpen but _if_ robust, it's the kind of ConsumerReports™ for medicine needed.
English
This feature was designed and built with @TravisZack2 , @lehmer16 , @evanqed and other incredible engineers on the OpenEvidence team. Such a blast to work with such thoughtful and productive people.
English
See the blog post for some more extended musings. I’m sure we’ll be getting a lot of feedback on this and I look forward to improving it over time if the community appears to find it worth keeping around
openevidence.com/blog/introduci…
English
One of my white whales at @EvidenceOpen is for us to do a better job conveying the level (un)certainty inherent in the set of refs we’ve retrieved for a given answer. A step in that direction is rolling out in product today!
Musings on the problem and how we are approached it here:
openevidence.com/blog/introduci…
English
@vintweeta @EvidenceOpen Thank you! More details in a blog post coming soon.
English
Kudos @IAmSamFin and @EvidenceOpen team!
"EvidenceGrade surfaces, quantifies, grades and visualizes the quality of the evidence behind OpenEvidence answers...
...The model is trained to weigh study design strength, consistency and precision across sources, and how directly the evidence applies to the question—mirroring how an expert methodologist would appraise a body of evidence"
fiercehealthcare.com/ai-and-machine…
English
Herculean effort by Jean Feng and colleagues to run this analysis so fast while still being so thoughtful and rigorous in their approach.
(COI notice: All authors on the paper are unpaid and unaffiliated with OE in any way; same does not apply to me).
Jean Feng@Jean_J_Feng
For the past two weeks, our independent team of statisticians, AI evaluation experts, clinical AI researchers, and clinicians was given a unique opportunity to test one question: “How well do different AI tools answer user questions on the OpenEvidence (OE) platform?”
English
@PierreEliasMD @bby_duno @batkatebush Depends on the angle haha. He has an insane level of strength and flexibility. (Also an autistic focus to rival an Anthropic employee.) Watch the silence video if you get a chance some time, it’s poetry in motion.
English
@IAmSamFin @bby_duno @batkatebush if i saw this dude with a hoodie on i wouldn't believe they let him out of the anthropic office in the last 6 months. so humbling how many amazing people are out there.
English
OK since we're doing men screaming discourse, one time I went to an unfamiliar climbing gym and this guy would release blood curdling screams every time he missed a move while top roping. It was very distracting and made my experience less pleasant. Later I saw a bunch of people greeting him and being friendly, so I figured he must be a regular. This made me wonder if he has tourette's or some kind of similar condition, because if the screams were voluntary I would expect him to be shunned by the community until he reformed his behavior. I will note that he was also not wearing a shirt and I am not aware of any disease that would produce that outcome.
Grace@singinghomeward
If my husband screamed, I would be dialing 911 while preparing myself for the sight of a lost limb. Screaming in frustration is something we're actively working on discouraging in our six-year-old. A grown man should never.
English
Adam Ondra, widely considered the best rock climber in the world. Notorious for his screams as he climbs. Eg 9:32
youtu.be/b6OvrRbGU68?is…
His screams are so quintessential that when he climbed the hardest climb in history and *didnt* scream he named it Silence
youtu.be/ZRTNHDd0gL8?is…
YouTube
YouTube
English
@varma_ashwin97 @nikillinit Depends very heavily on area of clinical specialty on the one hand and personal background on the other. That said, I’d at least hope that most folks didn’t go into medicine to blindly maximize their paycheck, else they’d have forked off ahead of time.
English
I get why a lot of doctors come to me about careers outside of medicine, but I really do think it's a grass is greener situation
the salary and stability of the median doctor job is far higher than the median stability in tech/finance/consulting right now IMO
and regardless of what the internet thinks docs still hold higher social status than any of those positions
English
@VerbinnenAndrew but even without aggregating across multiple people, you can design N=1 experiments so they are statistically robust for that individual (e.g. ABAB crossover). Much better than than the ad-hoc way people typically self-experiment with their health
English
n=1 self-experimentation can be incredibly compelling, even if not statistically significant.
it gets the ball rolling for more people to run similar n=1s until, together, they become a meaningful dataset.
the more people understood this, the faster science would move.
Crémieux@cremieuxrecueil
Interesting: Turns out people like my inane self-experimentation that I do on the basis of random papers. Maybe I should talk about all the nutty experiments I do more often.
English
If I ever win the lottery I won't tell anyone but there will be signs
www-cs-faculty.stanford.edu/%7Eknuth/email…
English
The last error I made as a doctor was because of a software bug (saved the notes in the wrong patient’s encounter)
waiting.@GreenOnionDuck
As a software engineer I have to unfortunately inform you that if medicine was held to the standard of software engineering there would be mass death.
English
@hnautiyal @sebgehr The key insight: this plot isn’t just about trends in AI slop. It’s about trends in the human writing habits that it rewires.
English
COLM meta-review writing thought: Is there any valid reason to use the word "genuine" in a review? Or is it a 100% precision Claude detector?
English
@fluxtheorist TBC I’m not saying *the whole of the Valley* has caused net societal harm. But I feel that many individual products have, and SV folks as a group do have a crappy track record of assessing for or caring about this.
English
@IAmSamFin I’m not sure about net harm. I certainly think has done more overall good than bad, and only recent memory colors that. But the pervasive belief that tech/innovation is holistically good or done with best intent is a very bad that people fall into.
English
I understand that the tech class is self pleasuring to the idea that they are so much smarter than doctors after yesterday’s announcement, and while many physicians have not put their best foot forwards, it would behoove many of you to realize you are the first to fall for scams.
English