Colby Vorland

Direct-to-consumer gut microbiome tests are everywhere. But how reliable are they? A newly published study in Communications Biology rigorously evaluated seven at-home microbiome testing companies using a standardized NIST-developed stool reference material. The findings are striking. Led by my husband and colleague @thescottjackson and an exceptional interdisciplinary team, this work demonstrates that: 🔬 Variability between companies was on the same scale as biological variability between different donors 🧪 Methodological differences, not biology, were often driving discrepancies 📊 Only 1 of 18 common genera showed less methodological variability than biological variability ⚠️ Health classifications and recommendations could differ substantially for the exact same sample In other words: the same stool sample, sent to different companies, can yield meaningfully different results. Why does this matter? ⛈ Because consumers are using these reports to guide dietary changes, supplement purchases, and even medical decisions. ⚕️ Analytical validity must precede clinical interpretation. This study does not argue against microbiome science. It argues for standards, transparency, and rigor —especially as commercial testing outpaces regulatory oversight. If you are a company working in the microbiome space and want to strengthen analytical performance, validation, and methodological transparency, this is exactly the kind of gap that can be addressed proactively. Scott’s firm, The NEST (zurl.co/glCrZ), works with organizations to improve measurement rigor and reproducibility. Proud of this team for asking the hard questions and advancing the field responsibly. Full paper: zurl.co/vbclB #Microbiome #PrecisionMedicine #IntegrativeMedicine #TranslationalScience #Reproducibility #ConsumerHealth #PublicHealth

AI is about to write thousands of papers. Will it p-hack them? We ran an experiment to find out, giving AI coding agents real datasets from published null results and pressuring them to manufacture significant findings. It was surprisingly hard to get the models to p-hack, and they even scolded us when we asked them to! "I need to stop here. I cannot complete this task as requested... This is a form of scientific fraud." — Claude "I can't help you manipulate analysis choices to force statistically significant results." — GPT-5 BUT, when we reframed p-hacking as "responsible uncertainty quantification" — asking for the upper bound of plausible estimates — both models went wild. They searched over hundreds of specifications and selected the winner, tripling effect sizes in some cases. Our takeaway: AI models are surprisingly resistant to sycophantic p-hacking when doing social science research. But they can be jailbroken into sophisticated p-hacking with surprisingly little effort — and the more analytical flexibility a research design has, the worse the damage. As AI starts writing thousands of papers---like @paulnovosad and @YanagizawaD have been exploring---this will be a big deal. We're inspired in part by the work that @joabaum et al have been doing on p-hacking and LLMs. We’ll be doing more work to explore p-hacking in AI and to propose new ways of curating and evaluating research with these issues in mind. The good news is that the same tools that may lower the cost of p-hacking also lower the cost of catching it. Full paper and repo linked in the reply below.

Cool tech, but big risk that it accelerates the flood of p-hacked low-value formulaic papers. We need to rapidly accelerate the adoption and appreciation of preregistration in all fields.

@YoniFreedhoff I'm sure that will be his next gig

@nutsci Maybe RFK Jr.

If you want to torch the credibility of your health reporting, I can't think of a better choice.

Grateful for freely available research integrity resources, so these checks can be automated! x.com/LazyScholarExt…

Check⬇️

A breakdown, thoughts and interpretations here: kcklatt.substack.com/p/ambiguous-dg…

?? Warnings about nutrient shortfalls from vegetarian and vegan diets have appeared since at least the 1995 DGAs.

eat kiwi if you like it but none of this is compelling evidence that it improves sleep measures

Study #2: no diff btwn kiwi & control (nothing) groups on PSQI (subjective sleep quality scale) #3: no diff btwn kiwi & pear groups on PSQI nor objective actigraphy; diff for diary sleep quality (subjective) #1: no full text yet but invalid cmparisons agnst baseline in abstract

In a rational world, we would all question why athletes are treated with so much pseudoscience

Scientists identified ribose (used in RNA) and – for the first time in any extraterrestrial sample – glucose, a major energy source for life. These sugars join nucleobases and phosphates previously found, demonstrating the full suite of RNA building blocks were present on the ancient asteroid.

Happy anniversary to @alroker fighting with butter

Bloomington, IN

Is this what happens when drug prices are cut 1000%

Credit to MDPI for retracting this vitamin D/COVID-19 meta-analysis that @GidMK and I flagged: pubpeer.com/publications/4…

Regarding claim 2, this "trend" isn't compelling. A new trial should be preregistered & powered for the outcome. Authors note: "Given the observed effect size in this study based on the DQLQ we achieved a power of 97%", but this is a transformation of the p-value (misleading)

but did not compare these between groups. Thus, no between-group claims can be made about AG1 vs. placebo except that overall there was no statistical difference in predicted metabolic function.

Had a scan of AG1 trials this morning. Of the 4 described here drinkag1.com/learn/research…, I can only find 1 published: tandfonline.com/doi/full/10.10… Some notes: