Raphael Merz

🚀Very excited to finally see our paper on nonsignificance misinterpretations published! 📈 Together, @ste_lee_murphy, @aurelio_fdez, Linda Reimann and I investigated the prevalence of "p > .05 = absence of an effect" interpretations. (1/4) doi.org/10.1098/rsos.2…

I am hiring a PhD student on the meaningful interpretation of effect sizes as part of my VICI funded project. This is a 4 year paid position in a welcoming and collaborative environment. Find out more or apply at tue.nl/werken-bij-tue…

I think at some point we need to move from taking this as a serious ‘reason’ to be against preregistration, and call these people out for being too incompetent on research methods to take their opinion seriously. As we wrote doi.org/10.1080/283337…

The new workshop of Paul Meehl Graduate School, Causal Inference and Variable Control by Peder Isager, is open for registration. Check out the announcement below: paulmeehlschool.github.io/2025-04-02-cau…

@Liikennepsykol1 We don't want to sell the idea that these prevalences change in one direction or the other and I don't think we claim that there is an effect in a specific direction in the paper. I also don't have a clear-cut definition when a nonsig finding is informative. The CIs are for sure.

@RaphaelMerz @JakubTomek13 @ste_lee_murphy @aurelio_fdez I was referring to that specific trend effect. By using "suggesting" & "it's likely" implies this trend effect indeed exists. That was my question: why use "suggesting" & "it's likely" in relation to the nonsignificant trend effect. And in which cases others can do the same?

🚀Very excited to finally see our paper on nonsignificance misinterpretations published! 📈 Together, @ste_lee_murphy, @aurelio_fdez, Linda Reimann and I investigated the prevalence of "p > .05 = absence of an effect" interpretations. (1/4) doi.org/10.1098/rsos.2…

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez In this case, we would need a larger sample to reliably estimate the true effect/change. However, the estimates and CIs are still informative in showing that these misinterpretations remain high. (2/2)

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez We don't need to debate whether the effect exists. These prevalences won't be exactly the same across years. Still, we don't have strong enough evidence to rule out that the effect is exactly zero, but you can check the CIs to see what range of values the model predicts. (1/2)

@Liikennepsykol1 @JakubTomek13 @ste_lee_murphy @aurelio_fdez Thanks for the question! In the screenshot, you can replace "suggests" with "the model estimates" (see also the CIs). That said, I don't think the exact p values observed in our study are the most informative part of it. You can take these prevalences more as descriptives.

@JakubTomek13 @RaphaelMerz @ste_lee_murphy @aurelio_fdez I understand that. However, the authors write in their paper that their nonsignificant results suggest the existence of an effect. They even say 'it is likely.' Does that mean any result suggests the existence of an effect?

📢Clearly, we have a problem. Few articles acknowledge that a nonsignificant result may simply reflect an effect that could not be found. We urge authors to reflect on their interpretations and recommend analyses with specified alternatives (e.g., minimum effect tests). (4/4)

👫Of these, 23% suggested the absence of an effect at the sample level (e.g., 'groups were the same') and 58% at the population level (e.g., 'men and women are the same').

@AminSinichi @ste_lee_murphy @LindaEReimann @aurelio_fdez Thanks a lot! :)

@RaphaelMerz @ste_lee_murphy @LindaEReimann @aurelio_fdez Awesome work!

Very excited to share our new preprint! 📢 We investigated how often researchers misinterpret statistically nonsignificant results as the absence of an effect. Spoiler: it’s a lot (~81% of 599 articles)! Dive into our findings and their implications here: osf.io/preprints/psya…

⏳ Reminder: Sign-up for the PYMS Pre-Symposium on Dec 5th closes tomorrow! Don't miss out on this chance to connect with fellow early career researchers. 🗓️ Check out the preliminary program here: docs.google.com/document/d/1iG… Sign up while you still can! 👇

@BenCTurnbull @ste_lee_murphy @LindaEReimann @aurelio_fdez Personally, I am fine with terms like "statistically significant" IF we improve researchers statistical literacy AND educate them on approaches like equivalence testing that require them to also say what effects they would regard as practically relevant!

@lukaswallrich @F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez I don't think confidence intervals alone would suffice. The statement still strongly implies that the groups *are the same*, even with CIs. To truly convince me, I'd need a prespecified smallest effect size of interest and a corresponding Equivalence Test.

@F_Bethke @RaphaelMerz @ste_lee_murphy @LindaEReimann @aurelio_fdez @RaphaelMerz what would you need to see in an article to accept the interpretation? Would confidence intervals around the two percentages be enough? Or would you require more to suggest equivalence? I did not quite understand that when *skimming* the coding criteria.

@F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez @trumanfrancis To strengthen the argument that there's no practically relevant effect of the intervention, you might consider Equivalence Testing with a predefined smallest effect size of interest (see @trumanfrancis response) (2/2)

@F_Bethke @ste_lee_murphy @LindaEReimann @aurelio_fdez I would probably side with @trumanfrancis on this one and caution against concluding 'no substantial effect' based on nonsignificance –since statistical significance doesn't inform practical relevance. (1/2)

@JakubTomek13 Nice! Excited to read about it! We were also left wondering how some of these really obvious misinterpretations made it through peer review and the editorial process… It’s surprising how often they "slip through"...

@RaphaelMerz I'm finishing writing a book on statistics for biomedical researchers, where we discuss this issue - your paper is going straight in as a reference. Very real issue, unfortunately :( I wonder why editors don't do more about this. It's so bad and so blatant.

Would be interesting to know if physiology is doing better than psychology, but I somehow doubt it. Quite shocking numbers, and great work to quantify the issue of "there was no effect".

@JakubTomek13 100% my experience! We actually considered including examples of these 'misinterpretation chains' - where one misinterpretation gets cited, then cited again, and so on. Didn’t make it into the final paper, but I think it’s definitely a real issue!

@RaphaelMerz Yeah, it's terrible. And when I raise this with people, often I heard "ah, it's just about wording, we of course know what it means". Except that not. And the distinction gets even weaker when "proof of null" results are cited in reviews, perpetuating the misinterpretation.