Forecasting Research Institute

We're hiring our first Data Engineer at FRI! 🚀 We apply forecasting science to the world's most consequential decisions, including AI progress, nuclear risk, and biosecurity. Our work has been featured in The Economist, Bloomberg, TIME, and Vox. This is a rare chance to build a data platform from scratch with genuine autonomy and mission impact. Know someone? Please share: forecastingresearch.org/roles/data-eng…

Update: We've added an addendum to the original post to clarify a technical error. Thank you to @EgeErdil2 for pointing out our error. You can read the full addendum here: forecastingresearch.substack.com/i/189859724/ad…

Read the full post: forecastingresearch.substack.com/p/introducing-… The Brier Index is now the primary metric on the ForecastBench leaderboard. Underlying Brier scores remain available on the Datasets page. forecastbench.org

The Brier score is the standard metric for evaluating forecasting accuracy. But it has a major drawback: it’s difficult to interpret and communicate. For example, superforecasters have a Brier score of 0.086 on ForecastBench, our AI forecasting benchmark. Technically, this means that on average superforecasters are “0.086 squared probability units from the truth.” Confused? You’re not alone. We propose a simple fix: the Brier Index. 🧵

Thank you for spotting this technical error! A monotone transformation of a proper scoring rule remains proper for affine transformations, but the Brier Index is a nonlinear transformation. We’ve just published an addendum clarifying this issue on our original Substack post. One clarification: “always predict the most likely outcome with 100%” is the optimal strategy for N = 1 (i.e., you’re predicting a single question). As the number of questions grows, the incentive to distort towards the most likely outcome shrinks rapidly. We include a short numerical experiment in the addendum: for N = 10 independent questions, where the true forecast is close to 0.5, reporting your true belief is already close to optimal. Finally, the ForecastBench leaderboard ranking is unaffected by this point: we apply the transformation after averaging across all questions, so it’s a monotonic function of a single Brier score. Full correction here: forecastingresearch.substack.com/p/introducing-…

@Research_FRI monotonic transformations of proper scoring rules are not necessarily proper and that specific rule is also not proper. the optimal strategy under that rule is to always predict the most likely outcome with 100% probability

To explore more Wave 5 forecasts from LEAP, including question-by-question details, visit: leap.forecastingresearch.org/reports/wave5 Find out more about LEAP, our panelists, and analysis from early waves on the LEAP website: leap.forecastingresearch.org If you have ideas for questions, submit your ideas here: airtable.com/appGCchUyUTPvT…

Experts expect $4 billion in regulatory fines for AI companies by 2030 💰 Median expert predictions on cumulative large-scale regulatory fines for AI companies By 2027: $1 billion By 2030: $4 billion By 2040: $10 billion Is this a lot of money? For comparison, the FTC fined Facebook $5 billion in 2019—the largest ever for privacy violations. On the other hand, Google's total revenue in 2025 topped $400 billion.

What are the geopolitical and military implications of AI progress? In Wave 5 of the Longitudinal Expert AI Panel (LEAP), we asked AI experts for their forecasts on U.S.–China AI polarity, chip manufacturing, cyberweapons, regulatory fines, and AI forecasting accuracy. Key finding: experts expect the U.S.–China AI capabilities gap to narrow and Taiwan's frontier chip manufacturing dominance to erode. 🧵 Read on for more:

For a more detailed exploration of these results, head to the FRI Substack: forecastingresearch.substack.com/p/how-well-did… And to learn more about the original RCT, read the Active Site paper: arxiv.org/pdf/2602.16703 Or the Active Site blog: activesite.substack.com/p/rct

🔮 Although the Active Site RCT did not show uplift in mid-2025, we were interested in how people would react if future work discovered a larger effect. We asked forecasters to imagine a repeat of the Active Site RCT in 2026, yielding various hypothetical results The below chart shows how their outbreak risk estimate changed depending on the hypothetical RCT results. The resulting “risk curves” sloped upwards for all forecaster groups, indicating that participants perceived the RCT as a relevant signal of increased outbreak risk.

In mid-2025, @ActiveSiteBio told us about an RCT it was working on that would test how access to LLMs affected novices' ability to do molecular biology work in a wet lab. We ran a forecasting study alongside this RCT, to find out what forecasters predicted for the RCT outcome and to understand how the study results connected to real world biorisk. So, how well did superforecasters and experts predict wet lab skill uplift from LLMs? 🧵 Read on to find out…