Forecasting Research Institute

243 posts

Forecasting Research Institute banner
Forecasting Research Institute

Forecasting Research Institute

@Research_FRI

We advance the science of forecasting to improve decision-making on high stakes issues. Co-founded by chief scientist Philip Tetlock.

Katılım Mayıs 2023
26 Takip Edilen3.3K Takipçiler
Sabitlenmiş Tweet
Forecasting Research Institute
We're working to improve ForecastBench! Upcoming changes: • Fresh superforecaster round: fall 2026. • Continuous (non-binary) forecasting questions: summer 2026. • External submissions on continuous questions: summer 2026. • Revamped dataset questions: fall 2026.
English
0
0
5
310
Forecasting Research Institute
Does parity on dataset questions mean ForecastBench is “solved”? No. On market questions that require judgment about novel, one-off events, LLMs are still behind. Even on dataset questions, the 64.9% achieved by Google DeepMind is unlikely to be the ceiling of what’s possible. ForecastBench doesn’t stop being useful at human parity. It will continue tracking LLM progress even if they surpass expert human forecasters.
English
1
0
2
351
Forecasting Research Institute
On a subset of ForecastBench questions, an LLM has matched superforecaster performance for the first time. A submission from @GoogleDeepMind, named “green tree,” is now #1 on dataset questions on ForecastBench, our AI forecasting benchmark. Superforecasters remain #1 overall.
Forecasting Research Institute tweet media
English
5
11
60
2.3K
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
💲Experts forecast Amazon's e-commerce sales per distribution employee to rise The median expert forecasts Amazon's real e-commerce net sales per distribution employee to rise from its 2024 baseline of $464,771. They expect: 2026: $480,000 2030: $549,842 2040: $800,000 Superforecasters are even more bullish, predicting over $1 million by 2040. Forecasters cited the suitability of warehouses for robotic deployment, Amazon's track-record in automation and a decoupling of sales from labor as key factors driving their predictions.
English
1
0
0
343
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
How fast do experts and superforecasters expect progress in robotics to be? In Wave 7 of the Longitudinal Expert AI Panel (LEAP) we asked forecasters to predict when robots will pass the "Coffee Test", and when they will perform the first autonomous appendectomy. We also asked for forecasts on the supply of industrial robots, future U.S. labor share, and Amazon's e-commerce sales per distribution employee, to try to understand how fast one of the U.S.’s largest employers will automate. 🧵 Read on for more:
Forecasting Research Institute tweet media
English
2
2
23
1.3K
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
Thanks for the engagement and thoughtful questions, Arvind! 1) Yes, the intention was to clearly distinguish AI capability from diffusion, and the scenario descriptions and questions are consistent with this. 2) Per the above, the rapid scenario description is intended to mean that AI is *as capable as* "highly competent CEOs" at the relevant tasks--it does not mean to say anything about social preferences. So, it is left to forecasters to think about what that capability would lead to in terms of actual economic impacts. Underlying views about rates of diffusion are likely a key factor driving forecasts and are mentioned in many rationales. 3) This is a great question and something we'll take a closer look at. We don't have preexisting analysis directly on this point, but it is a type of analysis we should be able to do. Hope to have more to say soon!
English
0
0
1
132
Arvind Narayanan
Arvind Narayanan@random_walker·
A few more thoughts on this. I'd love for the authors to weigh in because I'm frankly pretty confused. 1) The survey wording clearly distinguishes AI capability from diffusion.¹ But a thread by one of the authors, @BasilHalperin, seems to imply that that was not the intention.² So which is it? Was there not consensus among the authors? (FWIW I'm an advisor to a related project by the Forecasting Research Institute, the Longitudinal Expert AI Panel (LEAP). I repeatedly provided feedback to clearly separate capability from adoption and my understanding is that the the LEAP leads, who are also among the leads for this effort, were on board.) 2) Despite the explicit capability/adoption distinction made in the survey, the wording of the scenarios is confusing IMO. In the rapid progress scenario AI is supposed to be able to be at the level of a "highly competent CEO". But most people don't like to be managed by a bot.³ Is the scenario meant to imply that this preference has somehow changed? How did respondents interpret this? Halperin calls the collective survey responses "insane" in the thread. Is it possible that he understands the scenarios differently than the respondents did? 3) One author said my rationale about political backlash was frequently given by the respondents, but Halperin's thread says this was rare (and "underrated"). Which is it? Could the authors provide a quantitative breakdown of the rationales? ————————————————— ¹ "In the following scenarios, we consider the development of AI capabilities, not adoption. Regulation, social norms, or extended integration processes could all prevent the application of AI to all tasks of which it is capable." ² "Yes bottlenecks will slow progress, this is extremely important. But the rapid scenario under consideration has already chewed through most bottlenecks" ³ "Indeed, models have become so capable that AI can create an album of the same caliber as the Grammy Album of the Year. Additionally, a single AI agent can generate a Pulitzer(or Booker Prize-) caliber novel according to current (2025) standards, adapt the book into an engaging two-hour movie, negotiate the resulting book and movie contracts, and launch the marketing campaigns for both while its sibling agents manage the book publishing company and movie studio at the level of highly competent CEOs."
Stefan Schubert@StefanFSchubert

A recent @Research_FRI survey found that experts predict muted growth even under rapid AI progress. Why is that? @testingham, @BasilHalperin, @random_walker, @paulnovosad, and @alexolegimas provided competing explanations. I break down the debate: update.news/p/how-much-wil…

English
4
9
43
29.4K
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
We built a tool so that you can compare your own forecasts on AI’s economic impact to economists’ forecasts: explore.forecastingresearch.org/participate/ee… We hope this can be a useful tool for informing discourse on the economic impacts of AI. When you submit your forecasts, you will get a shareable image that compares your predictions to economists’ predictions. See example image below.
Forecasting Research Institute tweet media
English
2
5
40
10.6K
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
Thank you to all of our coauthors: @EzraKarger, Otto Kuusela, @Jabaluck, @Afinetheorem, @BasilHalperin, @toddrjones, @connacher_, @pawtrammell, @mattsreynolds1, @danmayland, Ria Viswanathan, Ananaya Mittal, Rebecca Ceppas de Castro, Josh Rosenberg, and @PTetlock For full results, including the impact on sectors and occupations, read our full paper here: forecastingresearch.org/s/forecasting-… For a summary of the results, see our policy briefing here: forecastingresearch.org/s/forecasting-… And visit our Substack: forecastingresearch.substack.com/forecasting-th…
English
3
2
28
5.9K
Forecasting Research Institute
Forecasting Research Institute@Research_FRI·
We completed the most comprehensive study of how economists and AI experts think AI will affect the U.S. economy. They predict major AI progress—but no dramatic break from economic trends: GDP growth rates similar to today's and a moderate decline in labor force participation. However, when asked to consider what would happen in a world with extremely rapid progress in AI capabilities by 2030, they predict significant economic impacts by 2050: • Annualized GDP growth of 3.5% (compared to 2.4% in 2025) • A labor force participation rate of 55% (roughly 10 million fewer jobs) • 80% of wealth held by the top 10% (highest since 1939) 🧵 Here's what we found:
Forecasting Research Institute tweet mediaForecasting Research Institute tweet media
English
20
192
644
377.9K