This experiment is part of the Metascience Challenge Grant. Browse more projects

FROcasting: A pilot of forecasting in the scientific review of focused research organizations

$5,605
Raised of $5,000 Goal
112%
Funded on 10/28/23
Successfully Funded
  • $5,605
    pledged
  • 112%
    funded
  • Funded
    on 10/28/23

About This Project

Scientific grant review consists of experts assessing the approach and impact of a given proposal. The process is thus an implicit exercise in forecasting – reviewers are betting on which projects are likely to accomplish their goals and produce valuable outcomes.

Explicit and grounded forecasts could produce better funding decisions, but this approach has yet to be widely tested. We aim to launch a pilot forecasting challenge to build understanding of forecasting in scientific review.

Ask the Scientists

Join The Discussion

What is the context of this research?

Merit review of scientific proposals is ubiquitous across the federal R&D enterprise. But the review process is arguably too conservative, focusing on feasibility over potential impact and limiting the uptake of new research models. Focused research organizations (FROs) – standalone, time-bound, non-profit organizations designed to tackle projects not incentivized by academia or industry – are a relatively untested model with high potential value. FROs aim to produce public goods: processes, tools, and datasets that unlock new methods and accelerate the pace of research.

Federal funders are tentatively curious about this model, but are often wary of (a) the likelihood of proposals achieving their goals and (b) the value of funding one large project rather than several small projects.

What is the significance of this project?

Forecasting can be a valuable tool for assessing the risk-reward profile of novel research models. Because of their bounded time span and technical milestones, FROs are particularly well-suited to forecasting.

The intersection of FROs and forecasting has the potential to yield more than the sum of its parts. By piloting this approach to review, we can (a) generate quantitative predictions of the feasibility and potential impact of real proposals, which funders can use to inform decision-making; (b) provide actionable insight to proposal authors regarding their approach; and (c) obtain metascientific understanding of how forecasting does – or does not – work in a scientific review context.

Follow-up research could entail a formal study comparing forecasting to traditional peer review.

What are the goals of the project?

FAS and Metaculus will run a pilot forecasting tournament on five high-quality FRO proposals sourced by partners at Convergent Research. The forecasting tournament will ask participants to make quantitative forecasts in response to three questions on the probability of success, the social impact, and the scientific impact, of achieving the technical milestones in each proposal. We will recruit and compensate subject matter experts and professional forecasters to participate in the tournament, which will be hosted on Metaculus' platform. After the tournament, we will use the forecasts to calculate the expected value of achieving each technical milestone and categorize the proposals based on their risk-reward profile. The results will be published in a public report.

Budget

Please wait...

This budget will enable us to provide financial compensation to subject matter experts for participating in the forecasting tournament. Since we would be asking subject matter experts to dedicate time and effort to evaluating a FRO proposal and making forecasts, we expect that it will be difficult to recruit participants without compensation. Our partner organization Metaculus will match this amount with $5,000 to fund professional forecasters to participate in the forecasting tournament.

Endorsed by

Paul Niehaus, Caleb Watney, and I are very excited to support this project, which we see as tacking an innovative and important research question that could inform how organizations fund and support scientific research.

Project Timeline

We plan to work on the design of the forecasting tournament and recruit participants throughout the summer of 2023. We will launch the forecasting tournament in mid-September 2023 and close the tournament in mid-December 2023. A report detailing the results and learnings from this pilot will be published in January 2024.

Sep 28, 2023

Launch the online forecasting tournament on Metaculus

Sep 28, 2023

Project Launched

Dec 31, 2023

Close the forecasting tournament

Jan 15, 2024

Extract and analyze forecasting data from the tournament

Jan 31, 2024

Produce public-facing and funder-facing reports describing the results and implications of forecasts

Meet the Team

Alice Wu
Alice Wu
Senior Associate

Affiliates

Federation of American Scientists
View Profile
Jordan Dworkin
Jordan Dworkin
Program Lead, Impetus Institute for Metascience

Affiliates

Federation of American Scientists
View Profile
Peter Mühlbacher
Peter Mühlbacher
Research Scientist

Affiliates

Metaculus
View Profile

Alice Wu

Alice Wu works on metascience and climate policy at the Federation of American Scientists. Prior, she conducted graduate research on solar cell technologies at the Massachusetts Institute of Technology, where she received her Master's in Electrical Engineering.

Jordan Dworkin

Jordan Dworkin is the metascience program lead at the Federation of American Scientists, where he aims to improve scientific funding, institutions, and incentive structures through experimentation. Before joining FAS, he was an assistant professor of clinical biostatistics at Columbia University and the New York State Psychiatric Institute, where his research touched on statistical methodology, medical imaging, and mental health.

Peter Mühlbacher

Peter Mühlbacher is a research scientist at Metaculus, where he works on aggregation algorithms and forecasting analyses, among other things. Prior, he spent some time in Vienna and Cambridge before obtaining his PhD on probabilistic methods for quantum spin systems from Warwick.


Project Backers

  • 3Backers
  • 112%Funded
  • $5,605Total Donations
  • $1,868.33Average Donation
Please wait...