Incorporating costs and benefits to the evaluation of uncertain research results: applications to cancer research funding

How should costs and benefits affect the evaluation of uncertain research?

A study's chance of being correct is only one part of deciding whether a research program is worth pursuing. This paper develops a decision-theoretic framework that makes potential costs and benefits explicit.

What the study found

• Acceptable uncertainty changes with the potential consequences of a research decision.
• The cancer-research applications illustrate why exploratory research can be worth supporting despite uncertain results.

How the study works

The framework uses Bayesian decision theory to derive thresholds involving prestudy odds and positive predictive values, then applies the calculations to cancer research and funding.

Scope and limitations

• The conclusions depend on the costs, benefits, probabilities, and alternatives supplied to the model.
• The framework supports deliberation about research programs; it does not supply universal funding thresholds.

Abstract

Abstract Correctness is a key aspiration of the scientific process, yet recent studies suggest that many high-profile findings may be difficult to replicate or require considerable evidence for verification. Proposals to fix these issues typically ask for tighter statistical controls (e.g., stricter p-value thresholds or higher statistical power). However, these approaches often overlook the importance of contemplating research outcomes’ potential costs and benefits. Here, we develop a framework grounded in Bayesian decision theory that seamlessly integrates cost-benefit analysis into evaluating research programs with potentially uncertain results. We derive minimally acceptable prestudy odds and positive predictive values for cost and benefit levels. We show that tolerance to inaccurate results changes dramatically due to uncertainties posed by research. We also show that reducing uncertainties (e.g., by recruiting more subjects) may have limited effects on the expected benefit of continuing specific research programs. We apply our framework to several types of cancer research and their funding. Our analysis shows that highly exploratory research designs are easily justifiable due to their potential benefits, even when probabilistic models suggest otherwise. We discuss how the cost and benefit of research could and should always be part of the toolkit used by scientists, institutions, or funding agencies.

Abstract from the original work, reproduced under its Creative Commons license. The overview above summarizes the study.

Cite this work

Han Zhuang, Daniel E. Acuna (2024). Incorporating costs and benefits to the evaluation of uncertain research results: applications to cancer research funding. Quantitative Science Studies. https://doi.org/10.1162/qss_a_00332

@article{10.1162/qss_a_00332,
  title = {{Incorporating costs and benefits to the evaluation of uncertain research results: applications to cancer research funding}},
  author = {Zhuang, Han and Acuna, Daniel E.},
  year = {2024},
  publication_date = {2024},
  journal = {Quantitative Science Studies},
  pages = {1-27},
  doi = {10.1162/qss_a_00332},
  url = {https://doi.org/10.1162/qss_a_00332},
  issn = {2641-3337}
}