A Note on Improving the Measurement of the Quality of Forecasts in Prediction Tournaments

Item Type Monograph (Working Paper)
Abstract

We begin by arguing that the quality of forecasts consists of two aspects: accuracy and timing. Existing conceptualizations and operationalizations of the quality of forecasts, however, appear to focus more on the accuracy and do not incorporate sufficiently the timing of forecasts. To improve the so-called Accuracy Score of Cultivate Labs (the backend of GJopen and our St. Gallen Forecasting platform), we propose what we call the “Quality Score”, which also considers the timing of bad forecasts. We believe that Cultivate Labs should consider adopting a Quality Score instead of its current Accuracy Score. Building on Merkle et al. (2017), we then move into a second important proposal by noting that research is frequently interested in measuring as precisely as possible the forecasting skills of persons. We argue that IRT models should be the preferred tool for measuring the forecasting skills of persons as they usually allow researchers to measure the forecasting skills more accurately than the Accuracy Score or Quality Score. To allow researchers to estimate forecasting skills, we refine an earlier IRT model to implement our definition of the quality of forecasts in the context of forecasting tournaments. Unlike earlier IRT models, which only captured the timing of one forecast per tournament question, our proposed model makes it possible to assess the timing of multiple forecast per tournament question, as is common on the GJopen and St. Gallen forecasting platforms. With our refined IRT model one can analyze experimental settings in which forecasters are encouraged to update their probability forecasts every time they obtain relevant new information.

Authors Schuler, Benedikt Alexander; Murmann, Johann Peter Horst Wilhelm & Beisemann, Marie
Language English
Subjects business studies
other research area
Date 9 November 2020
Depositing User Benedikt Alexander Schuler
Date Deposited 13 Jul 2020 17:40
Last Modified 10 Nov 2020 20:08
URI: https://www.alexandria.unisg.ch/publications/260670

Download

[img] Text
Research Note Quality Forecasts.pdf - Updated Version

Download (381kB)

Citation

Schuler, Benedikt Alexander; Murmann, Johann Peter Horst Wilhelm & Beisemann, Marie: A Note on Improving the Measurement of the Quality of Forecasts in Prediction Tournaments. , 2020,

Statistics

https://www.alexandria.unisg.ch/id/eprint/260670
Edit item Edit item
Feedback?