A Note on Improving the Measurement of the Quality of Forecasts in Prediction Tournaments

We begin by arguing that the quality of forecasts consists of two aspects: accuracy and timing. Existing conceptualizations and operationalizations of the quality of forecasts, however, appear to focus more on the accuracy and do not incorporate sufficiently the timing of forecasts. To improve the so-called Accuracy Score of Cultivate Labs (the backend of GJopen and our St. Gallen Forecasting platform), we propose what we call the “Quality Score”, which also considers the timing of bad forecasts. We believe that Cultivate Labs should consider adopting a Quality Score instead of its current Accuracy Score. Building on Merkle et al. (2017), we then move into a second important proposal by noting that research is frequently interested in measuring as precisely as possible the forecasting skills of persons. We argue that IRT models should be the preferred tool for measuring the forecasting skills of persons as they usually allow researchers to measure the forecasting skills more accurately than the Accuracy Score or Quality Score. To allow researchers to estimate forecasting skills, we refine an earlier IRT model to implement our definition of the quality of forecasts in the context of forecasting tournaments. Unlike earlier IRT models, which only captured the timing of one forecast per tournament question, our proposed model makes it possible to assess the timing of multiple forecast per tournament question, as is common on the GJopen and St. Gallen forecasting platforms. With our refined IRT model one can analyze experimental settings in which forecasters are encouraged to update their probability forecasts every time they obtain relevant new information.