How Good are your Counterfactuals? Assessing Quantitative Macro-Comparative Welfare State Research with Qualitative Criteria

Abstract: All causal statements based on historical data - both in qualitative and quantitative social research - rely on counterfactuals. In quantitative research, scholars attempt to arrive at valid counterfactuals by emulating an experimental design. However, because of treatments that are impossible to manipulate and the non-random assignment of data to treatment and control groups, causal statements are often based on invalid counterfactuals. In qualitative research, scholars attempt to arrive at valid counterfactuals by probing the historical and logical consistency of counterfactuals and by acknowledging the interconnectedness of events. Criteria to evaluate counterfactuals have been developed that allow for a discussion of the quality of counterfactuals used in causal statements. In this article, we suggest using these qualitative criteria to evaluate counterfactuals in quantitative macro-comparative welfare state research. We argue that these criteria can help us identify erroneous causal inferences in quantitative research based on historical data.