New publication | Heterogeneity in effect size estimates
Dreber and Johannesson with co-authors provide a framework for studying heterogeneity in effect sizes and the generalizability of empirical findings in the social sciences. Heterogeneity is divided into population heterogeneity, design heterogeneity and analytical heterogeneity. They also estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multi-analyst studies. The results suggest that population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. A conservative interpretation of the design and analytical heterogeneity implies doubling standard errors and confidence intervals to incorporate the added uncertainty. The results imply that the generalizability of individual empirical studies in the social sciences is typically low, and that we need to move towards much larger studies systematically varying populations, research designs and analysis paths; referred to as preregistered prospective meta-analysis.
Link to the publication
Abstract
A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.
