Marco Mendoza Aviña co-authored a paper published in The Journal of Politics. The paper examines the replicability crisis in political science by analyzing over 16,000 hypothesis tests from nearly 2,000 articles, revealing that most studies are severely underpowered while also showing that experts significantly overestimate typical power levels in the discipline.
The social sciences face a replicability crisis. A key determinant of replication success is statistical power. We assess the power of political science research by collating over 16,000 hypothesis tests from about 2,000 articles in 46 areas of the discipline. Under generous assumptions, we show that quantitative research in political science is greatly underpowered: the median analysis has about 10% power, and only about 1 in 10 tests have at least 80% power to detect the consensus effects reported in the literature.
We also find substantial heterogeneity in tests across research areas, with some being characterized by high power but most having very low power. To contextualize our findings, we survey political methodologists to assess their expectations about power levels. Most methodologists greatly overestimate the statistical power of political science research.
Read the full paper here.