In 2015, Comparative Political Studies embarked on a landmark pilot study in research transparency in the social sciences. The editors issued an open call for submissions of manuscripts that contained no mention of their actual results, incentivizing reviewers to evaluate manuscripts based on their theoretical contributions, research designs, and analysis plans. The three papers in this special issue are the result of this process that began with 19 submissions. In this article, we describe the rationale for this pilot, expressly articulating the practices of preregistration and results-free review. We document the process of carrying out the special issue with a discussion of the three accepted papers, and critically evaluate the role of both preregistration and results-free review. Our main conclusions are that results-free review encourages much greater attention to theory and research design, but that it raises thorny problems about how to anticipate and interpret null findings. We also observe that as currently practiced, results-free review has a particular affinity with experimental and cross-case methodologies. Our lack of submissions from scholars using qualitative or interpretivist research suggests limitations to the widespread use of results-free review.

References

Bates, R. H. ( 1996 ). Letter from the president: Area studies and the discipline . American Political Science Association–Comparative Politics, 7(1), 1 - 2 .

Google Scholar

Berger, J. O., Sellke, T. ( 1987 ). Testing a point null hypothesis: The irreconcilability of p values and evidence . Journal of the American Statistical Association, 82, 112 - 122 .

Google Scholar ISI

Broockman, D. E., Kalla, J. L. ( 2016 ). Durably reducing transphobia: A field experiment on door-to-door canvassing . Science, 352, 220 - 224 .

Google Scholar Crossref | Medline | ISI

Broockman, D. E., Kalla, J. L., Aronow, P. ( 2014 ). Irregularities in LaCour (2014). Unpublished manuscript, University of California , Berkeley .

Google Scholar

Bush, S., Erlich, A., Prather, L., Zeira, Y. ( 2016 ). The effects of authoritarian iconography: An experimental test . Comparative Political Studies, 49, 1704 - 1738 .

Google Scholar SAGE Journals | ISI

Carey, B. ( 2011 , November ). Fraud case seen as a red flag for psychology research . The New York Times. Retrieved from http://www.nytimes.com/2011/11/03/health/research/noted-dutch-psychologist-stapel-accused-of-research-fraud.html?_r=1

Google Scholar

Casey, K., Glennerster, R., Miguel, E. ( 2012 ). Reshaping institutions: Evidence on aid impacts using a preanalysis plan . Quarterly Journal of Economics, 127, 1755 - 1812 .

Google Scholar Crossref | ISI

Cassidy, J. ( 2013 , April ). The Reinhart and Rogoff controversy: A summing up . The New Yorker. Retrieved from http://www.newyorker.com/rational-irrationality/the-reinhart-and-rogoff-controversy-a-summing-up

Google Scholar

Clarke, K., Primo, D. ( 2012 ). A model discipline: Political science and the logic of representations. Oxford, UK : Oxford University Press .

Google Scholar Crossref

De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., …Van Der Weyden, M. B. ( 2004 ). Clinical trial registration: A statement from the international committee of medical journal editors . The New England Journal of Medicine, 351, 1250 - 1251 .

Google Scholar Crossref | Medline | ISI

Dickersin, K. ( 1990 ). The existence of publication bias and risk factors for its occurrence . Journal of the American Medical Association, 263, 1385 - 1389 .

Google Scholar Crossref | Medline | ISI

Dickersin, K., Rennie, D. ( 2003 ). Registering clinical trials . Journal of the American Medical Association, 290, 516 - 523 .

Google Scholar Crossref | Medline | ISI

Findley, M. G., Nielson, D. L., Sharman, J. C. ( 2013 ). Using field experiments in international relations: A randomized study of anonymous incorporation . International Organization, 67, 657 - 693 .

Google Scholar Crossref | ISI

Findley, M. G., Nielson, D. L., Sharman, J. C. ( 2014 ). Global shell games: Experiments in transnational relations, crime, and terrorism. Cambridge, UK : Cambridge University Press .

Google Scholar

Findley, M. G., Nielson, D. L., Sharman, J. C. ( 2015 ). Causes of non-compliance with international law: Evidence from a field experiment on financial transparency . American Journal of Political Science, 59, 146 - 161 .

Google Scholar Crossref | ISI

Finkel, S. E., Pérez-Liñán, A., Seligson, M. A. ( 2007 ). The effects of U.S. foreign assistance on democracy building, 1990-2003 . World Politics, 59, 404 - 439 .

Google Scholar Crossref | ISI

Franco, A., Malhotra, N., Simonovits, G. ( 2014 ). Publication bias in the social sciences: Unlocking the file drawer . Science, 345, 1502 - 1505 .

Google Scholar Crossref | Medline | ISI

Gelman, A., Carlin, J. ( 2014 ). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors . Perspectives on Psychological Science, 9, 641 - 651 .

Google Scholar SAGE Journals | ISI

Gerber, A., Malhotra, N. ( 2008 ). Do statistical reporting standards affect what is published: Publication bias in two leading political science journals . Quarterly Journal of Political Science, 3, 313 - 326 .

Google Scholar Crossref | ISI

Gerber, A., Malhotra, N., Dowling, C., Doherty, D. ( 2010 ). Publication bias in two political behavior literatures . American Politics Research, 38, 591 - 613 .

Google Scholar SAGE Journals | ISI

Gill, J. ( 1999 ). The insignificance of null hypothesis significance testing . Political Research Quarterly, 52, 647 - 674 .

Google Scholar SAGE Journals | ISI

Glewwe, P., Kremer, M. ( 2006 ). Schools, teachers, and education outcomes in developing countries . In Hanushek, E., Welch, F. (Eds.), Handbook of the economics of education (Vol. 2, pp. 945 - 1017 ). New York : Elsevier .

Google Scholar

Gottlieb, J. ( 2016 ). Greater expectations? A field experiment to improve accountability in Mali . American Journal of Political Science, 60, 143 - 157 .

Google Scholar Crossref | ISI

Hartman, E., Hidalgo, F. D. ( 2015 ). What’s the alternative? An equivalence approach to balance and placebo tests. Unpublished manuscript, Princeton University , NJ .

Google Scholar

Herndon, T., Ash, M., Pollin, R. ( 2014 ). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff . Cambridge Journal of Economics, 38, 257 - 279 .

Google Scholar Crossref | ISI

Hidalgo, F. D., Canello, J., Lima-de-Oliveira, R. ( 2016 ). Can politicians police themselves? Natural experimental evidence from Brazil’s audit courts . Comparative Political Studies, 49, 1739 - 1773 .

Google Scholar SAGE Journals | ISI

Huber, J. ( 2013 , June ). Is theory getting lost in the “identification revolution”? The Political Economist, pp. 1 - 3 .

Google Scholar

Huff, C., Kruszewska, D. ( 2016 ). Banners, barricades, and bombs: The tactical choices of social movements and public opinion . Comparative Political Studies, 49, 1774 - 1808 .

Google Scholar SAGE Journals | ISI

Humphreys, M., de laSierra, R. S., van der Windt, P. ( 2013 ). Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration . Political Analysis, 21, 1 - 20 .

Google Scholar Crossref | ISI

Ioannidis, J. ( 1998 ). Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials . Journal of the American Medical Association, 279, 281 - 286 .

Google Scholar Crossref | Medline | ISI

LaCour, M., Green, D. ( 2014 ). When contact changes minds: An experiment on transmission of support for gay equality . Science, 346, 1366 - 1369 .

Google Scholar Crossref | Medline | ISI

Laitin, D. ( 2013 ). Fisheries management . Political Analysis, 21, 42 - 47 .

Google Scholar Crossref | ISI

Lieberman, E. S. ( 2005 ). Nested analysis as a mixed-method strategy for comparative research . American Political Science Review, 99, 435 - 452 .

Google Scholar Crossref | ISI

Malesky, E. J. ( 2008 ). Battling onward: The debate over field research in developmental economics and its implications for comparative politics . Qualitative Methods, 6(2), 17 - 21 .

Google Scholar

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., … Van der Laan, M. ( 2014 ). Promoting transparency in social science research . Science, 343, 30 - 31 .

Google Scholar Crossref | Medline | ISI

Monogan, J. E. ( 2015 ). Research preregistration in political science: The case, counterarguments, and a response to critiques . Political Science & Politics, 48, 425 - 429 .

Google Scholar Crossref | ISI

Moravcsik, A. ( 2014 ). Transparency: The revolution in qualitative research . Political Science in Politics, 47, 48 - 53 .

Google Scholar Crossref | ISI

Neumark, D. ( 2001 ). The employment effects of minimum wages: Evidence from a prespecified research design the employment effects of minimum wages . Industrial Relation, 40, 121 - 144 .

Google Scholar ISI

Nielsen, R., Findley, M. G., Candland, T., Davis, Z., Nielson, D. L. ( 2011 ). Foreign aid shocks as a cause of violent armed conflict . American Journal of Political Science, 55, 219 - 232 .

Google Scholar Crossref | ISI

Nuzzo, R. ( 2014 ). Scientific method: Statistical errors . Nature. Retrieved from http://www.nature.com/news/scientific-method-statistical-errors-1.14700

Google Scholar

Nyhan, B. ( 2015 ). Increasing the credibility of political science research: A proposal for journal reforms . PS: Political Science and Politics 48( S1 ): 78 – 83 .

Google Scholar Crossref | ISI

O’Connor, A. ( 2016 , April 13 ). A decades-old study, rediscovered, challenges advice on saturated fat . The New York Times Well. Retrieved from http://nyti.ms/1SynN1M

Google Scholar

Open Science Collaboration . ( 2015 ). Estimating the reproducibility of psychological science . Science, 349. doi:10.1126/science.aac4716

Google Scholar Crossref | ISI

Pepinsky, T. B. ( 2015 ). Context and method in southeast Asian politics . Pacific Affairs, 87, 441 - 461 .

Google Scholar Crossref | ISI

Rayhill, M., Sharon, R., Burch, R., Loder, E. ( 2015 ). Registration status and outcome reporting of trials published in core headache medicine journals , Neurology, 85, 1789 - 1794 . doi:10.1212/WNL.0000000000002127

Google Scholar Crossref | Medline | ISI

Reinhart, C. M., Rogoff, K. S. ( 2010 ). Growth in a time of debt . American Economic Review: Papers and Proceedings, 100, 573 - 578 .

Google Scholar Crossref | ISI

Simmons, J. P., Nelson, L. D., Simonsohn, U. ( 2011 ). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant . Psychological Science, 22, 1359 - 1366 .

Google Scholar SAGE Journals | ISI

Simonsohn, U., Nelson, L. D., Simmons, J. P. ( 2014 ). P-curve: A key to the file-drawer . Journal of Experimental Psychology: General, 143, 534 - 547 .

Google Scholar Crossref | Medline | ISI

Sterling, T. ( 1959 ). Publication decisions and their possible effects on inferences drawn from statistical tests—Or vice versa . Journal of the American Statistical Association, 54, 30 - 34 .

Google Scholar ISI

Trouble at the lab. ( 2013 , October 19 ). The Economist. Retrieved from http://www.economist.com/news/briefing/21588057-scientists-think-science-self-correcting-alarming-degree-it-not-trouble

Google Scholar

Welzel, C., Inglehart, R. ( 2009 ). Mass beliefs and democratic institutions . In Boix, C., Stokes, S. C. (Eds.), The Oxford handbook of comparative politics (pp. 297 - 316 ). New York, NY : Oxford University Press .

Google Scholar

Yom, S. ( 2015 ). From methodology to practice: Inductive iteration in comparative research . Comparative Political Studies, 48, 616 - 644 .

Google Scholar SAGE Journals | ISI