Press release from the Stockholm School of Economics

Replications of results in experimental economics

A collaborative study between research groups at Caltech, the National University of Singapore, Stockholm School of Economics and the University of Innsbruck replicated experimental economics studies and found that published results in this field are actually quite reliable.

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol and achieve the same result in her laboratory. However, in recent years many results in a variety of disciplines have been questioned for their lack of reproducibility. A new study suggests that published results from experimental economics are better than average when it comes to reproducibility.

The work was published in the March 3 online issue of the journal Science.

"Trying to reproduce previous results is not glamorous or creative, so it is rarely done. But being able to get the same result over and over is part of the definition of what makes knowledge scientific," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech and one of the lead authors on the paper.

The study was based on a previous method used to assess the replication of psychology experiments. In the earlier technique, called the Reproducibility Project Psychology (RPP), researchers replicated 100 original studies published in three of the top journals in psychology--and found that although 97 percent of the original studies reported so-called "positive findings" (meaning a significant change compared to control conditions), such positive findings were reliably reproduced only 36 percent of the time.

Inspired by the RPP project, this international team of researchers reproduced 18 laboratory experimental papers published in two top-tier economics journals between 2011 and 2014. Eleven of the 18--roughly 61 percent--showed a "significant effect in the same direction as in the original study." The researchers also found that the sample size and p-values--a standard measure of statistical confidence--of the original studies were good predictors for the success of replication, meaning they could serve as good indicators for the reliability of results in future experiments.

"This project was inspired by the path-breaking reproducibility project in psychology led by Brian Nosek. It is extremely important to investigate to what extent we can trust published scientific findings and to implement institutions that promotes scientific reproducibility,” says Magnus Johannesson from the Stockholm School of Economics and one of the lead authors.

Anna Dreber, another lead author from the Stockholm School of Economics, adds, “In the future I hope that there will be less of these big replication projects simply because the original papers will already contain replications. There are a few recent examples of such papers in psychology, but we need more, and they are not common in experimental economics yet”.

The work was published in a paper titled, " Evaluating Replicability of Laboratory Experiments in Economics." Other coauthors are: Taisuke Imai and Gideon Nave from Caltech; Johan Almenberg from Sveriges Riksbank in Stockholm; Eskil Forsell, Adam Altmejd, Emma Heikensten, and Siri Isaksson from the Stockholm School of Economics; Teck-Hua Ho, Taizan Chan, and Hang Wu from the National University of Singapore; Felix Holzmeister, Jurgen Huber, Michael Kirchler and Michael Razen from the University of Innsbruck; and Thomas Pfeiffer from the New Zealand Institute for Advanced Study.

###

The study was funded by the Austrian Science Fund, the Austrian National Bank, the Behavioral and Neuroeconomics Discovery Fund, the Jan Wallander and Tom Hedelius Foundation, the Knut and Alice Wallenberg Foundation, the Swedish Foundation For Humanities and Social Sciences, and the Sloan Foundation.

For further information contact:

Professor Magnus Johannesson
Magnus.Johannesson@hhs.se
+46-8-736 9443

Associate Professor Anna Dreber Almenberg
Anna Dreber@hhs.se
+46-8-736 9646