In a unique study, the researchers show that experts in psychology are to a large extent correct when betting on which scientific results are reliable. The betting, which takes the form of a prediction market involving real money, can also be used to demonstrate the probability with which the outcome of an experiment is reliable or not.

“Reporting research results based solely on a single experiment is often not enough to indicate a connection. To be certain that the results are sustainable, they must be repeated multiple times. Nonetheless, individual research results that cannot be reproduced in repeated studies are often published in respected journals,” says Anna Dreber, Associate Professor in Economics at the Stockholm School of Economics.
Alongside others including Magnus Johannesson and Siri Isaksson at the School of Economics, Anna Dreber used betting to calculate the extent to which research studies results can be replicated in other studies.

Participants bet on whether different results could be repeated in other studies, and were given real money with which they could buy and sell shares, with the shares representing whether or not specific results could be repeated. The psychology experts’ bets managed to predict replication by 71 percent, which was significantly better than the outcome of a survey of the same participants.

“Well reputed psychology journals seem to focus on publishing surprising results rather than reliable results,” says Anna Dreber. “And these surprising results do not always hold up under re-testing. At the same time, there is some form of collective wisdom among researchers as to which results are reliable, raising the hope that the proportion of invalid results published could be reduced.”
The research results have been published in The Proceedings of the National Academy of Sciences (PNAS) and have been funded by the Jan Wallander and Tom Hedelius Foundation, the Knut and Alice Wallenberg Foundation and Riksbankens Jubileumsfond (the Swedish Foundation for the Humanities and Social Sciences). Other researchers in the project include Thomas Pfeiffer at the New Zealand Institute for Advanced Study, Johan Almenberg at the Riksbank (Sweden’s central bank), Brad Wilson at Consensus Point, Yiling Chen at Harvard University and Brian A. Nosek at Virginia University.

For further information, please contact:
Anna Dreber, Associate Professor in Economics at the Stockholm School of Economics
Anna.Dreber@hhs.se
+46 8 736 90 00