(C)
“I see a train wreck looming,” warned Daniel Kahneman, an eminent psychologist,
in an open letter last year. The premonition concerned research on a phenomenon known
as “priming.” Priming studies suggest that decisions can be influenced by apparently
irrelevant actions or events that took place just before the cusp of choice. They have
been a boom area in psychology over the past decade, and some of their insights have
already made it out of the lab and into the toolkits of policy wonks keen on “nudging”
the populace. Dr. Kahneman and a growing number of his colleagues fear that a lot of this
priming research is poorly founded. Over the past few years various researchers have
made systematic attempts to replicate some of the more widely cited priming
experiments. Many of these replications have failed. In April, for instance, a paper in
PLoS ONE, a journal, reported that nine separate experiments had not managed to
reproduce the results of a famous study from 1998 purporting to show that thinking
about a professor before taking an intelligence test leads to a higher score than
imagining a football hooligan.
The idea that the same experiments always get the same results, no matter who
performs them, is one of the cornerstones of science’s claim to objective truth. If a
systematic campaign of replication does not lead to the same results, then either the
original research is flawed (as the replicators claim) or the replications are (as many of
the original researchers on priming contend). Either way, something is awry.
It is tempting to see the priming fracas as an isolated case in an area of
science—psychology—easily marginalized as soft and wayward. But irreproducibility is
much more widespread. A few years ago scientists at Amgen, an American drug
company, tried to replicate 53 studies that they considered landmarks in the basic
science of cancer, often co-operating closely with the original researchers to ensure that
their experimental technique matched the one used first time round. According to a
piece they wrote last year in Nature, a leading scientific journal, they were able to
reproduce the original results in just six. Months earlier Florian Prinz and his colleagues
at Bayer HealthCare, a German pharmaceutical giant, reported in Nature Reviews Drug
Discovery, a sister journal, that they had successfully reproduced the published results in
just a quarter of 67 seminal studies.
The governments of the OECD, a club of mostly rich countries, spent $59 billion
on biomedical research in 2012, nearly double the figure in 2000. One of the
justifications for this is that basic-science results provided by governments form the
basis for private drug-development work. If companies cannot rely on academic
research, that reasoning breaks down. When an official at America’s National Institutes
of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce
at least three-quarters of all published biomedical findings, the public part of the process
seems to have failed.
Academic scientists readily acknowledge that they often get things wrong. But they
also hold fast to the idea that these errors get corrected over time as other scientists try
to take the work further. Evidence that many more dodgy results are published than are
subsequently corrected or withdrawn calls that much-vaunted capacity for
self-correction into question. There are errors in a lot more of the scientific papers being
published, written about and acted on than anyone would normally suppose, or like to
think.
Various factors contribute to the problem. Statistical mistakes are widespread. The
peer reviewers who evaluate papers before journals commit to publishing them are much
worse at spotting mistakes than they or others appreciate. Professional pressure,
competition and ambition push scientists to publish more quickly than would be wise. A
career structure which lays great stress on publishing copious papers exacerbates all
these problems. “There is no cost to getting things wrong,” says Brian Nosek, a
psychologist at the University of Virginia who has taken an interest in his discipline’s
persistent errors. “The cost is not getting them published.”