The Decline And Fall of Effects In Science

Nature has a piece called Unpublished results hide the decline effect.
This refers to the fact that many scientific findings which seem to indicate something big is happening, end up getting smaller and smaller as more people try to replicate them until they, eventually, may vanish entirely.

The Last Psychiatrist's take is that "The Decline Effect" just represents sloppy thinking, treating different things as if they were all instances of The One True Phenomenon. Someone does a study about something and finds an effect. Then someone else comes along and does a new study, of a related but different topic, and finds a different result. Both are right: there's a difference. Only if you, sloppily, decide that both studies were measuring the same thing does the "Decline Effect" appear.

This is perfectly true and I've touched on it before, but I think it's a bit optimistic. It assumes that the first study was true. Sometimes they are. But because of the way science is published at the moment, a lot of results that get published are flukes. Some even say that the majority are.

The problem is that there are so many ways to statistically analyze any given body of data that it's easy to test and retest it until you find a "positive result" - and then publish that, without saying (or only saying in the small print) that your original tests all came out negative. Combine this with selective publication of only the best data, and other scientific sins, and you can pull positive results out the hat of mere random noise.

In the Nature article, Jonathan Schooler discusses this and suggests that an open-access repository of findings (meaning raw data rather than the end product of analyses) would be A Good Thing. I agree. However, he seems to think that if we did this, we might still observe the "Decline Effect", and would be able to find out more about it. He even seems to suggest that some kind of weird quantum effect might mean that scientists are actually changing the laws of reality by observing them

Perhaps, just as the act of observation has been suggested to affect quantum measurements, scientific observation could subtly change some scientific effects. Although the laws of reality are usually understood to be immutable, some physicists, including Paul Davies, director of the BEYOND: Center for Fundamental Concepts in Science at Arizona State University in Tempe, have observed that this should be considered an assumption, not a foregone conclusion.

Hmm. Maybe. But there is really no need to posit such magical mysteries when plain old statistical conjuring tricks seem like a perfectly good explanation. On my view a raw result repository would not explain the decline effect, but just make it disappear.

Schooler doesn't go into detail as to how this repository would be set up, but he does cite the fact that we already have a pretty good one for clinical trials of medicines conducted in the USA. Anyone running a clinical trial is required to register it in advance, saying what they're planning to do and crucially, to spell out which statistics they are going to run on the data when it arrives.

What's really silly is that most scientists already do this when applying for funding: most grant applications include detailed statistical protocols. The problem is that these are not made public so people can ignore them when it comes to publication. Back in 2008 I suggested that scientific journals should require all studies, not just clinical trials, to be publicly pre-registered if they're to be considered for publication. This would be eminently do-able if there was a will to make it happen.

Schooler, J. (2011). Unpublished results hide the decline effect Nature, 470 (7335), 437-437 DOI: 10.1038/470437a