The Unreliability of Scientific Studies

Jonah Lehrer has an important piece in the New Yorker on the "decline effect" -- the fact that over time attempts to replicate the results of many well-established scientific studies increasingly fail. It is important to note that these studies have been thoroughly reviewed, were published in top journals, and were successfully replicated shortly after the time of their publication. Many of them have made their way into textbooks, and some are considered classics of their field. Yet, they have all been falsified in subsequent trials. This is a serious problem for science because it calls into question its fundamental tenets. How can an experimental result can be confirmed time and again, but later consistently falsified? In part, the answer would seem to be that science is not, and can never be, a completely objective undertaking. Subjectivity and bias enter into even the most rigorously designed studies.

But the importance of the decline effect is greater than a mere demonstration of scientific subjectivity. Lehrer writes:

[T]he decline effect is... troubling. Not because it reveals the human fallibility of science, in which data are tweaked and beliefs shape perceptions. (Such shortcomings aren’t surprising, at least for scientists.) And not because it reveals that many of our most exciting theories are fleeting fads and will soon be rejected. (That idea has been around since Thomas Kuhn.) The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.

Indeed! 

Read the whole article here.