Re: the scientific method (and `peer review') - this seems
interesting (thanks to Billy Cox at Herald-Tribune Dec 20th)
Richard Palmer, biologist (University of Alberta) - "We cannot
escape the troubling conclusion that some perhaps many cherished
generalities are at best exaggerated in their biological
ANNALS OF SCIENCE: THE TRUTH WEARS OFF - Is there something wrong
with the scientific method?
by Jonah Lehrer
DECEMBER 13, 2010
Jonah Lehrer, Annals of Science, “The Truth Wears Off,” The New
Yorker, December 13, 2010, p. 52
Read the full text of this article at http://crayz.org/science.pdf
Scientific Experiments; Decline Effect; Replicability;
Scientists; Statistics; Jonathan Schooler; Scientific Theories
ABSTRACT: ANNALS OF SCIENCE about the decline effect.
On September 18, 2007, a few dozen neuroscientists,
psychiatrists, and drug-company executives gathered in a hotel
conference room in Brussels to hear some startling news. It had
to do with a class of drugs known as atypical or
second-generation antipsychotics, which came on the market in the
early nineties. The therapeutic power of the drugs appeared to be
steadily falling. A recent study showed an effect that was less
than half of that documented in the first trials, in the early
Before the effectiveness of a drug can be confirmed, it must be
tested again and again. The test of replicability, as it’s known,
is the foundation of modern research. It’s a safeguard for the
creep of subjectivity.
But now all sorts of well-established, multiply confirmed
findings have started to look increasingly uncertain. It’s as if
our facts are losing their truth. This phenomenon doesn’t yet
have an official name, but it’s occurring across a wide range of
fields, from psychology to ecology. When Jonathan Schooler was a
graduate student at the University of Washington, he discovered a
surprising phenomenon having to do with language and memory that
he called verbal overshadowing. While Schooler was publishing his
results in journals, he noticed that it was proving difficult to
replicate his earlier findings.
Mentions psychologist Joseph Banks Rhine, who conducted several
experiments dealing with E.S.P.
In 2004, Schooler embarked on an imitation of Rhine’s research
in an attempt to test the decline effect.
The most likely explanation for the decline is an obvious one:
regression to the mean. Yet the effect’s ubiquity seems to
violate the laws of statistics. Describes Anders Møller’s
discovery of the theory of fluctuating asymmetry in sexual
Mentions Leigh Simmons and Theodore Sterling.
Biologist Michael Jennions argues that the decline effect is
largely a product of publication bias.
Biologist Richard Palmer suspects that an equally significant
issue is the selective reporting of results—that is, the subtle
omissions and unconscious misperceptions, as researchers struggle
to make sense of their results.
Mentions John Ioannidis.
In the late nineteen-nineties, neuroscientist John Crabbe
investigated the impact of unknown chance events on the test of
replicability. The disturbing implication of his study is that a
lot of extraordinary scientific data is nothing but noise.
This suggests that the decline effect is actually a decline of
illusion. Many scientific theories continue to be considered true
even after failing numerous experimental tests. The decline
effect is troubling because it reminds us how difficult it is to