Most Useless Phrase in the Political Lexicon: "Peer Reviewed"
Eric Gisin
just love to quote. See next post.
Paging Captain Renault - Research Journal Out for Access Fees
June 10, 2009, 13:58:28 | David Bruggeman
The Scientist is the source of our Casablanca flashback, with its report that an open access
journal published by Bentham was willing to publish a 'nonsense' paper that supposedly passed peer
review. A Ph.D. student in science communications and a staffer at The New England Journal of
Medicine have been testing journals peer review practices by submitting papers generated by
computer program. They document this particular incident on their blog. In short, the journal
agreed to publish the article, if the authors paid the fee, and asserted it had passed peer review.
At a minimum the publisher Bentham is guilty of allowing journals to assert peer review when none
had taken place. The scamming conclusion is reasonable, given the reports. I'm not in agreement
that open access journals are necessarily more suspect of putting out supposedly peer-reviewed
articles that weren't so reviewed. Yes, they do charge more fees than traditional journals (who
could be scamming authors for photo and chart fees, amongst other things), but an open access
journal is not more likely to skimp on peer review than any other journal.
What bothers me is that it has to take generating obviously lousy articles to ferret out derelict
peer review. Given the volume of scientific publishing, there's an enormous amount of implicit
trust in the processes behind these articles that people will continue to exploit. I wish I had
even the germ of a possible solution here.
Eric Gisin
http://www.climate-skeptic.com/2009/02/most-useless-phrase-in-the-polical-lexicon-peer-reviewed.html
Most Useless Phrase in the Political Lexicon: "Peer Reviewed"
February 11, 2009, 08:49:20 | admin
Last week, while I was waiting for my sandwich at the deli downstairs, I was applying about 10% of
my consciousness to CNN running on the TV behind the counter. I saw some woman, presumably in the
Obama team, defending some action of the administration as being based on "peer reviewed" science.
This may be a legacy of the climate debate. One of the rhetorical tools climate alarmists have
latched onto is to inflate the meaning of peer review. Often, folks, like the person I saw on TV
yesterday, use "peer review" as a synonym for "proven correct and generally accepted in its
findings by all right-thinking people who are not anti-scientific wackos." Sort of the scientific
equivalent of "USDA certified."
Here is a great example of that, from the DailyKos via Tom Nelson:
Contact NBC4 and urge them to send weatherman Jym Ganahl to some climate change conferences with
peer-reviewed climatologists. Let NBC4 know that they have a responsibility to have expert
climatologists on-air to debunk Ganahl's misinformation and the climate change deniers don't
deserve an opportunity to spread their propaganda:
NBC 4 phone # 614-263-4444
NBC 4 VP/GM Rick Rogala email: rrogala(ATSIGN)wcmh.com
By the way, is this an over-the-top attack on heresy or what? Let's all deluge a TV station with
complaints because their weatherman has the temerity to have a different scientific opinion than
ours? Seriously guys, its a freaking local TV weatherman in central Ohio, and the fate of mankind
depends on burning this guy at the stake? I sometimes get confused about what leftists really
think about free speech, but this sure sounds more like a bunch of good Oklahoma Baptists reacting
to finding out their TV minister is pro-abortion. But it is we skeptics who are anti-science?
Anyway, back to peer review, you can see in this example again the use of "peer review" as some
kind of impremateur of correctness and shield against criticism. The author treats it as if it
were a sacrament, like baptism or ordination. This certification seems to be so strong in their
mind that just having been published in a peer-reviewed journal seems to be sufficient to complete
the sacrament - the peer review does not necessarily seem to even have to be on the particular
topic being discussed.
But in fact peer review has a much narrower function, and certainly is not, either in intent or
practice, any real check or confirmation of the study in question. The main goals of peer review
are:
a.. Establish that the article is worthy of publication and consistent with the scope of the
publication in question. They are looking to see if the results are non-trivial, if they are new
(ie not duplicative of findings already well-understood), and in some way important. If you think
of peer-reviewers as an ad hoc editorial board for the publication, you get closest to intent
b.. Reviewers will check, to the extent they can, to see if the methodology and its presentation
is logical and clear - not necessarily right, but logical and clear. Their most frequent comments
are for clarification of certain areas of the work or questions that they don't think the authors
answered. They do not check all the sources, but if they are familiar with one of the sources
references, may point out that this source is not referenced correctly, or that some other source
with which they are familiar might be referenced as well. History has proven time and again that
gross and seemingly obvious math and statistical errors can easily clear peer review.
c.. Peer review is not in any way shape or form a proof that a study is correct, or even likely
to be correct. Enormous numbers of incorrect conclusions have been published in peer-reviewed
journals over time. This is demonstrably true. For example, at any one time in medicine, for
every peer-reviewed study I can usually find another peer-reviewed study with opposite or wildly
different findings. The fraud in the "peer reviewed" Lancet on MMR vaccines and autism by Andrew
Wakefield is a good example.
d.. Studies are only accepted as likely correct a over time after the community has tried as hard
as it can to poke holes in the findings. Future studies will try to replicate the findings, or
disprove them. As a result of criticism of the methodology, groups will test the findings in new
ways that respond to methodological criticisms. It is the accretion of this work over time that
solidifies confidence (Ironically, this is exactly the process that climate alarmists want to
short-circuit, and even more ironically, they call climate skeptics "anti-scientific" for wanting
to follow this typical scientific dispute and replication process).
So, typical peer review comments might be:
a.. I think Smith, 1992 covered most of this same ground. I am not sure what is new here
b.. Jones, 1996 is fairly well accepted and came up with opposite conclusions. The authors need
to explain why they think they got different results from Jones.
A typical peer review comment would not be:
a.. The results here looked suspicious so I organized a major effort here at my university and we
spent 6 months trying to replicate their work and cuold not duplicate their findings.
That latter is a follow-up article, not a peer review comment.
Further, the quality and sharpness of peer review depends a lot on the reviewers chosen. For
example, a peer review of Rush Limbaugh by the folks at LGF, Free Republic, and Powerline might not
be as compelling as a peer review by Kos or Kevin Drum.
But instead of this, peer review is used by folks, particularly in political settings, as a shield
against criticism, usually for something they don't understand and probably haven't even read
themselves. Here is an example dialog:
Politician or Activist: "Mann's hockey stick proves humans are warming the planet"
Critic: "But what about Mann's cherry-picking of proxy groups; or the divergence problem in the
data; or the fact that he routinely uses proxy's as a positive correlation in one period and
different, even negative, correlation in another; or the fact that the results are most driven by
proxys that have been manually altered; or the fact that trees really make bad proxies, as they
seldom actually display the assumed linear positive relationship between growth and temperature?"
Politician or Activist, who 99% of the time has not even read the study in question and
understands nothing of what critic is saying: "This is peer-reviewed science! You can't question
that."
Postscript: I am not trying to offend anyone or make a point about religion per se in the
comparisons above. I am not religious, but I don't have a problem with those that are. However,
alarmists on the left often portray skepticism as part-and-parcel of what they see as
anti-scientific ideas tied to the religious right. I get this criticism all the time, which is
funny since I am not religious and not a political conservative. But I find parallels between
climate alarmist and religion to be interesting, and a particularly effective criticism given some
of the left's foaming-at-the-mouth disdain for religion.
Ouroboros Rex
> All the serious climate science blogs have articles debunking the
> "peer review" that alarmists just love to quote.
That must be why you are quoting a denialist lie mill instead. lol
See next post.
>
> http://sciencepolicy.colorado.edu/prometheus/paging-captain-renault-research-journal-out-for-access-fees-5517
>
> Paging Captain Renault - Research Journal Out for Access Fees
> June 10, 2009, 13:58:28 | David Bruggeman
>
> The Scientist is the source of our Casablanca flashback, with its
> report that an open access journal published by Bentham was willing
> to publish a 'nonsense' paper that supposedly passed peer review.
Now where will denialists post their nonsense papers?
Ouroboros Rex
> All the foaming-at-the-mouth trolls who use these tactics put up your
> hand!
> http://www.climate-skeptic.com
Sorry, k00ksite.
Eric Gisin
> Eric Gisin wrote:
>> All the serious climate science blogs have articles debunking the
>> "peer review" that alarmists just love to quote.
>
> That must be why you are quoting a denialist lie mill instead. lol
>
>
> See next post.
>>
>> http://sciencepolicy.colorado.edu/prometheus/paging-captain-renault-research-journal-out-for-access-fees-5517
>>
>> Paging Captain Renault - Research Journal Out for Access Fees
>> June 10, 2009, 13:58:28 | David Bruggeman
>>
>> The Scientist is the source of our Casablanca flashback, with its
>> report that an open access journal published by Bentham was willing
>> to publish a 'nonsense' paper that supposedly passed peer review.
>
> Now where will denialists post their nonsense papers?
Ouroboros Rex
> Skeptics are not denialists.
I'm not talking about skeptics, I'm talking about lying denialists, like
you. Any questions?
Eric Gisin
recovery.
"Ouroboros Rex" <i...@casual.com> wrote in message news:h0r5a...@news3.newsguy.com...
columbiaaccidentinvestigation
serious climate science blogs have articles debunking the "peer
review" that alarmists just love to quote. See next post."
http://sciencepolicy.colorado.edu/prometheus/paging-captain-renault-r...
"Paging Captain Renault - ...Bentham was willing publish a 'nonsense'
paper ...A Ph.D. student in science communications and a staffer at
The New England Journal of Medicine have been testing journals peer
review practices by submitting papers generated by computer program.
In short, the journal agreed to publish the article, if the
authors ...asserted it had passed peer review"
so the authors of the "nonsense paper" were dishonest to the
publishing company, interesting, as the author of the article eric has
posted mentions the one example, but does not publish the methods and
actual results of the so called "tests". (what a bs article)
Lone Ranger
wrote:
>Skeptics are not denialists.
Those denying AGW are indeed denialists.
--
Hi-Yo, Silver! Away!
--
Jerry Okamura
"Lone Ranger" <snowba...@bigfoot.com.spamalamadingdong> wrote in message
news:j9ec655i2ijpsf7u1...@4ax.com...
> On Thu, 11 Jun 2009 07:54:48 -0700, "Eric Gisin" <gi...@uniserve.com>
> wrote:
>
>>Skeptics are not denialists.
>
> Those denying AGW are indeed denialists.
>
>
Jerry Okamura
"Lone Ranger" <snowba...@bigfoot.com.spamalamadingdong> wrote in message
news:j9ec655i2ijpsf7u1...@4ax.com...
> wrote:
>
>>Skeptics are not denialists.
>
> Those denying AGW are indeed denialists.
>
>
Lone Ranger
wrote:
>All the serious climate science blogs have articles debunking
>the "peer review" that alarmists just love to quote.
I guess you don't know what "peer review" means.
Peer review is a process of subjecting an author's scholarly work or
ideas to the scrutiny of others who are experts in the field. The peer
review process aims to make authors meet the standards of their
discipline, and of science in general. Publications that have not
undergone peer review are likely to be regarded with suspicion
by scholars and professionals in many fields.
Put simply, peer review is supposed to weed out poor science.
However, it is not foolproof and a deeply flawed paper can end up
being published under a number of different potential circumstances:
1. The work is submitted to a journal outside the relevant field where
the reviewers lack the expertise to properly review the paper.
2. Too few or too unqualified reviewers are chosen by the editor.
3. The reviewers or editor (or both) have agendas, and overlook flaws
that invalidate the paper's conclusions.
A blog may provide *comments* on peer reviewed papers, and many
blogs do so. But a blog is *not* a scientific publication. That's why
blogs shouldn't be your primary source for "debunking" a peer reviewed
paper.
Lone Ranger
wrote:
>But instead of this, peer review is used by folks, particularly in
>political settings, as a shield against criticism, usually for something
>they don't understand and probably haven't even read themselves.
>
>Here is an example dialog:
>
>Politician or Activist: "Mann's hockey stick proves humans are
>warming the planet"
>
>Critic: "But what about Mann's cherry-picking of proxy groups; or the
>divergence problem in the data; or the fact that he routinely uses proxy's
>as a positive correlation in one period and different, even negative,
>correlation in another; or the fact that the results are most driven by
>proxys that have been manually altered; or the fact that trees really
>make bad proxies, as they seldom actually display the assumed linear
>positive relationship between growth and temperature?"
Criticism of Mann's work takes place in peer reviewd papers. Although
there may be legitimate reasons to criticize some parts of Mann's
work, this criticism has not in any way invalidated his work. On the
contrary, the so-called 'hockey stick' has in fact been validated.
The National Academy of Sciences (NAS) pointed out uncertainties
present in the quantitative assessment of large-scale surface
temperature changes prior to about A.D. 1600, and that the
uncertainties inherent in temperature reconstructions for individual
years and decades are larger than those for longer time periods.
The NAS also agreed that there were statistical shortcomings in
Mann's analysis, but concluded that they were small in effect.
Their conclusion is quite clear:
http://books.nap.edu/openbook.php?record_id=11676&page=115
The basic conclusion of Mann et al. (1998, 1999) was that the late
20th century warmth in the Northern Hemisphere was unprecedented
during at least the last 1,000 years. This conclusion has subsequently
been supported by an array of evidence that includes the additional
large-scale surface temperature reconstructions and documentation
of the spatial coherence of recent warming described above (Cook et
al. 2004, Moberg et al. 2005b, Rutherford et al. 2005, D�Arrigo et al.
2006, Osborn and Briffa 2006, Wahl and Ammann in press) and also the
pronounced changes in a variety of local proxy indicators described in
previous chapters (e.g., Thompson et al. in press).
Based on the analyses presented in the original papers by Mann et al.
and this newer supporting evidence, the committee finds it plausible
that the Northern Hemisphere was warmer during the last few decades of
the 20th century than during any comparable period over the preceding
millennium.
...
More from the NAS report:
Surface Temperature Reconstructions for the Last 2,000 Years (2006)
http://www.nap.edu/catalog.php?record_id=11676
Large-Scale Multiproxy Reconstruction Techniques
http://books.nap.edu/openbook.php?record_id=11676&page=110
EVOLUTION OF MULTIPROXY RECONSTRUCTION TECHNIQUES
http://books.nap.edu/openbook.php?record_id=11676&page=111
Criticisms and Advances of Reconstruction Techniques
http://books.nap.edu/openbook.php?record_id=11676&page=112
http://books.nap.edu/openbook.php?record_id=11676&page=113
http://books.nap.edu/openbook.php?record_id=11676&page=114
STRENGTHS AND LIMITATIONS OF LARGE-SCALE SURFACE
TEMPERATURE RECONSTRUCTIONS
http://books.nap.edu/openbook.php?record_id=11676&page=115
http://books.nap.edu/openbook.php?record_id=11676&page=116
OVERALL FINDINGS AND CONCLUSIONS
http://books.nap.edu/openbook.php?record_id=11676&page=117
WHAT COMMENTS CAN BE MADE ON THE VALUE OF
EXCHANGING INFORMATION AND DATA?
http://books.nap.edu/openbook.php?record_id=11676&page=118
WHAT MIGHT BE DONE TO IMPROVE OUR UNDERSTANDING
OF CLIMATE VARIATIONS OVER THE LAST 2,000 YEARS?
http://books.nap.edu/openbook.php?record_id=11676&page=119
...
In the latest IPCC report you will also find that Mann's work and the
so-called 'hockey stick' is validated. They go through many papers,
some of them critical studies, and point out the uncertainties
associated with temperature reconstructions.
6.6.1.1 What do reconstructions based on palaeoclimatic proxies show?
http://www.ipcc.ch/pdf/assessment-report/ar4/wg1/ar4-wg1-chapter6.pdf#page=34
Their conclusion is similar to that of the NAS:
http://www.ipcc.ch/pdf/assessment-report/ar4/wg1/ar4-wg1-chapter6.pdf#page=42
The weight of current multi-proxy evidence, therefore, suggests
greater 20th-century warmth, in comparison with temperature levels
of the previous 400 years, than was shown in the TAR. On the evidence
of the previous and four new reconstructions that reach back more than
1 kyr, it is likely that the 20th century was the warmest in at least
the past 1.3 kyr. Considering the recent instrumental and longer proxy
evidence together, it is very likely that average NH temperatures
during the second half of the 20th century were higher than for any
other 50-year period in the last 500 years.
More information:
http://en.wikipedia.org/wiki/Hockey_stick_controversy