Auditing the auditors #1

107 views
Skip to first unread message

Josh Halpern

unread,
Dec 14, 2003, 11:04:18 PM12/14/03
to
Well, time consuming though I knew it would be I have started to audit
the M&M audit. Anyone who seriously wants to join in should go to
http://www.climate2003.com/ to about the middle of the page where it
says Audit Issues. I will be going through these as time allows, but
fair warning, it took about four hours (including a visit to a local
library that had the necessary journals, they were not on line) to get
through #1. It's taking about 1.5 hours to do this, because I have to
relocate a number of web sites, etc.

Some preliminaries. The file sent by Mann's group to McIntyre is called
pcproxy.txt and can be found at http://www.climate2003.com. To save
space, I will call this site C2300. Also to save space, unless
specifically noted, I will use pcproxy to refer to the file pcproxy.txt
as found at C2300. The file was compiled by Scott Rutherford.
Rutherford also sent a roster showing which proxys were in each column
of pcproxy. You can find this at
http://www.climate2003.com/data/backto1820.txt. This file is a
modification of the file multiproxy.inf sitting on
ftp://holocene.evsc.virginia.edu/pub/MBH98 which I will call /pub/MBH98
to save space. multiproxy.inf lists the input files for the multiproxy
analysis. backto1820.txt simply adds a serial number to the file names
in order.

The issues of C2300 refer to the file pcproxy and the paper MBH98 as
published in Nature, we will call that MBH98 and take care to
differentiate it from the ftp site /pub/MBH98. We will call the
published supplemental materials NSM.

OK, from C2300 inquiring minds wish to know.

#1 is Does the database contain truncations of series 10, 11 and 100?
(and of the version of series 65 used by MBH98)?

which is actually three questions, but wth. Let us call them 1a, 1b and
1c. Note that this is a point that has been heavily relied on in the
criticism of MBH98.

1a. #10 is the Central England historical record, originally published
by Manley. pcproxy lists values between 1730 and 1987.

The data base is maintained by the Hadley center and can be downloaded
from
http://www.met-office.gov.uk/research/hadleycentre/CR_data/Monthly/HadCET_act.txt
and the series runs from 1659 to the present.

The reference given in the supplemental materials of MBH98 are
sufficient to locate this source. The header on the data file is:

MONTHLY MEAN CENTRAL ENGLAND TEMPERATURE (DEGREES C)
1659-1973 MANLEY (Q.J.R.METEOROL.SOC., 1974)
1974ON PARKER ET AL. (INT.J.CLIM., 1992)


There is at least one Hadley Center web page which links to the data set
http://www.met-office.gov.uk/research/hadleycentre/obsdata/CET.html

That page has the following references at the bottom

Manley, G., 1953: The mean temperature of Central England, 1698 to 1952.
Quart J Roy Meteorol Soc, 79, 242-261.

Manley, G. 1974: Central England Temperatures: monthly means 1659 to
1973. Quart J Roy Meteorolol Soc, 100, 389-405.

Parker, D. E., T. P. Legg, and C. K. Folland, 1992: A new daily Central
England Temperature Series, 1772-1991. Int J Climatol, 12, 317-342.

I went and got the last two today, and read them. The reason that MBH98
used does not use the data before 1730 is clearly explained in the first
paragraph of the 1992 Parker, Legg and Folland paper:

"Manley1953) published a time series of monthly mean temperatures
representative of central England for 1698-1952, followed (Manley 1974)
by an extended and revised series for 1659-1973. Up to 1814 his data
are based mainly on overlapping sequences of observations from a variety
of carefully chosen and documented locations. Up to 1722, available
instrumental records fail to overlap and Manley needs to use
non-instrumental series for Utrecht compiled by Labrijn (1945), in order
to mate the monthly central England temperature (CET) series
complete.Between 1723 and the 1760s there are no gaps in the composite
instrumental record, but the observations generally were taken in
unheated rooms rather than with a truly outdoor exposure...."

Which means that the Manley reconstruction is only continuous from 1722
on, but the information upon which it relies from 1723-28 has further
difficulties, essentially absolute values were not reliable, and the
series was constructed by taking the difference between measurements
made by those thermometers and ones thought to be more reliable after
1727, and then repeatedly differenced to get values before 1727.

In the light of this, it is perfectly reasonable to truncate the CET
series at 1730. On the other end, 1987 is the cut off for the
analysis. We note that the cut offs were listed in the NSM.

On balance this illustrates the principal of RTFR (Read the effing
references) and the danger of someone unfamiliar with an area trying to
do an "audit". To use the CET before 1730 would clearly have been a
mistake, and I think you could make a good argument that it should have
been cut off at 1772.

1b. #11 is the Central Europe historical record, originally published
by Pfister. pcproxy lists values between 1550 and 1987. The series
goes to 1525. The data can be picked up from
ftp://ftp.ngdc.noaa.gov/paleo/historical/switzerland/clinddef.txt
and information on the compilation can be found in the same folder as
readme_swissindices.txt

The references for this data are:

Pfister, C. (1984): Das Klima der Schweiz von 1525-1860 und seine Bedeutung
in der Geschichte von Bevoelkerung und Landwirtschaft. Bern

Pfister, C. (1992): Monthly temperature and precipitation patterns in Central
Europe from 1525 to the present. A methodology for quantifying man made
evidence on weather and climate. In: Bradley R.S., Jones P.D. (eds.)
Climate since 1500A.D., pp. 118-143. London

Pfister C., Kington J., Kleinlogel G., Schuele H., Siffert E. (1994):
The creation of high resolution spatio- temporal reconstructions of past
climate from direct meteorological observations and proxy data.
Methodological considerations and results. In: Frenzel, B., Pfister C.,
Glaeser, B. (eds), Climate in Europe 1675-1715.


of which I could locate the second. On page 121, second paragraph from
the bottom, one reads:

"The evidence increases in volume, density and diversity over time. For
the period 1525-1549 the entries originate mainly from chronicles and
annals. Accordingly, weather sequences are mainly described at a
seasonal level; information is missing for 43% of the months and the
enphasis is on anomalous rather than ordinary weather. In the second
period 1550-1658 monthly data from weather diaries and personal papers
are abundant....."

Again, On balance this illustrates the principal of RTFR and the danger
of someone unfamiliar with an area trying to do an "audit". To use the
central European index before 1550 would clearly have been a mistake.
Note also that Bradley was one of the two book editors, so he surely
knew a great deal about these series. Phil Jones was the other.

1c #100 is a chinese tree ring series which can be found from the
NOAA paleoclimate site
ftp://ftp.ngdc.noaa.gov/paleo/treering/chronologies/asia/chin004.crn
that runs from 1540-1989. You can also find this data set from the noaa
paleoclimate site http://www.ngdc.noaa.gov/paleo/.

1540-1989 is the period for this data set that one finds in the NSM.
In pcproxy the first two years (1540 and 1541) are absent. Two files,
called chin04.txt and chin04-orig.txt are found in the
/pub/MBH98/TREE/ITRDB/MISC folder. They are identical to each other and
appear identical to the data set at the noaa paleo site.

In short, this is almost certainly a copying error in the construction
of pcproxy.

The urge to snark is strong, beware the jabberwock my son.

josh halpern
<ftp://holocene.evsc.virginia.edu/pub/MBH98/TREE/ITRDB/MISC/chin04-orig.txt>

Phil Hays

unread,
Dec 14, 2003, 11:39:26 PM12/14/03
to
Josh Halpern wrote:

> Well, time consuming though I knew it would be I have started to audit
> the M&M audit. <Rest Snipped>

Thank you for your effort.


--
Phil Hays

Jim Norton

unread,
Dec 15, 2003, 9:11:45 AM12/15/03
to

Thank you very much.


Roger Coppock

unread,
Dec 15, 2003, 9:32:21 AM12/15/03
to
Why all the interest in Mann et all 1998, in 2003?
Mann has published several times since then, even
this year. Why aren't M&M 'auditing' more recent
work?


--

"One who joyfully guards his mind
And fears his own confusion
Can not fall.
He has found his way to peace."

-- Buddha, in the "Pali Dhammapada,"
~5th century BCE


-.-. --.- Roger Coppock (rcop...@adnc.com)


-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----

Steve Schulin

unread,
Dec 15, 2003, 10:09:10 AM12/15/03
to
In article <6paDb.15160$gk1....@nwrddc01.gnilink.net>,
Josh Halpern <j.ha...@incoming.verizon.net> wrote:

Your abc breakdown ignores the M&M question about series 65?

Interesting info, Josh. Thanks. From what you've reported, it looks like
some other truncation point would have been more perfect than 1730,
however: 1723, 1728, or somewhere in 1760s. Your use of the root perfect
seems an exaggeration in this case. I agree that the supplemental info
lists 1730, but note that the column is described as showing the "first
year of record available". That does not seem like an accurate
description of the start date used by Mann et al. for this series.

I think you're on the right track in answering M&M's question 1a. It
seems likely that Mann et al did indeed decide to truncate the early
years, including a quite cold series of values, from this series. I
don't recall Mann et al discussing any systematic approach to validating
the data in all the 112 or 159 or however many there were series.

I think you're again on the right track in answering M&M's question 1b.
It seems likely that Mann et al did indeed decide to truncate the early
years from this series.

>
> 1c #100 is a chinese tree ring series which can be found from the
> NOAA paleoclimate site
> ftp://ftp.ngdc.noaa.gov/paleo/treering/chronologies/asia/chin004.crn
> that runs from 1540-1989. You can also find this data set from the noaa
> paleoclimate site http://www.ngdc.noaa.gov/paleo/.
>
> 1540-1989 is the period for this data set that one finds in the NSM.
> In pcproxy the first two years (1540 and 1541) are absent. Two files,
> called chin04.txt and chin04-orig.txt are found in the
> /pub/MBH98/TREE/ITRDB/MISC folder. They are identical to each other and
> appear identical to the data set at the noaa paleo site.

chin04.txt is not identical to chin04-orig.txt -- the first two years of
the series, high values both, appear in chin04-orig.txt but not
chin04.txt

>
> In short, this is almost certainly a copying error in the construction
> of pcproxy.

It's not clear whether you think Mann et al actually used the 1540-1541
values for this time series in producing the hockey stick.


>
> The urge to snark is strong, beware the jabberwock my son.

M&M ask reasonable questions under the heading of issue #1. Some of what
you write here is illuminatory, some is obfuscatory, and some looks just
snarky itself.


>
> josh halpern
> <ftp://holocene.evsc.virginia.edu/pub/MBH98/TREE/ITRDB/MISC/chin04-orig.txt>
>

David Ball

unread,
Dec 15, 2003, 11:08:27 AM12/15/03
to

Josh spent 4 hours getting the information he posted. What are
you doing, besides posting "what have you done lately" comments? Is
there anything preventing you from getting off your ass and doing some
of the legwork or do you plan to just sit here and complain about
someone else's efforts?
What I find illuminating about Josh's post is that the basic
information about why certain series were truncated is available in
the reference material. When the answer to each question posed by M&M
boils down to their inability to RTFR, there is a problem. That likely
stems from the unseemly rush they were in to get something in print. I
can't think of a researcher anywhere who can knock off a peer-reviewed
paper in a matter of weeks, without taking some serious shortcuts and
at least in the case of their question 1, it appears that those
shortcuts caused them to overlook basic information needed to do their
"audit".

>>
>> Which means that the Manley reconstruction is only continuous from 1722
>> on, but the information upon which it relies from 1723-28 has further
>> difficulties, essentially absolute values were not reliable, and the
>> series was constructed by taking the difference between measurements
>> made by those thermometers and ones thought to be more reliable after
>> 1727, and then repeatedly differenced to get values before 1727.
>>
>> In the light of this, it is perfectly reasonable to truncate the CET
>> series at 1730. On the other end, 1987 is the cut off for the
>> analysis. We note that the cut offs were listed in the NSM.
>>
>> On balance this illustrates the principal of RTFR (Read the effing
>> references) and the danger of someone unfamiliar with an area trying to
>> do an "audit". To use the CET before 1730 would clearly have been a
>> mistake, and I think you could make a good argument that it should have
>> been cut off at 1772.
>
>Interesting info, Josh. Thanks. From what you've reported, it looks like
>some other truncation point would have been more perfect than 1730,
>however: 1723, 1728, or somewhere in 1760s. Your use of the root perfect
>seems an exaggeration in this case. I agree that the supplemental info
>lists 1730, but note that the column is described as showing the "first
>year of record available". That does not seem like an accurate
>description of the start date used by Mann et al. for this series.

And you are basing this on what? Josh's point about the
problematic nature of the 1723-1728 data is well taken. Using any of
the data in that period would be a mistake. There is nothing
inherently wrong with using 1730 - it's a nice round number - and as
Josh pointed out other truncation points could have been adopted.
Let's face it, the proxy data are all reconstructed back to nice round
numbers. In the tree ring data, why go back to 1700? Why not 1699? Or
1697? It's a computational convenience, nothing more.
Your arbitrary choice of a point in the 1760's is noted. In
addition, your quibbling about the description of 1730 being the first
year or record is just that: a quibble. When an analysis is done, bad
data are routinely thrown out. That's a fact of life. The paper in
question is NOT meant for the general public but for other experts in
the field, experts who persumably are, unlike M&M, aware of the
problematic nature of some of the data. That being said, such a
notation can be considered a given.

>
>>
>> 1c #100 is a chinese tree ring series which can be found from the
>> NOAA paleoclimate site
>> ftp://ftp.ngdc.noaa.gov/paleo/treering/chronologies/asia/chin004.crn
>> that runs from 1540-1989. You can also find this data set from the noaa
>> paleoclimate site http://www.ngdc.noaa.gov/paleo/.
>>
>> 1540-1989 is the period for this data set that one finds in the NSM.
>> In pcproxy the first two years (1540 and 1541) are absent. Two files,
>> called chin04.txt and chin04-orig.txt are found in the
>> /pub/MBH98/TREE/ITRDB/MISC folder. They are identical to each other and
>> appear identical to the data set at the noaa paleo site.
>
>chin04.txt is not identical to chin04-orig.txt -- the first two years of
>the series, high values both, appear in chin04-orig.txt but not
>chin04.txt
>
>>
>> In short, this is almost certainly a copying error in the construction
>> of pcproxy.
>
>It's not clear whether you think Mann et al actually used the 1540-1541
>values for this time series in producing the hockey stick.
>
>>
>> The urge to snark is strong, beware the jabberwock my son.
>
>M&M ask reasonable questions under the heading of issue #1. Some of what
>you write here is illuminatory, some is obfuscatory, and some looks just
>snarky itself.
>

And the reasonable responses given by Josh show that the
information M&M are inquiring about was available in the paper's
reference material, material that they apparently didn't bother to
read. That isn't a snark, nor is it obfuscatory. It is simply the
truth. Nor are your minor quibbles adding anything to the discussion,
except noise.

Steve Schulin

unread,
Dec 15, 2003, 12:54:41 PM12/15/03
to
In article <qjlrtvg04lhu6ghd2...@4ax.com>,
David Ball <wra...@mb.sympatico.ca> wrote:

> you doing, besides posting "what have you done lately" comments? ...

My question was prompted because I got the impression that Josh was
attempting to fully address the questions in issue #1. His use of the
phrase "to get through #1" contributed to that impression. My question
was politely stated. If you had been asking question to me under similar
circumstances, it would have been no surprise to see you be quite
abusive about only seeing abc instead of abcd.

> ... Is


> there anything preventing you from getting off your ass and doing some
> of the legwork or do you plan to just sit here and complain about
> someone else's efforts?

There's plenty of legwork to be done, that's for sure. I had done enough
legwork previously to recognize that Josh was incorrect in stating,
below, that the chin04.txt and chin04-orig.txt files at ftp site were
identical. Are you not grateful to have that error corrected?

> What I find illuminating about Josh's post is that the basic
> information about why certain series were truncated is available in
> the reference material. When the answer to each question posed by M&M

> boils down to their inability to RTFR, there is a problem. ...

It seems clear enough that Mann et al chose to use subsets of series 10
and 11. There is no indication that they systematically applied some
evaluative methodology to all the series. And your suggestion that "the

answer to each question posed by M&M boils down to their inability to

RTFR" seems quite an exaggeration.

> ... That likely


> stems from the unseemly rush they were in to get something in print. I
> can't think of a researcher anywhere who can knock off a peer-reviewed
> paper in a matter of weeks, without taking some serious shortcuts and
> at least in the case of their question 1, it appears that those
> shortcuts caused them to overlook basic information needed to do their
> "audit".

You have often mistaken question marks for exclamation points, so your
take on the current matter comes as no surprise. Your apparent argument
that "of course Mann et al modified the source data" seems quite at odds
with the impression given by all those who have been echoing Mann's
recent claim that the data necessary to replicate their work is
available from the referenced sources of the data.

>
> >>
> >> Which means that the Manley reconstruction is only continuous from 1722
> >> on, but the information upon which it relies from 1723-28 has further
> >> difficulties, essentially absolute values were not reliable, and the
> >> series was constructed by taking the difference between measurements
> >> made by those thermometers and ones thought to be more reliable after
> >> 1727, and then repeatedly differenced to get values before 1727.
> >>
> >> In the light of this, it is perfectly reasonable to truncate the CET
> >> series at 1730. On the other end, 1987 is the cut off for the
> >> analysis. We note that the cut offs were listed in the NSM.
> >>
> >> On balance this illustrates the principal of RTFR (Read the effing
> >> references) and the danger of someone unfamiliar with an area trying to
> >> do an "audit". To use the CET before 1730 would clearly have been a
> >> mistake, and I think you could make a good argument that it should have
> >> been cut off at 1772.
> >
> >Interesting info, Josh. Thanks. From what you've reported, it looks like
> >some other truncation point would have been more perfect than 1730,
> >however: 1723, 1728, or somewhere in 1760s. Your use of the root perfect
> >seems an exaggeration in this case. I agree that the supplemental info
> >lists 1730, but note that the column is described as showing the "first
> >year of record available". That does not seem like an accurate
> >description of the start date used by Mann et al. for this series.
>

> And you are basing this on what? ...

On the meaning of the phrase "first year of record available", as
opposed to the meaning of "first year of record chosen to be included
although prior years were available".

> ... Josh's point about the
> problematic nature of the 1723-1728 data is well taken. ...

I too think it may very well have been reasonable to truncate this
series. I find it quite silly that you apparently chastise M&M for
asking the question, however.

> ... Using any of


> the data in that period would be a mistake. There is nothing
> inherently wrong with using 1730 - it's a nice round number - and as

> Josh pointed out other truncation points could have been adopted...

And once its verified what Mann et al actually did, the effects of other
treatments will be interesting to explore.

> ...


> Let's face it, the proxy data are all reconstructed back to nice round
> numbers. In the tree ring data, why go back to 1700? Why not 1699? Or

> 1697? It's a computational convenience, nothing more. ...

Are you saying that other time series were truncated by Mann et al
besides the four mentioned in issue #1?

> ...
> Your arbitrary choice of a point in the 1760's is noted. ...

I quite purposefully used the precision quoted from the reference by
Josh. How would you improve upon my choice of language?

> ... In


> addition, your quibbling about the description of 1730 being the first

> year or record is just that: a quibble. ...

I thought about elaborating, but didn't want to clutter. In the body of
the Nature paper, Mann et al claim to have included five series in their
analysis that apparently were not included. It is in the context of such
apparent sloppiness that M&M's questions warrant clear answers, untinged
by the insults and invective that have become rote from Mann and his
apologists.

> ... When an analysis is done, bad


> data are routinely thrown out. That's a fact of life. The paper in
> question is NOT meant for the general public but for other experts in
> the field, experts who persumably are, unlike M&M, aware of the
> problematic nature of some of the data. That being said, such a
> notation can be considered a given.

So you think it inappropriate that IPCC used the hocky stick graph in
Fig 1 of the WG1 TAR policymaker summary?

I've seen how prone some folks here are to exaggerate. I thought it
worthwhile to nip some of that in the bud in this case.

David Ball

unread,
Dec 15, 2003, 10:00:35 PM12/15/03
to
On Mon, 15 Dec 2003 04:04:18 GMT, Josh Halpern
<j.ha...@incoming.verizon.net> wrote:

Hi, Josh,

I took a quick look at M&M's second question:

2. Are the 1980 values of series #73 through #80 identical to
7 decimal places? Similarly for the 1980 values of series #81-83? And
for the 1980 values of series #84 and #90-92? What is the reason for
this?

I went to the PC data on Mann's website in

ftp://holocene.evsc.virginia.edu/pub/MBH98/TREE/STAHLE/SWM/BACKTO_1700

which encompasses series 73 through 80 and compared the series to
those highlighted by M&M in their excel spreadsheet. The short answer
is that the spreadsheet data are shifted one row up on each of the
PC's. In other words, the 1980 data given on the above ftp site are
shown in the spreadsheet as being in 1979. The 1979 are shifted to
1978 and so on. This is true for all of the PC's. I'm not sure where
the 0.023030 that appears in the 1980 position on the spreadsheet
comes from, but it looks like an error in putting the spreadsheet
together. Certainly, the data on the Mann website are consistent with
what is on the spreadsheet except for this point.
The same thing appears to have happened with series 81-83 in
the spreadsheet when compared to the data at:

ftp://holocene.evsc.virginia.edu/pub/MBH98/TREE/VAGANOV/BACKTO_1750

as well as series 84 and 90-92 at:

ftp://holocene.evsc.virginia.edu/pub/MBH98/TREE/ITRDB/NOAMER/BACKTO_1750

Each series is shifted down a year in the spreadsheet. Again, I don't
know where the 1980 data come from, but it looks like an artifact of
putting the spreadsheet together and nothing nefarious at all.

Dave.


Josh Halpern

unread,
Dec 15, 2003, 10:37:39 PM12/15/03
to

David Ball wrote:

Thanks, I was going to do that..........

Well that was the point of starting.

While I am at this, let me acknowledge that Steve Sculin was correct
about the chin004.txt and chin004-orig.txt. Which means that we still
have to understand if there was any reason for deleting the first two
years of the series, which probably means that I am going to have to
hunt down the original reference. The state of play wrt to that point
is that MBH HAD the complete file but they also had a file with two
years truncated at the beginning

josh halpern

>
>
>
>

Nigel Persaud

unread,
Dec 15, 2003, 11:05:33 PM12/15/03
to
> 1c #100 is a chinese tree ring series which can be found from the
> NOAA paleoclimate site
> ftp://ftp.ngdc.noaa.gov/paleo/treering/chronologies/asia/chin004.crn
> that runs from 1540-1989. You can also find this data set from the noaa
> paleoclimate site http://www.ngdc.noaa.gov/paleo/.
>
> 1540-1989 is the period for this data set that one finds in the NSM.
> In pcproxy the first two years (1540 and 1541) are absent. Two files,
> called chin04.txt and chin04-orig.txt are found in the
> /pub/MBH98/TREE/ITRDB/MISC folder. They are identical to each other and
> appear identical to the data set at the noaa paleo site.
>
> In short, this is almost certainly a copying error in the construction
> of pcproxy.

As Steve Schulin pointed out, your information here is false. Steve is
correct that the first two years of chin04-orig.txt are deleted in
chin.04 txt, exactly as stated by M&M. I hope your other research is
more accurate than what you've stated above.

BTW have you figured out the 159 series yet? I guess not or you'd have
told us. Why don't you finish that before you start something else?

David Ball

unread,
Dec 15, 2003, 11:08:52 PM12/15/03
to

I took a look at that as well and he is right, but I think we
have to keep things in perspective. We're talking about two years from
a single series. Steve makes much of the fact that the missing years
were apparently warm, but we're really talking about nothing more than
noise. If two warm years in one record from a single part of the world
is sufficient to make MBH's results do a 180 turn there is a serious
problem somewhere.

David Ball

unread,
Dec 15, 2003, 11:24:08 PM12/15/03
to
While we're at it, is M&M's question 3 -

>Where are the calculations of principal components for series in the range
>#73-92 that would show that these have been collated into the correct year?
>Do you have any working papers that show these, and if so, would you make
>them FTP or otherwise publicly available?

much of an issue anymore. The software used for calculating the PC's
used in series 72-83 are publically available on the Mann website.
I've yet to be able to reproduce Mann's results, but I'm making
progress. I haven't had much time in more than a week to even look at
it. Having said that, M&M are not asking about the validity of the
code, but rather whether the code exists and it does. The one thing I
do want to check is whether the fortran code used with the SWM and
Vaganov series is the functionally the same as that used in producing
the PC's for series 84-92.

David Ball

unread,
Dec 16, 2003, 7:52:39 AM12/16/03
to
On 15 Dec 2003 20:05:33 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>> 1c #100 is a chinese tree ring series which can be found from the
>> NOAA paleoclimate site
>> ftp://ftp.ngdc.noaa.gov/paleo/treering/chronologies/asia/chin004.crn
>> that runs from 1540-1989. You can also find this data set from the noaa
>> paleoclimate site http://www.ngdc.noaa.gov/paleo/.
>>
>> 1540-1989 is the period for this data set that one finds in the NSM.
>> In pcproxy the first two years (1540 and 1541) are absent. Two files,
>> called chin04.txt and chin04-orig.txt are found in the
>> /pub/MBH98/TREE/ITRDB/MISC folder. They are identical to each other and
>> appear identical to the data set at the noaa paleo site.
>>
>> In short, this is almost certainly a copying error in the construction
>> of pcproxy.
>
>As Steve Schulin pointed out, your information here is false. Steve is
>correct that the first two years of chin04-orig.txt are deleted in
>chin.04 txt, exactly as stated by M&M. I hope your other research is
>more accurate than what you've stated above.

Might I suggest you change your tone. It is completely
uncalled for, especially since you don't seem to want to get off your
ass and do any of the work. Josh has already given Steve credit for
this point, as have I. I guess you missed that part. Let's keep things
in perspective, too. So far, M&M have not come across as very
competent, especially when much of the information that they were
looking for was in the reference material, material they obviously
didn't read.

>
>BTW have you figured out the 159 series yet? I guess not or you'd have
>told us. Why don't you finish that before you start something else?

That's what YOU are after. Do some of the work for yourself
and look for the answers. I guess you simply don't like seeing M&M
cast in an unfavourable light. Why not address the rest of Josh's
post? The information was available in the references. Why didn't M&M
read them? Seems like a pretty simple thing to do, especially since it
only took Josh 4 hours or so.

James Acker

unread,
Dec 16, 2003, 10:16:13 AM12/16/03
to
I'd like to commend Josh on this effort -- one wonders how
much work M&M are doing to figure out the differences (as well as
Mann et al.)

One question, Josh: what library did you go to for all the
references? UMD-College Park?

Jim Acker

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Jim Acker
jac...@gl.umbc.edu
"Since we are assured that an all-wise Creator has observed the
most exact proportions, of number, weight, and measure, in the
make of all things, the most likely way therefore, to get any
insight into the nature of those parts of the creation, which
come within our observation, must in all reason be to number,
weigh, and measure." - Stephen Hales

Josh Halpern

unread,
Dec 16, 2003, 9:49:31 PM12/16/03
to

James Acker wrote:

SNIP...

> One question, Josh: what library did you go to for all the
>references? UMD-College Park?
>

Yep. Easily the best university library in the neighborhood. Open on
Sunday unlike the Library of Congress (We are too poor)

josh halpern

>
>
>
>

David Ball

unread,
Dec 16, 2003, 11:31:07 PM12/16/03
to
>From: James Acker (jac...@linux1.gl.umbc.edu)
>Subject: Re: Auditing the auditors #1
>
>View this article only
>Newsgroups: sci.environment
>Date: 2003-12-16 07:16:14 PST

>
>I'd like to commend Josh on this effort -- one wonders how
>much work M&M are doing to figure out the differences (as well as
>Mann et al.)

I've been wondering why M&M, when they noticed the problems,
assumed that there was something nefarious going on with MBH and also
why they went ahead and published their article? When one sees gross
errors in data analysis, one doesn't immediately leap to the most
extreme explanation as the reason. The first thing I do is check my
own work. I certainly wouldn't go ahead and publish something until I
was absolutely sure that my results were above reproach.
The frustrating thing is that there are actually a few people
taking shots at Josh for attempting to clarify this instead of
helping. That helps do nothing except perpetuate the discrepency
between the two positions.

David Ball

unread,
Dec 17, 2003, 9:31:47 AM12/17/03
to
On Mon, 15 Dec 2003 04:04:18 GMT, Josh Halpern
<j.ha...@incoming.verizon.net> wrote:

>ftp://holocene.evsc.virginia.edu/pub/MBH98

A clarification on M&M's question 6....

What is the source for your data for series #37 (precipitation in
grid-box 42.5N, 72.5W)? Did you use the data from Jones-Bradley
Paris, France and if so, in which series? More generally, please
provide, identifications of the exact Jones-Bradley locations for each
of the series #21-42. Where are the original source data?

M&M appear to have gotten the first part of the question
screwed up. According to:

ftp://holocene.evsc.virginia.edu/pub/MBH98/INSTR/PREC/prec.loc

1813 11 12.5 82.5 prec-1820-01.dat
1817 11 17.5 72.5 prec-1820-03.dat
1809 11 37.5 282.5 prec-1820-04.dat
1749 11 42.5 2.5 prec-1820-05.dat
1804 11 42.5 7.5 prec-1820-06.dat
1770 11 42.5 287.5 prec-1820-07.dat
1813 11 47.5 2.5 prec-1820-08.dat
1805 11 47.5 12.5 prec-1820-09.dat
1697 11 52.5 12.5 prec-1820-10.dat
1800 11 52.5 357.5 prec-1820-11.dat
1785 11 57.5 352.5 prec-1820-12.dat

series 37 ( prec-1820-07.dat ) corresponds to a lat/lon of 42.5 N
287.5E (72.5W) or which would put it somewhere in North America. (I
don't have an atlas handy so maybe someone could pin-point exactly
where this is). As near as I can tell, it appears to be somewhere
northwest of Boston (42'22N 71'02W). I don't see how they could have
uesd a European series to construct this. The data from series 34
looks to be in the same area.
Indeed the locations of the series correspond to series 32 to
42. The locations of series 21 to 31 can be found in:

ftp://holocene.evsc.virginia.edu/pub/MBH98/INSTR/TEMP/temp.loc

1820 10 42.5 267.5 temp-1820-01.dat
1757 10 47.5 2.5 temp-1820-02.dat
1753 10 47.5 7.5 temp-1820-03.dat
1767 10 47.5 12.5 temp-1820-04.dat
1775 10 47.5 17.5 temp-1820-05.dat
1792 10 52.5 17.5 temp-1820-07.dat
1756 10 57.5 17.5 temp-1820-09.dat
1752 10 57.5 32.5 temp-1820-10.dat
1816 10 62.5 7.5 temp-1820-11.dat
1761 10 62.5 12.5 temp-1820-12.dat
1814 10 62.5 42.5 temp-1820-13.dat

Most of these appear to be in Western Europe, except for the first
one.

Nigel Persaud

unread,
Dec 17, 2003, 9:43:46 AM12/17/03
to


Josh, this is an ingenious rationalization.

However, first of all, it is common ground now that MBH deleted the
first 25 years of this series. Prior to M&M, no one knew that. (The
undeleted series are also at ftp/MBH98. Secondly, MBH did not disclose
that they deleted the first 25 years.) The deletion may or may not be
justifiable in climatological terms, but people relying on MBH should
be advised MBH did this without having to parse through original
sources. The deletion should have been annotated in the Supplementary
Information directly and justified. Some readers might not agree with
this approach and are entitled to know. Third, it is far from obvious
that it would be a "mistake" to use the first 25 years. After all, MBH
is using PROXIES. The first 25 years of the historical record are a
better PROXY than some tree ring. Fourth, there is no way of knowing
whether your rationalization is the same as Mann's. The periods which
he deleted in Central England do not match exactly to the periods
which you purport to rationalize. Finally, while you're at this, can
you figure out why MBH used summer versions instead of annual versions
for these series?

Nigel.

David Ball

unread,
Dec 17, 2003, 11:28:47 AM12/17/03
to
On 17 Dec 2003 06:43:46 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>> 1b. #11 is the Central Europe historical record, originally published

Nigel, you appear desperate to try and rationalize M&M's
position. Why? First of all, the author's are under no obligation to
show all their work. A peer-reviewed paper is not intended for the
masses but for experts in the field. As such, many things may be taken
as given, espeicially since the experts should be familiar with the
reference material used in the study. If I write a paper on imaging
satellite products, I sure as hell don't need to put vast amounts of
filler in explaining how the satellite sensors work. It is assumed
that anyone reading the paper will have the expertise to understand
the basics. The real question is why didn't M&M read the reference
material. Care to answer that one?
Secondly, bad data are routinely thrown out of any analysis. I
do it all the time. Indeed many robust estimation techniques require
that outliers NOT be considered. As long as this is done in a rigorous
fashion it is entirely acceptable. No data were DELETED. They simply
were not used in the analysis. Big difference. Your intentional use of
this particular phrasing is perjorative and unnecessary. ANY analysis
requires that data be pre-processed prior to the analysis being done.
If there is any chance that the data are suspect, they should never be
used.
Thirdly, your claims about the first 25 years of the record
being better are wishful thinking on your part and nothing more. Prove
it! Josh's arguments are entirely appropriate. Better to use a
slightly smaller subset of the data than risk contaminating it. Your
argument has no justification that I can see.

Nigel Persaud

unread,
Dec 17, 2003, 6:18:16 PM12/17/03
to
David, the MBH proxy series #37 corresponds to ftp/MBH98 series
prec-1820-07.dat. This doesn't prove anything about the source data.
MBH98 refers to Bradlye-Jones 1992 as a source but fail to identify
the corresponding locations. M&M note that the correlation between #37
and the precipitation from Bradley-Jones Paris, France (archived at
WDCP) is extraordinarily high and the start years coincide. You've
not dealt with this issue at all and it is still outstanding. Nigel

David Ball <wra...@mb.sympatico.ca> wrote in message news:<n4p0uvgaj8jb102d6...@4ax.com>...

Nigel Persaud

unread,
Dec 17, 2003, 6:29:02 PM12/17/03
to
Secondly, bad data are routinely thrown out of any analysis. I
> do it all the time. Indeed many robust estimation techniques require
> that outliers NOT be considered. As long as this is done in a rigorous
> fashion it is entirely acceptable. No data were DELETED. They simply
> were not used in the analysis. Big difference. Your intentional use of
> this particular phrasing is perjorative and unnecessary. ANY analysis
> requires that data be pre-processed prior to the analysis being done.
> If there is any chance that the data are suspect, they should never be
> used.

David, your comments about robust estimation are a reasonable way of
approaching this issue and your phrase "in a rigorous fashion" is a
good criterion. However, there's no indication in MBH98 that they
carried out such a procedure. Even if the publication was directed at
the classes and not the masses, some description is surely required.
Nigel

David Ball

unread,
Dec 17, 2003, 6:50:00 PM12/17/03
to
On 17 Dec 2003 15:18:16 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>David, the MBH proxy series #37 corresponds to ftp/MBH98 series

David Ball

unread,
Dec 17, 2003, 6:51:44 PM12/17/03
to
On 17 Dec 2003 15:18:16 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>David, the MBH proxy series #37 corresponds to ftp/MBH98 series


>prec-1820-07.dat. This doesn't prove anything about the source data.
>MBH98 refers to Bradlye-Jones 1992 as a source but fail to identify
>the corresponding locations. M&M note that the correlation between #37
>and the precipitation from Bradley-Jones Paris, France (archived at
>WDCP) is extraordinarily high and the start years coincide. You've
>not dealt with this issue at all and it is still outstanding. Nigel
>

Not according to the latitude/longitude of the points in the
data files. Try not to be so single-minded. Look at the information,
not your dogmatic interpretation of the information. There are a
number of points in the data files that are CLEARLY not located in
Europe.

David Ball

unread,
Dec 17, 2003, 7:00:13 PM12/17/03
to
On 17 Dec 2003 15:29:02 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>Secondly, bad data are routinely thrown out of any analysis. I

Why? It is relatively easy to see from the reference material
why the data in the early years was deemed suspect. In the absence of
anything suggesting that these data are valid, they should be treated
as suspect, not the other way around. I ask the question again: why
were M&M unable to read the reference material?

David Ball

unread,
Dec 17, 2003, 9:38:24 PM12/17/03
to
>From: Steve Schulin (steve....@nuclear.com)

>Subject: Re: Auditing the auditors #1
>View: Complete Thread (18 articles)
>Original Format
>Newsgroups: sci.environment
>Date: 2003-12-15 09:56:27 PST

>
>> >
>> >Your abc breakdown ignores the M&M question about series 65?
>>
>> Josh spent 4 hours getting the information he posted. What are
>> you doing, besides posting "what have you done lately" comments? ...
>
>My question was prompted because I got the impression that Josh was
>attempting to fully address the questions in issue #1. His use of the
>phrase "to get through #1" contributed to that impression. My question
>was politely stated. If you had been asking question to me under similar
>circumstances, it would have been no surprise to see you be quite
>abusive about only seeing abc instead of abcd.

What exactly is preventing you from investigating the question
of series 65? Laziness? Indifference?

>
>> ... Is
>> there anything preventing you from getting off your ass and doing some
>> of the legwork or do you plan to just sit here and complain about
>> someone else's efforts?
>
>There's plenty of legwork to be done, that's for sure. I had done enough
>legwork previously to recognize that Josh was incorrect in stating,
>below, that the chin04.txt and chin04-orig.txt files at ftp site were
>identical. Are you not grateful to have that error corrected?

Most of the legwork seems to stem from M&M's inability to read
reference material, get the proper data in the proper format and a
number of other housekeeping issues. It doesn't seem to have much to
do with Mann et al so far.
As for being grateful, it was nice to have that pointed out. I
wouldn't make too much out of it. I doubt very much it has a lot to do
with anything in the grand scheme of things.

>
>> What I find illuminating about Josh's post is that the basic
>> information about why certain series were truncated is available in
>> the reference material. When the answer to each question posed by M&M
>> boils down to their inability to RTFR, there is a problem. ...
>
>It seems clear enough that Mann et al chose to use subsets of series 10
>and 11. There is no indication that they systematically applied some
>evaluative methodology to all the series. And your suggestion that "the
>answer to each question posed by M&M boils down to their inability to
>RTFR" seems quite an exaggeration.

If they had bothered to read the references, they would have
phrased their questions in a completely different way. Your posturing
on the matter has been noted for some time as has your penchant for
overstating the value of this paper - as one of the most important...
As for retaining/throwing away suspect data, it is done all
the time. In the event that data are suspect, and in this case, the
references clearly show they are suspect, common practice is to throw
them out unless some compelling reason can be found to retain them.
Since there is no way to judge their worth, it makes sense to get rid
of them, but retain as much information as possible, so carving the
series back to the 60's or 70's doesn't make a lot of sense.

>
>> ... That likely
>> stems from the unseemly rush they were in to get something in print. I
>> can't think of a researcher anywhere who can knock off a peer-reviewed
>> paper in a matter of weeks, without taking some serious shortcuts and
>> at least in the case of their question 1, it appears that those
>> shortcuts caused them to overlook basic information needed to do their
>> "audit".
>
>You have often mistaken question marks for exclamation points, so your
>take on the current matter comes as no surprise. Your apparent argument
>that "of course Mann et al modified the source data" seems quite at odds
>with the impression given by all those who have been echoing Mann's
>recent claim that the data necessary to replicate their work is
>available from the referenced sources of the data.

LOL. Spin in any way you want, Steve, but as of now, it's M&M
that are looking pretty bad.

>
>>
>> >>
>> >> Which means that the Manley reconstruction is only continuous from 1722
>> >> on, but the information upon which it relies from 1723-28 has further
>> >> difficulties, essentially absolute values were not reliable, and the
>> >> series was constructed by taking the difference between measurements
>> >> made by those thermometers and ones thought to be more reliable after
>> >> 1727, and then repeatedly differenced to get values before 1727.
>> >>
>> >> In the light of this, it is perfectly reasonable to truncate the CET
>> >> series at 1730. On the other end, 1987 is the cut off for the
>> >> analysis. We note that the cut offs were listed in the NSM.
>> >>
>> >> On balance this illustrates the principal of RTFR (Read the effing
>> >> references) and the danger of someone unfamiliar with an area trying to
>> >> do an "audit". To use the CET before 1730 would clearly have been a
>> >> mistake, and I think you could make a good argument that it should have
>> >> been cut off at 1772.
>> >

>> >Interesting info, Josh. Thanks. From what you've reported, it looks like
>> >some other truncation point would have been more perfect than 1730,
>> >however: 1723, 1728, or somewhere in 1760s. Your use of the root perfect
>> >seems an exaggeration in this case. I agree that the supplemental info
>> >lists 1730, but note that the column is described as showing the "first
>> >year of record available". That does not seem like an accurate
>> >description of the start date used by Mann et al. for this series.
>>
>> And you are basing this on what? ...
>
>On the meaning of the phrase "first year of record available", as
>opposed to the meaning of "first year of record chosen to be included
>although prior years were available".

More spin and posturing, I'm afraid. Only you could take
standard practice and something very innocent and try to make
something nefarious out of it.

>
>> ... Josh's point about the
>> problematic nature of the 1723-1728 data is well taken. ...
>
>I too think it may very well have been reasonable to truncate this
>series. I find it quite silly that you apparently chastise M&M for
>asking the question, however.

Asking the question? Never. I chastise them for their rush to
print because it is a likely cause in their overlooking crucial
information (i.e. reading the references).

>
>> ... Using any of
>> the data in that period would be a mistake. There is nothing
>> inherently wrong with using 1730 - it's a nice round number - and as
>> Josh pointed out other truncation points could have been adopted...
>
>And once its verified what Mann et al actually did, the effects of other
>treatments will be interesting to explore.

We're making progress. Certainly question 2, 3 and 12 have
been put to bed. Question 6 is still a bit of a problem, not because
of the data used but because some of the positions appear to be in the
US. There's nothing inherently wrong with that, although Nigel seems
to feel that isn't likely because some of the series resemble some
European series.


>
>> ...
>> Let's face it, the proxy data are all reconstructed back to nice round
>> numbers. In the tree ring data, why go back to 1700? Why not 1699? Or
>> 1697? It's a computational convenience, nothing more. ...
>
>Are you saying that other time series were truncated by Mann et al
>besides the four mentioned in issue #1?

I'm saying that if you look at all 22 SWM series for example,
they don't all stop at 1700, yet the author's chose 1700 as a
break-point. 1600 was also chosen. It was a convenience. They could
have used 1699, but 1700 is a nice round number.

>
>> ...
>> Your arbitrary choice of a point in the 1760's is noted. ...
>
>I quite purposefully used the precision quoted from the reference by
>Josh. How would you improve upon my choice of language?

1760 is no better or worse than 1730. Again, in any analysis
containing suspect data you want to get of the suspect data, while
retaining as much good data as possible. As Josh said, an argument
could be made that 1770 was a better starting point. The author's
apparently didn't agree.

>
>> ... In
>> addition, your quibbling about the description of 1730 being the first
>> year or record is just that: a quibble. ...
>
>I thought about elaborating, but didn't want to clutter. In the body of
>the Nature paper, Mann et al claim to have included five series in their
>analysis that apparently were not included. It is in the context of such
>apparent sloppiness that M&M's questions warrant clear answers, untinged
>by the insults and invective that have become rote from Mann and his
>apologists.

ROTFL. You've got a couple of hack who knock off a paper,
submit it completely full of errors to a third-rate "journal" then
whine when they are shown to be less than competent and you are
posturing about Mann et al's sloppiness? Too funny.

>
>> ... When an analysis is done, bad
>> data are routinely thrown out. That's a fact of life. The paper in
>> question is NOT meant for the general public but for other experts in
>> the field, experts who persumably are, unlike M&M, aware of the
>> problematic nature of some of the data. That being said, such a
>> notation can be considered a given.
>
>So you think it inappropriate that IPCC used the hocky stick graph in
>Fig 1 of the WG1 TAR policymaker summary?

No, I think it entirely appropriate. Mann et al have done
nothing wrong and I've looked at this far harder than you have. All
you've done is posture and try to spin and frankly it's wearing thin.
As for your perjorative terminology, it's what I've come to expect
from you: no facts and lots of snarkiness.


>> >
>> >M&M ask reasonable questions under the heading of issue #1. Some of what
>> >you write here is illuminatory, some is obfuscatory, and some looks just
>> >snarky itself.
>> >
>> And the reasonable responses given by Josh show that the
>> information M&M are inquiring about was available in the paper's
>> reference material, material that they apparently didn't bother to
>> read. That isn't a snark, nor is it obfuscatory. It is simply the
>> truth. Nor are your minor quibbles adding anything to the discussion,
>> except noise.
>
>I've seen how prone some folks here are to exaggerate.

Look in the mirror for the single worst offender.

> I thought it
>worthwhile to nip some of that in the bud in this case.

See that you don't resort to any more of it and this thread
will remain quite pleasant.

Nigel Persaud

unread,
Dec 17, 2003, 11:46:03 PM12/17/03
to
> Not according to the latitude/longitude of the points in the
> data files. Try not to be so single-minded. Look at the information,
> not your dogmatic interpretation of the information. There are a
> number of points in the data files that are CLEARLY not located in
> Europe.
>

I agree that ftp/MBH98 locates the series in North America. So what?
ftp/MBH98 is not source data. Can you show where the data in #37 comes
from? Nigel

David Ball

unread,
Dec 18, 2003, 6:58:19 AM12/18/03
to
On 17 Dec 2003 20:46:03 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>> Not according to the latitude/longitude of the points in the

Huh? That's funny, but we've been crunching the numbers from
that site, comparing M&M's spreadsheet with the numbers from that
site, recalculating the PC's from that site and now that there's some
discrepencies your answer is that it doesn't contain source data? We
know series 37 is prec-1820-07.dat posted to:

ftp://holocene.evsc.virginia.edu/pub/MBH98/INSTR/PREC

and in that directory is a file called:

prec.loc

which contains weighting factors and lat/lon points for the various
series, 2 of which are NOT in Europe, affecting series 34 and 37.
Similarly, for the temperature series, it appears that series 21 is
not located in Europe but in the US midwest. Now, do you have
something specific that precludes these three sites from being in the
US? Give something specific, not just a correlation between...

Steve Schulin

unread,
Dec 18, 2003, 10:00:38 AM12/18/03
to
In article <dg02uvc3ele66v652...@4ax.com>,
David Ball <wra...@mb.sympatico.ca> wrote:

> >From: Steve Schulin (steve....@nuclear.com)
> >Subject: Re: Auditing the auditors #1
> >View: Complete Thread (18 articles)
> >Original Format
> >Newsgroups: sci.environment
> >Date: 2003-12-15 09:56:27 PST
> >
> >> >
> >> >Your abc breakdown ignores the M&M question about series 65?
> >>
> >> Josh spent 4 hours getting the information he posted. What are
> >> you doing, besides posting "what have you done lately" comments? ...
> >
> >My question was prompted because I got the impression that Josh was
> >attempting to fully address the questions in issue #1. His use of the
> >phrase "to get through #1" contributed to that impression. My question
> >was politely stated. If you had been asking question to me under similar
> >circumstances, it would have been no surprise to see you be quite
> >abusive about only seeing abc instead of abcd.
>
> What exactly is preventing you from investigating the question
> of series 65? Laziness? Indifference?

Priorities.

>
> >
> >> ... Is
> >> there anything preventing you from getting off your ass and doing some
> >> of the legwork or do you plan to just sit here and complain about
> >> someone else's efforts?
> >
> >There's plenty of legwork to be done, that's for sure. I had done enough
> >legwork previously to recognize that Josh was incorrect in stating,
> >below, that the chin04.txt and chin04-orig.txt files at ftp site were
> >identical. Are you not grateful to have that error corrected?
>
> Most of the legwork seems to stem from M&M's inability to read
> reference material, get the proper data in the proper format and a
> number of other housekeeping issues. It doesn't seem to have much to
> do with Mann et al so far.

The world already knows much more about Mann et al's methodology,
because of McIntyre and McKitrick's efforts, than it knew for the 5
years after Nature published the article. Your exclusive praise for
Josh's efforts is quite a contrast to your exclusive derision of M&M's.
At this point, we don't actually even know what data Mann et al used.
McIntyre, using the error-filled pcproxy.txt file, without corrections,
came up with a graph quite similar to the Mann et al hockey stick
product. I wouldn't be surprised if Mann's peers let him avoid ever
publicly providing enough info to allow his work to be replicable.

> As for being grateful, it was nice to have that pointed out. I
> wouldn't make too much out of it. I doubt very much it has a lot to do
> with anything in the grand scheme of things.

That reminds me of a comment made by one of the relief pitchers of my
beloved Philadelphia Phillies: "Ten million years from now, when the sun
burns out and the Earth is just a frozen iceball hurtling through space,
nobody's going to care whether or not I got this guy out."

> >> What I find illuminating about Josh's post is that the basic
> >> information about why certain series were truncated is available in
> >> the reference material. When the answer to each question posed by M&M
> >> boils down to their inability to RTFR, there is a problem. ...
> >
> >It seems clear enough that Mann et al chose to use subsets of series 10
> >and 11. There is no indication that they systematically applied some
> >evaluative methodology to all the series. And your suggestion that "the
> >answer to each question posed by M&M boils down to their inability to
> >RTFR" seems quite an exaggeration.
>
> If they had bothered to read the references, they would have
> phrased their questions in a completely different way. Your posturing
> on the matter has been noted for some time as has your penchant for
> overstating the value of this paper - as one of the most important...

That was quite appropriate phrase to use as title of that
thread-starting message. The initial post contained excerpts from a
hot-off-the-press op ed by paleoclimate researcher Bob Carter:

* " ... it is clear that the McIntyre and McKitrick paper is set to
become one of the most important that has been published in recent
years. Apart from the light that it throws on the climate debate, the
paper raises profound issues to do with the integrity of scientific
publication, how data which underpins published papers should be
archived and made available, and whether science advice given to
governments on policy matters should be rigorously audited."

* "... unlike Mann, Bradley and Hughes for their 1998 paper, McIntyre
and McKitrick have made full disclosure of all the assumptions made and
techniques used in their manipulation of the data, have posted the data
they used on a freely-accessible website, and have invited other
scientists to comment on or check their conclusions."

* "Mann, Bradley and Hughes used complex statistical averaging methods
to combine no fewer than 112 such proxy records into an inferred climate
curve for 1400-1902. Controversially, for the period between 1902 and
1980 they then spliced on an averaged curve of actual thermometer
temperature measurements. The result was a combined curve which showed
little evidence of either the Mediaeval Warm Period or the Little Ice
Age and a dramatic upturn to higher temperatures after 1900 and
continuing to 1980. Thus was born the famous 'hockey stick curve' of
recent climate change. Though it was immediately adopted as the received
truth by global warming lobbyists, many scientists were sharply critical
of the conclusions of Mann, Bradley and Hughes. Critics pointed out that
the graph was based on datasets which were heavily manipulated
statistically and, in combining at the year 1902 two datasets of
different derivation, Mann, Bradley and Hughes had transgressed good
statistical practice."

* "In an initial response, Mann has asserted that the dataset which he
provided McIntyre and McKitrick is not the same dataset he used in his
Nature paper, and anyway contains errors. Such a response does not
inspire confidence in Mann's other work, and, anyway, McIntyre and
McKitrick used for their analysis a recompiled, accurate dataset. It
will obviously be some time before the argument is concluded and the
dust settles."

* "Australia should consider following Denmark's example and set up a
science audit unit to verify the soundness of advice the federal
government receives. Such an audit unit can be funded with the money
saved by closing down the Australian Greenhouse Office."

[Source: Bob Carter (prof geology, Marine Geophysical Laboratory, James
Cook U), "Global Warming May Turn Out To Be Just Hot Air", Australian
Financial Review, November 3, 2003, p. 71]

> As for retaining/throwing away suspect data, it is done all
> the time. In the event that data are suspect, and in this case, the
> references clearly show they are suspect, common practice is to throw
> them out unless some compelling reason can be found to retain them.
> Since there is no way to judge their worth, it makes sense to get rid
> of them, but retain as much information as possible, so carving the
> series back to the 60's or 70's doesn't make a lot of sense.

In the 1998 Nature paper, Mann et al note what looks to be a general
principle: "... the mutual information contained in a diverse and widely
distributed set of independent climate indicators can more faithfully
capture the consistent climate signal that is present, reducing the
compromising effects of biases and weaknesses in the individual
indicators." Finding reason to drop cold-year data from Little Ice Age
and warm-year data from prior begs the question of whether all the
series were treated the same.

> >> ... That likely
> >> stems from the unseemly rush they were in to get something in print. I
> >> can't think of a researcher anywhere who can knock off a peer-reviewed
> >> paper in a matter of weeks, without taking some serious shortcuts and
> >> at least in the case of their question 1, it appears that those
> >> shortcuts caused them to overlook basic information needed to do their
> >> "audit".
> >
> >You have often mistaken question marks for exclamation points, so your
> >take on the current matter comes as no surprise. Your apparent argument
> >that "of course Mann et al modified the source data" seems quite at odds
> >with the impression given by all those who have been echoing Mann's
> >recent claim that the data necessary to replicate their work is
> >available from the referenced sources of the data.
>
> LOL. Spin in any way you want, Steve, but as of now, it's M&M
> that are looking pretty bad.

They have raised many interesting points.

And they have discovered facts which prompt other interesting points,
too. Take the five series (fran003, ital015, ital015x, spai026, and
spai047) that were listed amongst the proxies used in the 1998 paper but
which were not amongst the pcproxy.txt file. M&M ask Mann et al if those
series were in fact used. Mann hasn't answered that specifically yet,
but he does say that all the data for the 1998 paper is available in the
newly disclosed ftp site. Those five series don't appear in the
directories at the ftp site. I'm curious as to how this matter plays
out. If it turns out that these series were not used, I'm sure I'm not
the only one who would be interested to learn the particulars as to why
they are listed. Did Mann et al intend to use them originally? If so,
what changed their mind?

Having seen you often misrepresent stuff, I can understand why you find
such to be standard practice. Mann et al wrote their own paper. Nobody
forced them to use that word "available" in describing the column.

> >> ... Josh's point about the
> >> problematic nature of the 1723-1728 data is well taken. ...
> >
> >I too think it may very well have been reasonable to truncate this
> >series. I find it quite silly that you apparently chastise M&M for
> >asking the question, however.
>
> Asking the question? Never. I chastise them for their rush to
> print because it is a likely cause in their overlooking crucial
> information (i.e. reading the references).

As it turns out, we're lucky that M&M did exactly what they did, and
that Mann reacted as he did. Otherwise, we wouldn't have been able to
contemplate on the sight of Mann's newly disclosed ftp site being
vacuumed like Hillary Clinton's former law firm's files were.

> >> ... Using any of
> >> the data in that period would be a mistake. There is nothing
> >> inherently wrong with using 1730 - it's a nice round number - and as
> >> Josh pointed out other truncation points could have been adopted...
> >
> >And once its verified what Mann et al actually did, the effects of other
> >treatments will be interesting to explore.
>
> We're making progress. Certainly question 2, 3 and 12 have

> been put to bed. ...

The recent revelations by Mann in response to M&M have already enabled
much to be known that was not known before.

> ... Question 6 is still a bit of a problem, not because


> of the data used but because some of the positions appear to be in the
> US. There's nothing inherently wrong with that, although Nigel seems
> to feel that isn't likely because some of the series resemble some
> European series.
> >> ...
> >> Let's face it, the proxy data are all reconstructed back to nice round
> >> numbers. In the tree ring data, why go back to 1700? Why not 1699? Or
> >> 1697? It's a computational convenience, nothing more. ...
> >
> >Are you saying that other time series were truncated by Mann et al
> >besides the four mentioned in issue #1?
>
> I'm saying that if you look at all 22 SWM series for example,
> they don't all stop at 1700, yet the author's chose 1700 as a
> break-point. 1600 was also chosen. It was a convenience. They could
> have used 1699, but 1700 is a nice round number.

If that's what you were saying, then it wasn't analogous to the
truncation issue.

> >> ...
> >> Your arbitrary choice of a point in the 1760's is noted. ...
> >
> >I quite purposefully used the precision quoted from the reference by
> >Josh. How would you improve upon my choice of language?
>

> 1760 is no better or worse than 1730. ...

Neither one warrants the root "perfect" in his context, fer sure.

> ... Again, in any analysis


> containing suspect data you want to get of the suspect data, while
> retaining as much good data as possible. As Josh said, an argument

> could be made that 1770 was a better starting point. ...

He said 1772. I agree with him.

> ... The author's
> apparently didn't agree.

Nor did they think it appropriate to give any insight into whether they
subjected all the series to the same criteria.

> >> ... In
> >> addition, your quibbling about the description of 1730 being the first
> >> year or record is just that: a quibble. ...
> >
> >I thought about elaborating, but didn't want to clutter. In the body of
> >the Nature paper, Mann et al claim to have included five series in their
> >analysis that apparently were not included. It is in the context of such
> >apparent sloppiness that M&M's questions warrant clear answers, untinged
> >by the insults and invective that have become rote from Mann and his
> >apologists.
>
> ROTFL. You've got a couple of hack who knock off a paper,
> submit it completely full of errors to a third-rate "journal" then
> whine when they are shown to be less than competent and you are
> posturing about Mann et al's sloppiness? Too funny.

I'm happy to disagree with every bit of your opinion on this.

> >> ... When an analysis is done, bad
> >> data are routinely thrown out. That's a fact of life. The paper in
> >> question is NOT meant for the general public but for other experts in
> >> the field, experts who persumably are, unlike M&M, aware of the
> >> problematic nature of some of the data. That being said, such a
> >> notation can be considered a given.
> >
> >So you think it inappropriate that IPCC used the hocky stick graph in
> >Fig 1 of the WG1 TAR policymaker summary?
>

> No, I think it entirely appropriate. ...

LOL - you just got through saying the paper "is NOT meant for the
general public but for other experts in the field".

> ... Mann et al have done
> nothing wrong and I've looked at this far harder than you have. ...

But you're blinded by your bias, having long ago drank the Mann-made
Kool-aid. You may not even recognize when you're displaying how little
regard you have for the scientific method.

> ... All


> you've done is posture and try to spin and frankly it's wearing thin.
> As for your perjorative terminology, it's what I've come to expect
> from you: no facts and lots of snarkiness.

I've been quite restrained in my comments about Mann et al 1998 and most
of its apologists. Probably too easy-going on some aspects so far.

>
>
> >> >
> >> >M&M ask reasonable questions under the heading of issue #1. Some of what
> >> >you write here is illuminatory, some is obfuscatory, and some looks just
> >> >snarky itself.
> >> >
> >> And the reasonable responses given by Josh show that the
> >> information M&M are inquiring about was available in the paper's
> >> reference material, material that they apparently didn't bother to
> >> read. That isn't a snark, nor is it obfuscatory. It is simply the
> >> truth. Nor are your minor quibbles adding anything to the discussion,
> >> except noise.
> >
> >I've seen how prone some folks here are to exaggerate.
>
> Look in the mirror for the single worst offender.

There you go exaggerating again.


>
> > I thought it
> >worthwhile to nip some of that in the bud in this case.
>
> See that you don't resort to any more of it and this thread
> will remain quite pleasant.

The thread included unpleasantness before I joined in. I can understand
your nostalgia, however, for the unpleasantness in which you revel.

Very truly,

BallB...@nuclear.com

David Ball

unread,
Dec 18, 2003, 9:03:09 PM12/18/03
to
On Thu, 18 Dec 2003 10:00:38 -0500, Steve Schulin
<steve....@nuclear.com> wrote:

>In article <dg02uvc3ele66v652...@4ax.com>,
> David Ball <wra...@mb.sympatico.ca> wrote:
>
>> >From: Steve Schulin (steve....@nuclear.com)
>> >Subject: Re: Auditing the auditors #1
>> >View: Complete Thread (18 articles)
>> >Original Format
>> >Newsgroups: sci.environment
>> >Date: 2003-12-15 09:56:27 PST
>> >
>> >> >
>> >> >Your abc breakdown ignores the M&M question about series 65?
>> >>
>> >> Josh spent 4 hours getting the information he posted. What are
>> >> you doing, besides posting "what have you done lately" comments? ...
>> >
>> >My question was prompted because I got the impression that Josh was
>> >attempting to fully address the questions in issue #1. His use of the
>> >phrase "to get through #1" contributed to that impression. My question
>> >was politely stated. If you had been asking question to me under similar
>> >circumstances, it would have been no surprise to see you be quite
>> >abusive about only seeing abc instead of abcd.
>>
>> What exactly is preventing you from investigating the question
>> of series 65? Laziness? Indifference?
>
>Priorities.

Incompetence more likely.


>>
>> Most of the legwork seems to stem from M&M's inability to read
>> reference material, get the proper data in the proper format and a
>> number of other housekeeping issues. It doesn't seem to have much to
>> do with Mann et al so far.
>
>The world already knows much more about Mann et al's methodology,
>because of McIntyre and McKitrick's efforts, than it knew for the 5
>years after Nature published the article.

No, people in this forum are farther ahead today because of
people like Josh who are doing the legwork necessary to correct M&M's
screw-ups. M&M haven't clarified anything, especially since they were
looking at the wrong bloody data.

>Your exclusive praise for
>Josh's efforts is quite a contrast to your exclusive derision of M&M's.
>At this point, we don't actually even know what data Mann et al used.
>McIntyre, using the error-filled pcproxy.txt file, without corrections,
>came up with a graph quite similar to the Mann et al hockey stick
>product. I wouldn't be surprised if Mann's peers let him avoid ever
>publicly providing enough info to allow his work to be replicable.

Incorrect, I'm afraid. You can't use bad data, jiggle it and
then claim that it resembles the original author's work. I looked very
hard and I don't recall seeing strong warming during the LIA, which
shows up on M&M's "analysis".

>
>> As for being grateful, it was nice to have that pointed out. I
>> wouldn't make too much out of it. I doubt very much it has a lot to do
>> with anything in the grand scheme of things.
>
>That reminds me of a comment made by one of the relief pitchers of my
>beloved Philadelphia Phillies: "Ten million years from now, when the sun
>burns out and the Earth is just a frozen iceball hurtling through space,
>nobody's going to care whether or not I got this guy out."

LOL. No, it comes from knowing what PCA techniques do, Steve,
and realizing that two years of missing data are doing to do jack.

>
>> >> What I find illuminating about Josh's post is that the basic
>> >> information about why certain series were truncated is available in
>> >> the reference material. When the answer to each question posed by M&M
>> >> boils down to their inability to RTFR, there is a problem. ...
>> >
>> >It seems clear enough that Mann et al chose to use subsets of series 10
>> >and 11. There is no indication that they systematically applied some
>> >evaluative methodology to all the series. And your suggestion that "the
>> >answer to each question posed by M&M boils down to their inability to
>> >RTFR" seems quite an exaggeration.
>>
>> If they had bothered to read the references, they would have
>> phrased their questions in a completely different way. Your posturing
>> on the matter has been noted for some time as has your penchant for
>> overstating the value of this paper - as one of the most important...
>
>That was quite appropriate phrase to use as title of that
>thread-starting message. The initial post contained excerpts from a
>hot-off-the-press op ed by paleoclimate researcher Bob Carter:
>

<silly op-ed deleted>

Sorry, Steve, M&M's paper is a mess and anything based on it
is necessarily flawed. I wonder why Mr. Carter didn't notice the
warming that M&M showed during the LIA. That should have set off alarm
bells for anyone with an ounce of understanding of the problem.

>
>> As for retaining/throwing away suspect data, it is done all
>> the time. In the event that data are suspect, and in this case, the
>> references clearly show they are suspect, common practice is to throw
>> them out unless some compelling reason can be found to retain them.
>> Since there is no way to judge their worth, it makes sense to get rid
>> of them, but retain as much information as possible, so carving the
>> series back to the 60's or 70's doesn't make a lot of sense.
>
>In the 1998 Nature paper, Mann et al note what looks to be a general
>principle: "... the mutual information contained in a diverse and widely
>distributed set of independent climate indicators can more faithfully
>capture the consistent climate signal that is present, reducing the
>compromising effects of biases and weaknesses in the individual
>indicators." Finding reason to drop cold-year data from Little Ice Age
>and warm-year data from prior begs the question of whether all the
>series were treated the same.

Apparently, you didn't understand what I said to you. Quality
control of any data MUST be done. It is a fact of life. If there is a
suspicion that data are compromised, YOU DON'T USE THEM! Pretty
straight forward, eh?

>
>> >> ... That likely
>> >> stems from the unseemly rush they were in to get something in print. I
>> >> can't think of a researcher anywhere who can knock off a peer-reviewed
>> >> paper in a matter of weeks, without taking some serious shortcuts and
>> >> at least in the case of their question 1, it appears that those
>> >> shortcuts caused them to overlook basic information needed to do their
>> >> "audit".
>> >
>> >You have often mistaken question marks for exclamation points, so your
>> >take on the current matter comes as no surprise. Your apparent argument
>> >that "of course Mann et al modified the source data" seems quite at odds
>> >with the impression given by all those who have been echoing Mann's
>> >recent claim that the data necessary to replicate their work is
>> >available from the referenced sources of the data.
>>
>> LOL. Spin in any way you want, Steve, but as of now, it's M&M
>> that are looking pretty bad.
>
>They have raised many interesting points.

Really? From a screwed up data set? How do you figure? Aren't
you the one who often preaches garbage in, garbage out? A wonderful
example of it here. Does this mean that Mann et al are perfect? I
doubt it.

>
>And they have discovered facts which prompt other interesting points,
>too. Take the five series (fran003, ital015, ital015x, spai026, and
>spai047) that were listed amongst the proxies used in the 1998 paper but
>which were not amongst the pcproxy.txt file. M&M ask Mann et al if those
>series were in fact used. Mann hasn't answered that specifically yet,
>but he does say that all the data for the 1998 paper is available in the
>newly disclosed ftp site.

You mean the ftp site that was available BEFORE M&M ever
started working on their paper?

>Those five series don't appear in the
>directories at the ftp site. I'm curious as to how this matter plays
>out. If it turns out that these series were not used, I'm sure I'm not
>the only one who would be interested to learn the particulars as to why
>they are listed. Did Mann et al intend to use them originally? If so,
>what changed their mind?

I don't know. The answer will be interesting.


>>
>> More spin and posturing, I'm afraid. Only you could take
>> standard practice and something very innocent and try to make
>> something nefarious out of it.
>
>Having seen you often misrepresent stuff, I can understand why you find
>such to be standard practice. Mann et al wrote their own paper. Nobody
>forced them to use that word "available" in describing the column.

Posturing again, Steve. You haven't got a clue what the
analysis is doing, but you're absolutely sure that it's something bad.
When told that it's pretty common, it becomes a grand conspiracy. Your
comments would carry a wee bit more weight if you could understand
what was happening.

>
>> >> ... Josh's point about the
>> >> problematic nature of the 1723-1728 data is well taken. ...
>> >
>> >I too think it may very well have been reasonable to truncate this
>> >series. I find it quite silly that you apparently chastise M&M for
>> >asking the question, however.
>>
>> Asking the question? Never. I chastise them for their rush to
>> print because it is a likely cause in their overlooking crucial
>> information (i.e. reading the references).
>
>As it turns out, we're lucky that M&M did exactly what they did, and
>that Mann reacted as he did. Otherwise, we wouldn't have been able to
>contemplate on the sight of Mann's newly disclosed ftp site being
>vacuumed like Hillary Clinton's former law firm's files were.

LOL. Paul Farrar has already pointed out that the ftp site in
question was available long ago as have I. Your ridiculous attempts to
show it otherwise are quite pathetic.

>
>> >> ... Using any of
>> >> the data in that period would be a mistake. There is nothing
>> >> inherently wrong with using 1730 - it's a nice round number - and as
>> >> Josh pointed out other truncation points could have been adopted...
>> >
>> >And once its verified what Mann et al actually did, the effects of other
>> >treatments will be interesting to explore.
>>
>> We're making progress. Certainly question 2, 3 and 12 have
>> been put to bed. ...
>
>The recent revelations by Mann in response to M&M have already enabled
>much to be known that was not known before.

Yeah verily, and Schulin laid his holy message upon the
peasants of usenet and he saw that all was good.

>
>> ... Question 6 is still a bit of a problem, not because
>> of the data used but because some of the positions appear to be in the
>> US. There's nothing inherently wrong with that, although Nigel seems
>> to feel that isn't likely because some of the series resemble some
>> European series.
>> >> ...
>> >> Let's face it, the proxy data are all reconstructed back to nice round
>> >> numbers. In the tree ring data, why go back to 1700? Why not 1699? Or
>> >> 1697? It's a computational convenience, nothing more. ...
>> >
>> >Are you saying that other time series were truncated by Mann et al
>> >besides the four mentioned in issue #1?
>>
>> I'm saying that if you look at all 22 SWM series for example,
>> they don't all stop at 1700, yet the author's chose 1700 as a
>> break-point. 1600 was also chosen. It was a convenience. They could
>> have used 1699, but 1700 is a nice round number.
>
>If that's what you were saying, then it wasn't analogous to the
>truncation issue.

I never said it was. I said that there is a certain amount of
fluidity available in the data (i.e. analyst preferences) and choosing
break-points is up to them.

>
>> >> ...
>> >> Your arbitrary choice of a point in the 1760's is noted. ...
>> >
>> >I quite purposefully used the precision quoted from the reference by
>> >Josh. How would you improve upon my choice of language?
>>
>> 1760 is no better or worse than 1730. ...
>
>Neither one warrants the root "perfect" in his context, fer sure.

But both are vastly superior to using the flawed data prior to
1730. Live with it, unless you are willing to show a detailed analysis
of your own backing up your contention that they should have been
included.

>
>> ... Again, in any analysis
>> containing suspect data you want to get of the suspect data, while
>> retaining as much good data as possible. As Josh said, an argument
>> could be made that 1770 was a better starting point. ...
>
>He said 1772. I agree with him.

Now back it up. Your opinion matters little. Josh stated that
it was arguable whether 1772 was a better date. Fine. That statement I
can agree with. Your absolute statement has zero to back it up.

>
>> ... The author's
>> apparently didn't agree.
>
>Nor did they think it appropriate to give any insight into whether they
>subjected all the series to the same criteria.

Spin again.

>
>> >> ... In
>> >> addition, your quibbling about the description of 1730 being the first
>> >> year or record is just that: a quibble. ...
>> >
>> >I thought about elaborating, but didn't want to clutter. In the body of
>> >the Nature paper, Mann et al claim to have included five series in their
>> >analysis that apparently were not included. It is in the context of such
>> >apparent sloppiness that M&M's questions warrant clear answers, untinged
>> >by the insults and invective that have become rote from Mann and his
>> >apologists.
>>
>> ROTFL. You've got a couple of hack who knock off a paper,
>> submit it completely full of errors to a third-rate "journal" then
>> whine when they are shown to be less than competent and you are
>> posturing about Mann et al's sloppiness? Too funny.
>
>I'm happy to disagree with every bit of your opinion on this.

The facts state otherwise. Live with it.

>
>> >> ... When an analysis is done, bad
>> >> data are routinely thrown out. That's a fact of life. The paper in
>> >> question is NOT meant for the general public but for other experts in
>> >> the field, experts who persumably are, unlike M&M, aware of the
>> >> problematic nature of some of the data. That being said, such a
>> >> notation can be considered a given.
>> >
>> >So you think it inappropriate that IPCC used the hocky stick graph in
>> >Fig 1 of the WG1 TAR policymaker summary?
>>
>> No, I think it entirely appropriate. ...
>
>LOL - you just got through saying the paper "is NOT meant for the
>general public but for other experts in the field".

The TAR is a summary of the science. Big difference, fool.

>
>> ... Mann et al have done
>> nothing wrong and I've looked at this far harder than you have. ...
>
>But you're blinded by your bias, having long ago drank the Mann-made
>Kool-aid. You may not even recognize when you're displaying how little
>regard you have for the scientific method.

ROTFL. Spoken like someone who decided his preferences a long
time ago. The facts say otherwise, but you've never let facts get in
the way before.

>>
>> > I thought it
>> >worthwhile to nip some of that in the bud in this case.
>>
>> See that you don't resort to any more of it and this thread
>> will remain quite pleasant.
>
>The thread included unpleasantness before I joined in. I can understand
>your nostalgia, however, for the unpleasantness in which you revel.
>

You're the one trolling, Steve. Don't blame me for catching
you at again.

David Ball

unread,
Dec 18, 2003, 11:38:19 PM12/18/03
to
Hi, Josh,

I finally found some time to go through a PCA analysis of
Mann's data in a fairly rigorous fashion. As I mentioned in an earlier
post, I downloaded their data, preprocessed it to get common starting
points as they did and then ran it through a completely different
algorithm. I didn't bother filling in the missing years at the end of
each series as MBH just used persistence and the PCA technique would
likely treat this as noise. I'd feel more comfortable extrapolating
values using polynomial or rational functions, but that's a lot of
work for a couple of points, at least in the basic series I've
processed so far.
Given the slightly different datasets and the different
algorithm - using Jacobi rotations to iterate toward the correct
eigenvalues and vectors - there are slight differences in the output
EOF's and eigenvalues in most of the cases I've looked at, but the
results are quite similar. The main difference between Mann et al and
my routine is in the number of PC's retained. Since I'm using Kaiser's
criterion - retaining only those PC's having eigenvalues greater than
or equal to 1 - there are differences. For the OK series in /STAHLE/OK
I retain 3 PC's as well as for the SOAMER series in /ITRDB/SOAMER. The
SWM series I retain far few PC's - only 4 - but there are a number
with eigenvalues hovering just below 1 so this slight difference is
not surprising. For the Australia series, the reverse happens. Where
MBH retained 3 PC's I actually retain 6, but the last 3 are hovering
just above 1.0. From what I've seen so far, there is not a huge
difference, certainly not the kind of difference hinted at by M&M.
M&M were not very honest with their Table 7 on page 762 of E&E
when they didn't disclose that the number of PC's listed in MBH were
THE NUMBER RETAINED and not the number of PC's. There should be a PC
for each dataset, but only a subset of them are retained. They make it
appear as if Mann et al were doing something wrong and they weren't.
That is the whole point of doing the PCA analysis in the first place:
to reduce the dimensionality of the problem. In addition, the 6% of
the variance explained that they list for the SWM series is clearly
wrong as is their 76%. That is way too high, but they were using
corrupt data. MBH's numbers are much different and agree within a few
percent of mine. I haven't finished processing the NOAMER data yet - I
didn't malloc an array large enough and got a segmentation fault when
I attempted to read in the array. I'll fix that up tomorrow. I'm also
going to compare some of Mann's PC's with my own and hopefully with
some of M&M's and plot them up. It will be interesting to see what
shakes out when I plot them.

Steve Schulin

unread,
Dec 19, 2003, 5:01:57 PM12/19/03
to
In article <4sl4uv0qpjnm4hj1o...@4ax.com>,
David Ball <wra...@mb.sympatico.ca> wrote:

> On Thu, 18 Dec 2003 10:00:38 -0500, Steve Schulin
> <steve....@nuclear.com> wrote:
>
> >In article <dg02uvc3ele66v652...@4ax.com>,
> > David Ball <wra...@mb.sympatico.ca> wrote:
> >
> >> >From: Steve Schulin (steve....@nuclear.com)
> >> >Subject: Re: Auditing the auditors #1
> >> >View: Complete Thread (18 articles)
> >> >Original Format
> >> >Newsgroups: sci.environment
> >> >Date: 2003-12-15 09:56:27 PST
> >> >
> >> >> >
> >> >> >Your abc breakdown ignores the M&M question about series 65?
> >> >>
> >> >> Josh spent 4 hours getting the information he posted. What are
> >> >> you doing, besides posting "what have you done lately" comments? ...
> >> >
> >> >My question was prompted because I got the impression that Josh was
> >> >attempting to fully address the questions in issue #1. His use of the
> >> >phrase "to get through #1" contributed to that impression. My question
> >> >was politely stated. If you had been asking question to me under similar
> >> >circumstances, it would have been no surprise to see you be quite
> >> >abusive about only seeing abc instead of abcd.
> >>
> >> What exactly is preventing you from investigating the question
> >> of series 65? Laziness? Indifference?
> >
> >Priorities.
>
> Incompetence more likely.

You're free to speculate all you want, bub. What particular competency
do you imagine would be needed to investigate the of series 65?

BTW, from the many quite obvious errors you make in this post, and some
not for the first time, it's clear enough that you aren't yet interested
in discovering whether Mann et al 1998 is replicable as published.

> >> Most of the legwork seems to stem from M&M's inability to read
> >> reference material, get the proper data in the proper format and a
> >> number of other housekeeping issues. It doesn't seem to have much to
> >> do with Mann et al so far.
> >
> >The world already knows much more about Mann et al's methodology,
> >because of McIntyre and McKitrick's efforts, than it knew for the 5
> >years after Nature published the article.
>
> No, people in this forum are farther ahead today because of
> people like Josh who are doing the legwork necessary to correct M&M's
> screw-ups. M&M haven't clarified anything, especially since they were
> looking at the wrong bloody data.

If M&M hadn't published, and Mann hadn't responded, how would Josh know
to look for 159 columns-worth of the so-called multiproxy series?


>
> >Your exclusive praise for
> >Josh's efforts is quite a contrast to your exclusive derision of M&M's.
> >At this point, we don't actually even know what data Mann et al used.
> >McIntyre, using the error-filled pcproxy.txt file, without corrections,
> >came up with a graph quite similar to the Mann et al hockey stick
> >product. I wouldn't be surprised if Mann's peers let him avoid ever
> >publicly providing enough info to allow his work to be replicable.
>
> Incorrect, I'm afraid. You can't use bad data, jiggle it and
> then claim that it resembles the original author's work. I looked very
> hard and I don't recall seeing strong warming during the LIA, which
> shows up on M&M's "analysis".

You're ballocksing things up in confusion. I agree that there's a big
difference between the graphs to which you refer. However, I was
speaking of yet a different graph. i was quite clear in describing it.

> >> As for being grateful, it was nice to have that pointed out. I
> >> wouldn't make too much out of it. I doubt very much it has a lot to do
> >> with anything in the grand scheme of things.
> >
> >That reminds me of a comment made by one of the relief pitchers of my
> >beloved Philadelphia Phillies: "Ten million years from now, when the sun
> >burns out and the Earth is just a frozen iceball hurtling through space,
> >nobody's going to care whether or not I got this guy out."
>
> LOL. No, it comes from knowing what PCA techniques do, Steve,
> and realizing that two years of missing data are doing to do jack.

Your peculiar juxtaposition of condescension and misstatement is quite
typical of you. The series we have discussed with two years of missing
data were not PC series.

>
> >
> >> >> What I find illuminating about Josh's post is that the basic
> >> >> information about why certain series were truncated is available in
> >> >> the reference material. When the answer to each question posed by M&M
> >> >> boils down to their inability to RTFR, there is a problem. ...
> >> >
> >> >It seems clear enough that Mann et al chose to use subsets of series 10
> >> >and 11. There is no indication that they systematically applied some
> >> >evaluative methodology to all the series. And your suggestion that "the
> >> >answer to each question posed by M&M boils down to their inability to
> >> >RTFR" seems quite an exaggeration.
> >>
> >> If they had bothered to read the references, they would have
> >> phrased their questions in a completely different way. Your posturing
> >> on the matter has been noted for some time as has your penchant for
> >> overstating the value of this paper - as one of the most important...
> >
> >That was quite appropriate phrase to use as title of that
> >thread-starting message. The initial post contained excerpts from a
> >hot-off-the-press op ed by paleoclimate researcher Bob Carter:
> >
> <silly op-ed deleted>
>
> Sorry, Steve, M&M's paper is a mess and anything based on it
> is necessarily flawed. I wonder why Mr. Carter didn't notice the
> warming that M&M showed during the LIA. That should have set off alarm
> bells for anyone with an ounce of understanding of the problem.

Of all the mistakes you make in this exchange, it's this one that really
displays how incredibly out-of-touch with reality you are. McIntyre and
McKitrick don't claim that Mann's method will result in a useful
product.

> >> As for retaining/throwing away suspect data, it is done all
> >> the time. In the event that data are suspect, and in this case, the
> >> references clearly show they are suspect, common practice is to throw
> >> them out unless some compelling reason can be found to retain them.
> >> Since there is no way to judge their worth, it makes sense to get rid
> >> of them, but retain as much information as possible, so carving the
> >> series back to the 60's or 70's doesn't make a lot of sense.
> >
> >In the 1998 Nature paper, Mann et al note what looks to be a general
> >principle: "... the mutual information contained in a diverse and widely
> >distributed set of independent climate indicators can more faithfully
> >capture the consistent climate signal that is present, reducing the
> >compromising effects of biases and weaknesses in the individual
> >indicators." Finding reason to drop cold-year data from Little Ice Age
> >and warm-year data from prior begs the question of whether all the
> >series were treated the same.
>
> Apparently, you didn't understand what I said to you. Quality
> control of any data MUST be done. It is a fact of life. If there is a
> suspicion that data are compromised, YOU DON'T USE THEM! Pretty
> straight forward, eh?

You and i apparently agree that Mann et al decided not to use some of
the data in some of the series. You seem to think it not necessary to
document (nor bother to answer questions about) what was decided to be
dropped and what criteria was used in coming to that decision.

> >> >> ... That likely
> >> >> stems from the unseemly rush they were in to get something in print. I
> >> >> can't think of a researcher anywhere who can knock off a peer-reviewed
> >> >> paper in a matter of weeks, without taking some serious shortcuts and
> >> >> at least in the case of their question 1, it appears that those
> >> >> shortcuts caused them to overlook basic information needed to do their
> >> >> "audit".
> >> >
> >> >You have often mistaken question marks for exclamation points, so your
> >> >take on the current matter comes as no surprise. Your apparent argument
> >> >that "of course Mann et al modified the source data" seems quite at odds
> >> >with the impression given by all those who have been echoing Mann's
> >> >recent claim that the data necessary to replicate their work is
> >> >available from the referenced sources of the data.
> >>
> >> LOL. Spin in any way you want, Steve, but as of now, it's M&M
> >> that are looking pretty bad.
> >
> >They have raised many interesting points.
>

> Really? From a screwed up data set? How do you figure? ...

For starters, I happily refer you back to the op ed by paleoclimate
researcher Bob Carter. You remember, the one which included

" ... it is clear that the McIntyre and McKitrick paper is set to
become one of the most important that has been published in recent
years. Apart from the light that it throws on the climate debate, the
paper raises profound issues to do with the integrity of scientific
publication, how data which underpins published papers should be
archived and made available, and whether science advice given to
governments on policy matters should be rigorously audited."

> ... Aren't
> you the one who often preaches garbage in, garbage out? ...

No. Perhaps you're misconstruing my oft-repeated theme that, for
purposes of public policy, climate models are of dubious predictive
value.

> ... A wonderful


> example of it here. Does this mean that Mann et al are perfect? I
> doubt it.

I can't imagine you not doubting it, given that the garbage to which you
appear to be referring was produced and distributed by Mann or his
designee.

> >And they have discovered facts which prompt other interesting points,
> >too. Take the five series (fran003, ital015, ital015x, spai026, and
> >spai047) that were listed amongst the proxies used in the 1998 paper but
> >which were not amongst the pcproxy.txt file. M&M ask Mann et al if those
> >series were in fact used. Mann hasn't answered that specifically yet,
> >but he does say that all the data for the 1998 paper is available in the
> >newly disclosed ftp site.
>
> You mean the ftp site that was available BEFORE M&M ever
> started working on their paper?

It was apparently so "available" that nary a single googled web page or
usenet post referenced it as of the day of Mann's recent disclosure.

> >Those five series don't appear in the
> >directories at the ftp site. I'm curious as to how this matter plays
> >out. If it turns out that these series were not used, I'm sure I'm not
> >the only one who would be interested to learn the particulars as to why
> >they are listed. Did Mann et al intend to use them originally? If so,
> >what changed their mind?
>
> I don't know. The answer will be interesting.
>
> >> More spin and posturing, I'm afraid. Only you could take
> >> standard practice and something very innocent and try to make
> >> something nefarious out of it.
> >
> >Having seen you often misrepresent stuff, I can understand why you find
> >such to be standard practice. Mann et al wrote their own paper. Nobody
> >forced them to use that word "available" in describing the column.
>
> Posturing again, Steve. You haven't got a clue what the
> analysis is doing, but you're absolutely sure that it's something bad.

> When told that it's pretty common, it becomes a grand conspiracy. ...

What are you babbling about? Does it in any way relate to the matter of
whether or not Mann et al were accurate in describing the contents of
the column?

> ... Your


> comments would carry a wee bit more weight if you could understand
> what was happening.

I understand enough to recognize at least some of your ignorances.

> >> >> ... Josh's point about the
> >> >> problematic nature of the 1723-1728 data is well taken. ...
> >> >
> >> >I too think it may very well have been reasonable to truncate this
> >> >series. I find it quite silly that you apparently chastise M&M for
> >> >asking the question, however.
> >>
> >> Asking the question? Never. I chastise them for their rush to
> >> print because it is a likely cause in their overlooking crucial
> >> information (i.e. reading the references).
> >
> >As it turns out, we're lucky that M&M did exactly what they did, and
> >that Mann reacted as he did. Otherwise, we wouldn't have been able to
> >contemplate on the sight of Mann's newly disclosed ftp site being
> >vacuumed like Hillary Clinton's former law firm's files were.
>
> LOL. Paul Farrar has already pointed out that the ftp site in
> question was available long ago as have I. Your ridiculous attempts to
> show it otherwise are quite pathetic.

I recall his claims as less than persuasive, especially given
Rutherford's description of the existance of several less than
comprehensive ftp sites. As for whatever activities of your own you're
referring to here, I confess to not recalling them at all. I'll keep an
eye out for details if I ever have the misfortune to review old posts of
yours.

> >> >> ... Using any of
> >> >> the data in that period would be a mistake. There is nothing
> >> >> inherently wrong with using 1730 - it's a nice round number - and as
> >> >> Josh pointed out other truncation points could have been adopted...
> >> >
> >> >And once its verified what Mann et al actually did, the effects of other
> >> >treatments will be interesting to explore.
> >>
> >> We're making progress. Certainly question 2, 3 and 12 have
> >> been put to bed. ...
> >
> >The recent revelations by Mann in response to M&M have already enabled
> >much to be known that was not known before.
>
> Yeah verily, and Schulin laid his holy message upon the
> peasants of usenet and he saw that all was good.

It's, uh, very interesting to see such fruits of your cognitive
workings. Despite your character flaws and whatnot, I'd like to stress
my hope that you aren't seriously mentally ill.


>
> >
> >> ... Question 6 is still a bit of a problem, not because
> >> of the data used but because some of the positions appear to be in the
> >> US. There's nothing inherently wrong with that, although Nigel seems
> >> to feel that isn't likely because some of the series resemble some
> >> European series.
> >> >> ...
> >> >> Let's face it, the proxy data are all reconstructed back to nice round
> >> >> numbers. In the tree ring data, why go back to 1700? Why not 1699? Or
> >> >> 1697? It's a computational convenience, nothing more. ...
> >> >
> >> >Are you saying that other time series were truncated by Mann et al
> >> >besides the four mentioned in issue #1?
> >>
> >> I'm saying that if you look at all 22 SWM series for example,
> >> they don't all stop at 1700, yet the author's chose 1700 as a
> >> break-point. 1600 was also chosen. It was a convenience. They could
> >> have used 1699, but 1700 is a nice round number.
> >
> >If that's what you were saying, then it wasn't analogous to the
> >truncation issue.
>
> I never said it was. I said that there is a certain amount of
> fluidity available in the data (i.e. analyst preferences) and choosing
> break-points is up to them.

We were discussing truncation. Your "computational convenience" line of
argument seems pretty irrelevant to the truncated series.

> >> >> ...
> >> >> Your arbitrary choice of a point in the 1760's is noted. ...
> >> >
> >> >I quite purposefully used the precision quoted from the reference by
> >> >Josh. How would you improve upon my choice of language?
> >>
> >> 1760 is no better or worse than 1730. ...
> >
> >Neither one warrants the root "perfect" in his context, fer sure.
>
> But both are vastly superior to using the flawed data prior to
> 1730. Live with it, unless you are willing to show a detailed analysis
> of your own backing up your contention that they should have been
> included.

All the series were from work by other researchers. Mann et al. did not
hint at any systematic effort to independently evaluate the series for
internal quality. Are you not curious as to whether Mann et al applied
truncation criteria in systematic fashion?

>
> >
> >> ... Again, in any analysis
> >> containing suspect data you want to get of the suspect data, while
> >> retaining as much good data as possible. As Josh said, an argument
> >> could be made that 1770 was a better starting point. ...
> >
> >He said 1772. I agree with him.
>
> Now back it up. Your opinion matters little. Josh stated that
> it was arguable whether 1772 was a better date. Fine. That statement I
> can agree with. Your absolute statement has zero to back it up.

LOL - you sure are confused. My absolute statements, including "I agree
with him" seems clear enough. It was your claim (your claim that that
Josh said 1770) that I was contrasting with the other, BTW.

> >> ... The author's
> >> apparently didn't agree.
> >
> >Nor did they think it appropriate to give any insight into whether they
> >subjected all the series to the same criteria.
>
> Spin again.

Spinning a web of truth, as I attempt, is preferable to the purposeful
lies of yours.

> >> >> ... In
> >> >> addition, your quibbling about the description of 1730 being the first
> >> >> year or record is just that: a quibble. ...
> >> >
> >> >I thought about elaborating, but didn't want to clutter. In the body of
> >> >the Nature paper, Mann et al claim to have included five series in their
> >> >analysis that apparently were not included. It is in the context of such
> >> >apparent sloppiness that M&M's questions warrant clear answers, untinged
> >> >by the insults and invective that have become rote from Mann and his
> >> >apologists.
> >>
> >> ROTFL. You've got a couple of hack who knock off a paper,
> >> submit it completely full of errors to a third-rate "journal" then
> >> whine when they are shown to be less than competent and you are
> >> posturing about Mann et al's sloppiness? Too funny.
> >
> >I'm happy to disagree with every bit of your opinion on this.
>
> The facts state otherwise. Live with it.

It's you who has been ignoring the material facts in this case, Mr. Ball.

> >> >> ... When an analysis is done, bad
> >> >> data are routinely thrown out. That's a fact of life. The paper in
> >> >> question is NOT meant for the general public but for other experts in
> >> >> the field, experts who persumably are, unlike M&M, aware of the
> >> >> problematic nature of some of the data. That being said, such a
> >> >> notation can be considered a given.
> >> >
> >> >So you think it inappropriate that IPCC used the hocky stick graph in
> >> >Fig 1 of the WG1 TAR policymaker summary?
> >>
> >> No, I think it entirely appropriate. ...
> >
> >LOL - you just got through saying the paper "is NOT meant for the
> >general public but for other experts in the field".
>
> The TAR is a summary of the science. Big difference, fool.

I'm sure you have a point in mind, but it's not clear what it could be.
I can understand why you think of foolishness shortly after calling the
TAR a summary. It brings to mind your own revealed foolishness in the
thread some time ago where it was appropriate to remind you that the A
in TAR stands for 'Assessment', not 'Abridgement'.

By the way, I notice that the IPCC's acronym for Fourth Assessment
Report is AR4. I'm not trying to infer that you don't already know this,
but I don't think it's been mentioned here in usenet yet, and perhaps
someone else will be glad to know this.

> >> ... Mann et al have done
> >> nothing wrong and I've looked at this far harder than you have. ...
> >
> >But you're blinded by your bias, having long ago drank the Mann-made
> >Kool-aid. You may not even recognize when you're displaying how little
> >regard you have for the scientific method.
>
> ROTFL. Spoken like someone who decided his preferences a long
> time ago. The facts say otherwise, but you've never let facts get in
> the way before.

I've never tried to hide behind a veneer of objectivity. I'm quite open
to learning from others who are interested in climate. I've even learned
stuff about weather and climate from you, although your signal-to-noise
ratio is pretty poor. Fact is, I know that you are nowhere near the
worst of the lying bastards promoting notions similar to your expression
of "the horror" of "what we've done" climate-wise. Not even at your
worst have I seen you say something as outrageous as the comment by Karl
and Trenberth in recent Science: "In the absence of climate mitigation
policies, the 90% probability interval for warming from 1990 to 2100 is
1.7? to 4.9?C."

>
> >>
> >> > I thought it
> >> >worthwhile to nip some of that in the bud in this case.
> >>
> >> See that you don't resort to any more of it and this thread
> >> will remain quite pleasant.
> >
> >The thread included unpleasantness before I joined in. I can understand
> >your nostalgia, however, for the unpleasantness in which you revel.
> >
> You're the one trolling, Steve. Don't blame me for catching
> you at again.

I've forgotten your peculiar definition of trolling. What do you mean
here?

Very truly,

BallB...@nuclear.com

David Ball

unread,
Dec 19, 2003, 8:29:20 PM12/19/03
to
On Fri, 19 Dec 2003 17:01:57 -0500, Steve Schulin
<steve....@nuclear.com> wrote:


>> >
>> >Priorities.
>>
>> Incompetence more likely.
>
>You're free to speculate all you want, bub. What particular competency
>do you imagine would be needed to investigate the of series 65?

You tell us.

>
>BTW, from the many quite obvious errors you make in this post, and some
>not for the first time, it's clear enough that you aren't yet interested
>in discovering whether Mann et al 1998 is replicable as published.

LOL. That's why I'm re-doing Mann et al from the ground up,
including re-doing their PCA calculations. Given that you lack that
wit to even understand what Mann is doing and lack the integrity to
acknowledge that M&M's amatuerish efforts, with a corrupt dataset no
less, show little of significance, it's pretty hard to take you
seriously.

>
>> >> Most of the legwork seems to stem from M&M's inability to read
>> >> reference material, get the proper data in the proper format and a
>> >> number of other housekeeping issues. It doesn't seem to have much to
>> >> do with Mann et al so far.
>> >
>> >The world already knows much more about Mann et al's methodology,
>> >because of McIntyre and McKitrick's efforts, than it knew for the 5
>> >years after Nature published the article.
>>
>> No, people in this forum are farther ahead today because of
>> people like Josh who are doing the legwork necessary to correct M&M's
>> screw-ups. M&M haven't clarified anything, especially since they were
>> looking at the wrong bloody data.
>
>If M&M hadn't published, and Mann hadn't responded, how would Josh know
>to look for 159 columns-worth of the so-called multiproxy series?

If M&M hadn't published, the media wouldn't have gotten in a
lather about a "study" that shows nothing.

>>
>> >Your exclusive praise for
>> >Josh's efforts is quite a contrast to your exclusive derision of M&M's.
>> >At this point, we don't actually even know what data Mann et al used.
>> >McIntyre, using the error-filled pcproxy.txt file, without corrections,
>> >came up with a graph quite similar to the Mann et al hockey stick
>> >product. I wouldn't be surprised if Mann's peers let him avoid ever
>> >publicly providing enough info to allow his work to be replicable.
>>
>> Incorrect, I'm afraid. You can't use bad data, jiggle it and
>> then claim that it resembles the original author's work. I looked very
>> hard and I don't recall seeing strong warming during the LIA, which
>> shows up on M&M's "analysis".
>
>You're ballocksing things up in confusion. I agree that there's a big
>difference between the graphs to which you refer. However, I was
>speaking of yet a different graph. i was quite clear in describing it.

Garbage in, garbage out, Steve.

>
>> >> As for being grateful, it was nice to have that pointed out. I
>> >> wouldn't make too much out of it. I doubt very much it has a lot to do
>> >> with anything in the grand scheme of things.
>> >
>> >That reminds me of a comment made by one of the relief pitchers of my
>> >beloved Philadelphia Phillies: "Ten million years from now, when the sun
>> >burns out and the Earth is just a frozen iceball hurtling through space,
>> >nobody's going to care whether or not I got this guy out."
>>
>> LOL. No, it comes from knowing what PCA techniques do, Steve,
>> and realizing that two years of missing data are doing to do jack.
>
>Your peculiar juxtaposition of condescension and misstatement is quite
>typical of you. The series we have discussed with two years of missing
>data were not PC series.

ROTFL. Posturing again, I see. Two years OF ANY DATA, are not
going to affect the results. Hell, they fill in missing data at the
end of some of the series with persistence values. No result there
either.


>> >>
>> >> If they had bothered to read the references, they would have
>> >> phrased their questions in a completely different way. Your posturing
>> >> on the matter has been noted for some time as has your penchant for
>> >> overstating the value of this paper - as one of the most important...
>> >
>> >That was quite appropriate phrase to use as title of that
>> >thread-starting message. The initial post contained excerpts from a
>> >hot-off-the-press op ed by paleoclimate researcher Bob Carter:
>> >
>> <silly op-ed deleted>
>>
>> Sorry, Steve, M&M's paper is a mess and anything based on it
>> is necessarily flawed. I wonder why Mr. Carter didn't notice the
>> warming that M&M showed during the LIA. That should have set off alarm
>> bells for anyone with an ounce of understanding of the problem.
>
>Of all the mistakes you make in this exchange, it's this one that really
>displays how incredibly out-of-touch with reality you are. McIntyre and
>McKitrick don't claim that Mann's method will result in a useful
>product.

LOL. It is M&M's own graphs that show strong warming during
the LIA. Mann's does not. The LIA was characterized by COLD
conditions. That should have set of alarm bells for M&M, especially
since a number of other researchers have built upon Mann's results, up
to and including a recent JofC article. In addition, vast numbers of
other papers have discussed the LIA. Given the problems M&M
encountered with the input data and the obvious output problem, the
author's should have said, "Oh, oh, we've got a problem with our input
data". Instead, they bulled forward in an attempt to make Mann et al
look bad and ended up with egg on their faces.

>
>> >> As for retaining/throwing away suspect data, it is done all
>> >> the time. In the event that data are suspect, and in this case, the
>> >> references clearly show they are suspect, common practice is to throw
>> >> them out unless some compelling reason can be found to retain them.
>> >> Since there is no way to judge their worth, it makes sense to get rid
>> >> of them, but retain as much information as possible, so carving the
>> >> series back to the 60's or 70's doesn't make a lot of sense.
>> >
>> >In the 1998 Nature paper, Mann et al note what looks to be a general
>> >principle: "... the mutual information contained in a diverse and widely
>> >distributed set of independent climate indicators can more faithfully
>> >capture the consistent climate signal that is present, reducing the
>> >compromising effects of biases and weaknesses in the individual
>> >indicators." Finding reason to drop cold-year data from Little Ice Age
>> >and warm-year data from prior begs the question of whether all the
>> >series were treated the same.
>>
>> Apparently, you didn't understand what I said to you. Quality
>> control of any data MUST be done. It is a fact of life. If there is a
>> suspicion that data are compromised, YOU DON'T USE THEM! Pretty
>> straight forward, eh?
>
>You and i apparently agree that Mann et al decided not to use some of
>the data in some of the series. You seem to think it not necessary to
>document (nor bother to answer questions about) what was decided to be
>dropped and what criteria was used in coming to that decision.

Not for the masses, no.

>> >>
>> >> LOL. Spin in any way you want, Steve, but as of now, it's M&M
>> >> that are looking pretty bad.
>> >
>> >They have raised many interesting points.
>>
>> Really? From a screwed up data set? How do you figure? ...
>
>For starters, I happily refer you back to the op ed by paleoclimate
>researcher Bob Carter.

Geologist, I thought.

>You remember, the one which included
>" ... it is clear that the McIntyre and McKitrick paper is set to
>become one of the most important that has been published in recent
>years. Apart from the light that it throws on the climate debate, the
>paper raises profound issues to do with the integrity of scientific
>publication, how data which underpins published papers should be
>archived and made available, and whether science advice given to
>governments on policy matters should be rigorously audited."

It addresses none of those points, but feel free to use your
vivid imagination any way you want.

>
>> ... Aren't
>> you the one who often preaches garbage in, garbage out? ...
>
>No. Perhaps you're misconstruing my oft-repeated theme that, for
>purposes of public policy, climate models are of dubious predictive
>value.

LOL. Wouldn't it have been simpler just to admit that you've
said it instead of dissembling?

>
>> ... A wonderful
>> example of it here. Does this mean that Mann et al are perfect? I
>> doubt it.
>
>I can't imagine you not doubting it, given that the garbage to which you
>appear to be referring was produced and distributed by Mann or his
>designee.

Funny, I've found no garbage on their website. It looks very
good.

>
>> >And they have discovered facts which prompt other interesting points,
>> >too. Take the five series (fran003, ital015, ital015x, spai026, and
>> >spai047) that were listed amongst the proxies used in the 1998 paper but
>> >which were not amongst the pcproxy.txt file. M&M ask Mann et al if those
>> >series were in fact used. Mann hasn't answered that specifically yet,
>> >but he does say that all the data for the 1998 paper is available in the
>> >newly disclosed ftp site.
>>
>> You mean the ftp site that was available BEFORE M&M ever
>> started working on their paper?
>
>It was apparently so "available" that nary a single googled web page or
>usenet post referenced it as of the day of Mann's recent disclosure.

Paul found it. I found it and it took just a couple of
minutes. I guess M&M just weren't trying very hard.


>> >
>> >Having seen you often misrepresent stuff, I can understand why you find
>> >such to be standard practice. Mann et al wrote their own paper. Nobody
>> >forced them to use that word "available" in describing the column.
>>
>> Posturing again, Steve. You haven't got a clue what the
>> analysis is doing, but you're absolutely sure that it's something bad.
>> When told that it's pretty common, it becomes a grand conspiracy. ...
>
>What are you babbling about? Does it in any way relate to the matter of
>whether or not Mann et al were accurate in describing the contents of
>the column?

It is standard practice to throw out suspect data in any
analysis, Steve. Think you can grasp that simple concept? You've only
been told repeatedly.

>
>> ... Your
>> comments would carry a wee bit more weight if you could understand
>> what was happening.
>
>I understand enough to recognize at least some of your ignorances.

So when can we expect your audit of Mann? LOL.

>> >As it turns out, we're lucky that M&M did exactly what they did, and
>> >that Mann reacted as he did. Otherwise, we wouldn't have been able to
>> >contemplate on the sight of Mann's newly disclosed ftp site being
>> >vacuumed like Hillary Clinton's former law firm's files were.
>>
>> LOL. Paul Farrar has already pointed out that the ftp site in
>> question was available long ago as have I. Your ridiculous attempts to
>> show it otherwise are quite pathetic.
>
>I recall his claims as less than persuasive, especially given
>Rutherford's description of the existance of several less than
>comprehensive ftp sites. As for whatever activities of your own you're
>referring to here, I confess to not recalling them at all. I'll keep an
>eye out for details if I ever have the misfortune to review old posts of
>yours.

More posturing. Is that all you can do?

>> >The recent revelations by Mann in response to M&M have already enabled
>> >much to be known that was not known before.
>>
>> Yeah verily, and Schulin laid his holy message upon the
>> peasants of usenet and he saw that all was good.
>
>It's, uh, very interesting to see such fruits of your cognitive
>workings. Despite your character flaws and whatnot, I'd like to stress
>my hope that you aren't seriously mentally ill.

I'm fine. Thanks for asking. And you? How's life in
Schulin-land. Still the cesspool it's always been?


>> >
>> >If that's what you were saying, then it wasn't analogous to the
>> >truncation issue.
>>
>> I never said it was. I said that there is a certain amount of
>> fluidity available in the data (i.e. analyst preferences) and choosing
>> break-points is up to them.
>
>We were discussing truncation. Your "computational convenience" line of
>argument seems pretty irrelevant to the truncated series.

On the contrary. As Josh pointed out, an argument COULD have
been made that the series should have been truncated in the 1770's.
You claim 1760 - without anything to support your POV naturally -
would have been better. The author's clearly thought a third thing and
acted upon that. It was their choice to make and since you have been
unable to show anything wrong with that choice other than you dislike
absolutely everything to do with the paper, without having to get into
the touchy area of actually understanding it, their choice is the one
that matters. Your hubris doesn't enter into the picture anywhere.


>> But both are vastly superior to using the flawed data prior to
>> 1730. Live with it, unless you are willing to show a detailed analysis
>> of your own backing up your contention that they should have been
>> included.
>
>All the series were from work by other researchers. Mann et al. did not
>hint at any systematic effort to independently evaluate the series for
>internal quality. Are you not curious as to whether Mann et al applied
>truncation criteria in systematic fashion?

With series of this type, it would be impossible to apply
systematic criteria to all the series. They need to be evaluated one
by one.

>
>>
>> >
>> >> ... Again, in any analysis
>> >> containing suspect data you want to get of the suspect data, while
>> >> retaining as much good data as possible. As Josh said, an argument
>> >> could be made that 1770 was a better starting point. ...
>> >
>> >He said 1772. I agree with him.
>>
>> Now back it up. Your opinion matters little. Josh stated that
>> it was arguable whether 1772 was a better date. Fine. That statement I
>> can agree with. Your absolute statement has zero to back it up.
>
>LOL - you sure are confused. My absolute statements, including "I agree
>with him" seems clear enough. It was your claim (your claim that that
>Josh said 1770) that I was contrasting with the other, BTW.

I could have sworn you said: From what you've reported, it


looks like some other truncation point would have been more perfect

than 1730, however: 1723, 1728, or somewhere in 1760s. An absolute
statement, Steve.


>> >> ... The author's
>> >> apparently didn't agree.
>> >
>> >Nor did they think it appropriate to give any insight into whether they
>> >subjected all the series to the same criteria.
>>
>> Spin again.
>
>Spinning a web of truth, as I attempt, is preferable to the purposeful
>lies of yours.

LOL. Steve, you wouldn't know the truth if it walked up and
bit you on the ass. Lies, now. Those are your specialty and you're
very good at telling them.

>
>> >> >> ... In
>> >> >> addition, your quibbling about the description of 1730 being the first
>> >> >> year or record is just that: a quibble. ...
>> >> >
>> >> >I thought about elaborating, but didn't want to clutter. In the body of
>> >> >the Nature paper, Mann et al claim to have included five series in their
>> >> >analysis that apparently were not included. It is in the context of such
>> >> >apparent sloppiness that M&M's questions warrant clear answers, untinged
>> >> >by the insults and invective that have become rote from Mann and his
>> >> >apologists.
>> >>
>> >> ROTFL. You've got a couple of hack who knock off a paper,
>> >> submit it completely full of errors to a third-rate "journal" then
>> >> whine when they are shown to be less than competent and you are
>> >> posturing about Mann et al's sloppiness? Too funny.
>> >
>> >I'm happy to disagree with every bit of your opinion on this.
>>
>> The facts state otherwise. Live with it.
>
>It's you who has been ignoring the material facts in this case, Mr. Ball.

Please enlighten me, Perfesser. And please, no nonsense about
how M&M's results show what Mann's results would have been if they'd
done their analysis properly.


>
>> >> ... Mann et al have done
>> >> nothing wrong and I've looked at this far harder than you have. ...
>> >
>> >But you're blinded by your bias, having long ago drank the Mann-made
>> >Kool-aid. You may not even recognize when you're displaying how little
>> >regard you have for the scientific method.
>>
>> ROTFL. Spoken like someone who decided his preferences a long
>> time ago. The facts say otherwise, but you've never let facts get in
>> the way before.
>
>I've never tried to hide behind a veneer of objectivity.

Nor one of honesty.

> I'm quite open
>to learning from others who are interested in climate. I've even learned
>stuff about weather and climate from you, although your signal-to-noise
>ratio is pretty poor. Fact is, I know that you are nowhere near the
>worst of the lying bastards promoting notions similar to your expression
>of "the horror" of "what we've done" climate-wise. Not even at your
>worst have I seen you say something as outrageous as the comment by Karl
>and Trenberth in recent Science: "In the absence of climate mitigation
>policies, the 90% probability interval for warming from 1990 to 2100 is
>1.7? to 4.9?C."

Actually, the worst lying bastard in this forum, is you. Have
a good look in the mirror.


David Ball

unread,
Dec 20, 2003, 12:01:17 AM12/20/03
to
Nigel,

You mentioned some time back that you'd found a problem with
the Vaganov series principle components. Do you recall what they were?
I've finished processing all the PC's for Mann's data and have
retrieved all of the remaining proxy datasets from their website. I
haven't found any problems with the PC's, though my method retains
many more than Mann does (10 in the case of the Vaganov series). I
just want to make sure that I haven't overlooked anything in the
meantime.


Josh Halpern

unread,
Dec 20, 2003, 12:17:19 AM12/20/03
to
Thanks David, this is well beyond me, although I ran into a strange
problem today which probably will require my learning principal
component analysis....Never say never.

Josh Halpern

(It's also a very busy time for me so posting is slow)

Josh Halpern

unread,
Dec 20, 2003, 12:24:57 AM12/20/03
to

Nigel Persaud wrote:

Bradley? (Actually Jones and Bradley) as in Mann, Bradley and Hughes.

josh halpern

Josh Halpern

unread,
Dec 20, 2003, 12:33:26 AM12/20/03
to

Nigel Persaud wrote:
SNIP....

>>Again, On balance this illustrates the principal of RTFR and the danger
>>of someone unfamiliar with an area trying to do an "audit". To use the
>>central European index before 1550 would clearly have been a mistake.
>>Note also that Bradley was one of the two book editors, so he surely
>>knew a great deal about these series. Phil Jones was the other.
>>
>>
>
>
>Josh, this is an ingenious rationalization.
>
>However, first of all, it is common ground now that MBH deleted the
>first 25 years of this series. Prior to M&M, no one knew that. (The
>undeleted series are also at ftp/MBH98. Secondly, MBH did not disclose
>that they deleted the first 25 years.)
>

How about anyone who was familiar with the two historical series, and
BTW, by stating the years of the Manley and Pfister records that they
used MBH did disclose. The only people who would not know are those not
in the field (Manley and Pfister are very famous records) and those who
did not know should have read the references.

>The deletion may or may not be
>justifiable in climatological terms, but people relying on MBH should
>be advised MBH did this without having to parse through original
>sources. The deletion should have been annotated in the Supplementary
>Information directly and justified. Some readers might not agree with
>this approach and are entitled to know. Third, it is far from obvious
>that it would be a "mistake" to use the first 25 years. After all, MBH
>is using PROXIES. The first 25 years of the historical record are a
>better PROXY than some tree ring. Fourth, there is no way of knowing
>whether your rationalization is the same as Mann's. The periods which
>he deleted in Central England do not match exactly to the periods
>which you purport to rationalize. Finally, while you're at this, can
>you figure out why MBH used summer versions instead of annual versions
>for these series?
>
>

An underlying issue is that M&M did their reconstruction using the full
record. However, as we have seen this is almost certainly a bad choice
and would therefore make their reconstruction less accurate.

josh halpern

Nigel Persaud

unread,
Dec 20, 2003, 10:28:48 AM12/20/03
to
> >
> >I agree that ftp/MBH98 locates the series in North America. So what?
> >ftp/MBH98 is not source data. Can you show where the data in #37 comes
> >from? Nigel
> >
>
> Bradley? (Actually Jones and Bradley) as in Mann, Bradley and Hughes.
>

Can you identify which series in Jones and Bradley?

David Ball

unread,
Dec 20, 2003, 11:56:55 AM12/20/03
to
On Sat, 20 Dec 2003 05:17:19 GMT, Josh Halpern
<j.ha...@incoming.verizon.net> wrote:

>Thanks David, this is well beyond me, although I ran into a strange
>problem today which probably will require my learning principal
>component analysis....Never say never.
>
>Josh Halpern
>
>(It's also a very busy time for me so posting is slow)

I understand completely. I finished processing the NOAMER data
and retain WAAAY more PC's than Mann et al, 40 to be exact, though
many are hovering just above the noise level.
I'm going take a stab at a complete reconstruction a-la Mann
et al, using my input data and see what I come up with. In inspecting
some of the PC's there are some noticable differences, primarily at
the ends of the series. At the start of some series, the PC's show
more interannual variability, while it is difficult to gauge how the
ends of the series will impact the consolidated series because I've
made no attempt at all to fill in the missing years pre-1980 with
persistence values.

Nigel Persaud

unread,
Dec 20, 2003, 6:37:31 PM12/20/03
to
David, when I re-did the Vaganov PCs, the order of some PCs flipped
and so there was little correlation between the ftp/MBH98 PC2 and my
PC2 and I think I may have posted that up. But I went back and
compared the PC2 and PC3 and those correlated.

In my calculations, as I mentioned before, I find discrepancies
between the NOAMER, SOAMER and AUSTRAL PCs and ftp/MBH98 PCs, while
the Vaganov and Stahle tie together much closer. Nigel

David Ball <wra...@mb.sympatico.ca> wrote in message news:<mnl7uvo5jlr3p6fdh...@4ax.com>...

David Ball

unread,
Dec 20, 2003, 7:00:46 PM12/20/03
to
On 20 Dec 2003 15:37:31 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>David, when I re-did the Vaganov PCs, the order of some PCs flipped


>and so there was little correlation between the ftp/MBH98 PC2 and my
>PC2 and I think I may have posted that up. But I went back and
>compared the PC2 and PC3 and those correlated.
>
>In my calculations, as I mentioned before, I find discrepancies
>between the NOAMER, SOAMER and AUSTRAL PCs and ftp/MBH98 PCs, while
>the Vaganov and Stahle tie together much closer. Nigel
>

I've found a lot of differences, but those, I think, stem from
the different approach I'm using to determining them. My eigenvalues
are very similar, though the first two PC's from the AUSTRAL series
are much smaller, and the percentage of variance explained is very
close - 17% to 16%. Visual inspection of the PC's is not going to tell
much, however, so I have to complete the entire MBH procedure to see
what can be seen. Since I've not used any fills I should be able to
address some of Mr. Schulin's concerns directly.

Joshua Halpern

unread,
Dec 20, 2003, 7:08:51 PM12/20/03
to
pers...@yahoo.com (Nigel Persaud) wrote in message news:<2860b88e.03122...@posting.google.com>...

No, not at this point, Frankly I have not looked, but I can identify
the Bradley, which is why I think this issue is almost certainly a
non-issue. It is somewhat amusing the way Mann has been picked on,
Bradley and Hughes are if anything more prominent and associated with
better known climate research organizations than Mann, who was a
post-doc at the time of publication.

Kind of like Rasool and Schneider V2.1 :)

josh halpern

Nigel Persaud

unread,
Dec 21, 2003, 2:03:30 AM12/21/03
to
> No, not at this point, Frankly I have not looked, but I can identify
> the Bradley, which is why I think this issue is almost certainly a
> non-issue. It is somewhat amusing the way Mann has been picked on,
> Bradley and Hughes are if anything more prominent and associated with
> better known climate research organizations than Mann, who was a
> post-doc at the time of publication.
>
> Kind of like Rasool and Schneider V2.1 :)
>
> josh halpern


Then this problem is still outstanding.

David Ball

unread,
Dec 22, 2003, 7:54:59 PM12/22/03
to
Has anyone figured out which gridboxes from the CRU dataset
MBH used as their 1080+ series? I've downloaded the data, truncated it
into the 1902-1995 period but need to select a subset. MBH are not
clear how this is done and M&M merely refer to data file containing
the points which I haven't been able to locate.

David Ball

unread,
Dec 22, 2003, 10:11:12 PM12/22/03
to
OK, I think I'm getting somewhere on Question 6. The two sets
of series appearing in series 21 to 42 come from "Climate Since AD
1500" Collected reconstructions, 500 Years, Bradley and Jones 1992.
Subsets can be viewed at:

ftp://ftp.ngdc.noaa.gov/paleo/climate1500ad/

The readme file:

ftp://ftp.ngdc.noaa.gov/paleo/climate1500ad/readme_climate1500ad.txt

details what each file in the directory contains. Pay special
attention to the file ch13.txt. The readme calls the file CH13.DAT so
I'm not sure if the two are one and the same. In that ch13.txt file
are a list of stations that are used and you can clearly see that
there are stations in North America - New Haven, Toronto and a site in
Minnesota. Now, getting from these data to the series listed in MBH
will take a little work, but I don't think it there is any requirement
that the Bradley and Jones series be in Europe. It would be
interesting to see if series 37 in the M&M spreadsheet could be
reconstructed from these data so that we could pinpoint exactly if
there is an error.

Josh Halpern

unread,
Dec 22, 2003, 10:58:26 PM12/22/03
to
I found those yesterday. Sorry for not writing. To some extent you can
identify which files are used by the starting dates.

Josh Halpern

Nigel Persaud

unread,
Dec 23, 2003, 12:34:28 AM12/23/03
to
Why don't you look at MM's http://www.climate2003.com/audit2.htm#37
and related series, which discusses the start dates, correlations of
Bradley-Jones series in much more detail than you've got to so far? It
has exact references to the WDCP locations. Nigel

Nigel Persaud

unread,
Dec 23, 2003, 12:41:08 AM12/23/03
to
M&M have a complete description of downloading Mann data: see
http://www.climate2003.com/scripts/read.mann.txt. If you go to
http://www.climate2003.com/computations.html, M&M also give
description of calculations to recover Jones data and a script in R to
do it. It's a pretty reasonable description. M&M do much more than
merely refer to the data file; they give you a script for getting it.
What more do you want? Nigel


David Ball <wra...@mb.sympatico.ca> wrote in message news:<4e4fuv003ccemmbh7...@4ax.com>...

David Ball

unread,
Dec 23, 2003, 10:47:33 AM12/23/03
to
On 22 Dec 2003 21:41:08 -0800, pers...@yahoo.com (Nigel Persaud)
wrote:

>M&M have a complete description of downloading Mann data: see


>http://www.climate2003.com/scripts/read.mann.txt. If you go to
>http://www.climate2003.com/computations.html, M&M also give
>description of calculations to recover Jones data and a script in R to
>do it. It's a pretty reasonable description. M&M do much more than
>merely refer to the data file; they give you a script for getting it.
>What more do you want? Nigel

Already done. Neither program specifies WHAT or HOW the subset
of the CRU data are selected. In

http://www.climate2003.com/scripts/read.jones.txt

the process is done by:

#PART 2 - TRUNCATING, SCALING AND LAT-WEIGHTING JONES DATA PER MBH98
RECIPE
#this requires prior downloading of Mann data from his FTP site. See
"read.mann"

base.out<-"c:/climate/data/mann"

#LOAD MANN GRIDPOINTS
load("c:/climate/data/mann/gridpoints.tab"); #loads grid2 with
grid2[,5] being 1082 Jones grid-box IDs

LOOK AT THE CODE ABOVE. I'M LOOKING THE 1082 JONES GRID-BOX ID'S.

tempmann<-sort(grid2[,5])##loads Mann grid-boxes as sorted Jones ID
numbers

#LOAD RAW JONES DATA AND MAKE 1902-95 SUBSET FOR MANN
load("c:/climate/data/jones/hadcruv.tab") ###month 553 is 1902 Jan;
v<-array(v,dim=c(dim(v)[1],36*72));####this puts the time series for
grid box (i,j) into