International Surface Temperature Initiative.

[David Jones]

I've just been tweaking ccc-gistemp to run with the ISTI data
http://data.giss.nasa.gov/gistemp/tabledata_v3/NH.Ts.txt

tool/run.py -p 'data_sources=isti.merged.dat;element=TAVG' -s 0-1,3-5

(no step 2, as there is no urban metadata, and it breaks anyway)

ultra-preliminary analysis (takes about 1.5 hours on my laptop):

Northern Hemisphere: no change
Southern Hemisphere: Differences of interest

drj

http://chart.apis.google.com/chart?cht=lxy&chds=-100,100,-100,150,-100,100,-100,150,-100,100,-100,150,-100,100,-100,150,-100,100,-100,150,-100,100,-100,150&chd=t:-999|-51,-37,-36,-49,-68,-50,-56,-64,-50,-20,-42,-38,-40,-45,-38,-34,-32,-18,-32,-16,-6,-10,-35,-29,-42,-28,-18,-56,-36,-35,-34,-33,-47,-44,-18,-4,-33,-61,-45,-28,-26,-9,-20,-14,-9,-7,6,-6,-1,-25,4,10,-2,-29,2,-9,-3,18,23,9,11,10,8,8,13,-6,5,14,5,-3,-19,3,6,25,-6,-14,-35,-1,15,11,10,8,11,11,-22,-20,-3,0,-10,-2,3,-13,-22,19,-15,-1,-22,19,8,17,24,50,7,38,15,7,19,37,48,35,61,51,17,25,47,75,41,66,98,69,69,83,92,92,85,105,98,107,78,86,114,92,96,94,-999|-999|-46,-35,-44,-59,-79,-60,-42,-59,-50,-24,-36,-38,-48,-54,-40,-42,-35,-20,-37,-12,-6,-8,-31,-28,-45,-24,-18,-54,-35,-39,-34,-28,-40,-31,-8,-2,-32,-60,-40,-26,-16,-2,-18,-12,-9,-1,10,-6,-1,-22,7,17,5,-23,9,-3,4,24,31,15,18,16,8,14,18,3,11,18,10,3,-14,6,8,32,3,-10,-30,1,14,14,11,10,11,11,-23,-20,-5,0,-12,-5,-1,-17,-23,13,-16,-5,-25,17,6,14,23,48,3,35,12,2,18,31,45,32,61,50,17,24,46,73,39,65,92,66,68,83,88,94,79,102,101,106,78,84,109,94,102,94,-999&chxt=x,y,r&chxl=0:|1880||||||||||1890||||||||||1900||||||||||1910||||||||||1920||||||||||1930||||||||||1940||||||||||1950||||||||||1960||||||||||1970||||||||||1980||||||||||1990||||||||||2000||||||||||2010||||2014|1:||-0.5|+0.0|+0.5|+1.0||2:||-0.5|+0.0|+0.5|+1.0|&chco=ff0000,000000,ff0000,ff0000,000000,000000&chls=1|1|1,8,2|1|1,8,2|1&chm=&chs=600x500&chtt=NH+land+stations&chdl=GHCN%2BSCAR%7CISTI&chdlp=t

http://chart.apis.google.com/chart?cht=lxy&chds=-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100,-100,100&chd=t:-999|-21,-22,-13,1,-36,-33,-52,-58,-25,-13,-65,-77,-59,-58,-45,-32,-30,-28,-48,-52,-26,-25,-37,-50,-59,-40,-30,-51,-41,-45,-32,-37,-23,-25,-1,-11,-26,-38,-39,-5,-28,-20,-18,-31,-29,-35,-6,-19,-15,-31,-28,-19,-16,-18,-19,-28,-10,-8,-3,-25,7,1,6,-7,-2,-9,-20,-2,-21,-21,-27,-16,-8,-11,-20,-9,-10,10,-3,-5,-16,7,-9,-9,-29,-14,-10,-2,-8,6,11,-3,15,26,7,5,-14,22,18,23,45,36,19,37,26,32,28,40,48,37,40,50,27,27,27,35,52,39,68,47,45,53,66,62,50,70,55,64,51,70,68,62,55,67,-999|-999|-5,-11,-21,-26,-50,-34,-61,-86,-70,-50,-80,-78,-74,-70,-45,-27,-20,-12,-41,-27,-11,-12,-22,-25,-54,-31,-23,-33,-30,-35,-22,-31,-7,-11,-1,4,-15,-32,-33,-4,-19,-14,-17,-28,-19,-30,-5,-18,-9,-23,-18,-9,-7,-6,-5,-15,-2,-39,3,-12,12,23,25,7,13,10,7,19,6,0,-3,-4,3,-1,-11,-10,-10,13,-2,-6,-17,7,-8,-9,-31,-12,-10,-4,-11,6,9,-5,14,25,4,1,-17,17,12,17,40,32,21,40,24,28,30,49,51,35,44,47,29,31,33,38,55,48,76,46,42,58,75,67,61,74,66,73,55,84,76,75,66,87,-999&chxt=x,y,r&chxl=0:|1880||||||||||1890||||||||||1900||||||||||1910||||||||||1920||||||||||1930||||||||||1940||||||||||1950||||||||||1960||||||||||1970||||||||||1980||||||||||1990||||||||||2000||||||||||2010||||2014|1:||-0.5|+0.0|+0.5||2:||-0.5|+0.0|+0.5|&chco=ff0000,000000,ff0000,ff0000,000000,000000&chls=1|1|1,8,2|1|1,8,2|1&chm=&chs=600x500&chtt=SH+land+stations&chdl=GHCN%2BSCAR%7CISTI&chdlp=t

[Filipe Pires Alvarenga Fernandes]

The differences for the SH deserve some attention!  I would love to see some error bars or something similar there.

Do you have the code for this in the main repository? In a branch?  Just checked out the latest source and I was unable to run.

IOError: [Errno 2] No such file or directory: 'input/isti.merged.dat'

Thanks,

-Filipe

--
You received this message because you are subscribed to the Google Groups "CCC GISTEMP discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ccc-gistemp-dis...@googlegroups.com.
To post to this group, send email to ccc-giste...@googlegroups.com.
Visit this group at http://groups.google.com/group/ccc-gistemp-discuss.
For more options, visit https://groups.google.com/d/optout.

[David Jones]

On 8 August 2014 15:33, Filipe Pires Alvarenga Fernandes

<oce...@gmail.com> wrote:
> The differences for the SH deserve some attention! I would love to see some
> error bars or something similar there.

Yes, I agree about the attention. Glad you're having a look.

> Do you have the code for this in the main repository? In a branch? Just
> checked out the latest source and I was unable to run.
>
> IOError: [Errno 2] No such file or directory: 'input/isti.merged.dat'

Sorry about that, I rushed out my email. But yes, main github repo,
master branch.

First you need to get the ISTI dataset which I've added to fetch:

tool/fetch.py isti

(annoyingly, this will also get all the data required for the usual
analysis, and none of that is required)

drj

[David Jones]

We should treat the "ultra-preliminary" analysis with extreme caution.

For example, the ISTI curve (black) in the Southern Hemisphere shows a
large (about 0.4) dip in 1937.

That turns out to be due to a huge negative anomaly for a single
station: CI000085469 -27.1670 -109.4330 69.0 ISLA_DE_PASCUA

which turns out to be just bogus data.

The Stage 3 ISTI dataset hasn't had any Quality Control yet (that's Stage 4).

drj

[David Jones]

On 12 August 2014 08:14, David Jones <d...@pobox.com> wrote:
> We should treat the "ultra-preliminary" analysis with extreme caution.
>
> For example, the ISTI curve (black) in the Southern Hemisphere shows a
> large (about 0.4) dip in 1937.
>
> That turns out to be due to a huge negative anomaly for a single
> station: CI000085469 -27.1670 -109.4330 69.0 ISLA_DE_PASCUA
>
> which turns out to be just bogus data.
>
> The Stage 3 ISTI dataset hasn't had any Quality Control yet (that's Stage 4).

In addition to the above record being bogus I've also found the
following to be bogus (and probably responsible for the deviations
since 2000 and between 1940 and 1950):

WAXLT556695 WALFISCH_BAY
AYM00089034 BELGRANO_II

and

AYXLT835318 MIZUHO

is suspect at best.

I'll see if I can do a run without those stations.

drj

[David Jones]

On 13 August 2014 22:02, David Jones <d...@pobox.com> wrote:
> On 12 August 2014 08:14, David Jones <d...@pobox.com> wrote:
>> We should treat the "ultra-preliminary" analysis with extreme caution.
>>
>

> I'll see if I can do a run without those stations.

i've now done a run with a QC'd dataset. Basically no different:

http://climatecode.org/blog/2014/08/ccc-gistemp-and-isti/

drj

[Filipe Pires Alvarenga Fernandes]

That is very interesting.  Maybe, in the name of openness, those findings could be logged in the github repository as issues.

-Filipe

drj

[David Jones]

On 29 August 2014 12:20, Filipe Pires Alvarenga Fernandes

<oce...@gmail.com> wrote:
> That is very interesting. Maybe, in the name of openness, those findings
> could be logged in the github repository as issues.

um, on what repo? I'm not sure what you're suggesting.

It's hardly a bug in ccc-gistemp that some stations in an unQC'd
dataset need QC.

I'm intending to write more about MADQC and show the effect on some
specific stations.

drj

[Filipe Pires Alvarenga Fernandes]

Maybe my interpretation of a "github" issue is different from yours.  I am not talking about code bugs, but use cases.

I believe that this *is* a discovery that goes together with the code.  If you write in the mailing list or  a blog post, the information might get lost.

I have the habit of logging github an issues for findings similar to that one.  Sometimes I just referenced another issue, that would be the case if the data is in another github repository and the QC issue is there.

The actual issue could be just a link to the mailing list or the blog where the discussion about the data took place.

-Filipe

drj

[Filipe Pires Alvarenga Fernandes]

Basically this is how I think:  It is a "data bug."  The data are used by ccc-gistemp and the QC was performed after running cc-gistemp.  So I do believe that this information should be at ccc-gistemp repository (as well as any other documentation related to this data).

That is my suggestion.  I find that reading code issues on github with that kind if information extremely helpful (and easier to find too).

-Filipe