I'm writing to let you know that we made a fresh release of
ccc-gistemp last night.  This has quite a bit of clarification,
especially in steps 1 and 2, and some new tools to help with comparing
results, both from different runs and also against the run you
provided in December.  We've removed most of the uses of intermediate
data files.  There are also work arounds for some bugs we encountered
in tar file handling on some platforms.

Here is the release:

<http://ccc-gistemp.googlecode.com/files/ccc-gistemp-0.3.0.tar.gz>

and here is a file comparing the results from the release to the ones
you provided:

<http://ccc-gistemp.googlecode.com/files/ccc-gistemp-0.3.0-comparison-...>

This latter file was generated by simply running

  python tool/regression.py

Please feel free to download it and run it, or just browse through the
code.  There's not much overall documentation yet (that's a bug!), but
anyone familiar with GISTEMP will find it obvious.  step2.py should
give you some indication of the sort of work we are doing.

Ongoing work will continue this type of clarification through more of
the code.  We're also going to modify all of the code to use a single
representation of a temperature series, and move any remaining
intermediate file I/O out into the tool/ directory.  That will also
include some code to round or truncate intermediate data, to match the
effect of GISTEMP intermediate files which have been removed.  For
instance, GISTEMP STEP2 reads the latitude and longitude of stations
from v2.inv (where it is in hundredths of degrees) and then rounds it
to tenths of degrees to write to an intermediate file.  Although we no
longer have the file, our code explicitly does the rounding step to
maintain GISTEMP compatibility (because it affects, in a small way,
the behaviour of both step 2 and step 3).

I'm hoping to get to a stage at which it is possible to run
ccc-gistemp either in a "simple" mode (in which there are no
intermediate files and no such rounding) or in a "GISTEMP
compatibility" mode - with some intermediate files, and some rounding.
If we compute and quantify the differences between these modes, then
maybe some future release will do without the compatibility mode
altogether.

As we arrive at a detailed understanding of the whole code base, we
will also write some accompanying documentation to describe the whole
algorithm.

Regards,

Nick Barnes
Clear Climate Code project