New members; plan to complete phase 1 by year-end

[Nick Barnes]

Please make our new members welcome. Daniel Rothenberg is an actual
climate scientist, at Cornell. Richard Brooksby is my fellow director
of Ravenbrook. Gareth Rees is an old friend and one-time colleague,
who now works in the games industry and has lots of Python experience.
Tucker Withington is a colleague of mine from the 1990s, an old-skool
hacker based in Massachusetts. Richard Hendricks has come out of the
blue and is already helping with some analysis of Windows issues.

I imagine some of you will have seen the various posts I've made in
the last few days to our shiny new blog <http://clearclimatecode.org/>.
They have mostly been a description of the project status and a
restatement, for newcomers, of the project goals and policies. In
particular I have reiterated that the project will welcome anyone who
can contribute positively, regardless of their views of climate
change.

I've been thinking about project progress. I'd very much like us to
make an up-to-date release which is entirely in Python, completing
phase 1, *by the end of the year*. Here's what remains to be done for
that:

- finish STEP5;
- do STEP4 (including the driver scripts);
- run.sh;
- test on Windows without cygwin;
- port recent GISTEMP changes.

David is working on STEP5, so should be the one to finish that. I
have suggested to Gareth that he do the driver scripts; is this OK
Gareth? Richard Hendricks, once the driver script is done, can you
test on a Windows machine without cygwin? Most of the recent GISTEMP
changes are in STEP0, concerning changes to data sources, and since I
did step0.py in the first place, I'll work on that.

That leaves STEP4, which is pretty straightforward. Any volunteers?

Does anyone have any objections or difficulty with this plan?

Once we are all in Python I expect things to move a bit more quickly,
because there's no configuration headache and it is much easier (for
non-Fortranistas) to debug and develop interactively. We will also
have opportunities for the more web-developer type of volunteer to
work on data presentation and so on.

Nick B

[Julian Todd]

Good to see this project is at last starting up again.

I feel it would be useful to not shy so much from the point of the
public goals.

In the context of the current swift-boating exercise of the Copenhagen
talks, I discovered the following email exchange between a skeptic and
the CRU that looks a lot like the exchange between Watts and Hansen
that got his climate code disclosed -- from 2001.

http://www.john-daly.com/cru/emails.htm


I'd like to argue that, had the CRU data and code been released then,
and had this clear climate code project been started 5 years ago, the
damage of the last few weeks might not have been possible.

A lot of hay is being made about the secrecy and unreliability of the
CRU data from people who don't have any data to contribute in its
stead, and are able to claim that they have been shut out for reasons
of their own (incredible) scientific integrity.


Julian.

PS, do you think it may eventually be possible to run the code on scraperwiki?

> --
>
> You received this message because you are subscribed to the Google Groups "CCC GISTEMP discussion" group.
> To post to this group, send email to ccc-giste...@googlegroups.com.
> To unsubscribe from this group, send email to ccc-gistemp-dis...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ccc-gistemp-discuss?hl=en.
>
>
>

[Paul Ollis]

On Dec 6, 4:49 pm, Nick Barnes <n...@ravenbrook.com> wrote:
>
> That leaves STEP4, which is pretty straightforward.  Any volunteers?
>

I am happy to do STEP4. I already have part of it converted, but not
tested.
I see no problem with getting it finished before Christmas.

Cheers,
Paul

[Gareth Rees]

Nick Barnes wrote:
> David is working on STEP5, so should be the one to finish that. I
> have suggested to Gareth that he do the driver scripts; is this OK
> Gareth?

Yes, that's fine with me. I understand that the requirements are:

1. A program that does the same thing as code/run.sh does now
2. But using only Python 2.4 and a Fortran compiler

so that the whole GISTEMP analysis can be run under Windows as well as
Unix.

--
Gareth Rees

[Nick Barnes]

Yes. Shortly to lose the "Fortran compiler" requirement, when David
finishes STEP5 and Paul does STEP4. You could coordinate with them
about the interface to their Python, and skip directly to all-Python.

Nick B

[Nick Barnes]

Further, some of the functionality of the existing code is not really
relevant. For instance, the exact progress messages output. Don't
trouble yourself overly about anything which doesn't end up in the
result/ directory.

Nick B

[Michael Tobis]

In theory I'm an existing member, though I haven't contributed.

I'm very happy to see momentum picking up and will be looking to
contribute in the new year.

mt
--
Michael Tobis

[Nick Barnes]

At 2009-12-06 17:08:59+0000, Julian Todd writes:

> I'd like to argue that, had the CRU data and code been released then,
> and had this clear climate code project been started 5 years ago, the
> damage of the last few weeks might not have been possible.

I agree that the particular "Harry Read Me" aspect of the CRU emails
could have been avoided, but most of the rest of it? I don't think
so. The rest of the attention is focussed on claims of "destroyed"
data, and on strange conspiracy theories about secret committees.

As just one example, consider what is being said about the emails
between Jones and Mann relating to peer review of a McKitrick and
Michaels paper. That has nothing to do with code quality, and some of
the commentary is deep into black helicopter territory.

Still, just about, on topic: the McKitrick and Michaels paper was
apparently an extended version of McKitrick and Michaels 2003.
Science and the world would have benefited if more attention had been
paid to that paper's code, either by the authors or in peer review.
It confuses degrees and radians and so all the results are invalid.

Nick B

[David Jones]

When you have a Python script that is run as a command, it is useful to:
- make a single entry point function called "main";
- have main take an optional argument which is the argument list to use.

Like this code from code/fetch.py:

def main(argv=None):
if argv is None:
argv = sys.argv

(these days I would be very tempted to push the "import sys" into
main, and try and get rid of it from anywhere else)

That way the Python script can be invoked from the command line (or a
shell script), and can also be called directly from another Python
program, without needing another process. As in this code from
preflight.py which calls fetch:

import fetch
fetch.main(argv=['fetch'] + missing)

(this is part of "Guido's main": http://www.artima.com/weblogs/
viewpost.jsp?thread=4829 )

drj

[David Jones]

On 6 Dec 2009, at 17:08, Julian Todd wrote:
>
> PS, do you think it may eventually be possible to run the code on
> scraperwiki?

I should think so, yes.

But I'm not sure what the point is, it produces a dataset that's
identical to one that already published. What did you have in mind?

It's also perhaps rather longer running than a lot a scrapers
(several minutes CPU time on a typical laptop).

Oh, and it also starts by downloading a few dozen megabytes of source
data, so that can take a while too.

drj

[David Jones]

That's excellent, thanks Paul.

drj

[Daniel Rothenberg]

I'm very excited to be hopping on board and contributing to this
project!

As soon as the semester finally peters out, I'll jump in wherever I
feel I can make positive contributions.

- Daniel Rothenberg

[Nick Barnes]

At 2009-12-07 23:14:25+0000, Gareth Rees writes:

> I spent some time this evening running CCC GISTEMP. On my very aged
> Powerbook G4, the whole process took 85 minutes. What kind of runtimes
> do you all get on your fancy modern kit?
>
> The rest of this e-mail is a write-up of random things I thought about
> while running it, to make sure I don't forget.

Thanks for all of this, Gareth. As always, you are asking the right
sorts of questions, and providing some useful answers.

David's replies are correct. However, and writing personally, I'd
like to emphasize two things:

Firstly, I'd like us to complete phase 1 and make a release this
month, while our attention is focussed on CCC-GISTEMP and the level of
outside interest is still high. I would hate for that release to slip
because (say) we are writing dependency-analysis code, or improving
the way in which we format our progress messages. I'd much rather
ship something sooner, even if it is clunky, slow, has ugly output,
lacks automated tests, etc.

Secondly, and looking forward beyond phase 1, I'm sure we will adopt
many of your suggestions, and I hope you will contribute them
yourself. I'd like if possible for features aside from the core data
processing to be separated out, for instance into a separate directory
or even perhaps a separate download. In that way the clarity effort
can be focused where it really matters, on the data processing code.
If interested parties come to CCC-GISTEMP and see a huge mass of
feature-rich code, with modules for everything from floating-point
printing to dependency-analysis, unit tests, regression tests, i18n
and l10n, that will obscure the clarity message.

Nick B

[Gareth Rees]

Nick Barnes wrote:
> Firstly, I'd like us to complete phase 1 and make a release this
> month, while our attention is focussed on CCC-GISTEMP and the level of
> outside interest is still high. I would hate for that release to slip
> because (say) we are writing dependency-analysis code, or improving
> the way in which we format our progress messages. I'd much rather
> ship something sooner, even if it is clunky, slow, has ugly output,
> lacks automated tests, etc.

Don't worry, Nick, I understand the relative priorities. Writing down
my "would be nice" ideas is a way of getting them out of my mind so I
can work on the important stuff. We should mark them as "dogwash" or
whatever and leave them for later, if ever.

The regression testing thing is probably the most important, because I
need to make sure that I'm not breaking anything. But for the moment
I'm doing ad-hoc regression testing using "diff -r".

--
Gareth Rees

[David Jones]

On 6 Dec 2009, at 17:08, Julian Todd wrote:

> Good to see this project is at last starting up again.
>
> I feel it would be useful to not shy so much from the point of the
> public goals.
>
> In the context of the current swift-boating exercise of the Copenhagen
> talks, I discovered the following email exchange between a skeptic and
> the CRU that looks a lot like the exchange between Watts and Hansen
> that got his climate code disclosed -- from 2001.
>
> http://www.john-daly.com/cru/emails.htm
>
>
> I'd like to argue that, had the CRU data and code been released then,
> and had this clear climate code project been started 5 years ago, the
> damage of the last few weeks might not have been possible.
>
> A lot of hay is being made about the secrecy and unreliability of the
> CRU data from people who don't have any data to contribute in its
> stead, and are able to claim that they have been shut out for reasons
> of their own (incredible) scientific integrity.

I agree, but this damage avoidance is not the point of the goals, but
more of a happy side effect (or it would be, if their code had been
public and clear to start with).

Would you like to write a post on the blog expanding a little?

drj

[Nick Barnes]

At 2009-12-10 20:58:19+0000, Gareth Rees writes:

> I ported the driver script (code/run.sh) to Python (code/run.py) and
> checked it in as revision 95: http://code.google.com/p/ccc-gistemp/source/detail?r=95

This is excellent, thank you Gareth. Is there a Windows user prepared
to test this on a machine without cygwin? I'm running it on
FreeBSD/GNU Fortran 4.4/Python 2.5.4 right now.

Paul Ollis, in your STEP4 work, please will you rewrite the /bin/sh
scripts as well as the Fortran, and update this run.py as necessary to
invoke your code?

Looking at the code, I am certain we will be rearranging this in
various ways as the project progresses. As a trivial example, if
STEP0 output sorting continues to be necessary - which it may not, as
step1.py evolves - then that should be incorporated into step0.py.
But I think this code is at least as clear as much of the rest, and it
eliminates a major dependency (The shell and various utilities,
i.e. cygwin on Windows).

Regarding path manipulations, I'm open to suggestions. We don't do
very much of it - and will do less as the Fortran disappears - and
we're unlikely to need to run very often on systems in which "foo/bar"
doesn't work. But portability is a Good Thing. I guess we could
stick it out in a module and forget about it.

Nick B

[John Keyes]

To test this on Windows what is required? Install Fortran and Python
and run 'run.py'?

I might get a chance to try this tomorrow night or Saturday morning on
a Windows 7 box.

<snip/>

> Regarding path manipulations, I'm open to suggestions.  We don't do
> very much of it - and will do less as the Fortran disappears - and
> we're unlikely to need to run very often on systems in which "foo/bar"
> doesn't work.  But portability is a Good Thing.  I guess we could
> stick it out in a module and forget about it.

I'd stick with os.path functionality. If you need paths for os.system
calls we can use os.path.abspath.

-John K

[John Keyes]

Does anyone know what's the best Fortran version to use on Windows 7?
Apologies about the possible high dumb factor, but I don't want to
waste my time with a version that doesn't do the job.

Thanks,
-John K

--
me: http://www.keyes.ie/

charities: http://www.oxfam.ie | http://www.bothar.org | http://www.ipcc.ie

[Nick Barnes]

[resent, to correct address this time!]

Gareth, do you have the energy and inclination to continue with
CCC-GISTEMP? If so, do you have a strong preference about what you
might like to do next? There are only a few outstanding items before
our phase 1 milestone:

- STEP4: assigned to Paul Ollis;
- STEP5: assigned to David Jones;
- porting the last year's GISTEMP changes: assigned to me;
- "acceptance testing": code to compare final result files, to allow
us to decide whether we are ready to release;
- release prep.

David has made a good start on STEP5. I'd be quite happy for you to
take over the update porting from me; I haven't started it yet.

So what do you think?

On the acceptance testing, we started out, and got quite a long way,
insisting on binary-identical intermediate files. Certainly it was a
useful strategy. However, we ended up struggling over matching float
formats (most of the Fortran is done in single, we're in double),
rounding modes, arithmetic order, and so on. So we relaxed that
requirement but we haven't really put anything else in place. Given
that what really matters is the final result files, we should have
some way of comparing two sets of those, given fixed input data, and
scoring the result, and then we can set some acceptance thresholds.
vischeck.py can already read some of the result data, although it only
extracts the global annual anomalies.

Reto Ruedy at GISS has said he will let us have a complete set of
input, intermediate, and result files when he has done their November
run in the next few days. So we will have a dataset against which to
test.

Nick B

[Nick Barnes]

At 2009-12-11 20:51:23+0000, Gareth Rees writes:

> I'll have a look at the acceptance testing.

Great.

> I guess the idea is to compare two sets of result files, one from GISS
> and one from CCC, and do something with the differences, perhaps
> plotting them on a graph?

Yes, exactly. vischeck.py already does a small amount of this.

> Which files need to be compared?

All of the files in result/. No intermediate files (comparing
intermediate files is also important, and we've done quite a bit of
that already, but it's a different kind of test). The box file will
be most sensitive.

> I'm not sure about "scoring" the differences. I guess I could compute
> some statistics, but would they be meaningful? The differences are a
> combination of (a) differing numerical accuracy, (b) reasonable
> implementation choices (like arithmetic order), and (c) bugs. We
> consider (a) and (b) acceptable, and (c) unacceptable. But I don't
> know how to separate out these components.

I appreciate your point, but experience sugggests to me that I think
that statistics here (especially at the box level) would still be
useful for us in finding bugs, and in determining whether we're "close
enough" to GISTEMP. Errors in (a) and (b) are rarely more than one
centiKelvin - and usually much less - once we get to the box file or
later stages. Errors in (c) are often much more than that.

If there is a monthly value for any box which is 10K different between
us and GISTEMP, that's a pretty clear indication of a bug (and things
have certainly been that bad before). Similarly if we have a monthly
value missing. If all the monthly and annual values for every box and
zone are identical, then whatever our code is doing is
indistinguishable (as a black box) from GISTEMP. We currently lie
somewhere in between (in fact, very close to the latter). I want to
know where, to have some figure or figures of merit. For instance,
maximum difference, RMS of differences, number of different values.

It's black-box testing, and fairly crude at that, but that's the first
level at which third parties are going to judge our results.

> Can I assume that the files from GISS will be in the same format as
> the files we're working with?

Yes.

> Do you have some result files that you generated back when CCC was in
> Fortran that I could use for testing?

Yes, or I can just point you at a pure /bin/sh+FORTRAN (very very
early) version of CCC, and you can make your own. There'll be one in
Ravenbrook's /project/ccc. I'll see if I can get to this tomorrow.

Nick B

[Nick Barnes]

At 2009-12-11 20:51:23+0000, Gareth Rees writes:

> Nick Barnes wrote:
> > There are only a few outstanding items before our phase 1 milestone:
> >
> > - STEP4: assigned to Paul Ollis;
> > - STEP5: assigned to David Jones;
> > - porting the last year's GISTEMP changes: assigned to me;
> > - "acceptance testing": code to compare final result files, to allow
> > us to decide whether we are ready to release;
> > - release prep.
>

> Since I don't yet understand what's involved in "release prep"

For what it's worth, "release prep" is my shorthand for:

(a) Writing and/or checking any release materials e.g. release notes,
readme, etc. Ideally including a "what's fixed", preferably one
automatically generated from the issue system. If so, this
includes making sure the issue system reflects whatever changes we
have made.

(b) Figuring out what should go in a release. Just a tarball? A
Python module? Installers? For what platforms? Making sure that
whatever it is can be reproduced from SVN.

(c) figuring out how to make the release on Google Code: just put it
in the downloads directory?

(d) thinking about release announcements. Is a blog post sufficient?

Nick B

[Nick Barnes]

At 2009-12-17 12:44:51+0000, Nick Barnes writes:

> I have made a good start on looking at changes to GISTEMP since last
> year. See the file doc/step0-update-notes for details.

Also doc/steps1-5-update-notes. Hardly any significant changes outside
STEP0: a typo fix in two station metadata files, a change to STEP2
which ensures Antarctic stations are treated as rural (!), a bugfix to
match our STEP5 SBBXotoBX.f bug fix, and a one-line change in STEP3
which I need to figure out.

Nick B

[Nick Barnes]

Reto has clarified some of these in email to me; I will update the
documents.

Nick B

[Nick Barnes]

At 2009-12-19 00:37:42+0000, Gareth Rees writes:

> I added a new Python program, compare_results.py, which compares the
> results from two runs of the GISTEMP algorithm and outputs a
> statistical report in HTML. See http://code.google.com/p/ccc-gistemp/source/detail?r=103
>
> I've attached some sample output. Please ignore the actual figures:
> it's a comparison of two runs on different sets of input data.

This is excellent.

Nick B

[Nick Barnes]

At 2010-01-15 13:33:32+0000, Nick Barnes writes:

> Now I would like to make a plan for ccc-gistemp version 0.3, and aim
> to release it by the end of February 2010. Does anyone on the list
> have any particular improvements in mind? Remember that our main
> objective is always to make the code more clear, so effort should
> probably be focussed on those areas in which the code is least clear.

Having spent some time looking over the code and thinking this
through, here's my own target for 0.3:

To remove all I/O code, and everything concerned with the structure
and format of Fortran data files, from the core code. Where possible,
this should be done by turning all I/O into iterables of Python
objects: ideally actual Python objects, with slots and everything.

This is certainly possible through steps 0, 1, and 2. Step 3, which
changes station records into subbox records, has to collect all the
station records coming out of step 2 before it can produce any subbox
records, but I think it can do that in memory just as well as in a
file. Steps 4 and 5 are both iterator-able.

Generating and reading inter-step intermediate files (v2.mean_comb,
Ts.txt, Ts.GHCN.CL.PA, SBBX1880...) is still very useful for
development and testing, but I think it should be separated out, and
invoked from a driver script (such as run.py) when necessary: if the
user wants to run just steps 0 and 1, the script should invoke the
step0.py generator, give the resulting iterable to the step1.py
generator, and pass the resulting iterable to the "output Ts.txt"
function. If the user wants to run the whole of GISTEMP, why generate
any files other than result/* ?

Comments?

Nick B

[Nick Barnes]

At 2010-02-04 17:36:57+0000, Nick Barnes writes:

> I am imagining a tool which presents as a map of the final subbox
> anomalies, in which the subboxes are clickable, allowing one to see
> the historical anomaly series for that sub-box and also to drill down,
> working back through the GISTEMP steps, ultimately to the original
> station records. There could be charts at each stage (e.g. click on an
> urban station and see a chart showing the different rural station
> records, the combined rural record, the anomaly difference series, the
> two-part linear fit, and before-and-after series for the urban
> station). Ideally with a "slippy map" interface, like Google Maps.

Do any of you ccc-gistemp volunteers have slippy map experience?

Nick B

[Ken Mankoff]

On Thu, 4 Feb 2010, Nick Barnes wrote:
> Do any of you ccc-gistemp volunteers have slippy map experience?

I have KML (Google Earth and Maps) experience.

-k.

[Nick Barnes]

At 2010-02-15 11:58:14+0000, Nick Barnes writes:
> At 2010-02-15 11:45:36+0000, Nick Barnes writes:
>
> > I've done this for step0 and step1. I'm already working on something
> > else in step2 and will do this at the same time.
>
> What I'm working on is removing the rounding. There are two main
> pieces of rounding going on:
>
> 1. station latitudes and longitudes - used for calculating distances,
> to choose rural stations close to urban stations - were rounded to
> the nearest tenth-degree.
>
> 2. Annual anomaly series - used to calculate the adjustment to apply -
> are calculated and combined in integral tenths of Kelvin.

I've checked this in now. The differences made by my most recent
changes can be seen (briefly) here:

<http://www.mad-ideas.org/compare.html>

This change seems to mostly affects results in the ex-USSR.

I often re-use this particular URL for looking at result comparisons,
as it's hosted on the machine I use for hacking on CCC. I think we
should probably have some really simple CGI for generating result
comparisons between any two revision numbers. It will require a bit
of work: to identify revisions which change the results at all, to
collect result directories, and to build a table connecting revision
numbers to result directories, but it's a SMOP. I might look into
that.

Nick B

[Richard Hendricks]

Would it be worthwhile to checkin the results of any changes into SVN
as a change-controlled file? That way any script can simply be given
two SVN numbers to compare between. And in my experience, calling
something a SMOP is almost dooming it to become a several week
project. :)

> --
> You received this message because you are subscribed to the Google Groups "CCC GISTEMP discussion" group.
> To post to this group, send email to ccc-giste...@googlegroups.com.
> To unsubscribe from this group, send email to ccc-gistemp-dis...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/ccc-gistemp-discuss?hl=en.
>
>

--
"Thus, from the war of nature, from famine and death, the most exalted
object of which we are capable of conceiving, namely, the production
of the higher animals, directly follows. There is grandeur in this
view of life, with its several powers, having been originally breathed
into a few forms or into one; and that, whilst this planet has gone
cycling on according to the fixed law of gravity, from so simple a
beginning endless forms most beautiful and most wonderful have been,
and are being, evolved." --Charles Darwin

[Nick Barnes]

At 2010-02-21 05:36:21+0000, Richard Hendricks writes:

> Would it be worthwhile to checkin the results of any changes into SVN
> as a change-controlled file?

Something like this, yes.

Nick B