Cube or CubeList?

[Adrian Matthews]

Hello,

I'm a newbie to Iris and I'm struggling with some basic concepts here.  I have some reanalysis data in a different file for each year (all with the same consistent base time, ie days since some fixed time).  I can read in a range of days that overlaps two years (ie two files) eg 1990,12,21 to 1991,1,15 and I then want to calculate the time average using the collapsed method.  However, to do this, the data has to be a single cube, not a cube list.  This is where I am stuck.  The concatenate_cube method fails.  I've tried loading the data in as a cubelist or  a cube, concatenating a cubelist to a cube,  and using iris.util.unify_time_units() but no joy.  What method is recommended for doing this?

I've copied my test script and output below.  Any help appreciated.

I am running the latest version of Iris: 1.7.2

print "# Import modules"

import iris,datetime

import iris.quickplot as qplt

import matplotlib.pyplot as plt

print "# Load cubes from input data files (single file for each year)"

cubes=iris.load('/esdata/env/e058/scratch/daily/ncepdoe/uwnd.199?.nc','eastward_wind')

print "cubes:\n",cubes

print "# Create time constraint that runs over two years (cubes)"

start_time=datetime.datetime(1990,12,21)

end_time=datetime.datetime(1991,1,15)

print "start_time:",start_time

print "end_time:",end_time

time_constraint=iris.Constraint(time=lambda cell: cell>=start_time and cell<=end_time)

print "# Extract data for this time constraint"

with iris.FUTURE.context(cell_datetime_objects=True):

    cubes1=cubes.extract(time_constraint)

print "cubes1:\n",cubes1

print "# Concatenate cube list to single cube along time dimension"

cube1=cubes1.concatenate_cube()

print "# Calculate time average"

cube2=cube1.collapsed('time',iris.analysis.MEAN)

print "cube2:",cube2

# Import modules

# Load cubes from input data files (single file for each year)

cubes:

0: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

1: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

2: eastward_wind / (m/s)               (time: 366; Level: 17; latitude: 73; longitude: 144)

3: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

4: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

5: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

6: eastward_wind / (m/s)               (time: 366; Level: 17; latitude: 73; longitude: 144)

7: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

8: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

9: eastward_wind / (m/s)               (time: 365; Level: 17; latitude: 73; longitude: 144)

# Create time constraint that runs over two years (cubes)

start_time: 1990-12-21 00:00:00

end_time: 1991-01-15 00:00:00

# Extract data for this time constraint

cubes1:

0: eastward_wind / (m/s)               (time: 11; Level: 17; latitude: 73; longitude: 144)

1: eastward_wind / (m/s)               (time: 15; Level: 17; latitude: 73; longitude: 144)

# Concatenate cube list to single cube along time dimension

Traceback (most recent call last):

  File "test_env3a91.py", line 26, in <module>

    cube1=cubes1.concatenate_cube()

  File "/gpfs/grace/anaconda-2.1.0-iris/lib/python2.7/site-packages/Iris-1.7.2-py2.7.egg/iris/cube.py", line 449, in concatenate_cube

    res = iris._concatenate.concatenate(self, error_on_mismatch=True)

  File "/gpfs/grace/anaconda-2.1.0-iris/lib/python2.7/site-packages/Iris-1.7.2-py2.7.egg/iris/_concatenate.py", line 249, in concatenate

    registered = proto_cube.register(cube, axis, error_on_mismatch)

  File "/gpfs/grace/anaconda-2.1.0-iris/lib/python2.7/site-packages/Iris-1.7.2-py2.7.egg/iris/_concatenate.py", line 680, in register

    match = self._cube_signature.match(cube_signature, error_on_mismatch)

  File "/gpfs/grace/anaconda-2.1.0-iris/lib/python2.7/site-packages/Iris-1.7.2-py2.7.egg/iris/_concatenate.py", line 442, in match

    raise iris.exceptions.ConcatenateError(msgs)

iris.exceptions.ConcatenateError: failed to concatenate into a single cube.

  Cube metadata differs for phenomenon: eastward_wind

  Dimension coordinates metadata differ: time != time

>>> cubes1[0].coord('time')

DimCoord(array([ 1674000.,  1674024.,  1674048.,  1674072.,  1674096.,  1674120.,

        1674144.,  1674168.,  1674192.,  1674216.,  1674240.]), standard_name=u'time', units=Unit('hours since 1800-1-1 00:00:0.0', calendar='gregorian'), long_name=u'Time', var_name='time', attributes={'delta_t': '0000-00-01 00:00:00', 'coordinate_defines': 'start', 'actual_range': array([ 1665504.,  1674240.]), 'avg_period': '0000-00-01 00:00:00'})

>>> cubes1[1].coord('time')

DimCoord(array([ 1674264.,  1674288.,  1674312.,  1674336.,  1674360.,  1674384.,

        1674408.,  1674432.,  1674456.,  1674480.,  1674504.,  1674528.,

        1674552.,  1674576.,  1674600.]), standard_name=u'time', units=Unit('hours since 1800-1-1 00:00:0.0', calendar='gregorian'), long_name=u'Time', var_name='time', attributes={'delta_t': '0000-00-01 00:00:00', 'coordinate_defines': 'start', 'actual_range': array([ 1674264.,  1683000.]), 'avg_period': '0000-00-01 00:00:00'})

>>>

--

Dr Adrian Matthews

Centre for Ocean and Atmospheric Sciences
School of Environmental Sciences / School of Mathematics
University of East Anglia
Norwich NR4 7TJ, UK
Tel: +44(0)1603 593733
Email: a.j.ma...@uea.ac.uk
Internet:  http://envam1.env.uea.ac.uk

UK Top 15 (14^th in the Guardian University Guide 2015; 14^th in the Times and Sunday Times Good University Guide 2015)
UK Top 3 for Student Experience (Times Higher Education Student Experience Survey 2014)
World top 1% (Times Higher Education World Rankings 2014-15)
World Top 100 (Leiden Ranking 2014)


       

IMPORTANT NOTICE - This email is intended for the named recipient only. It may contain privileged and confidential information. If you are not the intended recipient, notify the sender immediately and destroy this email. You must not copy, distribute or take action in reliance upon it. Whilst all efforts are made to safeguard emails, The School of Environmental Sciences cannot guarantee that attachments are virus free or compatible with your systems and does not accept liability in respect of viruses or computer problems experienced.

[Andrew Dawson]

Hi Adrian

Iris can't concatenate the cubes because the time dimensions are not compatible in their current form. It looks like each file probably define some attributes of the time dimension that includes information about the range of values (the 'actual_range' attribute). Iris determines whether coordinates can be treated as the same (i.e. is this time coordinates just another part of the same coordinate in another file?) by matching metadata.

In order for Iris to concatenate these cubes you will need to manually remove any conflicting metadata (in Iris) before attempting to concatenate the cubes. In this case you can either remove the specific attribute ('actual_range') or just nuke the whole attributes dictionary. The most useful metadata is directly attached to the coordinate, not held in the attributes dictionary as you can see from your printout, so this doesn't actually lose much of interest in your case). There are two common ways to do this. Method 1 loads the CubeList and then modifies each cube in the list to be suitable:

cubes = iris.load(['file1.nc', 'file2.nc', ...])

for cube in cubes:
    # Delete the problem attribute from the time coordinate:
    del cube.coord('time').attributes['actual_range']
    # Or I could have set the attributes dictionary of the time coordinate to empty:
    #cube.coord('time').attributes = {}

cube = cubes.concatenate_cube()

Method 2 modifies the cubes as they are loaded from file using a callback:

def clean_callback(cube, field, filename):
    # Delete the problem attribute from the time coordinate:
    del cube.coord('time').attributes['actual_range']
    # Or I could have set the attributes dictionary of the time coordinate to empty:
    #cube.coord('time').attributes = {}


cube = iris.load(['file1.nc', 'file2.nc', ...], callback=clean_callback).concatenate_cube()

The ability to specify a callback is part of the Iris load mechanism. You provide a function which takes 3 inputs, the first of which is the cube currently being loaded, and you may modify this cube as you see fit (I think you can even return a new cube if you need to, but you don't have to). The second argument is useful when loading from field based formats like PP or GRIB where access to a lower-level field object is useful (not relevant to NetCDF though). The last argument is the name of the file the cube was loaded from, which can be useful if there is metadata in the file name that isn't described within the file (yuck).

Personally I like the callback method. It allows you to make changes earlier in the load process allowing you to do some things that can't be done after the cubes are already loaded.

You may find that once you fix this issue, there are other issues preventing you from concatenating your data. These are likely to be simple attribute clashes (e.g. global history from ncatted contains the file name, and thus is different for each file). You can use the same approaches described here to fix similar problems. Let us know how you get on, I hope you enjoy using Iris!