Combining cubes together

[ScottHosking]

Hi,

I'm having trouble combining two (or more) similar cubes together in iris v1.7.1.  Ideally, I would like to end up with a cube that looks like this:

surface_air_pressure / (Pa)         (t: 2; surface: 1; latitude: 256; longitude: 512)
i.e., cubes.data.shape = (2, 1, 256, 512)

At the moment, I have two 6-hourly nc files which are loaded as a separate cubes making it harder to work with them, e.g., average over the time dimension (See code and output below)

Any advice would be greatly appreciated.

Scott



In [42]: def edit_attributes(cube, field, filename):
....: cube.attributes.pop('history', None)
....: cube.attributes.pop('time', None)


In [43]: cubes = iris.load_cube(['ggas197901010000.nc','ggas197901010600.nc'], 'surface_air_pressure', callback=edit_attributes)

---------------------------------------------------------------------------
ConstraintMismatchError Traceback (most recent call last)
/home/scott/ERAI_6hrly/<ipython-input-64-15757b72f2cc> in <module>()
----> 1 cubes = iris.load_cube(['ggas197901010000.nc','ggas197901010600.nc'], 'surface_air_pressure', callback=edit_attributes)

/usr/local/lib/python2.7/dist-packages/Iris-1.7.1-py2.7.egg/iris/__init__.pyc in load_cube(uris, constraint, callback)
312 cube = cubes.merge_cube()
313 except iris.exceptions.MergeError as e:
--> 314 raise iris.exceptions.ConstraintMismatchError(str(e))
315 except ValueError:
316 raise iris.exceptions.ConstraintMismatchError('no cubes found')

ConstraintMismatchError: failed to merge into a single cube.
Coordinates in cube.dim_coords differ: t.

In [44]: print cubes[0].coord('t')
DimCoord([1979-01-01 00:00:00], standard_name=None, calendar='gregorian', long_name=u't', var_name='t', attributes={'time_origin': '01-JAN-1979:00:00:00'})

In [45]: print cubes[1].coord('t')
DimCoord([1979-01-01 06:00:00], standard_name=None, calendar='gregorian', long_name=u't', var_name='t', attributes={'time_origin': '01-JAN-1979:06:00:00'})

[Niall Robinson]

Hi Scott,

Have you tried using CubeList.merge_cube()?

If you do iris.load() to load you data, you should get back a couple of separate cubes in a "cube list". If you then do my_cube_list.merge_cube(), Iris will try to merge them into a single cube, and should (if you're lucky) give you a reason why it can't, if it can't.

Actually, I can see one reason why your cubes are merging. The time_origin value is different for the two coords. That implies that your cube.coord('time').units is different too. Both of these will prevent the merge. I think there is a iris.util.merge_time_units() or something which should sort this (or you can do it manually in a couple of lines of code). Not sure the util function will sort the attributes value, you might have to do that manually.

Let us know if you are still having problems

Niall

[ScottHosking]

Thank you Niall,

After running "iris.util.unify_time_units(cubes)" I see that the time coordinate in cubes[1].dim_coords has changed from 0 to 0.25 (as these are 6 hourly fields this seems correct).  However, the cubes still won't merge.  I tried "cubes[1].dim_coords[0].attributes = cubes[0].dim_coords[0].attributes" but this sets the time coordinate in cubes[1] back to 0, and still no luck with merging.

Can you see what I should chance manually to get these cubes to merge? (see code below)

Thanks in advance,

Scott


In [67]: cubes[0].dim_coords[0]
Out[68]: DimCoord(array([ 0.], dtype=float32), standard_name=None, units=Unit('days since 1979-01-01 00:00:00', calendar='gregorian'), long_name=u't', var_name='t', attributes={'time_origin': '01-JAN-1979:00:00:00'})

In [68]: cubes[1].dim_coords[0]
Out[67]: DimCoord(array([ 0.25], dtype=float32), standard_name=None, units=Unit('days since 1979-01-01 00:00:00', calendar='gregorian'), long_name=u't', var_name='t', attributes={'time_origin': '01-JAN-1979:06:00:00'})

In [69]: cubes.merge_cube()
---------------------------------------------------------------------------
MergeError                                Traceback (most recent call last)
/home/scott/ERAI_6hrly/<ipython-input-69-7d5141b128cf> in <module>()
----> 1 cubes.merge_cube()

/usr/local/lib/python2.7/dist-packages/Iris-1.7.1-py2.7.egg/iris/cube.pyc in merge_cube(self)
    341         proto_cube = iris._merge.ProtoCube(self[0])
    342         for cube in self[1:]:
--> 343             proto_cube.register(cube, error_on_mismatch=True)
    344
    345         # Extract the merged cube from the ProtoCube.


/usr/local/lib/python2.7/dist-packages/Iris-1.7.1-py2.7.egg/iris/_merge.pyc in register(self, cube, error_on_mismatch)
   1235             coord_payload = self._extract_coord_payload(cube)
   1236             match = coord_payload.match_signature(self._coord_signature,
-> 1237                                                   error_on_mismatch)
   1238         if match:
   1239             # Register the cube as a source-cube for this ProtoCube.


/usr/local/lib/python2.7/dist-packages/Iris-1.7.1-py2.7.egg/iris/_merge.pyc in match_signature(self, signature, error_on_mismatch)
    257         match = not bool(msgs)
    258         if error_on_mismatch and not match:
--> 259             raise iris.exceptions.MergeError(msgs)
    260         return match
    261

MergeError: failed to merge into a single cube.

[Andrew Dawson]

The time_origin attributes of each t dimension are different. I'd suggest deleting these attributes, one of them is already invalid after you changed the reference time so doesn't seem much point in keeping it around.

You could try this out too: equalise_attributes. It is written and documented for cubes but having looked at the source code I'd say it should work for coordinates too.

[Richard Hattersley]

Hi Scott,

The merge operation only combines source cubes by creating new dimensions from scalar coordinates. These are coordinates of length one which are not attached to a dimension. The two cubes in this case aren't merging because "t" is attached to a dimension - albeit in this case with length one. (Perhaps it would be better if "merge" was called "stack" instead.)

To combine cubes along an existing dimension you can use `cubes.concatenate_cube()` instead.

Or, since the "t" dimension is length one, you could stick with `cubes.merge_cube()` but use indexing so the source cubes have "t" as a scalar coordinate. For example: cubes = [cube[0] for cube in cubes]

NB. Whichever technique you use, the other comments about unifying the attributes and units still apply.

Regards,
Richard

[Niall Robinson]

(Perhaps it would be better if "merge" was called "stack" instead.)

Interesting. This is a repeated problem for new users, so it might actually be worth thinking about changing the API for v 2.0
 

[ScottHosking]

Thanks Richard and Andrew,

That's very useful information, I hadn't appreciated the difference between merge_cube and concatenate_cube.

Another question I'm afraid, how do you delete an attribute?  I've tried `del cubes[0].dim_coords[0].attributes` but that gave me the error "AttributeError: can't delete attribute
".

Best,
Scott

On Tuesday, 26 August 2014 09:59:04 UTC+1, Richard Hattersley wrote:

[Andrew Dawson]

Scott, you can't (and shouldn't) be deleting the attributes dictionary. Instead you should delete the specific attribute you don't want, or if you really don't want any attributes you should set it to an empty dictionary. These options correspond to:

del cubes[0].dim_coords[0].attributes['my_attribute_name']

and

cubes[0].dim_coords[0].attributes = {}

respectively.

[ScottHosking]

Thanks everyone, I can now concatenate my cubes!

For those who maybe experiencing similar problems, here is my code....

def edit_attributes(cube, field, filename):
    cube.attributes.pop('history', None)
    cube.attributes.pop('time', None)

files = ['ggas197901010000.nc','ggas197901010600.nc','ggas197901011200.nc','ggas197901011800.nc']
   
cubes = iris.load(files, 'surface_air_pressure', callback=edit_attributes)

iris.util.unify_time_units(cubes)

for i in range(0,len(files)):
    del cubes[i].dim_coords[0].attributes['time_origin']

cubes = cubes.concatenate_cube()