merge_cube difficulties with different attribute 'file_name'

[colin talbert]

Hello all,

I've got some historical climate data that comes split up into separate netcdf files for each year.  (from: ftp://gdo-dcp.ucllnl.org/pub/dcp/archive/wwcra/monthly_observed/colo/)

I can load these cubes but then end up with a cube_list with a cube for each year.  The first thing I run into is that the time dimension is not a 'Scalar coordinate' So I have to create a list of slices to do the merge on. But even still I end up with an error because each original cube lists its file_name as an attribute:

MergeError: failed to merge into a single cube.
  cube.attributes values differ for keys: 'file_name'

I keep feeling like there must be an easier way to manage this?  Am I missing an obvious solution because I'm relatively new to Iris?  If that's just life in the city hopefully my code example can help others figure this out, as it wasn't at all obvious to me.

Thanks for any help or insight.

Colin 

My code for reference:

fname1 = r"J:\GIS_Layers\Climate\GDO\mauer\ftp_mirror\gdo-dcp.ucllnl.org\pub\dcp\archive\wwcra\monthly_observed\cali\obs.maurer_2002.cali.monthly.tmax.1994.nc" fname2 = r"J:\GIS_Layers\Climate\GDO\mauer\ftp_mirror\gdo-dcp.ucllnl.org\pub\dcp\archive\wwcra\monthly_observed\cali\obs.maurer_2002.cali.monthly.tmax.1995.nc" tmax_9495 = iris.load([fname1, fname2]) print "first data:", tmax_9495[0]

#tmax_9495.merge_cubes() #doesn't work that easily #first delete off the file_name attribute for cube in tmax_9495: del cube.metadata.attributes['file_name'] # print cube #now create a list of slices for each cube that we can merge on slices = [] for cube in tmax_9495: slices.extend(list(cube.slices(['longitude', 'latitude']))) #now we can merge those slices final_data = iris.cube.CubeList(slices).merge_cube() print "final data:", final_data

which gives:

first data: tmax / (C)                          (time: 12; latitude: 88; longitude: 79)
     Dimension coordinates:
          time                           x             -              -
          latitude                       -             x              -
          longitude                      -             -              x
     Attributes:
          Conventions: GDT-1.2
          History: Initially created in 2011
          SurfSgnConvention: Traditional
          associate: time lat lon
          authors: D.Raff,L.Brekke,S.Gangopadhyay,T.Pruitt,A.Wood
          creation_date: Jan-Mar, 2011
          desc: mean monthly maximum daily temperature
          description: BCSD surface water projections hydrologic data
          file_name: nc_mon/cali/obs.maurer_2002.cali.monthly.tmax.1995.nc
          institution: U.S. Bureau of Reclamation
          name: tmax
          references: U.S. Bureau of Reclamation, 2011, West-Wide Climate Risk Assessments: BCSD...
          vic_name: aggregated daily observed

final data: tmax / (C)                          (time: 24; longitude: 79; latitude: 88)
     Dimension coordinates:
          time                           x              -             -
          longitude                      -              x             -
          latitude                       -              -             x
     Attributes:
          Conventions: GDT-1.2
          History: Initially created in 2011
          SurfSgnConvention: Traditional
          associate: time lat lon
          authors: D.Raff,L.Brekke,S.Gangopadhyay,T.Pruitt,A.Wood
          creation_date: Jan-Mar, 2011
          desc: mean monthly maximum daily temperature
          description: BCSD surface water projections hydrologic data
          institution: U.S. Bureau of Reclamation
          name: tmax
          references: U.S. Bureau of Reclamation, 2011, West-Wide Climate Risk Assessments: BCSD...
          vic_name: aggregated daily observed

[Andrew Dawson]

Iris will insist that all attributes on variables that are merged must be the same. It will also require that cubes can only be merged over scalar coordinates. This is why slicing and recombining works.

I'd suggest something like the following:

def my_callback(cube, field, filename):
    del cube.attributes['file_name']

tmax_9495 = iris.load([fname1, fname2], callback=my_callback).concatenate_cube()
print tmax

which gives:

tmax / (C)                          (time: 24; latitude: 98; longitude: 81)

     Dimension coordinates:
          time                           x             -              -
          latitude                       -             x              -
          longitude                      -             -              x
     Attributes:
          Conventions: GDT-1.2
          History: Initially created in 2011
          SurfSgnConvention: Traditional
          associate: time lat lon
          authors: D.Raff,L.Brekke,S.Gangopadhyay,T.Pruitt,A.Wood
          creation_date: Jan-Mar, 2011
          desc: mean monthly maximum daily temperature
          description:

BCSD surface water projections hydrologic data
          institution: U.S. Bureau of Reclamation

          name: tmax
          references: U.S. Bureau of Reclamation, 2011, West-Wide Climate Risk Assessments: BCSD...
          vic_name: aggregated daily observed

I used a load-time callback function to eliminate the 'file_name' attribute, which is more convenient than doing it manually later on I find. I then used concatenate instead of merge. Concatenate is designed to combine cubes along existing dimension coordinates, whereas merge is slightly different, stacking cubes with matching scalar coordinates. The downside of concatenate is it will load all the data into memory if it isn't loaded already, this shouldn't be a problem for your use case because those files are very small, but it can be a headache for larger data sets.

[colin talbert]

Awesome!  I knew there had to be a cleaner way to accomplish this.  The loading might become an issue when I move onto the daily version but I'll worry about that then.

Thanks so much for the answer, and thanks for this library.

Colin