aggregated_by raising TypeError: cannot perform reduce with flexible type

[Niall Robinson]

Hello - I keep getting the above error raised by aggregated_by and I can't for the lift of me figure out what it means (I think its a numpy error) or what I should do to make it go away. It seemed that if I removed the month coord classification then it went away but I've done something that brings it back now...I know that's not very helpful but I have no idea what it means so I'm having trouble pinning down what is causing it. The coords look fine to me i.e. the season are year look to be applied correctly.

Has anyone else experienced this? Here is some more info:

for this cube

sea_water_density / kg m-3          (*ANONYMOUS*: 29; model_level_number: 20; latitude: 144; longitude: 288)
     Dimension coordinates:
          model_level_number                    -                       x             -               -
          latitude                              -                       -             x               -
          longitude                             -                       -             -               x
     Auxiliary coordinates:
          forecast_period                       x                       -             -               -
          season                                x                       -             -               -
          time                                  x                       -             -               -
          year                                  x                       -             -               -
          depth                                 -                       x             -               -
     Scalar coordinates:
          forecast_reference_time: 1924-11-01 00:00:00
     Attributes:
          Conventions: CF-1.5
          STASH: m02s00i???
          history: Mean of sea_water_density aggregated over season, year
          source: Data from Met Office Unified Model
     Cell methods:
          mean: season, year

get this stack trace:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/net/home/h02/nrobin/workspace/sandpit/spgcooling.py", line 147, in doAnal
    seasTempClim, seasSalClim, seasDensClim = bySeason(tempClim, salClim, densClim)
  File "/net/home/h02/nrobin/workspace/sandpit/spgcooling.py", line 135, in bySeason
    a = _indBySeason(temp)
  File "/net/home/h02/nrobin/workspace/sandpit/spgcooling.py", line 131, in _indBySeason
    a = cube.aggregated_by([cube.coord('season'), cube.coord('year')], analysis.MEAN)
  File "/project/avd/iris/live/testing/iris/cube.py", line 2348, in aggregated_by
    key[dimension_to_groupby] = (0,) * len(groupby)
  File "/project/avd/iris/live/testing/iris/analysis/__init__.py", line 937, in __len__
    value = len([s for s in self.group()])
  File "/project/avd/iris/live/testing/iris/analysis/__init__.py", line 834, in group
    self._compute_shared_coords()
  File "/project/avd/iris/live/testing/iris/analysis/__init__.py", line 920, in _compute_shared_coords
    new_points = np.array(new_bounds).mean(-1)
TypeError: cannot perform reduce with flexible type

[bjlittle]

Hi Niall,

This looks pretty involved. Could you contact me with the details so that we can recreate the problem.

Regards,
-Bill

[Niall Robinson]

OK - I we've figured out the issue now. This arises when iris tries to take the mean of a coordinate made up of strings, in this case month name i.e. there is no such thing as mean(['dec','jan','feb'). If you remove any coordinates on the same dimension which are text based, the problem goes away.

[oliver browne]

Thanks for reporting this Niall.

I've run into the same problem a couple of times, most recently whilst demonstrating the use of rolling_window to a class, e.g.

import iris
import iris.coord_categorisation

# grab some HadCRUT data and get the (loose) area-avg...
sat = iris.load_cube('HadCRUT*.nc', 'temperature')
sat_ts = sat.collapsed(['latitude', 'longitude'], iris.analysis.MEAN)

# create a smoothed version...
sat_ts.rolling_window('time', iris.analysis.MEAN, 12)

# add some additional coords (e.g. if we were thinking of doing some seasonal work)...
iris.coord_categorisation.add_season(sat_ts , 'time')

# repeat the smoothing..
sat_ts.rolling_window('time', iris.analysis.MEAN, 12)

/exports/home/obrowne/envs/geosSciTools/lib/python2.7/site-packages/iris/cube.pyc in rolling_window(self, coord, aggregator, window, **kwargs)
   3099             # bounds)
   3100             new_bounds = new_bounds[:, (0, -1)]
-> 3101             new_points = np.mean(new_bounds, axis=-1)
   3102
   3103             # wipe the coords points and set the bounds


/exports/home/obrowne/envs/geosSciTools/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in mean(a, axis, dtype, out, keepdims)
   2714
   2715     return _methods._mean(a, axis=axis, dtype=dtype,
-> 2716                             out=out, keepdims=keepdims)
   2717
   2718 def std(a, axis=None, dtype=None, out=None, ddof=0, keepdims=False):


/exports/home/obrowne/envs/geosSciTools/lib/python2.7/site-packages/numpy/core/_methods.pyc in _mean(a, axis, dtype, out, keepdims)
     60         dtype = mu.dtype('f8')
     61
---> 62     ret = um.add.reduce(arr, axis=axis, dtype=dtype, out=out, keepdims=keepdims)
     63     if isinstance(ret, mu.ndarray):
     64         ret = um.true_divide(

TypeError: cannot perform reduce with flexible type

It also causes the same problem had I used 'month'. Not being able to average over strings make sense; I'm guessing the attempt at averaging occurs because it is trying to work out what value to give the auxiliary coordinate at the new 'smoothed' positioned.

I suppose I might expect it to use centre of the window as the 'value' in this case.  But maybe there are situations where the 'right' choice is less obvious.

Whether there is a way to 'fix' this for certain cases or not, a better error message would be helpful, e.g.

  "Error!  Collapsing over dimension ('time') requires calculating new values for dependent auxiliary coordinates ('year','month'), which cannot be done for string coordinates ('month')."

to be triggered by a check done on the coordinates that are affected before processing is attempted.

[Andrew Dawson]

For collapsing and aggregation this problem is already solved. We serialize string coordinates when they are collapsed, which seems like the only sensible thing to do with them. The rolling_window method contains a special case at the moment, which should be changed to bring the behaviour in line with the aggregated_by and collapsed methods. Thanks for the report, I've created a Gtihub ticket #1108.

[Niall Robinson]

For the record - I don't really like the way aggregate_by deals with this currently ie. you coords look like this "[| Jan | Jan | Jan | Jan | Jan | Jan |, | Feb | Feb | Feb | Feb | Feb | Feb, etc etc]"

Although it was better than just crashing.

[Andrew Dawson]

A fix for the rolling window problem is now incorporated into the master branch of Iris and will be available in the next release (v1.7).