problem with aggregated_by and mdtol and masks with iris2.1 and numpy1.15

[Patrick McGuire]

When I run these commands in python2.7 with iris 2.1 and numpy 1.15, I get the error messages in traceback below. 

----------------

PYTHON commands:

 import iris

 cube = iris.load_cube('example.nc')

 #check to see if the data is there:

 cube.data

 cube = cube.aggregated_by(["year", "day_of_year"], iris.analysis.MEAN, mdtol=0.03)

----------------

The 20MB example.nc NETCDF file is available here: 

https://drive.google.com/open?id=1hZNWXs7wOtBq2dXWM1zbAJnVMjy_DXc7

-----------------

Traceback (most recent call last):

  File "test10.py", line 10, in <module>

    cube = cube.aggregated_by(["year", "day_of_year"], iris.analysis.MEAN, mdtol=0.03)

  File "/Users/pmcguire/fluxnet1/iris/cube.py", line 3414, in aggregated_by

    **kwargs)

  File "/Users/pmcguire/fluxnet1/iris/analysis/__init__.py", line 524, in aggregate

    result.mask = result.mask | mask_update

  File "/Users/pmcguire/.local/lib/python2.7/site-packages/numpy/ma/core.py", line 6339, in __setattr__

    "attributes of {!r} are not writeable".format(self))

AttributeError: attributes of masked are not writeable

-------------------

I have tracked this down I think.

I could get test_fluxnet_evaluation.py to work if I comment out the three lines below from the _Aggregator(object).aggregate method in iris/analysis/__init__.py (near line 514) of a copy of iris2.1. 

The result.mask below was not defined for some reason even though ma.isMaskedArray(result) is true.

        result = self.call_func(data, axis=axis, **kwargs)

        if (mdtol is not None and ma.isMaskedArray(data)):

            fraction_not_missing = data.count(axis=axis) / data.shape[axis]

            mask_update = 1 - mdtol > fraction_not_missing

            #if ma.isMaskedArray(result):

            #    result.mask = result.mask | mask_update

            #else:

            result = ma.array(result, mask=mask_update)

In version Iris1.13, this problem does not exist.
I can reproduce this problem if I upgrade from Iris1.13 to Iris2.2.0dev0, including all dependencies.

But then when I downgrade only Iris2.2 -> Iris.1.13, keeping the updated dependencies, the bug persists.

But then when I downgrade only Iris2.2 -> Iris.1.13 and numpy1.15->numpy1.12.1, keeping the updated dependencies, the bug goes away.

So I think it's something in numpy.

When it works with Iris1.13 and numpy1.12.1, Aggregrate (with iris.analysis.Mean) function tries to override the (non-array) mask value for the day, and it can do this, because the mask is writeable. But when it doesn't work with iris2.1 and numpy1.15, this function fails because this mask is not writeable.

2)A quick perusal of the numpy1.15 documentation suggests that they changed the mask behavior: https://docs.scipy.org/doc/numpy/release.html

"np.ma.masked is no longer writeable:

Attempts to mutate the masked constant now error, as the underlying arrays are marked readonly. In the past, it was possible to get away with...".

I am not sure if this is the change which is affecting us or not.

Can someone fix this problem in iris? Should I just maintain a separate iris branch (with those three lines commented out) for my own purposes till this is fixed? Or does anybody have a better solution? I am not sure if I should downgrade numpy or not.

Thanks,

Patrick McGuire

Department of Meteorology and Department of Geography & Environmental Science

University of Reading

Personal Web Page: http://www.personal.reading.ac.uk/~gn916173

[dkillick]

See https://github.com/SciTools/iris/issues/3190 for the report of this issue on GitHub.