accessing cube.data throws error

[Phil Morefield]

I'm working with statistically downscaled climate model output and something is clearly not working correctly. For example, if I grab two consecutive files from this ftp folder and load them into a cube:

import glob
import iris

future_folder = 'F:\\LOCA_2016-04-02\\EC-EARTH\\16th\\rcp85\\r2i1p1\\tasmax'
future_files = glob.glob(path.join(future_folder, '*.nc'))[:2]
future_cube_list = iris.load(future_files)
future_cube = future_cube_list.concatenate_cube()

future_cube.data

Traceback (most recent call last):
  Debug Probe, prompt 21, line 1
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\iris\cube.py", line 1695, in data
    data = data.masked_array()
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\biggus\_init.py", line 2063, in masked_array
    data[indices] = tile.masked_array()
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\biggus\_init.py", line 1820, in masked_array
    masked_array = self._stack[index].masked_array()
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\biggus\_init.py", line 1471, in masked_array
    array = self._apply_keys()
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\biggus\_init.py", line 1559, in _apply_keys
    array = self.concrete.__getitem__(self._keys)
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\iris\fileformats\netcdf.py", line 396, in __getitem__
    data = variable[keys]
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\netCDF4\_netCDF4.pyd", line 3753, in netCDF4._netCDF4.Variable.__getitem__ (netCDF4\_netCDF4.c:40529)
    \000\000‰ n
  File "C:\Users\pmorefi\Anaconda2\Lib\site-packages\netCDF4\_netCDF4.pyd", line 4472, in netCDF4._netCDF4.Variable._get (netCDF4\_netCDF4.c:51271)
    H‹C H‹ËÿP0J Lý H‰Œ$Ø\000\000\000H‹D$PH
ÀuUH‹ ¶Q \000H9E u'L;} Ä \000\000H‹E IÿÇJ‹\øøHÿ HƒÁ H‰Œ$Ø\000\000\000ë2L;} \000\000H‹ IÿÇHÿ HƒÁ H‰Œ$Ø\000\000\000ë H‹ÍÿÐH‹ØH
À „¢ \000\000I‹ÎH‰œ$È\000\000\000L‹óH
Ét
RuntimeError: NetCDF: HDF error

I've read the Common Issues with Merge and Concatenate section carefully, and tried the most obvious things such as equalise_attributes(). The issue only occurs when I'm using concatenate. I seem to be able to manipulate a single file cube just fine.

Any thoughts on how I can go about trouble shooting this?

[Andrew Dawson]

This looks like a problem when loading data from NetCDF, see https://github.com/SciTools/iris/issues/2063 for a similar problem. The solution at this time is either load the data first, by touching each data attribute (this assumes each cube comes from one file, I can't tell if that is true in your case), or to use a version of hdf that is compiled to be thread-safe.

[Phil Morefield]

Thanks Andrew. Finding a thread-safe version of hdf is not an option for me at this time. Can you clarify what you mean by "touching each data attribute"?

[Andrew Dawson]

I just mean accessing it to trigger loading data from file (as you are doing with the concatenated cube. Something along these lines I expect:

for cube in future_cube_list:
    cube.data

It looks like it doesn't do anything, but it will try and load the data from each cube from disk sequentially, and hopefully avoid the problem, depending on the layout of your files.

[Phil Morefield]

So after this idea didn't work, I did more testing. Turns out there is a corrupted times step in the file. Every slice is accessible except for that single time step. I did a bulk download of the entire archive and it looks like some errors crept in. 

Good news for iris. Bad news for me.

Thanks for your help.