Overwriting NetCDF files with iris.save

[Pagw]

Hello,

I'm trying to save a CubeList to a NetCDF file using a command like iris.save(cubelist,'fname.nc') in Iris 1.7.1. If the file already exists, this results in the error below, but it works fine if I delete the file. Is there a way to get Iris to overwrite the file?


/a/jupiter/local/home/jupiter/cpdn/watson/.virtualenvs/python/local/lib/python2.7/site-packages/Iris-1.7.1-py2.7.egg/iris/io/__init__.pyc in save(source, target, saver, **kwargs)
    345     # Single cube?

    346     if isinstance(source, iris.cube.Cube):
--> 347         saver(source, target, **kwargs)
    348
    349     # CubeList or sequence of cubes?


/a/jupiter/local/home/jupiter/cpdn/watson/.virtualenvs/python/local/lib/python2.7/site-packages/Iris-1.7.1-py2.7.egg/iris/fileformats/netcdf.pyc in save(cube, filename, netcdf_format, local_keys, unlimited_dimensions, zlib, complevel, shuffle, fletcher32, contiguous, chunksizes, endian, least_significant_digit)
   1603
   1604     # Initialise Manager for saving

-> 1605     with Saver(filename, netcdf_format) as sman:
   1606         # Iterate through the cubelist.

   1607         for cube in cubes:

/a/jupiter/local/home/jupiter/cpdn/watson/.virtualenvs/python/local/lib/python2.7/site-packages/Iris-1.7.1-py2.7.egg/iris/fileformats/netcdf.pyc in __init__(self, filename, netcdf_format)
    557         try:
    558             self._dataset = netCDF4.Dataset(filename, mode='w',
--> 559                                             format=netcdf_format)
    560         except RuntimeError:
    561             dir_name = os.path.dirname(filename)

/a/jupiter/local/home/jupiter/cpdn/watson/.virtualenvs/python/src/netcdf4/netCDF4.so in netCDF4.Dataset.__init__ (netCDF4.c:19738)()

RuntimeError: Can't add HDF5 file metadata

[Andrew Dawson]

I can't reproduce this issue. I'm able to overwrite a netcdf file as you describe without problems.

I have seen this error before. It comes from the netcdf/hdf5 C libraries via netCDF4-python, so isn't actually an iris issue. It typically means you don't have permission to write to the particular file, but has also appeared when trying to write a file to a directory that doesn't exist. I've seen mention of it cropping up when a failure to write causes netcdf (C) to be unable to modify the file further. See these for more details: https://code.google.com/p/netcdf4-python/issues/detail?id=153, https://code.google.com/p/netcdf4-python/issues/detail?id=141.

It has been suggested that this error is sometimes seen due to a bug in netcdf4/hdf5 so your problem might depend on which versions of these libraries your netcdf4 (python) is linked against, and unfortunately might mean you need an upgrade to these.

In the mean time you can check for the file before attempting to write and delete it if it already exists, something like this should do the trick:

import os
outfile = '/path/to/output/file/thisone.nc'
 ...
if os.path.exists(outfile):
    os.remove(outfile)
iris.save(cubelist, outfile)

[Pagw]

Thanks Andrew. I'll investigate whether the other modules can be fixed then, and until then use the os workaround.