closing all previously opened files for write

[Paul]

I'm not sure this is a well-formed question but I have always had this
issue with h5py that I haven't figured out.

Say I am debugging script.py.

1. In ipython, I %run script.py, which has somewhere h =
h5py.File('out.h5', 'w')
2. The code crashes or I interrupt it with pdb somewhere after h is
defined.
3. I modify script.py and do %run again.

h5py complains: IOError: 'unable to create file (File accessability:
Unable to open file)' when it tries to open h for writing again.

How can I avoid this? h is not a global variable and I can't do
h.close() after interrupting my code the first time. I don't
understand how it is managing to stick around.

Currently, I do 'reset' in ipython, sometimes several times to avoid
the IOError. Or restart ipython.

Thanks, Paul.

[Andrew Collette]

Hi Paul,

> 1. In ipython, I %run script.py, which has somewhere  h =
> h5py.File('out.h5', 'w')
> 2. The code crashes or I interrupt it with pdb somewhere after h is
> defined.
> 3. I modify script.py and do %run again.
> h5py complains: IOError: 'unable to create file (File accessability:
> Unable to open file)' when it tries to open h for writing again.

As I think you suspected, the problem is that you're trying to open
the file twice. HDF5 actually allows this, although I'm not sure it's
a good idea. What it can't do is open the file in 'w' mode twice; 'w'
means "truncate the file and open it", which is impossible if the file
is already open. H5py, at least modern versions, does not
automatically close files for you; unfortunately this is an
intentional design decision related to the internal complexities of
HDF5.

I would suggest one of the following:

1. Open the file in 'a' or 'r+' mode, in which multiple openings are
allowed. I'm not exactly sure what this does for data consistency,
but I suspect HDF5 does the "right thing" and gives you back the
already-open file handle.
2. Make sure the file is closed, even when KeyboardInterrupt or
similar goes off. You can do this by using try...finally or by using
the File object as a context manager:

with h5py.File('out.hdf5','w') as f:
# code here using f
# after "with" statement file is closed regardless of exceptions

HTH,
Andrew