NetCDF: HDF error: when importing a network

[Willem Dafoe]

Hi Group,

I have an issue with importing a pypsa-eur network.

When trying to import a network with my custom script solve_redispatch_network, it  throws the error for SOME of the networks. For context: I created multiple node networks for germany, ranging from 10 nodes to 200 nodes. Randomly, some of the this way created networks are not imported properly with the error:

INFO:pypsa.io:Imported network elec_s300_100_ec_lcopt_1H-noex.nc has buses, carriers, generators, lines, loads, storage_units

Traceback (most recent call last):

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info

    file = self._cache[self._key]

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__

    value = self._cache[key]

KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/cluster/home/wlaumen/Euler/pypsa-eur/networks_redispatch/solved/elec_s300_100_ec_lcopt_1H-noex.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "solve_redispatch_networks.py", line 577, in <module>

    main()

  File "solve_redispatch_networks.py", line 574, in main

    solve_all_redispatch_workflows(c_rate=0.25, flex_share=0.1)

  File "solve_redispatch_networks.py", line 552, in solve_all_redispatch_workflows

    n_optim = pypsa.Network(path_n_optim)

  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pypsa/components.py", line 285, in __init__

    self.import_from_netcdf(import_name)

  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pypsa/io.py", line 496, in import_from_netcdf

    with ImporterNetCDF(path=path) as importer:

  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/site-packages/pypsa/io.py", line 200, in __init__

    self.ds = xr.open_dataset(path)

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/api.py", line 554, in open_dataset

    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 378, in open

    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 326, in __init__

    self.format = self.ds.data_model

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 387, in ds

    return self._acquire()

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 381, in _acquire

    with self._manager.acquire_context(needs_lock) as root:

  File "/cluster/apps/nss/gcc-6.3.0/python/3.8.5/x86_64/lib64/python3.8/contextlib.py", line 113, in __enter__

    return next(self.gen)

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context

    file, cached = self._acquire_with_cache_info(needs_lock)

  File "/cluster/home/wlaumen/.local/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info

    file = self._opener(*self._args, **kwargs)

  File "src/netCDF4/_netCDF4.pyx", line 2330, in netCDF4._netCDF4.Dataset.__init__

  File "src/netCDF4/_netCDF4.pyx", line 1948, in netCDF4._netCDF4._ensure_nc_success

OSError: [Errno -101] NetCDF: HDF error: b'/cluster/home/wlaumen/Euler/pypsa-eur/networks_redispatch/solved/elec_s300_100_ec_lcopt_1H-noex.nc'

I tested whether my script might have errors by simply importing the .nc files in python, but in fact, I get the error for a 100 node network and a 190 node network whenever I try to load them via pypsa.Network(). All the others are imported correctly, and I have no idea why.

For the 100 node network, i get the error when inputting the solved network from /pypsa-eur/results/networks, for the 190 node network i am getting the error already for importing the prepared network from /pypsa-eur/networks. For each, the other network respectively is imported correctly:

Could anyone help me where this error might result from?

Thanks!

Willem

[Johannes Hampp]

Hi Willem,

Maybe the files are corrupted.
Check whether this works:

ncdump -h elec_s300_100_ec_lcopt_1H-noex.nc
ncdump -h elec_s300_190_ec_lcopt_1H-noex.nc

If yes you need to recreate the networks.

Best,
Johannes

> For the 100 node network, i get the error when inputting the *solved*

> network from /pypsa-eur/results/networks, for the 190 node network i am

> getting the error already for importing the *prepared *network from

> /pypsa-eur/networks. For each, the other network respectively is
> imported correctly:
>

> Screenshot 2021-04-06 182011.png
> Could anyone help me where this error might result from?
>
> Thanks!
>
> Willem
>

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/42d388a9-810b-41b1-8ffc-24e32b7b172en%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/42d388a9-810b-41b1-8ffc-24e32b7b172en%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Willem Dafoe]

Hi  Johannes,

thanks for your fast answer!

Without going to deep here: What exactly does it mean if the ncdump command works (file is corrupted), and how do you think this could happen?

Thanks,
Willem

[Johannes Hampp]

Hi Willem,

If the file is intact then the command will print information on the
file. Otherwise the command should fail and print something accordingly.

File corruption can have many reasons, not sure what your problem is.
I'd say the most common ones are incomplete or incorrect scripts which
are not correctly accessing or closing files. It could also be a race
condition from cluster execution or caused by your cluster environment
not properly handling the file. Or a temporary hick-up on the file
system which caused the files to be corrupted...

I'd recommend a practical approach:
Unless the problem occurs again repeatedly and is reproducible: Just
ignore it an rerun the rules.

Maybe someone else has a better idea?

Best,
Johannes

> <https://groups.google.com/d/msgid/pypsa/42d388a9-810b-41b1-8ffc-24e32b7b172en%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/pypsa/42d388a9-810b-41b1-8ffc-24e32b7b172en%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/ac30eb8f-58f6-4ae2-9043-357dbcdf1c2fn%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/ac30eb8f-58f6-4ae2-9043-357dbcdf1c2fn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Willem Dafoe]

Thx Johannes, 

this approach worked for me!

Willem