export and import & network / solution data to netCDF

[Tim Kittel]

Dear all,

I am running a simulation and would like to save the result so I can load it later again. And if I understood correctly, that's what the export methods are for. But for some reason, this is not working. What am I doing wrong?

Here is a minimal working example. During the loading, Python gives a warning INFO:pypsa.io:Exported network test.nc has loads, buses, generators and then the Code fails with a KeyError when accessing nu2.generators_t.p["generator"]. And ncdumps tells me that the structure is there but there seems to be no data in the file.

from matplotlib import pyplot as plt
import pandas as pd
import pypsa

test_file = "test.nc"

load = pd.Series(
    [0, 0, 0, 0, 0, 0, 0.2, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.2, 0, 0, 0, 0, 0, 0, 0],
    index=pd.date_range("0:00", "23:00", freq="1h")
)
load.index.name = "time"

nu = pypsa.Network()
nu.set_snapshots(load.index)

nu.add("Bus", "bus")

nu.add(
    "Generator", "generator", bus="bus",
    p_nom_extendable=True,
    capital_cost=1,
)

nu.add("Load", "load", bus="bus", p_set=load)

nu.consistency_check()

out = nu.lopf(nu.snapshots)

assert out == ("ok", "optimal")

print(f"writing output to {test_file} ... ", end="", flush=True)
nu.export_to_netcdf(test_file)
print("done")

nu.generators_t.p["generator"].plot()

del nu

nu2 = pypsa.Network()
print(f"loading input from {test_file} ... ", end="", flush=True)
nu2.import_from_netcdf(test_file)
print("done")

plt.figure()
nu2.generators_t.p["generator"].plot()
plt.plot()

[Tom Brown]

Dear Tim,

Thanks for report this. The NetCDF export/import was introduced very
recently and I've also been experiencing similar bugs with it. We'll
look into it very soon.

In the meantime, you can import/export from HDF5 (only works up to 1000
elements - doesn't seem to be a problem for you) or CSV.

We'll announce when it's fixed (could take a few weeks).

Best,

Tom

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To post to this group, send email to py...@googlegroups.com
> <mailto:py...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/13bb51aa-53bc-4f68-839b-6d9c6d982a3f%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/13bb51aa-53bc-4f68-839b-6d9c6d982a3f%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--
Tom Brown
Postdoctoral Researcher
Frankfurt Institute for Advanced Studies
https://fias.uni-frankfurt.de/physics/schramm/renewable-energy-system-and-network-analysis/
Personal site: https://www.nworbmot.org
Phone: +49 69 798 47693

[Jonas Hörsch]

Hi Tim,

thanks for that 10 minute break from my annoying writing tasks. The
netcdf exporter had a problem with components that were completely
standard except for their name. The trivial fix just landed on master.

Best,
Jonas

> --
> You received this message because you are subscribed to the Google Groups "pypsa" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
> To post to this group, send an email to py...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/13bb51aa-53bc-4f68-839b-6d9c6d982a3f%40googlegroups.com.

[Tom Brown]

Thanks Jonas,

I'll push a new bug fix version 0.13.1 tonight / tomorrow.

Best,

Tom

[kostas syranidis]

Deat all,

I have encountered a similar problem with pandas and hdf5 but I haven't tested how pypsa handles it; it is when there are columns with mixed datatypes, e.g. NaN and boolean. Then the corresponding table is simply not written.

By the way, I wanted to ask about the 1000 columns limit. I think that if you switch to 'fixed' format, there shouldn't be any problems, right? The main difference is that you cannot submit "queries" anymore.

Best,
Kostas

[Jonas Hörsch]

Hi Costas,

no, it's different. It's a limited storage space for column names type
restriction and actually depends on how long (as in no of characters)
your components with time-dependent data are.

We do use the fixed format, since it turned out to be faster; but it
does not protect from the problem.

Best,
Jonas

> --
> You received this message because you are subscribed to the Google Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
> To post to this group, send an email to py...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/e8cc5c5e-f1d0-47c2-8b3a-252ab9784575%40googlegroups.com.

[Tim Kittel]

Hi Tom and Jonas,

thanks for you quick reply. I just pulled the latest master and the problem persists. Actually, it's the same for hdf5. After loading, the time series (in my case nu2.generator_t.p) is empty. I modified the MWE to print the used PyPSA (to check I am using the correct one), to print the time series that's being saved and the one that's being loaded and finally to work for hdf5 and netCDF. The Output is attached, too. Could you check whether you get the same error on your system?

Cheers,
Tim

import pandas as pd
import pypsa

# check he uses the correct PyPSA, cloned from github
print(f"PyPSA location: {pypsa.__file__}")

# test_file = "test.nc" # test netCDF
test_file = "test.hdf5" # test hdf5

load = pd.Series(
    [0, 0, 0, 0, 0, 0, 0.2, 0.5, 0.7, 0.85, 0.95, 1.0, 0.95, 0.85, 0.7, 0.5, 0.2, 0, 0, 0, 0, 0, 0, 0],
    index=pd.date_range("0:00", "23:00", freq="1h")
)
load.index.name = "time"

nu = pypsa.Network()
nu.set_snapshots(load.index)

nu.add("Bus", "bus")

nu.add(
    "Generator", "generator", bus="bus",
    p_nom_extendable=True,
    capital_cost=1,
)

nu.add("Load", "load", bus="bus", p_set=load)

nu.consistency_check()

out = nu.lopf(nu.snapshots)

assert out == ("ok", "optimal")

print(f"writing output to {test_file} ... ", end="", flush=True)

if test_file.endswith(".nc"):
    nu.export_to_netcdf(test_file)
elif test_file.endswith(".hdf5"):
    nu.export_to_hdf5(test_file)
else:
    raise NotImplementedError(f"unknown file extension for {test_file!r}")
print("done")

print(nu.generators_t.p)

del nu

nu2 = pypsa.Network()
print(f"loading input from {test_file} ... ", end="", flush=True)

if test_file.endswith(".nc"):
    nu2.import_from_netcdf(test_file)
elif test_file.endswith(".hdf5"):
    nu2.import_from_hdf5(test_file)
else:
    raise NotImplementedError(f"unknown file extension for {test_file!r}")
print("done")

print(nu2.generators_t.p)


=======================================================
= Output
=======================================================

PyPSA location: **REMOVED BUT CORRECT**
INFO:pypsa.pf:Slack bus for sub-network 0 is bus
INFO:pypsa.opf:Performed preliminary steps
INFO:pypsa.opf:Building pyomo model using `angles` formulation
INFO:pypsa.opf:Solving model using glpk
# ==========================================================
# = Solver Results                                         =
# ==========================================================
# ----------------------------------------------------------
#   Problem Information
# ----------------------------------------------------------
Problem: 
- Name: unknown
  Lower bound: 1.0
  Upper bound: 1.0
  Number of objectives: 1
  Number of constraints: 97
  Number of variables: 50
  Number of nonzeros: 121
  Sense: minimize
# ----------------------------------------------------------
#   Solver Information
# ----------------------------------------------------------
Solver: 
- Status: ok
  Termination condition: optimal
  Statistics: 
    Branch and bound: 
      Number of bounded subproblems: 0
      Number of created subproblems: 0
  Error rc: 0
  Time: 0.009018421173095703
# ----------------------------------------------------------
#   Solution Information
# ----------------------------------------------------------
Solution: 
- number of solutions: 0
  number of solutions displayed: 0
INFO:pypsa.opf:Optimization successful
writing output to test.hdf5 ... WARNING:pypsa.io:HDF5 file format for PyPSA is now deprecated,
        because HDF5 fails for tables with more than 1000 columns.
        Please use netCDF instead.
INFO:pypsa.io:Exported network test.hdf5 has loads, buses, generators
done
name                 generator
time                          
2018-03-26 00:00:00       0.00
2018-03-26 01:00:00       0.00
2018-03-26 02:00:00       0.00
2018-03-26 03:00:00       0.00
2018-03-26 04:00:00       0.00
2018-03-26 05:00:00       0.00
2018-03-26 06:00:00       0.20
2018-03-26 07:00:00       0.50
2018-03-26 08:00:00       0.70
2018-03-26 09:00:00       0.85
2018-03-26 10:00:00       0.95
2018-03-26 11:00:00       1.00
2018-03-26 12:00:00       0.95
2018-03-26 13:00:00       0.85
2018-03-26 14:00:00       0.70
2018-03-26 15:00:00       0.50
2018-03-26 16:00:00       0.20
2018-03-26 17:00:00       0.00
2018-03-26 18:00:00       0.00
2018-03-26 19:00:00       0.00
2018-03-26 20:00:00       0.00
2018-03-26 21:00:00       0.00
2018-03-26 22:00:00       0.00
2018-03-26 23:00:00       0.00
loading input from test.hdf5 ... WARNING:pypsa.io:HDF5 file format for PyPSA is now deprecated,
        because HDF5 fails for tables with more than 1000 columns.
        Please use netCDF instead.
ERROR:pypsa.io:Error, no buses found
done
Empty DataFrame
Columns: []
Index: [2018-03-26 00:00:00, 2018-03-26 01:00:00, 2018-03-26 02:00:00, 2018-03-26 03:00:00, 2018-03-26 04:00:00, 2018-03-26 05:00:00, 2018-03-26 06:00:00, 2018-03-26 07:00:00, 2018-03-26 08:00:00, 2018-03-26 09:00:00, 2018-03-26 10:00:00, 2018-03-26 11:00:00, 2018-03-26 12:00:00, 2018-03-26 13:00:00, 2018-03-26 14:00:00, 2018-03-26 15:00:00, 2018-03-26 16:00:00, 2018-03-26 17:00:00, 2018-03-26 18:00:00, 2018-03-26 19:00:00, 2018-03-26 20:00:00, 2018-03-26 21:00:00, 2018-03-26 22:00:00, 2018-03-26 23:00:00]

[Tim Kittel]

PS: Happy to get you away from your annoying tasks, Jonas (:

[Jonas Hörsch]

Hi Tim,

it's unfortunately a no-brainer, albeit my fault. I accidently pushed to
our fias-internal repository instead of to github. So the fix was not
online, yet.

Your earlier script did and does run fine with that fix.

I am NOT going to check why it does not work with hdfstore, now.
I'll save it for tomorrow.

Best,
Jonas

>> an email to pypsa+un...@googlegroups.com <javascript:>.

>> > To post to this group, send an email to py...@googlegroups.com

>> <javascript:>.

>> > To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/pypsa/e8cc5c5e-f1d0-47c2-8b3a-252ab9784575%40googlegroups.com.
>>
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
> To post to this group, send an email to py...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/cd989c6b-a40a-4c3a-ac5a-7c4c0e73f80a%40googlegroups.com.

[Tim Kittel]

Haha, things happen. Thanks, it's working like a charm now (:

Cheers,
Tim

PS: want me to file an issue for the hdf5 thingy?

[Jonas Hörsch]

Hi Tim,

On Mon, Mar 26 2018, Tim Kittel wrote:

> PS: want me to file an issue for the hdf5 thingy?

No need. The HDF5 issue was basically similar and is solved now as well.
I understand Tom will prepare a bugfix release soon. Actually, gave me
an idea (which is in the same commit) to circumvent the limit on the
number of columns, so i lifted the deprecation.

I would nevertheless argue for using the netcdf format, which is

- structured more cleanly
- easier to use from other languages
- the ability to limit float rep precision to save space

and has support for

- lazy loading, i.e. there is a clear upgrade path by which we can avoid
loading time-series into memory until really needed (and discarding
again, when not needed any more), lessening the memory burden
(although the benefits will only be marginal as long as we're tied to
pyomo).

Best,
Jonas

P.S.: Thanks for indirectly pointing out PEP 498
https://www.python.org/dev/peps/pep-0498/
makes me want to ditch any python before 3.6.

>> https://groups.google.com/d/msgid/pypsa/cd989c6b-a40a-4c3a-ac5a-7c4c0e73f80a%40googlegroups.com.
>>
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
> To post to this group, send an email to py...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/8a8974da-6b43-4fcc-9a4b-b71f28e37392%40googlegroups.com.

[Tim Kittel]

Hi Jonas,

if I understand the discussion in the comments here https://stackoverflow.com/a/1131013 correctly then hdf5 files are now actually netCDF-4 files with additional structuring. So maybe hdf5 would be the way to go in the future? Here https://www.unidata.ucar.edu/software/netcdf/netcdf-4/newdocs/netcdf/Interoperability-with-HDF5.html is a short comparison from UCAR but I didn't find a date on it (:

Cheers, Tim

PS: Yes, PEP 498 is awesome (:

[Jonas Hörsch]

Hi Tim,

i actually understand it the other way around: netCDF4 uses HDF5 for
storage but imposes some additional conventions (and not passing through
the support of some of the hdf5 features, as in your second link).

But be that as it may, what I am unhappy with is the way that
pd.HDFStore uses to store the dataframes. They seem to be splitting
indices and columns into separate tables; so for each dataframe you have
3 different hdf5 tables with some '_something' suffixes. I don't think
there is a standard behind that, but something that the pandas
guys came up with to encode their dataframes. THAT is what will
complicate portability. If I understand correctly one could write im-
and exporters based on pytables, which would get around that, but i am
happy with netcdf for now!

Cheers,
Jonas

>> https://groups.google.com/d/msgid/pypsa/8a8974da-6b43-4fcc-9a4b-b71f28e37392%40googlegroups.com.
>>
>> > For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
> To post to this group, send an email to py...@googlegroups.com.

> To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/4a82bf98-cb19-46c4-ad3b-19bb11522c84%40googlegroups.com.

[Tim Kittel]

that is definitely a good argument and if netcdf works cleanly, I don't find a reason against it (:

Cheers,
Tim