Errors in build_cutout and build_hydro_profile

[Willem Dafoe]

Dear all,

I am encountering issues in build_cutout and in build_hydro_profiles, regardless of which network I specifically want to build.

I changed the config file so that I build the weather cutouts for 2017. To do so, I enabled build_cutout = true and set the year parameter & filenames to 2017. I also changed sarah to era5 and registered and installed the CDS API accordingly. You can find the config file attached.

When trying to execute for example:

snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc --keep-target-files (standard network from fabian's pypsa tutorial)

When doing so, I get the following error at build_cutout:

INFO:atlite.cutout:Cutout europe-2017-era5 not found in directory cutouts, building new one

Traceback (most recent call last):

  File "C:\Users\Willem\pypsa-eur\.snakemake\scripts\tmp_knou02c.build_cutout.py", line 117, in <module>

    cutout = atlite.Cutout(snakemake.wildcards.cutout,

  File "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\cutout.py", line 88, in __init__

    self.meta = self.get_meta(**cutoutparams)

  File "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\preparation.py", line 171, in cutout_get_meta

    ds = prepare_func(xs=xs, ys=ys, year=years.stop, month=months.stop, **meta_kwds)

TypeError: prepare_meta_era5() got an unexpected keyword argument 'resolution'

[Fri Mar  5 13:45:51 2021]

Error in rule build_cutout:

    jobid: 14

    output: cutouts/europe-2017-era5

    log: logs/build_cutout/europe-2017-era5.log (check log file(s) for error message)

Now, I did reset the config file to the default and ran the same rule again just to be sure the error does not occur again. However, after this, I now get another error in build_hydro_profile that I did not get before.

Convert and aggregate `runoff`: 100% (1 of 1) |###############################| Elapsed Time: 0:00:00 Time:  0:00:00

Traceback (most recent call last):

  File "C:\Users\Willem\pypsa-eur\.snakemake\scripts\tmp_72pdf9p.build_hydro_profile.py", line 91, in <module>

    inflow = cutout.runoff(shapes=country_shapes,

  File "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\convert.py", line 502, in runoff

    assert len(years), "Need at least a full year of data (more is better)"

AssertionError: Need at least a full year of data (more is better)

[Fri Mar  5 14:38:31 2021]

Error in rule build_hydro_profile:

    jobid: 18

    output: resources/profile_hydro.nc

    log: logs/build_hydro_profile.log (check log file(s) for error message)

Somehow, nothing seems to work anymore. Have you ever encountered those two errors?  

Thanks in advance!

Willem

[Fabian Hofmann]

Hey Willem,

when you look at the error mesage it points you to where the problem is. I am pretty sure it results from the EIA hydro statistics (in the data bundle EIA_hydro_generation_2000_2014.csv) which only cover years until 2000-2014. It would likely work if the data included 2017. You can try out to generate a new csv file from (probably) here

https://www.eia.gov/international/data/world/total-energy/total-energy-production?pd=44&p=004000000000000000000000000000000000000000000000000000000f00o&u=0&f=A&v=mapbubble&a=-&i=none&vo=value&&t=C&g=00000000000000000000000000000000000000000000000001&l=249-ruvvvvvfvtvnvv1vrvvvvfvvvvvvfvvvou20evvvvvvvvvvnvvvs0008&s=315532800000&e=1514764800000

and store it in the correct readable format.

In case you get along, you are welcome to draft a pull request, or send us the new data file.

Best,

Fabian H

--
You received this message because you are subscribed to the Google Groups "pypsa" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/970dcd65-1803-4788-bee6-824c2ac027c3n%40googlegroups.com.

[Johannes Hampp]

Hey Willem,

I don't see an attached config.yaml so I am guessing blindly here:

What Fabian H mentioned seems to apply to your second problem.

For your first problem:
When you changed the config entry for the cutout to 2017, did you copy
the SARAH entry and change it to ERA5 with the different year, while
keeping the 'resolution' entry?

If yes: The current version of pypsa-eur and atlite (which takes care of
reanalysis dataset handling and time-series generation) does not
understand a 'resolution' in config.yaml for the ERA5 dataset (only for
SARAH) and will fail with presumably this error.

Copying the ERA5 entry from config.yaml and adjusting it like this:

europe-2017-era5:
module: era5
xs: [-12., 35.]
ys: [72., 33.
years: [2017, 2017]

then calling snakemake to create the cutout:

snakemake -jall cutouts/europe-2017-era5

Should work.

HTH,
Johannes

Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Office 110
Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

Am 07/03/2021 um 12:19 schrieb Fabian Hofmann:
> Hey Willem,
>
>
> when you look at the error mesage it points you to where the problem is.
> I am pretty sure it results from the EIA hydro statistics (in the data
> bundle EIA_hydro_generation_2000_2014.csv) which only cover years until
> 2000-2014. It would likely work if the data included 2017. You can try
> out to generate a new csv file from (probably) here
>
> https://www.eia.gov/international/data/world/total-energy/total-energy-production?pd=44&p=004000000000000000000000000000000000000000000000000000000f00o&u=0&f=A&v=mapbubble&a=-&i=none&vo=value&&t=C&g=00000000000000000000000000000000000000000000000001&l=249-ruvvvvvfvtvnvv1vrvvvvfvvvvvvfvvvou20evvvvvvvvvvnvvvs0008&s=315532800000&e=1514764800000
>
> and store it in the correct readable format.
>
> In case you get along, you are welcome to draft a pull request, or send
> us the new data file.
>
>
> Best,
>
> Fabian H
>
>
>
> On 06.03.21 18:56, Willem Dafoe wrote:
>> Dear all,
>>
>> I am encountering issues in build_cutout and in build_hydro_profiles,
>> regardless of which network I specifically want to build.
>>
>> I changed the config file so that I build the weather cutouts for
>> 2017. To do so, I enabled build_cutout = true and set the year
>> parameter & filenames to 2017. I also changed sarah to era5 and
>> registered and installed the CDS API accordingly. You can find the
>> config file attached.
>>
>> When trying to execute for example:

>> *snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc
>> --keep-target-files* (standard network from fabian's pypsa tutorial)
>> When doing so, I get the following error at *build_cutout:*
>>
>> /INFO:atlite.cutout:Cutout europe-2017-era5 not found in directory
>> cutouts, building new one/
>> /Traceback (most recent call last):/
>> /  File

>> "C:\Users\Willem\pypsa-eur\.snakemake\scripts\tmp_knou02c.build_cutout.py",

>> line 117, in <module>/
>> /    cutout = atlite.Cutout(snakemake.wildcards.cutout,/
>> /  File

>> "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\cutout.py",

>> line 88, in __init__/
>> /    self.meta = self.get_meta(**cutoutparams)/
>> /  File

>> "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\preparation.py",

>> line 171, in cutout_get_meta/
>> /    ds = prepare_func(xs=xs, ys=ys, year=years.stop,
>> month=months.stop, **meta_kwds)/
>> /TypeError: prepare_meta_era5() got an unexpected keyword argument
>> 'resolution'/
>> /[Fri Mar  5 13:45:51 2021]/
>> /Error in rule build_cutout:/
>> /    jobid: 14/
>> /    output: cutouts/europe-2017-era5/
>> /    log: logs/build_cutout/europe-2017-era5.log (check log file(s)
>> for error message)/

>>
>>
>> Now, I did reset the config file to the default and ran the same rule
>> again just to be sure the error does not occur again. However, after

>> this, I now get another error in *build_hydro_profile *that I did not
>> get before.
>>
>>
>> /Convert and aggregate `runoff`: 100% (1 of 1)
>> |###############################| Elapsed Time: 0:00:00 Time:  0:00:00/
>> /Traceback (most recent call last):/
>> /  File

>> "C:\Users\Willem\pypsa-eur\.snakemake\scripts\tmp_72pdf9p.build_hydro_profile.py",

>> line 91, in <module>/
>> /    inflow = cutout.runoff(shapes=country_shapes,/
>> /  File

>> "C:\Users\Willem\.conda\envs\pypsa-eur\lib\site-packages\atlite\convert.py",

>> line 502, in runoff/
>> /    //assert len(years), "Need at least a full year of data (more is
>> better)"/
>> /AssertionError: Need at least a full year of data (more is better)/
>> /[Fri Mar  5 14:38:31 2021]/
>> /Error in rule build_hydro_profile:/
>> /    jobid: 18/
>> /    output: resources/profile_hydro.nc/
>> /    log: logs/build_hydro_profile.log (check log file(s) for error
>> message)/

>>
>>
>> Somehow, nothing seems to work anymore. Have you ever encountered
>> those two errors?  
>>
>> Thanks in advance!
>>
>> Willem
>> --
>> You received this message because you are subscribed to the Google
>> Groups "pypsa" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to pypsa+un...@googlegroups.com

>> <mailto:pypsa+un...@googlegroups.com>.

>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/pypsa/970dcd65-1803-4788-bee6-824c2ac027c3n%40googlegroups.com

>> <https://groups.google.com/d/msgid/pypsa/970dcd65-1803-4788-bee6-824c2ac027c3n%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com

> <mailto:pypsa+un...@googlegroups.com>.

> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/79b07b47-dfbe-873e-7764-9c7d66ed8da8%40gmail.com
> <https://groups.google.com/d/msgid/pypsa/79b07b47-dfbe-873e-7764-9c7d66ed8da8%40gmail.com?utm_medium=email&utm_source=footer>.

[Willem Dafoe]

Hi Both,

I am terribly sorry, I just realized that Google somehow dropped the attachment and does not allow me to attach one now as well, no idea why.

Regarding the errors:

@Fabian: Thanks for the heads up, however, the error also showed up for the 2013-14 network. I could resolve that one by deleting the entire resources folder and rebuilding it from 0. 

For 2017, I am now adjusting the hydro generation file with data up to 2018 (latest data from your source) and will revert upon further updates :)

@Johannes: Thanks for your proposed fix. Indeed it did resolve the error as required!

Best,

Willem

[Willem Dafoe]

Hi Fabian, 

I extracted all the data required and with a few pandas operations created a CSV file that has the exact same format as EIA_hydro_generation_2000_2014.csv (when being opened with notepad/editor, in excel however it looks a bit different, I had to add some ' " ' symbols in pandas here and there to truly recreate the format)

However, I screened the code a bit and it seems that the document's name is only specifically mentioned in the snakefile. In order to adjust the document's name in there accordingly, would it only be necessary to change the name in there? (I don't want to simply overwrite the old  EIA_hydro_generation_2000_2014  file, which would lead to confusion IMO).

Feel free to tell me your KIT-eMail, so I can send you the new csv file directly for double-checking.

Best,

Willem