Error building pypsa-eur-sec

[Ammar]

Hello everyone,

I am trying to build the pypsa-eur-sec model, but the snakemake process always breaks with the following error:

>snakemake prepare_sector_networks --keep-target-files --cores 4

Building DAG of jobs...

Executing subworkflow pypsaeur.

Building DAG of jobs...

MissingRuleException:

No rule to produce networks\elec_s_128_ec_lv1.0_.nc (if you use input functions make sure that they don't raise unexpected exceptions).

- I am working on a windows machine.

- Building the pypsa-eur model separately works just fine (with the use of --keep-target-files)

My guess is that (--keep-target-files) is only applying to the main snakemake workflow (pypsa-eur-sec), but not applying to the sub-workflow (pypsaeur).

I know that switching to Linux would make life easier, but does any of you have any workarounds that would work with windows?

Thanks in advance!

Ammar

[Ammar]

More context on the issue:

- I am using pypsa-eur version: 0.2.0

- If I try to run the pypsa-eur-sec snakemake multiple times in a row, the error keep changing between these three:

Building DAG of jobs...
Executing subworkflow pypsaeur.
Building DAG of jobs...
MissingRuleException:

No rule to produce networks\elec_s_128_lv1.0_.nc (if you use input functions make sure that they don't raise unexpected exceptions).

or

Building DAG of jobs...
Executing subworkflow pypsaeur.
Building DAG of jobs...
MissingRuleException:

No rule to produce resources\regions_onshore_elec_s_128.geojson (if you use input functions make sure that they don't raise unexpected exceptions).

or

Building DAG of jobs...
Executing subworkflow pypsaeur.
Building DAG of jobs...
MissingRuleException:

No rule to produce resources\nuts3_shapes.geojson (if you use input functions make sure that they don't raise unexpected exceptions).

This is why I think it is an issue related to (--keep-target-files) and path separators (maybe) ?

[Lisa Zeyen]

Hi Ammar,

is it working without the flag --keep-target-files? 

Since you are using the version 0.2 of pypsa-eur you should not modify your Snakefile in pypsa-eur-sec in the rule prepare_sector_network (so keep the 'ec').

On my linux machine I couldn't reproduce the error. 

Is your pypsa-eur directory parallel to the pypsa-eur-sec directory?

Best,

Lisa 

----------------------------------------------------

What I wrote before, sorry, ended in a private mail:

Hi Ammar,

which pypsa-eur version are you using? The error occurs because the input network from pypsa-eur can not be build.

PyPSA-Eur-Sec starts at a subworkflow pypsa-eur to generate the clustered electric network. I guess you use an older version of pypsa-eur which had a different name for the clustered network. There are 2 things you can do:

1.) update your pypsa-eur version

2.) change your pypsa-eur-sec Snakefile to work with your pypsa-eur version. Therefore you would have to change in rule prepare_sector_network the input   by removing the 'ec'
        network=pypsaeur('networks/{network}_s{simpl}_{clusters}_lv{lv}_{opts}.nc')

Hope that helps!

Best,

Lisa

----------------------------------

[Ammar]

Hi Lisa,

I do have both (pypsa-eur) and (pypsa-eur-sec) in parallel directories,

For pypsa-eur:

• snakemake -j 1 results/networks/elec_s_128_ec_lcopt_Co2L-3H.nc     # (does not work)
• snakemake -j 1 results/networks/elec_s_128_ec_lcopt_Co2L-3H.nc --keep-target-files     # (works fine)
• snakemake prepare all_elec_networks --cores 4     # (works fine without the keep-target-files flag)

For pypsa-eur-sec:

• Nothing works and I always get one of the (MissingRuleException) errors I mentioned above.
  
  

I would imagine the error would not occur with Linux because snakemake and the snakefiles were originally written for Linux (I would assume?).

This further supports my initial guess that it is related to (--keep-target-files) and path separators.

Best,

Ammar

[Tom Brown]

Hi Ammar,

I don't know what the cause is, but you could try playing with
"wildcard_constraints" in the Snakefile, since these might be sensitive
to the operating system. They're different in PyPSA-Eur and PyPSA-Eur-Sec.

Best,

Tom

On 22/07/2020 14:52, Ammar wrote:
> Hi Lisa,
>
> I do have both (pypsa-eur) and (pypsa-eur-sec) in parallel directories,
>
> For pypsa-eur:
>

> * snakemake -j 1 results/networks/elec_s_128_ec_lcopt_Co2L-3H.nc     #
> (does not work)
> * snakemake -j 1 results/networks/elec_s_128_ec_lcopt_Co2L-3H.nc
> --keep-target-files     # (works fine)
> * snakemake prepare all_elec_networks --cores 4     # (works fine

> without the keep-target-files flag)
>
> For pypsa-eur-sec:
>

> * Nothing works and I always get one of the (MissingRuleException)

> errors I mentioned above.
>
> I would imagine the error would not occur with Linux because snakemake
> and the snakefiles were originally written for Linux (I would assume?).
> This further supports my initial guess that it is related to
> (--keep-target-files) and path separators.
>
> Best,
>
> Ammar
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/ff69d5fc-a176-4980-b92a-73ccbba72a49o%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/ff69d5fc-a176-4980-b92a-73ccbba72a49o%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Karlsruhe Institute of Technology (KIT)
Institute for Automation and Applied Informatics (IAI)

Tom Brown (he/him)
Research Group Leader, Energy System Modelling

Phone: +49 721 608 25737
Fax: +49 721 608 22602
Group website: https://www.iai.kit.edu/english/ESM.php
Personal website: https://nworbmot.org/

Visitor Address:
Office 309
Campus North Building 445
Hermann-von-Helmholtz-Platz 1
76344 Eggenstein-Leopoldshafen

[Erik Rathje]

Hello everyone,

I don't think that Ammar need this suggestions anymore, but I managed to run PyPSA-Eur-Sec on Windows with the following Changes:

1) Windows and Linux use different delimiters by calling a path. When running pypsa-eur, Windows helps with this issue and "corrects" the Path by itself.

    In the Snakemake Subworkflow from PyPSa-Eur-Sec to PyPSA-Eur, Snakemake searches for the path with "/", but Windows uses "\".

    So  every Snakerule from PyPSA-Eur, which is called from PyPSA-Eur-Sec Snakefile with the keyword "pypsaeur" must be changed from the delimiter "/" to "\" with a "r" infront of the quotation marks

    for raw string. Furthermore the modified path from this rules must be changed in every Rule, where the Output from the modified path is an Input.

 

   For example: In the PyPSA-Eur-Sec-Snakefile rule "build_populations_layouts" the nuts3-shapes is an input from PyPSA-Eur  with "nuts_shapes=pypsaeur('resources/nuts3_shapes.geojson'),"

   so we change within the PyPSA-Eur Snakefile the Rule "build_shapes" in the Output Section from "nuts3_shapes='resources/nuts3_shapes.geojson' " to "nuts3_shapes= r'resources\nuts3_shapes.geojson' " .

  The nuts3_shapes is an input in the PyPSA-Eur Rule "add_electricity", so the path must be changed there the same way.

2) The Yaml File gets decoded with the default decodation of the OS. Our Yaml File is written in utf-8, which can cause an error on Windows       

    machines. But with an small modification of an py-script of Snakemake , this can be solved.

    In the py-script "io.py" from the snakemake package in the definition of the method "_load_configfile..." the line

    "with open(configpath) as f: " must be changed to "with open(configpath, encoding ="utf-8") as f:"

With this to Changes, I managed to run PyPSA-Eur Sec (Commit 5a7f0c2 and PyPSA-Eur Commit 08674a6).

I'm very sorry, if this post is hard to read, but I wanted to offer some help, if someone else is trying to run PyPSA-Eur-Sec on a Windows Machine.

Best Regards,

Erik

[Ammar]

Hi Erik,

I imagined it would have something to do with path separators, but I actually switched to Linux (which made things much easier) before going through with that direction.

Thanks for sharing your experience though!

Cheers,

Ammar