[PyPSA-Eur] solve_network

[Anna Peecock]

Hi All,

We're having an issue running the snakemake workflow on an HPC, both of which I'm very new to so apologies in advance if this is easily solvable. Even though there is a corresponding .nc file in the networks folder, when we run the 'snakemake -j 1 --use-conda  results/networks/elec_s_37_ec_lcopt_24H.nc', the process fails seemingly trying to execute all dependency rules (instead of just the solve_network rule).

Do you have any idea how to get around this?

Many thanks,

Anna

[Johannes Hampp]

Hi Anna,

Sounds like a snakemake issue.

If you just want to run the `solve_network` rule but snakemake is
trying to execute all other rules before as well, then you have to
remove the reason which triggers snakemake to run them.

Try find the reason which triggers snakemake

snakemake -j 1 --use-conda -n -r results/networks/elec_s_37_ec_lcopt_24H.nc

-n -> --dry-run
-r -> --reason

Check the snakemake docs for more details on those flags:
snakemake.readthedocs.io/

If you know why snakemake wants to rerun all preceeding rules then you
just have to fix it.

If I had to blindly guess, then adding `--rerun-trigger mtime` to your
snakemake call might fix it. (Stupid usability issue which was
introduced into snakemake a while back)


Best,
Johannes


Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Gailin Pease]

I ran into a similar issue when attempting to prepare and solve the networks from the Zenodo release (https://zenodo.org/record/7251657#.Y5D4goLMKrM). I'm recording my solution here in case it's useful to future users. Please let me know if there's a simpler way to do this that I missed -- I'm new to snakemake and PyPSA-Eur. I'm on an M1 mac and running the pypsa-eur master branch.

The command I used was snakemake -j 1 -n -r results/networks/elec_s_37_ec_lv1.0_CO2L-24H.nc --rerun-trigger mtime, which should just run the add_extra_component, prepare_network, and solve_network steps (using elec_s_37.nc from the Zenodo release as input). But snakemake wanted to run 19 steps:

Job stats:
job                         count    min threads    max threads
------------------------  -------  -------------  -------------
add_electricity                 1              1              1
add_extra_components            1              1              1
base_network                    1              1              1
build_bus_regions               1              1              1
build_hydro_profile             1              1              1
build_load_data                 1              1              1
build_powerplants               1              1              1
build_renewable_profiles        4              1              1
build_shapes                    1              1              1
build_ship_raster               1              1              1
cluster_network                 1              1              1
retrieve_cutout                 2              1              1

prepare_network                 1              1              1

simplify_network                1              1              1

solve_network                   1              1              1

total                          19              1              1

I fixed this by manually redating the output files that were triggering snakemake to re-run jobs, and was able to get the following (correct, I think) plan:

Job stats:
job                     count    min threads    max threads
--------------------  -------  -------------  -------------
add_extra_components        1              1              1
prepare_network             1              1              1
solve_network               1              1              1
total                       3              1              1

How I fixed it:

I used touch -d '<old date>' <file> for each file that snakemake was detecting as updated ("Updated input files" in the snakemake output). I then reran  snakemake -j 1 -n -r results/networks/elec_s_37_ec_lv1.0_CO2L-24H.nc --rerun-trigger mtime and used touch -d '<slightly newer old date>' <file>  on the new list of "Updated input files". I did this until I'd manually reordered all the relevant files:

I made the following files oldest:

• data/load_raw.csv
• data/bundle/NUTS_2013_60M_SH/data/NUTS_RG_60M_2013.shp
• data/bundle/eez/World_EEZ_v8_2014.shp
• data/bundle/naturalearth/ne_10m_admin_0_countries.shp
• data/bundle/nama_10r_3gdp.tsv.gz

Second oldest:

• data/entsoegridkit/buses.csv
• data/entsoegridkit/lines.csv

Third oldest:

• data/nuclear_p_max_pu.csv

[Johannes Hampp]

There's indeed an easier method for touch-ing files (if the reason for
snakemake trying to rerun rules is the modification date of files). In
your case:

snakemake -j 1 -r --touch
results/networks/elec_s_37_ec_lv1.0_CO2L-24H.nc --until cluster_network

would let snakemake automatically touch all output files of the rules
required to produce the output
'results/networks/elec_s_37_ec_lv1.0_CO2L-24H.nc' until including the
rule `cluster_network`.

See here for details:
https://snakemake.readthedocs.io/en/stable/executing/cli.html#execution

Best,
Johannes

Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

> *How I fixed it:*

> I used touch -d '<old date>' <file> for each file that snakemake was
> detecting as updated ("Updated input files" in the snakemake output). I
> then reran snakemake -j 1 -n -r
> results/networks/elec_s_37_ec_lv1.0_CO2L-24H.nc --rerun-trigger mtime
> and used touch -d '<slightly newer old date>' <file>  on the new list of
> "Updated input files". I did this until I'd manually reordered all the
> relevant files:
>
> I made the following files oldest:
>

> * data/load_raw.csv
> * data/bundle/NUTS_2013_60M_SH/data/NUTS_RG_60M_2013.shp
> * data/bundle/eez/World_EEZ_v8_2014.shp
> * data/bundle/naturalearth/ne_10m_admin_0_countries.shp
> * data/bundle/nama_10r_3gdp.tsv.gz
>
> Second oldest:
>
> * data/entsoegridkit/buses.csv
> * data/entsoegridkit/lines.csv
>
> Third oldest:
>
> * data/nuclear_p_max_pu.csv

>
>
> On Thursday, October 27, 2022 at 10:26:40 AM UTC-4 Johannes wrote:
>
> Hi Anna,
>
> Sounds like a snakemake issue.
>
> If you just want to run the `solve_network` rule but snakemake is
> trying to execute all other rules before as well, then you have to
> remove the reason which triggers snakemake to run them.
>
> Try find the reason which triggers snakemake
>
> snakemake -j 1 --use-conda -n -r
> results/networks/elec_s_37_ec_lcopt_24H.nc
>
> -n -> --dry-run
> -r -> --reason
>
> Check the snakemake docs for more details on those flags:

> snakemake.readthedocs.io/ <http://snakemake.readthedocs.io/>

>
> If you know why snakemake wants to rerun all preceeding rules then you
> just have to fix it.
>
> If I had to blindly guess, then adding `--rerun-trigger mtime` to your
> snakemake call might fix it. (Stupid usability issue which was
> introduced into snakemake a while back)
>
>
> Best,
> Johannes
>
>
> Best regards,
> Johannes Hampp (he/him)
>
> Justus Liebig University Giessen (JLU)
> Center for international Development and Environmental Research (ZEU)
>
> mailto: johanne...@zeu.uni-giessen.de
>
> Senckenbergstr. 3
> DE-35392 Giessen

> https://uni-giessen.de/zeu <https://uni-giessen.de/zeu>

>
> Am 27/10/2022 um 14:15 schrieb Anna Peecock:
> > Hi All,
> >
> > We're having an issue running the snakemake workflow on an HPC,
> both of
> > which I'm very new to so apologies in advance if this is easily
> > solvable. Even though there is a corresponding .nc file in the
> networks
> > folder, when we run the 'snakemake -j 1 --use-conda
> >  results/networks/elec_s_37_ec_lcopt_24H.nc', the process fails
> > seemingly trying to execute all dependency rules (instead of just
> the
> > solve_network rule).
> >
> > Do you have any idea how to get around this?
> >
> > Many thanks,
> > Anna
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pypsa" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pypsa+un...@googlegroups.com
> > <mailto:pypsa+un...@googlegroups.com>.
> > To view this discussion on the web, visit
> >

> https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com> <https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pypsa/1f887eb7-dba6-46e2-ae44-0f5d03b523e6n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/23bf6c4c-e4e3-44cd-97f1-0254522bf92en%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/23bf6c4c-e4e3-44cd-97f1-0254522bf92en%40googlegroups.com?utm_medium=email&utm_source=footer>.