PyPSA-EUR 0.81 Issue - base.py, build_industrial_distribution_key, build_biomass_potentials

[Michael Schulthoff]

Hi everyone,

I just cloned the pypsa-eur repository two days ago and updated the environment.

Now I am running into a couple of issues, but I am not sure if it is related tot he environment, my config, or if it is an issue within the code. Here is the Issue from the log (only a part). If you need more information, please let me know:

----------------------------

34 of 53 steps (64%) done
Traceback (most recent call last):
  File "/home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/.snakemake/scripts/tmp1m2zz742.build_industrial_distribution_key.py", line 158, in <module>
    keys = build_nodal_distribution_key(hotmaps, regions, countries)
  File "/home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/.snakemake/scripts/tmp1m2zz742.build_industrial_distribution_key.py", line 122, in build_nodal_distribution_key
    emissions = facilities["Emissions_ETS_2014"].fillna(
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/series.py", line 5145, in fillna
    return super().fillna(
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/generic.py", line 6893, in fillna
    value = value.reindex(self.index, copy=False)
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/series.py", line 4918, in reindex
    return super().reindex(
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/generic.py", line 5360, in reindex
    return self._reindex_axes(
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/generic.py", line 5375, in _reindex_axes
    new_index, indexer = ax.reindex(
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 4275, in reindex
    raise ValueError("cannot reindex on an axis with duplicate labels")
ValueError: cannot reindex on an axis with duplicate labels
[Mon Aug 21 23:09:57 2023]
Error in rule build_industrial_distribution_key:
    jobid: 46
    input: resources/Test-256-1H-allsectors/regions_onshore_elec_s_256.geojson, resources/Test-256-1H-allsectors/pop_layout_elec_s_256.csv, data/Industrial_Database.csv
    output: resources/Test-256-1H-allsectors/industrial_distribution_key_elec_s_256.csv
    log: logs/Test-256-1H-allsectors/build_industrial_distribution_key_s_256.log (check log file(s) for error details)
    conda-env: /home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/.snakemake/conda/e62942a1318ff32469016242196f3b08_

RuleException:
CalledProcessError in file /home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/rules/build_sector.smk, line 500:
Command 'set -euo pipefail;  /home/johnson/mambaforge/envs/pypsa-eur-v081/bin/python3.10 /home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/.snakemake/scripts/tmp1m2zz742.build_industrial_distribution_key.py' returned non-zero exit status 1.
  File "/home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/rules/build_sector.smk", line 500, in __rule_build_industrial_distribution_key
  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/concurrent/futures/thread.py", line 58, in run
[Mon Aug 21 23:09:58 2023]
Finished job 54.
35 of 53 steps (66%) done
[Mon Aug 21 23:09:58 2023]
Finished job 38.
36 of 53 steps (68%) done
INFO:pypsa.io:Imported network elec_s_256_ec.nc has buses, carriers, generators, lines, links, loads, storage_units, stores
INFO:__main__:N-1 security margin of lines set to 0.7
INFO:pypsa.io:Exported network elec_s_256_ec_lv1.5_.nc has buses, global_constraints, carriers, loads, storage_units, links, generators, stores, lines
[Mon Aug 21 23:10:01 2023]
Finished job 5.
37 of 53 steps (70%) done
/home/johnson/Projects/PyPSA_Michael/20230821/pypsa-eur/.snakemake/scripts/tmpw586lz1b.build_biomass_potentials.py:135: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  pop_nuts2["fraction"] = pop_nuts2.total / pop_nuts2.ct.map(by_country)
[Mon Aug 21 23:10:15 2023]
Finished job 39.
38 of 53 steps (72%) done
[Mon Aug 21 23:11:02 2023]
Finished job 59.
39 of 53 steps (74%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-08-21T225015.124629.snakemake.log

----------------------------------------------

Thanks for your help!

Best,

Michael

[lisaze...@gmail.com]

Hi Michael,

this problem should be resolved in the current version of the master branch (fixed in the PR https://github.com/PyPSA/pypsa-eur/pull/726). Try to update your repository and to rerun the workflow.

Best wishes,

Lisa

[Michael Schulthoff]

Hi Lisa,

Thanks for your quick response! 

I reran the workflow and run into a different issue now. 

Thanks for your help!

Best,

Michael

-----------------------------------------------------------

Set parameter Username
Academic license - for non-commercial use only - expires 2023-12-08
No parameters matching '_test' found
localrules directive specifies rules that are not present in the Snakefile:
copy_conda_env

Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 48
Rules claiming more threads will be scaled down.
Provided resources: mem_mb=250000, mem_mib=238419
Conda environments: ignored
Select jobs to execute...
Changing to shadow directory: /home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/.snakemake/shadow/tmp1yumf3pf
INFO:pypsa.io:Imported network elec_s_256_lv1.5__1H-T-B-H-I-A-solar+p3-dist1_2030.nc has buses, carriers, generators, global_constraints, lines, links, loads, storage_units, stores
Not cleaning up /home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/.snakemake/shadow/tmp1yumf3pf/.snakemake/scripts/tmpik_pwqps.solve_network.py
[Tue Aug 22 22:20:07 2023]
Error in rule solve_sector_network:
    jobid: 0
    input: results/Test-256-1H-allsectors/prenetworks/elec_s_256_lv1.5__1H-T-B-H-I-A-solar+p3-dist1_2030.nc, results/Test-256-1H-allsectors/config.yaml
    output: results/Test-256-1H-allsectors/postnetworks/elec_s_256_lv1.5__1H-T-B-H-I-A-solar+p3-dist1_2030.nc
    log: logs/Test-256-1H-allsectors/elec_s_256_lv1.5__1H-T-B-H-I-A-solar+p3-dist1_2030_solver.log, logs/Test-256-1H-allsectors/elec_s_256_lv1.5__1H-T-B-H-I-A-solar+p3-dist1_2030_python.log (check log file(s) for error details)
    conda-env: /home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/.snakemake/conda/9dc3b4a29ba5e68507dcec076ac7720d_

RuleException:
CalledProcessError in file /home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/rules/solve_overnight.smk, line 41:
Command 'set -euo pipefail;  /home/johnson/mambaforge/envs/pypsa-eur-v081/bin/python3.10 /home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/.snakemake/shadow/tmp1yumf3pf/.snakemake/scripts/tmpik_pwqps.solve_network.py' died with <Signals.SIGKILL: 9>.
  File "/home/johnson/Projects/PyPSA_Michael/20230822/pypsa-eur/rules/solve_overnight.smk", line 41, in __rule_solve_sector_network

  File "/home/johnson/mambaforge/envs/pypsa-eur-v081/lib/python3.10/concurrent/futures/thread.py", line 58, in run

Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

Complete log: .snakemake/log/2023-08-22T214832.788221.snakemake.log

----------------------------------------------------------------------