trouble in replicating pypsa-eur tutorial
440 views
Skip to first unread message
Jiangyi Huang
Sep 28, 2022, 11:25:16 AM9/28/22
to pypsa
Hi PyPSAers,
Following the official instruction, I still got stuck when trying to replicate the pypsa-eur tutorial case. Logs can be found below:
```
(pypsa-eur) hjy@mavt-esa-n105w:/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur$ snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc
Restricted license - for non-production use only - expires 2023-10-25
No parameters matching '_test' found
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
add_electricity 1 1 1
add_extra_components 1 1 1
build_renewable_profiles 4 1 1
build_ship_raster 1 1 1
cluster_network 1 1 1
prepare_network 1 1 1
simplify_network 1 1 1
solve_network 1 1 1
total 11 1 1
Select jobs to execute...
[Wed Sep 28 16:44:14 2022]
rule build_ship_raster:
input: data/shipdensity_global.zip, cutouts/be-03-2013-era5.nc
output: resources/shipdensity_raster.nc
log: logs/build_ship_raster.log
jobid: 14
benchmark: benchmarks/build_ship_raster
reason: Set of input files has changed since last execution
resources: tmpdir=/tmp, mem_mb=5000
/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp0khija2v.build_ship_raster.py:66: DeprecationWarning: open_rasterio is Deprecated in favor of rioxarray. For information about transitioning, see: https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html
with xr.open_rasterio("shipdensity_global.tif") as ship_density:
[Wed Sep 28 16:45:10 2022]
Finished job 14.
1 of 11 steps (9%) done
Select jobs to execute...
[Wed Sep 28 16:45:10 2022]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/be-03-2013-era5.nc
output: resources/profile_onwind.nc
log: logs/build_renewable_profile_onwind.log
jobid: 6
benchmark: benchmarks/build_renewable_profiles_onwind
reason: Set of input files has changed since last execution
wildcards: technology=onwind
resources: tmpdir=/tmp, mem_mb=20000
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (160.30s)
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:__main__:Calculating maximal capacity per bus (method 'simple')
INFO:__main__:Calculate average distances.
[Wed Sep 28 16:47:59 2022]
Finished job 6.
2 of 11 steps (18%) done
Select jobs to execute...
[Wed Sep 28 16:47:59 2022]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/shipdensity_raster.nc, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/be-03-2013-era5.nc
output: resources/profile_offwind-ac.nc
log: logs/build_renewable_profile_offwind-ac.log
jobid: 13
benchmark: benchmarks/build_renewable_profiles_offwind-ac
reason: Input files updated by another job: resources/shipdensity_raster.nc
wildcards: technology=offwind-ac
resources: tmpdir=/tmp, mem_mb=20000
INFO:__main__:Calculate landuse availabilities...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 534, in _process_func
return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
masked, transform = shape_availability(geometry, excluder)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 463, in shape_availability
exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (283,246) (113,103)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp_6if6ca0.build_renewable_profiles.py", line 286, in <module>
availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 610, in compute_availabilitymatrix
availability = list(pool.map(_process_func, shapes.index))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 534, in _process_func
return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
masked, transform = shape_availability(geometry, excluder)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 463, in shape_availability
exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (283,246) (113,103)
[Wed Sep 28 16:48:08 2022]
Error in rule build_renewable_profiles:
jobid: 13
output: resources/profile_offwind-ac.nc
log: logs/build_renewable_profile_offwind-ac.log (check log file(s) for error message)
RuleException:
CalledProcessErrorin line 377 of /mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/Snakefile:
Command 'set -euo pipefail; /home/hjy/miniconda3/envs/pypsa-eur/bin/python3.10 /mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp_6if6ca0.build_renewable_profiles.py' returned non-zero exit status 1.
File "/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/Snakefile", line 377, in __rule_build_renewable_profiles
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-09-28T164413.458072.snakemake.log
Restricted license - for non-production use only - expires 2023-10-25
No parameters matching '_test' found
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
add_electricity 1 1 1
add_extra_components 1 1 1
build_renewable_profiles 4 1 1
build_ship_raster 1 1 1
cluster_network 1 1 1
prepare_network 1 1 1
simplify_network 1 1 1
solve_network 1 1 1
total 11 1 1
Select jobs to execute...
[Wed Sep 28 16:44:14 2022]
rule build_ship_raster:
input: data/shipdensity_global.zip, cutouts/be-03-2013-era5.nc
output: resources/shipdensity_raster.nc
log: logs/build_ship_raster.log
jobid: 14
benchmark: benchmarks/build_ship_raster
reason: Set of input files has changed since last execution
resources: tmpdir=/tmp, mem_mb=5000
/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp0khija2v.build_ship_raster.py:66: DeprecationWarning: open_rasterio is Deprecated in favor of rioxarray. For information about transitioning, see: https://corteva.github.io/rioxarray/stable/getting_started/getting_started.html
with xr.open_rasterio("shipdensity_global.tif") as ship_density:
[Wed Sep 28 16:45:10 2022]
Finished job 14.
1 of 11 steps (9%) done
Select jobs to execute...
[Wed Sep 28 16:45:10 2022]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/be-03-2013-era5.nc
output: resources/profile_onwind.nc
log: logs/build_renewable_profile_onwind.log
jobid: 6
benchmark: benchmarks/build_renewable_profiles_onwind
reason: Set of input files has changed since last execution
wildcards: technology=onwind
resources: tmpdir=/tmp, mem_mb=20000
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (160.30s)
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:atlite.convert:Convert and aggregate 'wind'.
INFO:__main__:Calculating maximal capacity per bus (method 'simple')
INFO:__main__:Calculate average distances.
[Wed Sep 28 16:47:59 2022]
Finished job 6.
2 of 11 steps (18%) done
Select jobs to execute...
[Wed Sep 28 16:47:59 2022]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/shipdensity_raster.nc, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/be-03-2013-era5.nc
output: resources/profile_offwind-ac.nc
log: logs/build_renewable_profile_offwind-ac.log
jobid: 13
benchmark: benchmarks/build_renewable_profiles_offwind-ac
reason: Input files updated by another job: resources/shipdensity_raster.nc
wildcards: technology=offwind-ac
resources: tmpdir=/tmp, mem_mb=20000
INFO:__main__:Calculate landuse availabilities...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 534, in _process_func
return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
masked, transform = shape_availability(geometry, excluder)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 463, in shape_availability
exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (283,246) (113,103)
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp_6if6ca0.build_renewable_profiles.py", line 286, in <module>
availability = cutout.availabilitymatrix(regions, excluder, **kwargs)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 610, in compute_availabilitymatrix
availability = list(pool.map(_process_func, shapes.index))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 367, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 774, in get
raise self._value
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 534, in _process_func
return shape_availability_reprojected(shapes.loc[[i]], *args)[0]
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 509, in shape_availability_reprojected
masked, transform = shape_availability(geometry, excluder)
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/site-packages/atlite/gis.py", line 463, in shape_availability
exclusions = exclusions | masked_
ValueError: operands could not be broadcast together with shapes (283,246) (113,103)
[Wed Sep 28 16:48:08 2022]
Error in rule build_renewable_profiles:
jobid: 13
output: resources/profile_offwind-ac.nc
log: logs/build_renewable_profile_offwind-ac.log (check log file(s) for error message)
RuleException:
CalledProcessErrorin line 377 of /mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/Snakefile:
Command 'set -euo pipefail; /home/hjy/miniconda3/envs/pypsa-eur/bin/python3.10 /mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/.snakemake/scripts/tmp_6if6ca0.build_renewable_profiles.py' returned non-zero exit status 1.
File "/mnt/c/Projects/NorthSeaSectorCoupling/PyPSA_SysModel/pypsa-eur/Snakefile", line 377, in __rule_build_renewable_profiles
File "/home/hjy/miniconda3/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2022-09-28T164413.458072.snakemake.log
```
What is the problem here? The script was run on WSL.
And what if I want to run a specific snakemake rule with wildcards specified in config.yaml, e.g. solve_network, prepare_networks, make_summary, etc.? It would be way much helpful if there is some intruction on these general cases but I couldn't find one.
Max Parzen
Sep 28, 2022, 3:51:52 PM9/28/22
to pypsa
Hi Jiangyi,
I think this requires some debugging. The rule which is causing issues is related to the ship_density raster stuff. Removing eligible offshore areas due to shipping traffic was done recently here: https://github.com/PyPSA/pypsa-eur/pull/401 (quite cool!). A windows compatible fix was also introduced here: https://github.com/PyPSA/pypsa-eur/pull/403. There is either a small bug somewhere or the files that are downloaded are not compatible with the tutorial (hope you use the config.tutorial).
Here is an example of how to debug:
In case it is necessary to debug imported packages e.g. Atlite while being in PyPSA-Eur code, you might want to use the following debug config & debug with "Python: niet justmycode":
{
"version": "0.2.0",
"configurations": [
{
"name": "Python: justmycode",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true
},
{
"name": "Python: niet justmycode",
"type": "python",
"request": "launch",
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": false
}
]
}
Good luck and have fun! :)
Best wishes,
Max
Fabian Neumann
Sep 29, 2022, 1:14:01 AM9/29/22
to Jiangyi Huang, pypsa
Hi Jiangyi,
although I am not 100% certain of it, this looks a bit like you might
have changed the spatial extent of the model in the config.yaml in
between runs? It might be worth retrying once you have deleted resources
results and networks folders.
For general applications of workflow control with snakemake, the
snakemake documentation is a great resource:
snakemake.readthedocs.io/
Best wishes,
Fabian N
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/c79d9e84-8168-4fdb-acb9-988fa6695303n%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/c79d9e84-8168-4fdb-acb9-988fa6695303n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Dr.-Ing.
Fabian Neumann (he/him)
Postdoctoral Researcher
Department of Digital Transformation in Energy Systems
Institute of Energy Technology
Technische Universität Berlin
Group website: https://tub-ensys.github.io
Personal website: https://neumann.fyi
Schedule Meeting: https://calendly.com/fneum
Einsteinufer 25 (TA 8)
10587 Berlin
My working hours may not be your working hours. Do not feel pressure to
reply to this email outside your working hours.
although I am not 100% certain of it, this looks a bit like you might
have changed the spatial extent of the model in the config.yaml in
between runs? It might be worth retrying once you have deleted resources
results and networks folders.
For general applications of workflow control with snakemake, the
snakemake documentation is a great resource:
snakemake.readthedocs.io/
Best wishes,
Fabian N
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/c79d9e84-8168-4fdb-acb9-988fa6695303n%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/c79d9e84-8168-4fdb-acb9-988fa6695303n%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Dr.-Ing.
Fabian Neumann (he/him)
Postdoctoral Researcher
Department of Digital Transformation in Energy Systems
Institute of Energy Technology
Technische Universität Berlin
Group website: https://tub-ensys.github.io
Personal website: https://neumann.fyi
Schedule Meeting: https://calendly.com/fneum
Einsteinufer 25 (TA 8)
10587 Berlin
My working hours may not be your working hours. Do not feel pressure to
reply to this email outside your working hours.
Jiangyi Huang
Sep 29, 2022, 5:12:32 AM9/29/22
to pypsa
Hi Fabian and Max,
Many thanks for your reply! Cleaning the cache folders seems working for now.
Having solved the tutorial system, I also tried the following rules of result analysis and plot. Most of them worked except the "plot_summary" rule that raised errors. I post this here: pypsa-eur rule plot_summary error. Could you please take a look? Thanks again.
Best,
Jiangyi
Reply all
Reply to author
Forward
0 new messages