[PyPSA-Eur] Tutorial rule build_renewable_profiles error
282 views
Skip to first unread message
Hyemin Park
Feb 11, 2023, 3:38:06 PM2/11/23
to pypsa
Hi PyPSA team,
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
add_electricity 1 1 1
add_extra_components 1 1 1
base_network 1 1 1
build_bus_regions 1 1 1
build_load_data 1 1 1
build_powerplants 1 1 1
build_renewable_profiles 4 1 1
build_ship_raster 1 1 1
cluster_network 1 1 1
prepare_network 1 1 1
simplify_network 1 1 1
solve_network 1 1 1
total 15 1 1
Select jobs to execute...
[Sat Feb 11 20:20:05 2023]
rule base_network:
input: data/entsoegridkit/buses.csv, data/entsoegridkit/lines.csv, data/entsoegridkit/links.csv, data/entsoegridkit/converters.csv, data/entsoegridkit/transformers.csv, data/parameter_corrections.yaml, data/links_p_nom.csv, data/links_tyndp.csv, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/europe_shape.geojson
output: networks/base.nc
log: logs/base_network.log
jobid: 7
benchmark: benchmarks/base_network
reason: Missing output files: networks/base.nc
resources: tmpdir=/tmp, mem_mb=1500, mem_mib=1431
INFO:__main__:Removing buses with voltages Float64Index([132.0, 500.0, 750.0], dtype='float64')
INFO:__main__:TYNDP links already in the dataset (skipping): Italy-Montenegro, NordLink, COBRA cable, Anglo-Scottish -1, ALEGrO, North Sea Link, ElecLink, Estlink 1
INFO:__main__:Adding the following TYNDP links: Biscay Gulf, Italy-France, IFA2, Thames Estuary Cluster (NEMO-Link), HVDC SuedOstLink, HVDC Line A-North, France-Alderney-Britain, Viking DKW-GB, Greenconnector, Hansa PowerBridge I, NorthConnect, HVDC SuedLink, AQUIND Interconnector, HVDC Ultranet, Gridlink, NeuConnect, NordBalt, Greenlink, Celtic Interconnector
/home/hm_a_park/pypsa-eur/.snakemake/scripts/tmpsose2nxm.base_network.py:720: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.
n.set_snapshots(pd.date_range(freq="h", **config["snapshots"]))
INFO:__main__:Removing 60 unconnected network components with less than 4 buses. In total 93 buses.
WARNING:__main__:31 buses are not in any country or offshore shape, 21 have been assigned from the tag of the entsoe map, the rest from the next bus in terms of pathlength.
INFO:__main__:Replacing B2B converter `6342` together with bus `5544` and line `14746` by an HVDC tie-line LT-PL
INFO:pypsa.io:Exported network base.nc has links, transformers, buses, lines
[Sat Feb 11 20:20:20 2023]
Finished job 7.
1 of 15 steps (7%) done
Select jobs to execute...
[Sat Feb 11 20:20:20 2023]
rule build_powerplants:
input: networks/base.nc, data/custom_powerplants.csv
output: resources/powerplants.csv
log: logs/build_powerplants.log
jobid: 21
reason: Input files updated by another job: networks/base.nc
resources: tmpdir=/tmp, mem_mb=5000, mem_mib=4769
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
INFO:powerplantmatching.collection:Retrieving data from https://raw.githubusercontent.com/FRESNA/powerplantmatching/v0.5.5/powerplants.csv
INFO:powerplantmatching.collection:Store data at /home/hm_a_park/.local/share/powerplantmatching/data/out/default/matched_data_red.csv
WARNING:root:No powerplants known in: MK
WARNING:root:Couldn't find close bus for 398 powerplants. Removing them from the powerplants list.
[Sat Feb 11 20:20:46 2023]
Finished job 21.
2 of 15 steps (13%) done
Select jobs to execute...
[Sat Feb 11 20:20:47 2023]
rule build_bus_regions:
input: resources/country_shapes.geojson, resources/offshore_shapes.geojson, networks/base.nc
output: resources/regions_onshore.geojson, resources/regions_offshore.geojson
log: logs/build_bus_regions.log
jobid: 11
reason: Input files updated by another job: networks/base.nc
resources: tmpdir=/tmp, mem_mb=1000, mem_mib=954
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
I am trying to replicate the tutorial and getting an error. Could you help on this?
~/pypsa-eur$ snakemake -j 1 results/networks/elec_s_6_ec_lcopt_Co2L-24H.nc
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
add_electricity 1 1 1
add_extra_components 1 1 1
base_network 1 1 1
build_bus_regions 1 1 1
build_load_data 1 1 1
build_powerplants 1 1 1
build_renewable_profiles 4 1 1
build_ship_raster 1 1 1
cluster_network 1 1 1
prepare_network 1 1 1
simplify_network 1 1 1
solve_network 1 1 1
total 15 1 1
Select jobs to execute...
[Sat Feb 11 20:20:05 2023]
rule base_network:
input: data/entsoegridkit/buses.csv, data/entsoegridkit/lines.csv, data/entsoegridkit/links.csv, data/entsoegridkit/converters.csv, data/entsoegridkit/transformers.csv, data/parameter_corrections.yaml, data/links_p_nom.csv, data/links_tyndp.csv, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/europe_shape.geojson
output: networks/base.nc
log: logs/base_network.log
jobid: 7
benchmark: benchmarks/base_network
reason: Missing output files: networks/base.nc
resources: tmpdir=/tmp, mem_mb=1500, mem_mib=1431
INFO:__main__:Removing buses with voltages Float64Index([132.0, 500.0, 750.0], dtype='float64')
INFO:__main__:TYNDP links already in the dataset (skipping): Italy-Montenegro, NordLink, COBRA cable, Anglo-Scottish -1, ALEGrO, North Sea Link, ElecLink, Estlink 1
INFO:__main__:Adding the following TYNDP links: Biscay Gulf, Italy-France, IFA2, Thames Estuary Cluster (NEMO-Link), HVDC SuedOstLink, HVDC Line A-North, France-Alderney-Britain, Viking DKW-GB, Greenconnector, Hansa PowerBridge I, NorthConnect, HVDC SuedLink, AQUIND Interconnector, HVDC Ultranet, Gridlink, NeuConnect, NordBalt, Greenlink, Celtic Interconnector
/home/hm_a_park/pypsa-eur/.snakemake/scripts/tmpsose2nxm.base_network.py:720: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.
n.set_snapshots(pd.date_range(freq="h", **config["snapshots"]))
INFO:__main__:Removing 60 unconnected network components with less than 4 buses. In total 93 buses.
WARNING:__main__:31 buses are not in any country or offshore shape, 21 have been assigned from the tag of the entsoe map, the rest from the next bus in terms of pathlength.
INFO:__main__:Replacing B2B converter `6342` together with bus `5544` and line `14746` by an HVDC tie-line LT-PL
INFO:pypsa.io:Exported network base.nc has links, transformers, buses, lines
[Sat Feb 11 20:20:20 2023]
Finished job 7.
1 of 15 steps (7%) done
Select jobs to execute...
[Sat Feb 11 20:20:20 2023]
rule build_powerplants:
input: networks/base.nc, data/custom_powerplants.csv
output: resources/powerplants.csv
log: logs/build_powerplants.log
jobid: 21
reason: Input files updated by another job: networks/base.nc
resources: tmpdir=/tmp, mem_mb=5000, mem_mib=4769
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
INFO:powerplantmatching.collection:Retrieving data from https://raw.githubusercontent.com/FRESNA/powerplantmatching/v0.5.5/powerplants.csv
INFO:powerplantmatching.collection:Store data at /home/hm_a_park/.local/share/powerplantmatching/data/out/default/matched_data_red.csv
WARNING:root:No powerplants known in: MK
WARNING:root:Couldn't find close bus for 398 powerplants. Removing them from the powerplants list.
[Sat Feb 11 20:20:46 2023]
Finished job 21.
2 of 15 steps (13%) done
Select jobs to execute...
[Sat Feb 11 20:20:47 2023]
rule build_bus_regions:
input: resources/country_shapes.geojson, resources/offshore_shapes.geojson, networks/base.nc
output: resources/regions_onshore.geojson, resources/regions_offshore.geojson
log: logs/build_bus_regions.log
jobid: 11
reason: Input files updated by another job: networks/base.nc
resources: tmpdir=/tmp, mem_mb=1000, mem_mib=954
INFO:pypsa.io:Imported network base.nc has buses, lines, links, transformers
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
.......
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
[Sat Feb 11 20:20:55 2023]
Finished job 11.
3 of 15 steps (20%) done
Select jobs to execute...
[Sat Feb 11 20:20:55 2023]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 18
benchmark: benchmarks/build_renewable_profiles_solar
reason: Missing output files: resources/profile_solar.nc; Input files updated by another job: networks/base.nc, resources/regions_onshore.geojson
wildcards: technology=solar
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:distributed.scheduler:State start
INFO:distributed.scheduler: Scheduler at: tcp://127.0.0.1:46175
INFO:distributed.scheduler: dashboard at: 127.0.0.1:8787
INFO:distributed.nanny: Start Nanny at: 'tcp://127.0.0.1:34141'
2023-02-11 20:21:02,070 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space/worker-4cv85fec', purging
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:33645', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:33645
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:55480
INFO:distributed.scheduler:Receive client connection: Client-9a95f67b-aa49-11ed-a9dd-42010a800002
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:55494
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (466.07s)
[Sat Feb 11 20:29:01 2023]
Error in rule build_renewable_profiles:
jobid: 18
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Complete log: .snakemake/log/2023-02-11T202003.807215.snakemake.log
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/site-packages/shapely/set_operations.py:133: RuntimeWarning: invalid value encountered in intersection
return lib.intersection(a, b, **kwargs)
[Sat Feb 11 20:20:55 2023]
Finished job 11.
3 of 15 steps (20%) done
Select jobs to execute...
[Sat Feb 11 20:20:55 2023]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 18
benchmark: benchmarks/build_renewable_profiles_solar
reason: Missing output files: resources/profile_solar.nc; Input files updated by another job: networks/base.nc, resources/regions_onshore.geojson
wildcards: technology=solar
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:distributed.scheduler:State start
INFO:distributed.scheduler: Scheduler at: tcp://127.0.0.1:46175
INFO:distributed.scheduler: dashboard at: 127.0.0.1:8787
INFO:distributed.nanny: Start Nanny at: 'tcp://127.0.0.1:34141'
2023-02-11 20:21:02,070 - distributed.diskutils - INFO - Found stale lock file and directory '/tmp/dask-worker-space/worker-4cv85fec', purging
INFO:distributed.scheduler:Register worker <WorkerState 'tcp://127.0.0.1:33645', name: 0, status: init, memory: 0, processing: 0>
INFO:distributed.scheduler:Starting worker compute stream, tcp://127.0.0.1:33645
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:55480
INFO:distributed.scheduler:Receive client connection: Client-9a95f67b-aa49-11ed-a9dd-42010a800002
INFO:distributed.core:Starting established connection to tcp://127.0.0.1:55494
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (466.07s)
[Sat Feb 11 20:29:01 2023]
Error in rule build_renewable_profiles:
jobid: 18
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
/home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Complete log: .snakemake/log/2023-02-11T202003.807215.snakemake.log
Fabian Neumann
Feb 20, 2023, 3:24:08 AM2/20/23
to pypsa
Hi,
does the error still persist? We have fixed a few compatibility issues with some dependencies.
Best wishes,
Fabian
Message has been deleted
Swaechchha Dahal
Sep 24, 2023, 10:19:37 AM9/24/23
to pypsa
Hi,
INFO:__main__:Calculate landuse availabilities...
wildcards: technology=solar
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
I am trying to do the same (replicate the tutorial: electricity only) and got the similar error.
[Sun Sep 24 15:27:37 2023]
Finished job 19.
16 of 25 steps (64%) done
Select jobs to execute...
[Sun Sep 24 15:27:37 2023]
rule build_renewable_profiles:
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, data/bundle/GEBCO_2014_2D.nc, resources/shipdensity_raster.tif, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/europe-2013-era5.nc
output: resources/profile_offwind-dc.nc
log: logs/build_renewable_profile_offwind-dc.log
jobid: 18
benchmark: benchmarks/build_renewable_profiles_offwind-dc
reason: Missing output files: resources/profile_offwind-dc.nc; Input files updated by another job: resources/shipdensity_raster.tif, resources/country_shapes.geojson, data/bundle/GEBCO_2014_2D.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/offshore_shapes.geojson, resources/networks/base.nc, cutouts/europe-2013-era5.nc, resources/natura.tiff, resources/regions_offshore.geojson
wildcards: technology=offwind-dc
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:__main__:correction_factor is set as 0.8855
Finished job 19.
16 of 25 steps (64%) done
Select jobs to execute...
[Sun Sep 24 15:27:37 2023]
rule build_renewable_profiles:
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, data/bundle/GEBCO_2014_2D.nc, resources/shipdensity_raster.tif, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_offshore.geojson, cutouts/europe-2013-era5.nc
output: resources/profile_offwind-dc.nc
log: logs/build_renewable_profile_offwind-dc.log
jobid: 18
benchmark: benchmarks/build_renewable_profiles_offwind-dc
reason: Missing output files: resources/profile_offwind-dc.nc; Input files updated by another job: resources/shipdensity_raster.tif, resources/country_shapes.geojson, data/bundle/GEBCO_2014_2D.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/offshore_shapes.geojson, resources/networks/base.nc, cutouts/europe-2013-era5.nc, resources/natura.tiff, resources/regions_offshore.geojson
wildcards: technology=offwind-dc
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:__main__:correction_factor is set as 0.8855
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (65.46s)
INFO:atlite.convert:Convert and aggregate 'wind'.
[########################################] | 100% Completed | 107.24 s
INFO:atlite.convert:Convert and aggregate 'wind'.
[########################################] | 100% Completed | 108.14 s
INFO:__main__:Calculating maximal capacity per bus (method 'simple')
INFO:__main__:Calculate average distances.
INFO:__main__:Calculate underwater fraction of connections.
[Sun Sep 24 15:32:22 2023]
Finished job 18.
17 of 25 steps (68%) done
Select jobs to execute...
[Sun Sep 24 15:32:22 2023]
rule build_renewable_profiles:
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 6
benchmark: benchmarks/build_renewable_profiles_solar
reason: Missing output files: resources/profile_solar.nc; Input files updated by another job: resources/offshore_shapes.geojson, resources/country_shapes.geojson, resources/networks/base.nc, resources/natura.tiff, data/bundle/corine/g250_clc06_V18_5.tif, cutouts/europe-2013-sarah.nc, resources/regions_onshore.geojson
INFO:atlite.convert:Convert and aggregate 'wind'.
[########################################] | 100% Completed | 107.24 s
INFO:atlite.convert:Convert and aggregate 'wind'.
[########################################] | 100% Completed | 108.14 s
INFO:__main__:Calculating maximal capacity per bus (method 'simple')
INFO:__main__:Calculate average distances.
INFO:__main__:Calculate underwater fraction of connections.
[Sun Sep 24 15:32:22 2023]
Finished job 18.
17 of 25 steps (68%) done
Select jobs to execute...
[Sun Sep 24 15:32:22 2023]
rule build_renewable_profiles:
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 6
benchmark: benchmarks/build_renewable_profiles_solar
reason: Missing output files: resources/profile_solar.nc; Input files updated by another job: resources/offshore_shapes.geojson, resources/country_shapes.geojson, resources/networks/base.nc, resources/natura.tiff, data/bundle/corine/g250_clc06_V18_5.tif, cutouts/europe-2013-sarah.nc, resources/regions_onshore.geojson
wildcards: technology=solar
resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
INFO:__main__:Calculate landuse availabilities...
INFO:__main__:Completed availability calculation (91.23s)
INFO:atlite.convert:Convert and aggregate 'pv'.
[ ] | 0% Completed | 84.28 s[Sun Sep 24 15:35:22 2023]
Error in rule build_renewable_profiles:
jobid: 6
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
INFO:atlite.convert:Convert and aggregate 'pv'.
[ ] | 0% Completed | 84.28 s[Sun Sep 24 15:35:22 2023]
Error in rule build_renewable_profiles:
jobid: 6
input: resources/networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error details)
conda-env: /home/sdaha/pypsa-eur/.snakemake/conda/ccbebbbab5ec647bc4ed52e844531a6e_
RuleException:
CalledProcessError in file /home/sdaha/pypsa-eur/rules/build_electricity.smk, line 257:
Command 'set -euo pipefail; /home/sdaha/miniconda3/envs/pypsa-eur/bin/python3.10 /home/sdaha/pypsa-eur/.snakemake/scripts/tmpfn_8t_gm.build_renewable_profiles.py' died with <Signals.SIGKILL: 9>.
File "/home/sdaha/pypsa-eur/rules/build_electricity.smk", line 257, in __rule_build_renewable_profiles
File "/home/sdaha/miniconda3/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
RuleException:
CalledProcessError in file /home/sdaha/pypsa-eur/rules/build_electricity.smk, line 257:
Command 'set -euo pipefail; /home/sdaha/miniconda3/envs/pypsa-eur/bin/python3.10 /home/sdaha/pypsa-eur/.snakemake/scripts/tmpfn_8t_gm.build_renewable_profiles.py' died with <Signals.SIGKILL: 9>.
File "/home/sdaha/pypsa-eur/rules/build_electricity.smk", line 257, in __rule_build_renewable_profiles
File "/home/sdaha/miniconda3/envs/pypsa-eur/lib/python3.10/concurrent/futures/thread.py", line 58, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2023-09-24T132040.590936.snakemake.log
Johannes Hampp
Sep 25, 2023, 6:15:57 AM9/25/23
to Swaechchha Dahal, pypsa
Hi,
Something seems to be killing your process (SIGKILL 9). Is the workflow
exceeding local computer resources or are you exceeding allocated
resources in a shared computing environment (e.g. requesting not enough
ressources on a HPC)?
Best,
Johannes
> output: networks/base.nc <http://base.nc>
> resources: tmpdir=/tmp, mem_mb=5000, mem_mib=4769
>
> INFO:pypsa.io:Imported network base.nc <http://base.nc> has
> resources: tmpdir=/tmp, mem_mb=1000, mem_mib=954
>
> INFO:pypsa.io:Imported network base.nc <http://base.nc> has
> input: networks/base.nc <http://base.nc>,
> output: resources/profile_solar.nc <http://profile_solar.nc>
> networks/base.nc <http://base.nc>, resources/regions_onshore.geojson
> output: resources/profile_solar.nc <http://profile_solar.nc>
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/cd1bf94e-7fb5-4e6b-a8bf-af445a8faa15n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/cd1bf94e-7fb5-4e6b-a8bf-af445a8faa15n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Something seems to be killing your process (SIGKILL 9). Is the workflow
exceeding local computer resources or are you exceeding allocated
resources in a shared computing environment (e.g. requesting not enough
ressources on a HPC)?
Best,
Johannes
> log: logs/base_network.log
> jobid: 7
> benchmark: benchmarks/base_network
> reason: Missing output files: networks/base.nc <http://base.nc>
> jobid: 7
> benchmark: benchmarks/base_network
> resources: tmpdir=/tmp, mem_mb=1500, mem_mib=1431
>
> INFO:__main__:Removing buses with voltages Float64Index([132.0,
> 500.0, 750.0], dtype='float64')
> INFO:__main__:TYNDP links already in the dataset (skipping):
> Italy-Montenegro, NordLink, COBRA cable, Anglo-Scottish -1,
> ALEGrO, North Sea Link, ElecLink, Estlink 1
> INFO:__main__:Adding the following TYNDP links: Biscay Gulf,
> Italy-France, IFA2, Thames Estuary Cluster (NEMO-Link), HVDC
> SuedOstLink, HVDC Line A-North, France-Alderney-Britain, Viking
> DKW-GB, Greenconnector, Hansa PowerBridge I, NorthConnect, HVDC
> SuedLink, AQUIND Interconnector, HVDC Ultranet, Gridlink,
> NeuConnect, NordBalt, Greenlink, Celtic Interconnector
> /home/hm_a_park/pypsa-eur/.snakemake/scripts/tmpsose2nxm.base_network.py:720 <http://tmpsose2nxm.base_network.py:720>: FutureWarning: Argument `closed` is deprecated in favor of `inclusive`.
>
> INFO:__main__:Removing buses with voltages Float64Index([132.0,
> 500.0, 750.0], dtype='float64')
> INFO:__main__:TYNDP links already in the dataset (skipping):
> Italy-Montenegro, NordLink, COBRA cable, Anglo-Scottish -1,
> ALEGrO, North Sea Link, ElecLink, Estlink 1
> INFO:__main__:Adding the following TYNDP links: Biscay Gulf,
> Italy-France, IFA2, Thames Estuary Cluster (NEMO-Link), HVDC
> SuedOstLink, HVDC Line A-North, France-Alderney-Britain, Viking
> DKW-GB, Greenconnector, Hansa PowerBridge I, NorthConnect, HVDC
> SuedLink, AQUIND Interconnector, HVDC Ultranet, Gridlink,
> NeuConnect, NordBalt, Greenlink, Celtic Interconnector
> n.set_snapshots(pd.date_range(freq="h", **config["snapshots"]))
> INFO:__main__:Removing 60 unconnected network components with
> less than 4 buses. In total 93 buses.
> WARNING:__main__:31 buses are not in any country or offshore
> shape, 21 have been assigned from the tag of the entsoe map, the
> rest from the next bus in terms of pathlength.
> INFO:__main__:Replacing B2B converter `6342` together with bus
> `5544` and line `14746` by an HVDC tie-line LT-PL
> INFO:pypsa.io:Exported network base.nc <http://base.nc> has
> INFO:__main__:Removing 60 unconnected network components with
> less than 4 buses. In total 93 buses.
> WARNING:__main__:31 buses are not in any country or offshore
> shape, 21 have been assigned from the tag of the entsoe map, the
> rest from the next bus in terms of pathlength.
> INFO:__main__:Replacing B2B converter `6342` together with bus
> `5544` and line `14746` by an HVDC tie-line LT-PL
> links, transformers, buses, lines
> [Sat Feb 11 20:20:20 2023]
> Finished job 7.
> 1 of 15 steps (7%) done
> Select jobs to execute...
>
> [Sat Feb 11 20:20:20 2023]
> rule build_powerplants:
> input: networks/base.nc <http://base.nc>,
> [Sat Feb 11 20:20:20 2023]
> Finished job 7.
> 1 of 15 steps (7%) done
> Select jobs to execute...
>
> [Sat Feb 11 20:20:20 2023]
> rule build_powerplants:
> data/custom_powerplants.csv
> output: resources/powerplants.csv
> log: logs/build_powerplants.log
> jobid: 21
> reason: Input files updated by another job:
> networks/base.nc <http://base.nc>
> output: resources/powerplants.csv
> log: logs/build_powerplants.log
> jobid: 21
> reason: Input files updated by another job:
> resources: tmpdir=/tmp, mem_mb=5000, mem_mib=4769
>
> INFO:pypsa.io:Imported network base.nc <http://base.nc> has
> buses, lines, links, transformers
> INFO:powerplantmatching.collection:Retrieving data from
> https://raw.githubusercontent.com/FRESNA/powerplantmatching/v0.5.5/powerplants.csv <https://raw.githubusercontent.com/FRESNA/powerplantmatching/v0.5.5/powerplants.csv>
> INFO:powerplantmatching.collection:Retrieving data from
> INFO:powerplantmatching.collection:Store data at
> /home/hm_a_park/.local/share/powerplantmatching/data/out/default/matched_data_red.csv
> WARNING:root:No powerplants known in: MK
> WARNING:root:Couldn't find close bus for 398 powerplants.
> Removing them from the powerplants list.
> [Sat Feb 11 20:20:46 2023]
> Finished job 21.
> 2 of 15 steps (13%) done
> Select jobs to execute...
>
> [Sat Feb 11 20:20:47 2023]
> rule build_bus_regions:
> input: resources/country_shapes.geojson,
> resources/offshore_shapes.geojson, networks/base.nc <http://base.nc>
> /home/hm_a_park/.local/share/powerplantmatching/data/out/default/matched_data_red.csv
> WARNING:root:No powerplants known in: MK
> WARNING:root:Couldn't find close bus for 398 powerplants.
> Removing them from the powerplants list.
> [Sat Feb 11 20:20:46 2023]
> Finished job 21.
> 2 of 15 steps (13%) done
> Select jobs to execute...
>
> [Sat Feb 11 20:20:47 2023]
> rule build_bus_regions:
> input: resources/country_shapes.geojson,
> output: resources/regions_onshore.geojson,
> resources/regions_offshore.geojson
> log: logs/build_bus_regions.log
> jobid: 11
> reason: Input files updated by another job:
> networks/base.nc <http://base.nc>
> resources/regions_offshore.geojson
> log: logs/build_bus_regions.log
> jobid: 11
> reason: Input files updated by another job:
> resources: tmpdir=/tmp, mem_mb=1000, mem_mib=954
>
> INFO:pypsa.io:Imported network base.nc <http://base.nc> has
> data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff,
> resources/country_shapes.geojson,
> resources/offshore_shapes.geojson,
> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
> <http://europe-2013-sarah.nc>
> resources/country_shapes.geojson,
> resources/offshore_shapes.geojson,
> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
> output: resources/profile_solar.nc <http://profile_solar.nc>
> log: logs/build_renewable_profile_solar.log
> jobid: 18
> benchmark: benchmarks/build_renewable_profiles_solar
> reason: Missing output files: resources/profile_solar.nc
> <http://profile_solar.nc>; Input files updated by another job:
> jobid: 18
> benchmark: benchmarks/build_renewable_profiles_solar
> reason: Missing output files: resources/profile_solar.nc
> networks/base.nc <http://base.nc>, resources/regions_onshore.geojson
> wildcards: technology=solar
> resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
>
> INFO:distributed.scheduler:State start
> INFO:distributed.scheduler: Scheduler at:
> tcp://127.0.0.1:46175 <http://127.0.0.1:46175>
> resources: tmpdir=/tmp, mem_mb=20000, mem_mib=19074
>
> INFO:distributed.scheduler:State start
> INFO:distributed.scheduler: Scheduler at:
> INFO:distributed.scheduler: dashboard at: 127.0.0.1:8787
> <http://127.0.0.1:8787>
> INFO:distributed.nanny: Start Nanny at:
> 'tcp://127.0.0.1:34141 <http://127.0.0.1:34141>'
> 2023-02-11 20:21:02,070 - distributed.diskutils - INFO - Found
> stale lock file and directory
> '/tmp/dask-worker-space/worker-4cv85fec', purging
> INFO:distributed.scheduler:Register worker <WorkerState
> 'tcp://127.0.0.1:33645 <http://127.0.0.1:33645>', name: 0,
> stale lock file and directory
> '/tmp/dask-worker-space/worker-4cv85fec', purging
> INFO:distributed.scheduler:Register worker <WorkerState
> status: init, memory: 0, processing: 0>
> INFO:distributed.scheduler:Starting worker compute stream,
> tcp://127.0.0.1:33645 <http://127.0.0.1:33645>
> INFO:distributed.scheduler:Starting worker compute stream,
> INFO:distributed.core:Starting established connection to
> tcp://127.0.0.1:55480 <http://127.0.0.1:55480>
> INFO:distributed.scheduler:Receive client connection:
> Client-9a95f67b-aa49-11ed-a9dd-42010a800002
> INFO:distributed.core:Starting established connection to
> tcp://127.0.0.1:55494 <http://127.0.0.1:55494>
> Client-9a95f67b-aa49-11ed-a9dd-42010a800002
> INFO:distributed.core:Starting established connection to
> INFO:__main__:Calculate landuse availabilities...
> INFO:__main__:Completed availability calculation (466.07s)
> [Sat Feb 11 20:29:01 2023]
> Error in rule build_renewable_profiles:
> jobid: 18
> input: networks/base.nc <http://base.nc>,
> INFO:__main__:Completed availability calculation (466.07s)
> [Sat Feb 11 20:29:01 2023]
> Error in rule build_renewable_profiles:
> jobid: 18
> data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff,
> resources/country_shapes.geojson,
> resources/offshore_shapes.geojson,
> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
> <http://europe-2013-sarah.nc>
> resources/country_shapes.geojson,
> resources/offshore_shapes.geojson,
> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
> output: resources/profile_solar.nc <http://profile_solar.nc>
> log: logs/build_renewable_profile_solar.log (check log
> file(s) for error details)
>
> Shutting down, this might take some time.
> Exiting because a job execution failed. Look above for error message
> /home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
> warnings.warn('resource_tracker: There appear to be %d '
> Complete log: .snakemake/log/2023-02-11T202003.807215.snakemake.log
>
>
> --
> file(s) for error details)
>
> Shutting down, this might take some time.
> Exiting because a job execution failed. Look above for error message
> /home/hm_a_park/mambaforge/envs/pypsa-eur/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 6 leaked semaphore objects to clean up at shutdown
> warnings.warn('resource_tracker: There appear to be %d '
> Complete log: .snakemake/log/2023-02-11T202003.807215.snakemake.log
>
>
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/cd1bf94e-7fb5-4e6b-a8bf-af445a8faa15n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/cd1bf94e-7fb5-4e6b-a8bf-af445a8faa15n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Reply all
Reply to author
Forward
0 new messages