Error in rule build_renewable_profiles solar
Samira
snakemake -j1 resources/profile_solar.nc
Which caused the following error (whereas for profile_onwind.nc, profile_offwind-dc.nc and profile_offwind-ac.nc it worked without any error):
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 0
benchmark: benchmarks/build_renewable_profiles_solar
wildcards: technology=solar
resources: tmpdir=/var/folders/gn/gbwc08zs72gfh9kj_v4jddt40000gn/T, mem=20000
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job stats:
job count min threads max threads
------------------------ ------- ------------- -------------
build_renewable_profiles 1 1 1
total 1 1 1
Select jobs to execute...
[Thu Sep 30 15:59:03 2021]
rule build_renewable_profiles:
input: networks/base.nc, data/bundle/corine/g250_clc06_V18_5.tif, resources/natura.tiff, resources/country_shapes.geojson, resources/offshore_shapes.geojson, resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log
jobid: 0
benchmark: benchmarks/build_renewable_profiles_solar
wildcards: technology=solar
resources: tmpdir=/var/folders/gn/gbwc08zs72gfh9kj_v4jddt40000gn/T, mem=20000
INFO:__main__:correction_factor is set as 0.854337
Traceback (most recent call last):
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
file = self._cache[self._key]
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/cutouts/europe-2013-sarah.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/.snakemake/scripts/tmpkd2mx7bb.build_renewable_profiles.py", line 224, in <module>
cutout = atlite.Cutout(paths['cutout'])
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/atlite/cutout.py", line 176, in __init__
data = xr.open_dataset(str(path), chunks=chunks)
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/api.py", line 497, in open_dataset
backend_ds = backend.open_dataset(
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 551, in open_dataset
store = NetCDF4DataStore.open(
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 380, in open
return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 328, in __init__
self.format = self.ds.data_model
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 389, in ds
return self._acquire()
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 383, in _acquire
with self._manager.acquire_context(needs_lock) as root:
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/contextlib.py", line 113, in __enter__
return next(self.gen)
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
file, cached = self._acquire_with_cache_info(needs_lock)
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
file = self._opener(*self._args, **kwargs)
File "src/netCDF4/_netCDF4.pyx", line 2330, in netCDF4._netCDF4.Dataset.__init__
File "src/netCDF4/_netCDF4.pyx", line 1948, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/cutouts/europe-2013-sarah.nc'
[Thu Sep 30 15:59:10 2021]
Error in rule build_renewable_profiles:
jobid: 0
output: resources/profile_solar.nc
log: logs/build_renewable_profile_solar.log (check log file(s) for error message)
RuleException:
CalledProcessError in line 197 of /Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/Snakefile:
Command 'set -euo pipefail; /Users/samira/anaconda3/envs/pypsa-eur/bin/python3.8 '/Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/.snakemake/scripts/tmpkd2mx7bb.build_renewable_profiles.py'' returned non-zero exit status 1.
File "/Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/Snakefile", line 197, in __rule_build_renewable_profiles
File "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /Users/samira/OneDrive - Politecnico di Milano/MASTERTHESIS/PyPSA software/pypsa-eur/.snakemake/log/2021-09-30T155857.586506.snakemake.log
__________________________________
I am using a mac with Big Sur 11.6, python 4.10.3 and i cloned the git repository on Fri Aug 27 13:12:47 2021 +0200
I am happy about any hints on how to solve the problem! I already read some feeds in this list about the build_renewable_profiles rule, but they seem to be always a bit different to my problem.
Thanks in advance!
Best,
Samira
Fabian Hofmann
Hey Samira,
can you delete the sarah cutout (cutouts/europe-2013-sarah.nc)
and try runnning it again? It is just a guess but it might be that
your netcdf file is corrupted.
Best
Fabian H
--
You received this message because you are subscribed to the Google Groups "pypsa" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/be11ca84-34e4-4601-afbb-ec0d79742d8fn%40googlegroups.com.
Samira
Hi Fabian,
Fabian Hofmann
mmh, yes you can delete the old folder europe-2013-sarah. But
it's unlikely that this is the cause of the error...
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/96007f0c-4c35-4e1e-b51b-0f8fb5493384n%40googlegroups.com.
Johannes Hampp
Wild guess:
1. Delete the file cutouts/europe-2013-sarah.nc again, make sure is also
deleted from your OneDrive
2. Then rerun the rule to download the sarah cutout:
snakemake -j1 cutouts/europe-2013-sarah.nc
3. Wait for OneDrive to fully sync the file (or disable OneDrive syncing)
4. Try running again:
snakemake -j1 resources/profile_solar.nc
From the error message I'd also speculate that the file get's corrupted,
maybe through the background file syncing from OneDrive?
Best,
Johannes
> mmh, yes you can delete the old folder europe-2013-sarah. But it's
> unlikely that this is the cause of the error...
>
> On 30.09.21 17:03, Samira wrote:
>>
>> Hi Fabian,
>>
>> I tried that. Unfortunately it didn't change anything.
>> In my cutouts folder I have the following files. Should I also try to
>> Thanks for your help!
>> Best,
>> Samira
>>
>>
>> Fabian Hofmann schrieb am Donnerstag, 30. September 2021 um 16:45:55
>> UTC+2:
>>
>> Hey Samira,
>>
>>
>> can you delete the sarah cutout (cutouts/europe-2013-sarah.nc
>>
>>
>> Best
>>
>> Fabian H
>>
>>
>> On 30.09.21 16:41, Samira wrote:
>>> Hello,
>>> My name is Samira and I am writing my Master thesis at
>>> Politecnico di Milano about energy modeling also using PyPSA-Eur.
>>>
>>> While trying to run the one node per country setup, I have an
>>> error during the build_renewable_profiles of solar. So I ran only
>>> this:
>>>
>>> <http://profile_onwind.nc>, profile_offwind-dc.nc
>>> <http://profile_offwind-dc.nc> and profile_offwind-ac.nc
>>> <http://profile_offwind-ac.nc> it worked without any error):
>>>
>>> _________________________________________________________________________
>>>
>>> rule build_renewable_profiles:
>>>
>>> input: networks/base.nc <http://base.nc>,
>>> resources/country_shapes.geojson,
>>> resources/offshore_shapes.geojson,
>>> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
>>>
>>> output: resources/profile_solar.nc <http://profile_solar.nc>
>>> log: logs/build_renewable_profile_solar.log
>>>
>>> jobid: 0
>>>
>>> benchmark: benchmarks/build_renewable_profiles_solar
>>>
>>> wildcards: technology=solar
>>>
>>> resources:
>>> tmpdir=/var/folders/gn/gbwc08zs72gfh9kj_v4jddt40000gn/T, mem=20000
>>>
>>>
>>> Building DAG of jobs...
>>>
>>> Using shell: /bin/bash
>>>
>>> Provided cores: 1 (use --cores to define parallelism)
>>>
>>> Rules claiming more threads will be scaled down.
>>>
>>> Job stats:
>>>
>>> job count min threads max threads
>>>
>>> ------------------------ ------- ------------- -------------
>>>
>>> build_renewable_profiles 1 1 1
>>>
>>> total 1 1 1
>>>
>>>
>>> Select jobs to execute...
>>>
>>>
>>> [Thu Sep 30 15:59:03 2021]
>>>
>>> rule build_renewable_profiles:
>>>
>>> resources/country_shapes.geojson,
>>> resources/offshore_shapes.geojson,
>>> resources/regions_onshore.geojson, cutouts/europe-2013-sarah.nc
>>>
>>> output: resources/profile_solar.nc <http://profile_solar.nc>
>>> log: logs/build_renewable_profile_solar.log
>>>
>>> jobid: 0
>>>
>>> benchmark: benchmarks/build_renewable_profiles_solar
>>>
>>> wildcards: technology=solar
>>>
>>> resources:
>>> tmpdir=/var/folders/gn/gbwc08zs72gfh9kj_v4jddt40000gn/T, mem=20000
>>>
>>> INFO:__main__:correction_factor is set as 0.854337
>>>
>>> Traceback (most recent call last):
>>>
>>> File
>>> "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/file_manager.py",
>>> line 199, in _acquire_with_cache_info
>>>
>>> file = self._cache[self._key]
>>>
>>> File
>>> "/Users/samira/anaconda3/envs/pypsa-eur/lib/python3.8/site-packages/xarray/backends/lru_cache.py",
>>> line 53, in __getitem__
>>>
>>> value = self._cache[key]
>>>
>>> KeyError: [<class 'netCDF4._netCDF4.Dataset'>,
>>> ('/Users/samira/OneDrive - Politecnico di
>>> Milano/MASTERTHESIS/PyPSA
>>> software/pypsa-eur/cutouts/europe-2013-sarah.nc
>>>
>>>
>>> During handling of the above exception, another exception occurred:
>>>
>>>
>>> Traceback (most recent call last):
>>>
>>> File "/Users/samira/OneDrive - Politecnico di
>>> Milano/MASTERTHESIS/PyPSA
>>> software/pypsa-eur/.snakemake/scripts/tmpkd2mx7bb.build_renewable_profiles.py
>>> [Thu Sep 30 15:59:10 2021]
>>>
>>> Error in rule build_renewable_profiles:
>>>
>>> jobid: 0
>>>
>>> log: logs/build_renewable_profile_solar.log (check log
>>> file(s) for error message)
>>>
>>>
>>> RuleException:
>>>
>>> CalledProcessError in line 197 of /Users/samira/OneDrive -
>>> Politecnico di Milano/MASTERTHESIS/PyPSA
>>> software/pypsa-eur/Snakefile:
>>>
>>> Command 'set -euo pipefail;
>>> /Users/samira/anaconda3/envs/pypsa-eur/bin/python3.8
>>> '/Users/samira/OneDrive - Politecnico di
>>> Milano/MASTERTHESIS/PyPSA
>>> software/pypsa-eur/.snakemake/scripts/tmpkd2mx7bb.build_renewable_profiles.py
>> --
>> You received this message because you are subscribed to the Google
>> Groups "pypsa" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to pypsa+un...@googlegroups.com
>> <https://groups.google.com/d/msgid/pypsa/96007f0c-4c35-4e1e-b51b-0f8fb5493384n%40googlegroups.com?utm_medium=email&utm_source=footer>.
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/f7115389-8911-7e6a-5f07-dee17a1de7a0%40gmail.com?utm_medium=email&utm_source=footer>.
Samira Vogt
thanks for your answer. I tried what you suggested, at first disabling
the Onedrive syncronasation and second waiting for the syncronisation to
be finished, to try to run again. The same error as before occured.
Best,
Samira
Am 01.10.21 um 09:55 schrieb Johannes Hampp:
