build_cutout sarah data

[Uwe]

Dear community,

I am trying to get cutouts for several the period of several other years using the CDS API and the build_cutout method. This works just fine for the ERA5 data, but throws an error for the sarah data, which is used for PV. Did anyone get the same error and knows a solution for the problem?

Here is the error:

rule build_cutout:
    output: cutouts/europe-2013-sarah
    log: logs/build_cutout/europe-2013-sarah.log
    jobid: 0
    benchmark: benchmarks/build_cutout_europe-2013-sarah
    wildcards: cutout=europe-2013-sarah
    resources: mem=4000

INFO:atlite.cutout:Cutout europe-2013-sarah not found in directory cutouts, building new one
Traceback (most recent call last):
  File "/home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/.snakemake/scripts/tmpk7iz0o21.build_cutout.py", line 120, in <module>
    **cutout_params)
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/cutout.py", line 88, in __init__
    self.meta = self.get_meta(**cutoutparams)
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/preparation.py", line 171, in cutout_get_meta
    ds = prepare_func(xs=xs, ys=ys, year=years.stop, month=months.stop, **meta_kwds)
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 73, in prepare_meta_sarah
    for t in (template_sis, template_sid)]
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/datasets/sarah.py", line 73, in <listcomp>
    for t in (template_sis, template_sid)]
StopIteration
[Thu Aug  6 13:35:35 2020]
Error in rule build_cutout:
    jobid: 0
    output: cutouts/europe-2013-sarah
    log: logs/build_cutout/europe-2013-sarah.log (check log file(s) for error message)

RuleException:
CalledProcessError in line 142 of /home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/Snakefile:
Command 'set -euo pipefail;  /home/uwe/anaconda3/envs/pypsa-eur/bin/python3.7 /home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/.snakemake/scripts/tmpk7iz0o21.build_cutout.py' returned non-zero exit status 1.
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2135, in run_wrapper
  File "/home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/Snakefile", line 142, in __rule_build_cutout
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 529, in _callback
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
  File "/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2201, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/.snakemake/log/2020-08-06T133533.955909.snakemake.log

[Johannes Hampp]

Dear Uwe,

It looks like you did not download the raw SARAH data?
The rule `build_cutout` uses the module `atlite`.
`atlite` can automatically download data for ERA5 but not for SARAH.

You need to manually download the raw SARAH data for the years you are
interested in. After placing them in the appropriate location (path
configured in "config.py" in your atlite installation directory with
subfolder `SID/` and `SIS/`) you should be able to create the cutout(s).

Link describing the SARAH data to download and folder structure
(the rest, especially the configuration, is *not* applicable):
https://atlite.readthedocs.io/en/latest/examples/create_cutout_SARAH.html

I.e.:

1. Download raw data.
2. Extract into SID/ and SIS/ folders.
3. Configure `sarah_dir` in `config.py` in directory of `atlite.__path__`.
4. Try again.

HTH.

Best,
Johannes

Am 06/08/2020 um 14:39 schrieb Uwe:
> Dear community,
>
> I am trying to get cutouts for several the period of several other years
> using the CDS API and the build_cutout method. This works just fine for
> the ERA5 data, but throws an error for the sarah data, which is used for
> PV. Did anyone get the same error and knows a solution for the problem?
>
> Here is the error:
>
> |
> rule build_cutout:
>     output:cutouts/europe-2013-sarah
>     log:logs/build_cutout/europe-2013-sarah.log
>     jobid:0
>     benchmark:benchmarks/build_cutout_europe-2013-sarah
>     wildcards:cutout=europe-2013-sarah
>     resources:mem=4000
>
> INFO:atlite.cutout:Cutouteurope-2013-sarah notfound indirectory
> cutouts,building newone

> Traceback(most recent call last):

>  
> File"/home/uwe/PyPSA/pypsa-eur-0171/pypsa-eur/.snakemake/scripts/tmpk7iz0o21.build_cutout.py",line
> 120,in<module>
>     **cutout_params)
>  
> File"/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/cutout.py",line
> 88,in__init__
>     self.meta =self.get_meta(**cutoutparams)
>  
> File"/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/preparation.py",line
> 171,incutout_get_meta
>     ds
> =prepare_func(xs=xs,ys=ys,year=years.stop,month=months.stop,**meta_kwds)
>  
> File"/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/datasets/sarah.py",line
> 73,inprepare_meta_sarah
>     fort in(template_sis,template_sid)]
>  
> File"/home/uwe/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/datasets/sarah.py",line
> 73,in<listcomp>
>     fort in(template_sis,template_sid)]
> StopIteration
> [ThuAug 613:35:352020]

> Errorinrule build_cutout:

> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/56a88098-f9d2-4260-b747-61d6e3ff4de2o%40googlegroups.com
> <https://groups.google.com/d/msgid/pypsa/56a88098-f9d2-4260-b747-61d6e3ff4de2o%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Uwe]

Hi Johannes,

thanks for the detailed question. The files will be quite large. Do you know if I need to download whole Europe or is it sufficient if I only download the data for the considered region? E.g. just downloading the data for Germany if I specify Germany in the config file.

Best

Uwe

[Johannes Hampp]

Hi Uwe,

I believe that should work. Give it a try.

Be wary that the rule 'build_cutout' uses the parameters
'atlite'.'cutouts'.'europe-2013-sarah' and not the country boundaries.
So adjust 'xs' (lon) and 'ys' (lat) accordingly in the 'config.yaml'.


Best,
Johannes

Best regards,
Johannes Hampp

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

Online office hours (without appointment):
Every Thursday, 13:30-14:30 (UTC+2) at:
https://uni-giessen.webex.com/meet/johannes.hampp

mailto: johanne...@zeu.uni-giessen.de

Office 110
Senckenbergstr. 3
DE-35392 Giessen
https://www.uni-giessen.de/fbz/zentren/zeu?set_language=en

[Uwe]

Hi Johannes,

I am currently trying to create the sarah cutout but I have some troubles.

I have downloaded the sarah SIS and SID data for 2010-2017 and xs from -12 to 35, ys from 33 to 72. I built following folder structure: ~/sarah/SID and ~/sarah/SIS (alternatively I also used ~/sarah/sid and ~/sarah/sis) , where I put the downloaded and extractet .nc files, and adjusted the sarah_dir in the atlite config file to ~/sarah. I used the following code to build the cutout:

cutout = atlite.Cutout(name='Test',
                       module="sarah",
                       xs=slice(-12, 35),
                       ys=slice(72, 33),
                       years=slice(2010,2017))

But I get the following error.

---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-65-cd1b02675604> in <module>
      3                        xs=slice(-12, 35),
      4                        ys=slice(72, 33),
----> 5                        years=slice(2010,2017))

~/anaconda3/lib/python3.7/site-packages/atlite/cutout.py in __init__(self, name, cutout_dir, **cutoutparams)
     86             if {"xs", "ys", "years"}.difference(cutoutparams):
     87                 raise TypeError("Arguments `xs`, `ys` and `years` need to be specified")
---> 88             self.meta = self.get_meta(**cutoutparams)
     89
     90     def datasetfn(self, *args):

~/anaconda3/lib/python3.7/site-packages/atlite/preparation.py in cutout_get_meta(cutout, xs, ys, years, months, **dataset_params)
    169
    170     prepare_func = meta_kwds.pop('prepare_func')
--> 171     ds = prepare_func(xs=xs, ys=ys, year=years.stop, month=months.stop, **meta_kwds)
    172     ds.attrs.update(dataset_params)
    173

~/anaconda3/lib/python3.7/site-packages/atlite/datasets/sarah.py in prepare_meta_sarah(xs, ys, year, month, template_sis, template_sid, module, resolution)
     68 def prepare_meta_sarah(xs, ys, year, month, template_sis, template_sid, module, resolution=resolution):
     69
---> 70     fns = [next(glob.iglob(t.format(year=year, month=month)))
     71            for t in (template_sis, template_sid)]
     72

~/anaconda3/lib/python3.7/site-packages/atlite/datasets/sarah.py in <listcomp>(.0)
     68 def prepare_meta_sarah(xs, ys, year, month, template_sis, template_sid, module, resolution=resolution):
     69
---> 70     fns = [next(glob.iglob(t.format(year=year, month=month)))
     71            for t in (template_sis, template_sid)]
     72

StopIteration:

Do you have an idea why I get this error and how to solve the problem?

Best

Uwe

[Jonas Hörsch]

Hi Uwe,

The error is telling you, that it doesn’t find any files for one year, month combination.

You can use the ipython debugger by running the command in a Jupyter notebook or in the ipython console and then use `%debug` after you encountered the stop iteration, which will allow you to browse the stack with the debugger commands `up` and `down` and investigate individual variable names.

by looking up the actual value of `t`, you will most likely realise that python does not expand the `~` in the path name and you will have to write it as `/home/…/sarah/` instead.

Best,

--
You received this message because you are subscribed to the Google Groups "pypsa" group.

To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/1f67f03e-c9a0-42ed-b090-1e923f8202d5o%40googlegroups.com.

[Uwe]

Hi Jonas and Johannes,

thank you very much for the responses. I tried it today again and it worked even without changing anything. I am not sure what the problem was in the end. But thanks a lot for the help and suggestions. Really appreciate it!

As I had a bit an issue with the building of the sarah cutout, here a brief overview of the necessary steps. The documentation is a bit outdated so this might help others. Feel free to comment if something is wrong/missing.

Important: This is an approach independet from pypsa-eur (e.g., using a jupyter notebook instead of the build_cutouts method). If pypsa-eur's build_cutouts is prefered, please adjust atlite directory to pypsa-eur envs (I did not test it) -> ~/anaconda3/envs/pypsa-eur/lib/python3.7/site-packages/atlite/config.py.

• Download the sarah files
1. From: https://wui.cmsaf.eu/safira/action/viewDoiDetails?acronym=SARAH_V002_01
2. Surface incoming direct radiation (SID)
3. Surface incoming shortwave radiation (SIS)
4. Adjust time and coordinates
• Build the directory structure. E.g.,
1. ~/sarah
   
2. ~/sarah/SID
3. ~/sarah/SIS
• Extract the data
1. SID into ~/sarah/SID
2. SIS into ~/sarah/SIS
• Change directory in atlite package. Important: directories can differ
  
1. Here: ~/anaconda3/lib/python3.7/site-packages/atlite/config.py
2. Change: sarah_dir = '~/sarah'
• Set up cdsapi
1. More information here: https://cds.climate.copernicus.eu/api-how-to
   
• Use jupyter notebook to build cutout. Following code was used successfully.
1. Please adjust respectively
   

import atlite
import cdsapi

import logging
logging.basicConfig(level=logging.INFO)

cutout = atlite.Cutout(name='Test',
                       module="sarah",
                       xs=slice(-12, 35),

                       ys=slice(72, 33), #Imporant: slice(North, South)
                       years=slice(2010,2017))

cutout.prepare()

I hope this helps someone.

Best

Uwe

[Jonas Hörsch]

Thanks for the detailed description!

On 3. Sep 2020, at 14:58, Uwe <u.lang...@gmx.de> wrote:

• Change directory in atlite package. Important: directories can differ
  

1. Here: ~/anaconda3/lib/python3.7/site-packages/atlite/config.py
2. Change: sarah_dir = '~/sarah’

To find the correct path of execute in the python environment in which you installed alite:

```python

import atlite

print(atlite.__file__)

```

In Uwe’s case this would have printed:

/home/uwe/anaconda3/lib/python3.7/site-packages/atlite/__init__.py

And config.py lives right next to that.