Atlite - Grouping ERA5 cut-out by times

59 views
Skip to first unread message

Matthew Smith

unread,
Mar 2, 2023, 7:28:50 AM3/2/23
to pypsa
Hello,

Thanks again to this great community. 

I have a question about Atlite. The application of this would be to create season-specific or time specific analysis.

Is it possible to average specific times throughout the year? For example, if I'd like to create a generation map which shows the average annual generation only for the hours of 6pm to 6am, would it be possible to group the times in that way?

Related but perhaps different, are we able to download a cut-out for only those hours across all days of the year or multiple years?

Best regards,
Matt

Fabian Hofmann

unread,
Mar 2, 2023, 7:36:58 AM3/2/23
to py...@googlegroups.com

Hey Matt,


since atlite is mainly using slices, it is not perfectly designed for such selective time-series during the cutout creation. I would suggest to create the cutout for the whole year (or multiple) and select the relevant times after the calculation of the power generation. With pandas/xarray you can do powerful selection, groupby or resample operations (which to explain would be out of the scope here). Have a look at the example here https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html for getting started.


Hope that helps for the start.


Best

Fabian Hofmann

--
You received this message because you are subscribed to the Google Groups "pypsa" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com.
-- 
Fabian Hofmann 

Postdoctoral Researcher
Institute of Energy Technology
Technische Universität Berlin
http://fabianhofmann.org/

Group website: https://tub-ensys.github.io/

Matthew Smith

unread,
Mar 5, 2023, 11:55:49 PM3/5/23
to pypsa
Thanks Fabian. I'll come back to the group with results once I make some progress.

Matthew Smith

unread,
Mar 15, 2023, 2:30:12 AM3/15/23
to pypsa
Just as an update - I used Bing's chatGPT ai bot to help me come up with the following code. This should produce a country-wide plot showing the ratio of night time and daytime capacity factors. Potentially useful for identifying areas for wind which may have anti-correlation with Solar PV generation.

Note I haven't tested this - waiting for my cut-out to download! Will come back with corrections.

# Calculate day / night average and plot the day/night ratio

# Select only hours between 18:00 and 05:00
cap_factors_night = cap_factors.sel(time=cap_factors.time.dt.hour.isin(range(18,24)+range(0,6)))

# Calculate average capacity factor over time dimension
cap_factors_night_avg = cap_factors_night.mean(dim='time')

# Select only hours between 6:00 and 17:00
cap_factors_day = cap_factors.sel(time=cap_factors.time.dt.hour.isin(range(6,18)))

# Calculate average capacity factor over time dimension
cap_factors_day_avg = cap_factors_day.mean(dim='time')

# Calculate the ratio of night to day average capacity factors
cap_factors_ratio = cap_factors_night_avg / cap_factors_day_avg

# Plot the map
fig, ax = plt.subplots(subplot_kw={'projection': projection}, figsize=(9, 7))
cap_factors_ratio.name = 'Capacity Factor Ratio night/day'
cap_factors_ratio.plot(ax=ax, transform=plate(), alpha=0.8)
cells.plot(ax=ax, **plot_grid_dict)
#ax.outline_patch.set_edgecolor('white')
fig.tight_layout();

Matthew Smith

unread,
Mar 15, 2023, 5:38:26 AM3/15/23
to pypsa
Okay! So the previous code doesn't work, on testing. That's because the capacity factor object, generated by cap_factors = cutout.wind(turbine=turbine_config, capacity_factor=True), doesn't have a time element in it any more. It is the output of the average capacity factor.

I've managed to work out how to use .sel() for the raw windspeed data, but I don't know how to use the filtered wind speed data as an input into the capacity factor calculation. As cutout.data() is not the same as cutout.wind(). Does anyone know how I could do this?

Below is my code for producing maps of night / day ratio of wind speed at 100m, in case it helps anyone:

#### WIND SPEED 100m night / day ratio

# Select nighttime hours in the cutout data
night_hours = cutout.data.sel(time=cutout.data["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))
# Calculate the average wind speed for the night hours for all grid cells
night_ws = night_hours.wnd100m.mean('time')

# Select daytime hours in the cutout data
day_hours = cutout.data.sel(time=cutout.data["time"].dt.hour.isin(range(6,18)))
# Calculate the average wind speed for the night hours for all grid cells
day_ws = day_hours.wnd100m.mean('time')

# Calculate the ratio of night to day average wind speeds
ws_ratio = night_ws / day_ws


fig, ax = plt.subplots(subplot_kw={'projection': projection}, figsize=(9, 7))
ws_ratio.name = 'Wind Speed day night ratio'
ws_ratio.plot(ax=ax, transform=plate(), alpha=1)
cells.plot(ax=ax, **plot_grid_dict)
fig.tight_layout();



Johannes Hampp

unread,
Mar 17, 2023, 4:47:57 AM3/17/23
to Matthew Smith, pypsa
Hi Mathew,

Good work so far!

You can call the method which we also use internally to calculate
cutout.wind(..) and pass that method the reduced windspeed data.

The method is:

atlite.convert.convert_wind(ds, turbine)

https://github.com/PyPSA/atlite/blob/17c81f9bee46752a89e31d5c28dc9e0b5fb107b9/atlite/convert.py#L462

IIRC we did implement the .sel(...) method for the cutout itself, so you
could try producing "sub-cutouts" based on your original cutout and then
calling the .wind(...) method on those. The syntax should be the same.

HTH, Johannes


Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu
> cap_factors_ratio.name <http://cap_factors_ratio.name> = 'Capacity
> Factor Ratio night/day'
> cap_factors_ratio.plot(ax=ax, transform=plate(), alpha=0.8)
> cells.plot(ax=ax, **plot_grid_dict)
> #ax.outline_patch.set_edgecolor('white')
> fig.tight_layout();
>
> On Monday, 6 March 2023 at 10:25:49 UTC+5:30 Matthew Smith wrote:
>
> Thanks Fabian. I'll come back to the group with results once I
> make some progress.
>
> On Thursday, 2 March 2023 at 18:06:58 UTC+5:30 Fabian Hofmann wrote:
>
> Hey Matt,
>
>
> since atlite is mainly using slices, it is not perfectly
> designed for such selective time-series during the cutout
> creation. I would suggest to create the cutout for the whole
> year (or multiple) and select the relevant times after the
> calculation of the power generation. With pandas/xarray you
> can do powerful selection, groupby or resample operations
> (which to explain would be out of the scope here). Have a
> look at the example here
> https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html> for getting started.
>
>
> Hope that helps for the start.
>
>
> Best
>
> Fabian Hofmann
>
>
> On 02.03.23 13:28, Matthew Smith wrote:
>> Hello,
>>
>> Thanks again to this great community.
>>
>> I have a question about Atlite. The application of this
>> would be to create season-specific or time specific analysis.
>>
>> Is it possible to average specific times throughout the
>> year? For example, if I'd like to create a generation map
>> which shows the average annual generation only for the
>> hours of 6pm to 6am, would it be possible to group the
>> times in that way?
>>
>> Related but perhaps different, are we able to download a
>> cut-out for only those hours across all days of the year
>> or multiple years?
>>
>> Best regards,
>> Matt
>> --
>> You received this message because you are subscribed to
>> the Google Groups "pypsa" group.
>> To unsubscribe from this group and stop receiving emails
>> from it, send an email to pypsa+un...@googlegroups.com.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> Fabian Hofmann
>
> Postdoctoral Researcher
> Institute of Energy Technology
> Technische Universität Berlin
> http://fabianhofmann.org/ <http://fabianhofmann.org/>
>
> Group website:https://tub-ensys.github.io/ <https://tub-ensys.github.io/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Matthew Smith

unread,
Mar 21, 2023, 9:29:01 AM3/21/23
to pypsa
Hi  Johannes,

Thanks for the help!

I thought it would be best to apply the .sel() method to the cutout itself. However I wasn't able to. This line:
night_hours = cutout.sel(time=cutout["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))
Gave the error "TypeError: 'Cutout' object is not subscriptable" which I wasn't able to fix. Do you know what I did wrong?

I then tried your first suggestion:

#### CUF night / day ratio
WindCO = cutout.prepare(features='wind')

# Select nighttime hours in the cutout data
night_hours = WindCO.data.sel(time=cutout.data["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))

# Calculate the average wind speed for the night hours for all grid cells
night_CUF = atlite.convert.convert_wind(night_hours,turbine=turbine_config).mean('time')

This worked well. Though I had a strange occurance where atlite started downloading additional features like 2m_temperature, soil_temperatuvre_level_4, runnoff, which I hadn't downloaded for my cutout. They were downloaded as temporary files too: INFO:atlite.data:Storing temporary files in C:\Users\MATTHE~1\AppData\Local\Temp\tmp8d68xa_7
It didn't fully download and so my code executed as expected, eventually.

Thanks again for the help. 

Best regards,
Matt

Johannes Hampp

unread,
Mar 21, 2023, 11:29:08 AM3/21/23
to pypsa
Hi Mathew,


Close enough :)

Instead of cutout["time"] it should be cutout.coords["time"]:

night_hours =
cutout.sel(time=cutout.coords["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))

Depending on how you are developing, your IDE should've shown you the
cause of the problem. E.g. Jupyter Lab highlights the problematic part
of the code.


Nice to hear the alternative approach also worked. Strange downloads
shouldn't happen. If you observe the mysterious downloads again make
sure to open a bug report on GitHub.

Best,
Johannes

Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

Am 21/03/2023 um 14:29 schrieb Matthew Smith:
> Hi Johannes,
>
> Thanks for the help!
>
> I thought it would be best to apply the .sel() method to the cutout
> itself. However I wasn't able to. This line:
> /night_hours =
> cutout.sel(time=cutout["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))/
> Gave the error "TypeError/: 'Cutout' object is not subscriptable"/ which
> https://github.com/PyPSA/atlite/blob/17c81f9bee46752a89e31d5c28dc9e0b5fb107b9/atlite/convert.py#L462 <https://github.com/PyPSA/atlite/blob/17c81f9bee46752a89e31d5c28dc9e0b5fb107b9/atlite/convert.py#L462>
>
> IIRC we did implement the .sel(...) method for the cutout itself, so
> you
> could try producing "sub-cutouts" based on your original cutout and
> then
> calling the .wind(...) method on those. The syntax should be the same.
>
> HTH, Johannes
>
>
> Best regards,
> Johannes Hampp (he/him)
>
> Justus Liebig University Giessen (JLU)
> Center for international Development and Environmental Research (ZEU)
>
> mailto: johanne...@zeu.uni-giessen.de
>
> Senckenbergstr. 3
> DE-35392 Giessen
> https://uni-giessen.de/zeu <https://uni-giessen.de/zeu>
> > ws_ratio.name <http://ws_ratio.name> = 'Wind Speed day night ratio'
> <http://cap_factors_ratio.name <http://cap_factors_ratio.name>> =
> 'Capacity
> > Factor Ratio night/day'
> > cap_factors_ratio.plot(ax=ax, transform=plate(), alpha=0.8)
> > cells.plot(ax=ax, **plot_grid_dict)
> > #ax.outline_patch.set_edgecolor('white')
> > fig.tight_layout();
> >
> > On Monday, 6 March 2023 at 10:25:49 UTC+5:30 Matthew Smith wrote:
> >
> > Thanks Fabian. I'll come back to the group with results once I
> > make some progress.
> >
> > On Thursday, 2 March 2023 at 18:06:58 UTC+5:30 Fabian Hofmann wrote:
> >
> > Hey Matt,
> >
> >
> > since atlite is mainly using slices, it is not perfectly
> > designed for such selective time-series during the cutout
> > creation. I would suggest to create the cutout for the whole
> > year (or multiple) and select the relevant times after the
> > calculation of the power generation. With pandas/xarray you
> > can do powerful selection, groupby or resample operations
> > (which to explain would be out of the scope here). Have a
> > look at the example here
> >
> https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html> <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html <https://docs.xarray.dev/en/stable/generated/xarray.DataArray.groupby.html>> for getting started.
> >
> >
> > Hope that helps for the start.
> >
> >
> > Best
> >
> > Fabian Hofmann
> >
> >
> > On 02.03.23 13:28, Matthew Smith wrote:
> >> Hello,
> >>
> >> Thanks again to this great community.
> >>
> >> I have a question about Atlite. The application of this
> >> would be to create season-specific or time specific analysis.
> >>
> >> Is it possible to average specific times throughout the
> >> year? For example, if I'd like to create a generation map
> >> which shows the average annual generation only for the
> >> hours of 6pm to 6am, would it be possible to group the
> >> times in that way?
> >>
> >> Related but perhaps different, are we able to download a
> >> cut-out for only those hours across all days of the year
> >> or multiple years?
> >>
> >> Best regards,
> >> Matt
> >> --
> >> You received this message because you are subscribed to
> >> the Google Groups "pypsa" group.
> >> To unsubscribe from this group and stop receiving emails
> >> from it, send an email to pypsa+un...@googlegroups.com.
> >> To view this discussion on the web, visit
> >>
> https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com> <https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pypsa/355503e8-d73f-4ee3-803e-618ffe57b737n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >
> > --
> > Fabian Hofmann
> >
> > Postdoctoral Researcher
> > Institute of Energy Technology
> > Technische Universität Berlin
> > http://fabianhofmann.org/ <http://fabianhofmann.org/>
> <http://fabianhofmann.org/ <http://fabianhofmann.org/>>
> >
> > Group website:https://tub-ensys.github.io/
> <https://tub-ensys.github.io/> <https://tub-ensys.github.io/
> <https://tub-ensys.github.io/>>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "pypsa" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to pypsa+un...@googlegroups.com
> > <mailto:pypsa+un...@googlegroups.com>.
> > To view this discussion on the web, visit
> >
> https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com> <https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pypsa/85f10e65-716c-4f44-a637-8faefb06ff01n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.
> To view this discussion on the web, visit
> https://groups.google.com/d/msgid/pypsa/359b1a24-2d76-467e-8f97-a83b19111b18n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/359b1a24-2d76-467e-8f97-a83b19111b18n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Matthew Smith

unread,
Mar 23, 2023, 6:29:00 AM3/23/23
to pypsa
Dear Johannes,

Thank you very much again. I had seen the .coords() function when I first looked at .groupby(), but that didn't occur to me. I was able to make the cut-out method work:

# Select nighttime hours in the cutout data
night_hours = cutout.sel(time=cutout.coords["time"].dt.hour.isin(list(range(18,24))+list(range(0,6))))

# Calculate the average wind speed for the night hours for all grid cells
night_CUF = night_hours.wind(turbine=turbine_config, capacity_factor=True)


# Select daytime hours in the cutout data
day_hours = cutout.sel(time=cutout.coords["time"].dt.hour.isin(range(6,18)))
#Calculate the average wind speed for the night hours for all grid cells
day_CUF = day_hours.wind(turbine=turbine_config, capacity_factor=True)

                       
# Calculate the ratio of night to day average wind speeds
CUF_ratio = night_CUF / day_CUF

This method has a progress bar, and so is far superior! Haha.
Reply all
Reply to author
Forward
0 new messages