Atlite help

[Matthew Smith]

Hello,

I would like to do a few things in Atlite. I'd really appreciate it if you could point me in the right direction.

1) Add new turbine power curve: I have a turbine power curve which I can't share to any of the turbine libraries as it is confidential. I can't use a similar turbine in the library as this one is configured for India's low wind environment so there are quite significant differences in the capacity factor. Are there any code examples showing how to reference a user added power curve table?

2) Outputting data as GIS files: There's an example which shows how a gridded map of the UK showing capacity factors can be generated. Is there a way to output this data as a GIS compatible shape file? It would be great to be able to work with this data in QGIS.

3) Generating hourly time series to CSV: are there any examples showing how hourly time series for multiple points can be exported to CSV particularly?

Apologies for the basic questions. I'm a beginner at python but trying to learn. I'm conducting a site selection process starting with the whole country, with the aim of finding sites with complementary generation profiles to maximise consistency of generation across the portfolio. It seems like Atlite would be the best tool for this, at least during the first stage of the filtering process, but it is presenting a bit of a learning curve for me .

Thanks,

Matt

[Matthew Smith]

Oh, to add another question; is there a way to reference a downloaded GRIB cut-out of ERA5 data on my local machine? Instead of requesting CDS to prepare a new cut out.

Thank you,

Matt

--
You received this message because you are subscribed to a topic in the Google Groups "pypsa" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pypsa/7BpisV3Xd8E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pypsa+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com.

[Johannes Hampp]

Hi Matt,

Let's get you settled in :)

1. Yes, that is possible. The information used for converting wind data
to wind turbine CFs are stored as Python dict() in atlite.

As an example load the configuration for an Enercon turbine:

turbine_config =
atlite.resource.get_windturbineconfig(atlite.resource.windturbines.Enercon_E82_3000kW)

and have a look at the turbine_config. You can modify that config to
match your turbine's parameters and the power curve specified by the
entries in turbine_config["V"] and turbine_config["POW"].

If you call the function to determine the wind CF you simply pass your
modified turbine_config like this:

cutout.wind(turbine=turbine_config)

That's it.

Comfort option:
A very recent addition allows you to load the turbine config also
automatically from file. This code addition is not yet packaged, so you
need to install the master branch of atlite from GitHub:

pip install git+https://github.com/PyPSA/atlite.git@master

You then parametrize your turbine in a yaml file with the same
properties as you used above, see e.g. here:

https://github.com/PyPSA/atlite/blob/master/atlite/resources/windturbine/Enercon_E82_3000kW.yaml

With the atlite version from the master branch, you can then
automatically load a .yaml file if it follows the example format by calling:

from pathlib import Path

atlite.resource.get_windturbineconfig(Path("<path to your .yaml file>")


2. That's probably possible. I've never done that myself. I think you
can convert the DataArray and DataSets returned by atlite to a raster
file or tif which should be readable with *GIS using rioxarray:

First install rioxarray:
https://corteva.github.io/rioxarray/stable/installation.html

and then something like this:

import rioxarray
da = <DataSet or DataArray returned by atlite>
da.rio.to_raster(<path to save to>)

Please treat this as pointing you to a certain direction. I've never
followed that route before, so it might not be that straightforward.

Doc:
https://corteva.github.io/rioxarray/stable/rioxarray.html#rioxarray.raster_dataset.RasterDataset.to_raster


3. If you already have the time-series then I personally find it easiest
to switch to pandas and export to csv from there:

ds = <your DataSet from atlite with the timeseries>)

df = ds.to_pandas()
df.to_csv(<save path>)

Doc:
https://pandas.pydata.org/docs/getting_started/index.html#installation

https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html


4. If your local data is already an atlite cutout, then you can use it
by giving the path:

cutout = atlite.Cutout(path="<your local path>")

atlite will load it and use it.
If the data is ERA5 data downloaded but not a cutout created by atlite,
then you cannot use it and have to downloaded it via the CDS API.


HTH,

Best,
Johannes

Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

> <https://groups.google.com/d/topic/pypsa/7BpisV3Xd8E/unsubscribe>.

> To unsubscribe from this group and all its topics, send an email to
> pypsa+un...@googlegroups.com

> <mailto:pypsa+un...@googlegroups.com>.

> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "pypsa" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pypsa+un...@googlegroups.com
> <mailto:pypsa+un...@googlegroups.com>.

> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com <https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Matthew Smith]

Dear Johannes,

This is very helpful, thank you. Let me give it a good go. I'll be sure to share my results (if successful) with the group.

Best regards,

Matt 

[Matthew Smith]

Hi all,

Reporting back after successfully getting Atlite to work. The examples in the documentation were a big help, especially since I basically can’t code at all. Thanks Johannes for your support too.

 

Some key learnings, in case they are useful for future newbies:

Using custom turbines

Make sure to use numpy.array when entering the wind speed and power curve values:

turbine_config = atlite.resource.get_windturbineconfig(atlite.resource.windturbines.Enercon_E82_3000kW)

turbine_config['V'] = np.array([0, 0.25, 0.5, 0.75, 1, 30]) #Most wind speed values removed to save space

turbine_config['POW'] = np.array([0, 0, 0, 0, 0, 0])  #Note values removed for this example

turbine_config['hub_height'] = 140

turbine_config['P'] = 3.3

 

Exporting Capacity Factor grid to GIS

Rather than using Rasterio to produce a raster file directly. I just exported the values as csv, loaded it as a vector in QGIS and then converted it to a raster in that program.

Note that you need to flatten the capacity factor grid to a 1-dimensional table for it to be accepted in QGIS. This turned out to be quite simple in the end:

cap_factors = cutout.wind(turbine=turbine_config, capacity_factor=True)

df= cap_factors.to_dataframe()

df.to_csv('AllindiaCUFs.csv')

 

Having it as a Raster in QGIS let me filter minimum values, present it with different colours, etc. I’m sure you can do it in Python but this way was easier for me.

I ended up converting it back to a shape file though, so that I could join the capacity factor values to another layer I had which mapped district boundaries. Exporting this joined table to excel then allowed me to find the highest capacity factor grid square in my filtered list of districts. Which then became the source of sites to model in the next step. Quite cool!

 

Producing hourly generation time series from a csv of sites

In the ‘Plotting with Atlite’ example, the sites for which generation is calculated are generated in the code itself:

sites = gpd.GeoDataFrame([['Dist_Anantapur_Max', 77.25, 14.75, 1],['Dist_Barmer_Max', 70.75, 25.25, 1], ['Dist_Bijapur_Max', 76, 16.75, 1]], columns=['name', 'x', 'y', 'capacity']).set_index('name')

 

I wanted to import a bigger list of sites from CSV. It was pretty straightforward in the end:

df = pd.read_csv('Sites to model_Max CUFs.csv')

sites = gpd.GeoDataFrame(df)    

sites = gpd.GeoDataFrame(sites).set_index('Name')

 

Note that the code in the example broke when there were two or more sites with the same coordinates. So make sure that all the sites have unique coordinate combinations.

 

Now I’m analysing the generation time series to find the optimal combination of sites to meet a minimum load consistently. Do please let me know if you know any mathematical approaches that might be good to try.

 

Best regards,

Matt

--

Matthew Smith

[Johannes Hampp]

Hi Matt,

Thanks for reporting back and sharing your detailed description!

Regarding your last question:
Have your tried building a PyPSA model
(https://pypsa.readthedocs.io/en/latest/optimal_power_flow.html) for
that purpose? Where each site and time-series corresponds to one generator.

Best,
Johannes

Best regards,
Johannes Hampp (he/him)

Justus Liebig University Giessen (JLU)
Center for international Development and Environmental Research (ZEU)

mailto: johanne...@zeu.uni-giessen.de

Senckenbergstr. 3
DE-35392 Giessen
https://uni-giessen.de/zeu

Am 02/12/2022 um 08:30 schrieb Matthew Smith:
> Hi all,
>
> Reporting back after successfully getting Atlite to work. The examples
> in the documentation were a big help, especially since I basically can’t
> code at all. Thanks Johannes for your support too.
>
> Some key learnings, in case they are useful for future newbies:
>

> _Using custom turbines_

>
> Make sure to use numpy.array when entering the wind speed and power
> curve values:
>

> /turbine_config =
> atlite.resource.get_windturbineconfig(atlite.resource.windturbines.Enercon_E82_3000kW)/
>
> /turbine_config['V'] = np.array([0, 0.25, 0.5, 0.75, 1, 30]) #Most wind
> speed values removed to save space/
>
> /turbine_config['POW'] = np.array([0, 0, 0, 0, 0, 0])  #Note values
> removed for this example/
>
> /turbine_config['hub_height'] = 140/
>
> /turbine_config['P'] = 3.3/
>
> //
>
> _Exporting Capacity Factor grid to GIS_

>
> Rather than using Rasterio to produce a raster file directly. I just
> exported the values as csv, loaded it as a vector in QGIS and then
> converted it to a raster in that program.
>
> Note that you need to flatten the capacity factor grid to a
> 1-dimensional table for it to be accepted in QGIS. This turned out to be
> quite simple in the end:
>

> /cap_factors = cutout.wind(turbine=turbine_config, capacity_factor=True)/
>
> /df= cap_factors.to_dataframe()/
>
> /df.to_csv('AllindiaCUFs.csv')/

>
> Having it as a Raster in QGIS let me filter minimum values, present it
> with different colours, etc. I’m sure you can do it in Python but this
> way was easier for me.
>
> I ended up converting it back to a shape file though, so that I could
> join the capacity factor values to another layer I had which mapped
> district boundaries. Exporting this joined table to excel then allowed
> me to find the highest capacity factor grid square in my filtered list
> of districts. Which then became the source of sites to model in the next
> step. Quite cool!
>

> _Producing hourly generation time series from a csv of sites_

>
> In the ‘Plotting with Atlite’ example, the sites for which generation is
> calculated are generated in the code itself:
>

> /sites = gpd.GeoDataFrame([['Dist_Anantapur_Max', 77.25, 14.75,

> 1],['Dist_Barmer_Max', 70.75, 25.25, 1], ['Dist_Bijapur_Max', 76, 16.75,

> 1]], columns=['name', 'x', 'y', 'capacity']).set_index('name')/
>
> //

>
> I wanted to import a bigger list of sites from CSV. It was pretty
> straightforward in the end:
>

> /df = pd.read_csv('Sites to model_Max CUFs.csv')/
>
> /sites = gpd.GeoDataFrame(df) /
>
> /sites = gpd.GeoDataFrame(sites).set_index('Name')/

>
> Note that the code in the example broke when there were two or more
> sites with the same coordinates. So make sure that all the sites have
> unique coordinate combinations.
>
> Now I’m analysing the generation time series to find the optimal
> combination of sites to meet a minimum load consistently. Do please let
> me know if you know any mathematical approaches that might be good to try.
>
> Best regards,
>
> Matt
>
>
> On Wed, 30 Nov 2022 at 10:00, Matthew Smith <matts...@gmail.com
> <mailto:matts...@gmail.com>> wrote:
>
> Dear Johannes,
>
> This is very helpful, thank you. Let me give it a good go. I'll be
> sure to share my results (if successful) with the group.
>
> Best regards,
> Matt
>
> On Wed, 30 Nov 2022, 02:05 Johannes Hampp,
> <johanne...@zeu.uni-giessen.de

> https://github.com/PyPSA/atlite/blob/master/atlite/resources/windturbine/Enercon_E82_3000kW.yaml <https://github.com/PyPSA/atlite/blob/master/atlite/resources/windturbine/Enercon_E82_3000kW.yaml>

>
> With the atlite version from the master branch, you can then
> automatically load a .yaml file if it follows the example format
> by calling:
>
> from pathlib import Path
>
> atlite.resource.get_windturbineconfig(Path("<path to your .yaml
> file>")
>
>
> 2. That's probably possible. I've never done that myself. I
> think you
> can convert the DataArray and DataSets returned by atlite to a
> raster
> file or tif which should be readable with *GIS using rioxarray:
>
> First install rioxarray:
> https://corteva.github.io/rioxarray/stable/installation.html
> <https://corteva.github.io/rioxarray/stable/installation.html>
>
> and then something like this:
>
> import rioxarray
> da = <DataSet or DataArray returned by atlite>
> da.rio.to_raster(<path to save to>)
>
> Please treat this as pointing you to a certain direction. I've
> never
> followed that route before, so it might not be that straightforward.
>
> Doc:

> https://corteva.github.io/rioxarray/stable/rioxarray.html#rioxarray.raster_dataset.RasterDataset.to_raster <https://corteva.github.io/rioxarray/stable/rioxarray.html#rioxarray.raster_dataset.RasterDataset.to_raster>

>
>
> 3. If you already have the time-series then I personally find it
> easiest
> to switch to pandas and export to csv from there:
>
> ds = <your DataSet from atlite with the timeseries>)
>
> df = ds.to_pandas()
> df.to_csv(<save path>)
>
> Doc:

> https://pandas.pydata.org/docs/getting_started/index.html#installation <https://pandas.pydata.org/docs/getting_started/index.html#installation>
>
> https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html <https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_csv.html>

>
>
> 4. If your local data is already an atlite cutout, then you can
> use it
> by giving the path:
>
> cutout = atlite.Cutout(path="<your local path>")
>
> atlite will load it and use it.
> If the data is ERA5 data downloaded but not a cutout created by
> atlite,
> then you cannot use it and have to downloaded it via the CDS API.
>
>
> HTH,
>
> Best,
> Johannes
>
> Best regards,
> Johannes Hampp (he/him)
>
> Justus Liebig University Giessen (JLU)
> Center for international Development and Environmental Research
> (ZEU)
>
> mailto: johanne...@zeu.uni-giessen.de
> <mailto:johanne...@zeu.uni-giessen.de>
>
> Senckenbergstr. 3
> DE-35392 Giessen

> https://uni-giessen.de/zeu <https://uni-giessen.de/zeu>

>
> Am 29/11/2022 um 20:49 schrieb Matthew Smith:
> > Oh, to add another question; is there a way to reference a
> downloaded
> > GRIB cut-out of ERA5 data on my local machine? Instead of
> requesting CDS
> > to prepare a new cut out.
> >
> > Thank you,
> > Matt
> >
> > On Wed, 30 Nov 2022, 01:04 Matthew Smith,
> <matts...@gmail.com <mailto:matts...@gmail.com>

> > <mailto:matts...@gmail.com

> <mailto:pypsa%2Bunsu...@googlegroups.com>
> >     <mailto:pypsa+un...@googlegroups.com
> <mailto:pypsa%2Bunsu...@googlegroups.com>>.

> >     To view this discussion on the web, visit
> >

> https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com <https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com> <https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pypsa/f4209a78-1748-4d7b-98d4-1a03a2c07a56n%40googlegroups.com?utm_medium=email&utm_source=footer>>.

> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "pypsa" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send
> > an email to pypsa+un...@googlegroups.com

> <mailto:pypsa%2Bunsu...@googlegroups.com>
> > <mailto:pypsa+un...@googlegroups.com
> <mailto:pypsa%2Bunsu...@googlegroups.com>>.

> > To view this discussion on the web, visit
> >

> https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com <https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com> <https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/pypsa/CAMAWJS%3DOShuEwq463_y%2BNt08mRDtyAeVR7%2BmZBRbrvT7ZvPirQ%40mail.gmail.com?utm_medium=email&utm_source=footer>>.
>
>
>
> --
> Matthew Smith

[Matthew Smith]

Hi Johannes,

I think you are right, there's likely no short cut to reach the answer. I'll need to find the optimal combination of plants by using PyPSA or another tool.

Just to ask another question on Atlite. I'd like to show the average windspeed per grid square in the same style map as in this page https://atlite.readthedocs.io/en/latest/examples/plotting_with_atlite.html below the heading 'Plot capacity factors'. However I can't find the right object to represent the average windspeed at 100m.

cap_factors = cutout.wind(turbine='Vestas_V112_3MW', capacity_factor=True) # This should be replaced with something like cutout.data.wnd100m, but I can't find the right object

Thanks for the help.

Best regards,

Matt

[Matthew Smith]

Okay I figured it out! Just needed to use cutout.data.wind100m.mean('time') with time being the aspect of the data being averaged. In the previous code block in the example, mean is used to average all the sites to get a geographic cut out wide average for each time period.

wind_speed = cutout.data.wnd100m.mean('time') 

fig, ax = plt.subplots(subplot_kw={'projection': projection}, figsize=(9, 7))
wind_speed.name = 'Wind Speed'
wind_speed.plot(ax=ax, transform=plate(), alpha=1)
cells.plot(ax=ax, **plot_grid_dict)
fig.tight_layout();