GIS and OnSSET

[andreas.sahlberg]

Use this thread to discuss GIS-related OnSSET questions.

[hermaw...@gmail.com]

Hi Teams,

I am very happy finally the onsset forum work, and I am very grateful to meet this great program. Hopefully the onsset program will be the center of superior studies related to spatial-based electrification and become a reference for policy makers in making energy planning in their regions.

My name Hermawan from Indonesia, ordinary people, like spatial work, not familiar with programing. I have tried to use the program, initially it was very difficult to run the program because of the lack of full instructions (the process of creating a CSV file), but finally success too.

There are things I want to ask:

1. Onsset is designed for 1 km resolution data, if I have dencity population data with a resolution of 500 m or greater, will the onsets analysis be correct? how do you suggest?
2. Regarding land cover, classification is different from Indonesia (23 classes), what should be emphasized in the classification process so that the results make sense.
3. The nightlight time data will have a range of 0-63, I can get the latest data from the range more than that (0-247 my sample location), what's the solution?

I thank you very much for the kind onset team.

[andreas.sahlberg]

Dear Hermawan Nabil, 

Thank you for your interest in OnSSET! 

Regarding your questions:

1. For now we suggest to use 1 km2 population data. If you have higher resolution data, you can convert these into 1 km2 cells in GIS before creating the csv-file. In ArcGIS this can be done using the Aggregate tool and in QGIS this can be done using the r.resamp.stats tool.
   
2. In the OnSSET manual there is a description of how to work with different land cover maps:
   The model classifies the land cover in order to calculate the grid extension penalties. The default classification values are based on the MODIS dataset found here, where the legend ranges from 0-16 with the values and corresponding land cover type can be seen below. If land cover data is retrieved from other data sources with different classification values they should be reclassified in GIS (using the Reclassify tool in ArcGIS or r.reclass in QGIS) to match those below. Alternatively changes can be madein the Python code instead. If this reclassification is not performed it may lead to an incorrect grid penalty factor or, if the highest values are above 16, an error message while running the code.
   I suggest to make the changes in GIS so that your values are reclassified to those in the table found in the manual under Land Cover, i.e. match your land cover descriptions to the closest ones found in the table.
3. Which dataset are you using for the nightlights data? In case this is the VIIRS dataset: in the onsset.py code, line 633, change the value 5 to 0.5 or 0.1.
    

Let me know if you have any more questions!

[Hermawan Nabil]

Hi Andreas,

Thank you very much for the response. Regarding the nighttime light data source, I used the same source Viirs but the latest version I think ; https://www.ngdc.noaa.gov/eog/viirs/download_dnb_composites.html#NTL_2015 so the value range is greater from 63 and the resolution is 15 arc sec.

I still don't understand what is meant by the value of 0.5 / 0.1 (... change the value 5 to 0.5 or 0.1), after I aggregate (because of the resolution of 15 arc) the value is doubled, the highest value of 247 becomes 741, whether the change is 5 to 0.5 /0.1, Is it still relevant? is there a reference to understanding the values ​​in the phyonsset / onsset.py / runner.py file?

So far I have tried using the onsset model for two different locations, Riau Province and Jambi with Arcgis the code for creating CSV files, and it works. There are things I want to ask:

1. There is a message "WARNING 000728: NEAR_FC field does not exist within table", can it be ignored?

Start Time: Fri Nov 09 10:28:48 2018
Dropping NEAR_DIST from E: \ .........
Dropping NEAR_FID from E: \ .........
WARNING 000728: The NEAR_FC field does not exist within table
Succeeded at Fri Nov 09 10:28:51 2018 (Elapsed Time: 2.46 seconds)

2. GHI Data Unit which is used daily or annual average?

3. Regarding the results of the onsset analysis, why did the recommendations for MG PV and MG Wind not appear, always 0 for 5 tiers? even though in fact there are about  60 MG PV plants in that location? (FYI: value range for Solar data 2.89-4.75, Windvel 2.2-5.5, Traveltime 0-1212, etc)

Regards,

Hermawan Yulianto

[andreas.sahlberg]

Dear Hermawan,

Apologies for the slow response!

First of all, the value "5" represent the lowest limit a cell need to achieve to potentially be considered as electrified in the start year. For the VIIRS dataset this lower limit is better set as "0.1". The upper limit can still be kept as "60". There are some updates going on with the calibration, and a better description will follow in the OnSSET manual once those are complete.

Solar GHI should be annual values, so probably in the range of 1500-2000 kWh/year. This is likely the reason you do not get PV mini-grids in the model.
Your wind velocities seem correct, but travel time should maybe be divided by 60 (I suspect your values are in minutes, it should be hours in the model).
Also, which diesel price are you using? This also affects quite a lot the technology choice in the model.

[Wendell]

Hi,

Could you share how your created the csv file from layer data? This is something I have been struggling with.

Thanks

[Hermawan Nabil]

Hi Andreas,

Thanks for the explanation. Related diesel price, I used low and high price (0.4 & 0.8 ) for simulation.

Regards,

Hermawan

[Hermawan Nabil]

Hi Wendell,

Sure I sent you the csv and specs file. Please find the attached below.

Regards,

Hermawan

[Wendell]

Thanks Hermawan, but I am struggling to create the .csv from the 16 files. I have the various shapefiles and rasters for Zambia (my study region) arranged according to the instructions, but am running into execution problems.

I'd really appreciate it if someone could share a zipped file with their 16 folders (from "Admin" to "Wind") and the python code they used to generate their csv. That'd be a great help for my debugging.

Best,
Wendell

[Hermawan Nabil]

Hi Wendell,

I understand what you have been doing, I also experienced various problems when I first created the CSV file with ArcGis.  Hopefully the shp file and gis python code can help you (py code is the same as in the manual, just little adjustment)

FYI: after running code in arcgis, for the result file I fixed the column name manually because the conditioning of argis py files didn't work (sorry, I didn't understand programming).

Regards,

Hermawan

[Wendell]

Wow, thanks again, Hermawan! Your files and comments proved extremely helpful with my problem!!

Best,

Wendell

[amyna...@gmail.com]

Hi,

Please I want to kwow how to proceed if I have to run the ONSSET Model for several regions of a country with different rate of electrification.

[andreas.sahlberg]

Hi!

Many apologies for the very late reply. Can you please elaborate a bit further on your question. Have you run all the steps for each region (extracting from GIS to CSV, calibrating the file and running scenarios), or how far did you get?

[signer...@gmail.com]

Hello

I'm currently applying OnSSET to Guatemala. The documentation and the guides are really well done and make it way easier to understand. So many thanks to the developers for that!

Still, I have still some open questions concerning the GIS input:

1. The Hydro Power Potential: How are the points determined, for which to calculate the hydro power potential? Depending on what are they placed along the river network? And is there maybe a more detailed methodology, how this is done, than in Mentis et al 2017 "2.3. Spatial energy resource availability"?
2. Nighttime Lights: I got my raster file from the source mentioned in the manual. However, there are different data sets (i.e. with/without frequency of light sources included). Which do you recommend?
3. What are the needed units for Travel Time and Population Density? Is it minutes resp. households/km2?

[andreas.sahlberg]

Hi!

The hydropower datasets were created using the methodology described in this paper: https://www.mdpi.com/1996-1073/11/11/3100. Unfortunately there is no plugin or code developed that performs this whole process in one go (at least not yet). Hope this helps, and if you have any more questions please post them here. 

For the night-time lights I suggest you use the annual data avaialble here instead: https://eogdata.mines.edu/download_dnb_composites.html. Will update the manual accordingly. 

Finally, the Travel Time data should be in hours and the population should be total number of people, not population density (the population density can of course be derived then if you know the areas of the cell/cluster).

/Andreas 

[signer...@gmail.com]

Hey Andreas

Thank you for the quick response! 

I've looked at the paper. Apparently, they wrote some Python code to set the points on the rivers. Do you know whether it is possible to get access to it?

Best, Mario

Am Dienstag, 28. April 2020 09:49:05 UTC+2 schrieb andreas.sahlberg:

Hi!

The hydropower datasets were created using the methodology described in this paper: https://www.mdpi.com/1996-1073/11/11/3100. Unfortunately there is no plugin or code developed that performs this whole process in one go (at least not yet). Hope this helps, and if you have any more questions please post them here. 

For the night-time lights I suggest you use the annual data avaialble here instead: https://eogdata.mines.edu/download_dnb_composites.html. Will update the manual accordingly. 

Finally, the Travel Time data should be in hours and the population should be total number of people, not population density (the population density can of course be derived then if you know the areas of the cell/cluster).

/Andreas 

[signer...@gmail.com]

Hello Andreas

I've another question: How important is the map of the transformer data? It's really hard to get my hand on a usable map of the transformers. Also, in the example file on GitHub for Djibouti, TransformerDist  is 0 for all the squares which makes me wonder, whether it is even needed. Could you clarify please.

Thanks and best regards

Mario

Am Freitag, 19. Oktober 2018 09:52:50 UTC+2 schrieb andreas.sahlberg:

[signer...@gmail.com]

For everybody that wants to do the hydropotential calculations, there is now a much easier way to do it, using the code by Alexandros Korkovelos that you can find here, the process behind it neatly explained here.

Best, Mario

[Luis Carlos Romero]

Good Afternoon

my name is Luis Carlos Romero

I Currently enroll  on the open university course about onsset, is interesting for me because some time ago I try to do something similar on other GIS, ILWIS using neighborhood operations with rasters, I wanted to learn about OnSSET methodology that I found quite interesting, thanks for sharing it and make it available to the general public.

Right now I am at the hands-on 6  GEP Scenario Generator

I write to you  because I have found some throuble running the code as requested on the exercise

First I thought It was the Operating system, I am running it on MAC os , so I switch  to windows but still have the same problem, seems to me that for some reason is not reading the  modules in python on the subdirectory...

I Downloaded the codes a couple day ago from the location  given and using  Anaconda and jupyter as requested.

on every line a different error appear related to modules or names

for code line 1 section 0

Module not found error: no module name 'hybrids_pv'

for line 2, section 1

name 'SettlementProcessor' is not defined

for step 16, section 4

name 'Technology' is not defined ---> related to  grid_calc= Technology(...)

for step 17, section 5

name 'onsseter' is not defined ---> related to pop_modelled, urban_modelled= onsseter.calibrate....

I appreciate your guide lines to debug the code so I could continue the course

Best Regards

Luis Carlos

[Luis Carlos Romero]

Hello

Trying to solve the problem, I have added  this code

prior to the first line of code

import os

os.getcwd()

# ip to here I check where the current path is I have found that is ~ /gep-onsset-master

import sys

sys.path.append('my/path/to/gep-onsset-master/gep_onsset')

# just added the /gep_onsset  to  os.getcwd() code response

with this the program runed  up to step 5,

a warning appear when loading the Csv file

2022-02-10 09:02:01,636 NumExpr defaulting to 4 threads.

and then in the next line an error appear

onsseter.condition_df()
onsseter.grid_penalties()  <-----
onsseter.calc_wind_cfs()

TypeError: grid_penalties() missing 1 required positional argument: 'data_frame'

 then the program shows other  errors on the following lines, probably related to the previous

----> 8 elec_modelled, urban_internal_elec_ratio, rural_internal_elec_ratio = onsseter.elec_current_and_future(elec_ratio_start_year, 9 urban_elec_ratio, 10 rural_elec_ratio,

TypeError: elec_current_and_future() got multiple values for argument 'min_night_lights'

and

KeyError: 'ElecStart' on the following plots

If someone coud give some lights on where is the problem and how could I solved I appreciated

best regards

Luis Carlos

[Luis Carlos Romero]

Hello,  here is the solution

On the Hands on 6,  appear a link to download the On SEET code,  but it is an older version called master,   so you should download the version 1.1

if  not updated the link on the pdf for any reason  you should go to:

https://github.com/global-electrification-platform/gep-onsset/tree/v1.1

This code no need any intervention  as I mention and works perfectly!

Best for all

Luis Carlos