The OnSSET code

[andreas.sahlberg]

The OnSSET code is found here: https://github.com/KTH-dESA/PyOnSSET/. Use this thread to discuss issues relating to the code.

[Churchill Agutu]

Hello there, I am currently using the Onsset tool from my research, I am currently testing to see if the model works using Afghanistan data, Step one seems fine (I assume, I do not have to do this if it is only for one country), however for step 2, I am not sure how to go about this> Do I run the code, enter the country name and then select step 2? Also once I select step 2, it asks me for the directory to save the calibrated info, what is this purpose of this?

[Churchill Agutu]

Thanks Andreas,

My questions are:

1. I am trying to test the code using the Afghanistan CSV data and it seems there are runner codes:
• One from the Afghanistan Github – which has 3 options : Enter 1 to split, 2 to prepare the inputs, 3 to run a scenario
• One from the Onsset page – This one has only 2 options: 1 to calibrate and prep, 2 to run a scenario
Question: Of the above two runners? Is there one which supercedes another? Which one should I use?
2. When I run the code using option 1 Afghanistan Github code:
• I specify the spec file
• Then got to prepare inputs because the data is only for Afghanistan, it then prompts me to enter the csv file source for data (Afghanistan. Csv) and then to provide a directory to save the calibration data (I just name it @”Afg_calib”)
• The code ends
• So I run it again to get the list of options i.e Enter 1 to split, 2 to prepare the inputs, 3 to run a scenario , I choose option 2 to prepare inputs, Then I get an error “KeyError: 'PopStartYear'”.
Question: Why is this happening?
3. The third issue is more of a general question, if I want to use the model and apply it in a set of countries across sub-saharan Africa, do I have to generate the data for each country using the QGIS software or is there an existing database with this? i.e can I access the same GIS info that was on IEA’s Energy Access Outlook? What would you advise?

[andreas.sahlberg]

The updated code is now available at https://github.com/onsset 

[signer...@gmail.com]

Hi there

So, I've followed the manual quite strictly up the point where I have the country.csv file and the specs file ready. I dowloaded the code from Github, Master Branch. So, running the code would be the next step. However, doing it according to the manual doesn't really work for me as the "runner" file seems to not work the same way anymore (Step 1: Split, Step 2: Prep, Step 3: Run scenario(s) as the manual says it). It appears to me as if "gui_runner" needs to be run to run OnSSET. Correct?

When I tried to run "gui_runner" with my country.csv file, I got key errors and I realized that the columns that are created with the extraction code from the manual don't really fit the required variables in the code.

The Parameters that I have in my file are these:

Parameter	Description
Country	Name of the country
Pop	Population in base year
X	Longitude
Y	Latitude
GHI	Global Horizontal Irradiation (kWh/m2/year)
SolarRestriction	Defines if an areas is restricted to solar PV deployment (1: restricted, 0: non restricted)
WindVel	Wind speed (m/s)
TravelHours	Distance to the nearest town (hours)
NightLights	Nighttime light intensity (0-63)
Elevation	Elevation from sea level (m)
Slope	Ground surface slope gradient (degrees)
LandCover	Type of land cover as defined by the source data
GridDistCurrent	Distance from the existing electricity grid network (km)
GridDistPlan	Distance from the planned electricity grid network (km)
SubstationDist	Distance from the existing sub-stations (km)
RoadDist	Distance from the existing and planned road network (km)
HydropowerDist	Distance from identified hydropower potential (km)
Hydropower	Closest hydropower technical potential identified
HydropowerFID	ID of the nearest hydropower potential

So I'm a bit confused atm. Am I using the wrong code or do I have the wrong input? Is there something that I'm missing?

Best, Mario

[Babak Khavari]

Hello Mario,

We have updated the methods that we use for the data extraction. We are now using QGIS plugins to generate the input file.  

There are two plugins, one for generating population clusters and one for generating the input file itself. They both have to be run, first the clustering plugin and then the extraction one. I recommend that you download and install QGIS 3.10 (https://qgis.org/en/site/forusers/download.html) and then install the plugins. You can find the 3.10 version of the clustering plugin here: https://github.com/OnSSET/PopCluster/tree/master/Plugin/Option%201 and for the extraction plugin here: https://github.com/OnSSET/ClusterbasedExtraction/tree/master/Plugin/Option%201/ClusterbasedExtraction/tree/master/Plugin/Option%201. There are also instructions on how to install and run the plugins on the GitHub repositories connected to each plugin.

We will update the documentation on ReadTheDocs to reflect these changes.

Best, Babak 

[signer...@gmail.com]

Dear Babak

Thank you very much for your reply. That makes things much more clear to me.

However there are still some open questions to me:

- What exactly is the Custom Demand variable? It is not included in the instructions , so I guess it was added more recently.
- How do you recommend running the actual OnSSET code? I saw that there is a Jupyter Notebook in the folder. Should that one be used?

Best regards
Mario

[babak....@gmail.com]

Dear Mario

• The custom demand layer is a layer that we generate here at KTH using poverty and GDP maps following the methodology described in appendix F here. This layer is meant to provide a more dissagregated measure of demand, rather than just one tier for urban and one tier for rural. We are planning on making this an optional layer, but currently you have to put a raster layer in its place during the extraction in QGIS. If you want to skip this layer I suggest that you replace it with any other raster and then when the plugin is finished and you have received the final output CSV, go to the column named "ResidentialDemandTIerCustom" and manually replace these values with 0, since this column is not needed in OnSSET it-self.
  
• Yes I recommend you use the Jupyter Notebook
   
  

Best,

Babak

[signer...@gmail.com]

Hey Babak

Thank you for your answers. Let me just ask you two more:

• As the Custom Demand is not used in OnSSET itself, how difficult do you think is it to change the code, so that it is used instead of the 5 targets? Couldn't it be used to determine how much more electricity actually will be needed where? So rather than modeling for a policy goal, you would look at the economic predictions of the country and say in order to fulfill the el. demand of this future economy at say 100 % electrification, you would need this amount of money using these technologies?
• In the meantime I've managed to run the plugins and the main code. However if I include the hydro_points when running the GEP plugin, I get error messages. I could already solve some minor issues. However now I got stuck, and these error messages are not really detailed. F.e. at the moment qgis is telling me that the algorithm at line 1272 failed, but I don't see any obvious problems with the code. Do you have knowledge of issues in the hydro_points section? Or can you recommend any debugging tool for qgis that might help me to find out what's going wrong?
  

Best, Mario

[babak....@gmail.com]

Dear Mario,

• Let me clarify regarding the custom demand layer. What I meant to say is, that it is not a mandatory layer. If you have such a layer and include in the settlement file you can run it in OnSSET. If you do not have it, you can still run OnSSET you just have to use the normal approach of one tier for rural and one tier for urban. Do I understand your question correctly?
  
• I do not seem to be able to replicate your error for the plugin. I will look a bit more into it and post here when I have a solution. Do you receive a specific error message or does it simply stop? Usually a good way of debugging it is to go the code it-self, find out which tool is being used and then manually run it in QGIS with the same datasets to see if it still crashes. The tool being used at line is 1272 is "Join attribute by field value", this tool has been used earlier in the plugin so there is no problem with the tool it self. Should be the input data which is saved as a memory layer (so it might be difficult to run manually)
  

[signer...@gmail.com]

Dear Babak

• Ah yes, I think you understood me correctly. So in that case, if there is no Custom Demand layer, OnSSET assumes that everybody which is not electrified yet, will have the same/the scenario's electricity demand, correct?
• Oh, I'm so sorry. It is line 1277! The error message is a generic one: "QgsProcessingException: Error excuting algorithm". The problem is that I fail myself to replicate it using the python console in qgis or the toobox. It all works just fine there. Because of that I think the code of the method (rastercalculator) itself is fine. The one thing, I cannot really replicate however, is as you say the input and the output as these are memory layers. Do you have any suggestions for me how to continue forward? 

For completeness, I will list here the changes to the code that I've made until now:

• Line 709, 803, 890, 993, 1083: 'ADD         ' ----- deleted, as these spaces made my qgis freeze forever
• Line 1262:  'FIELD': 'AUTO', 'UNIT': 3, 'OUTPUT': 'memory: '})  ----- added missing colon, as it created error: "QgsProcessingException: Error excuting algorithm"

Thanks again for your time, 

Mario

[Babak Khavari]

Dear Mario,

• Yes that is correct.
• OK I will check this and see. The tool that you are referring to uses an input from the previous tool. Maybe the first step would be to change the output of the previous tool to a layer saved on your disc by replacing 'OUTPUT': 'memory:' with
  'OUTPUT': workspace + r'/Hydropower/Hydro/hydro_test.shp'. Then you run the plugin. It will still crash but it will give you the opportunity to use the output and run the tool on line 1277 with the correct data set directly in QGIS. Hopefully this tool will crash when ran manually as well with your input and give us a useful error code to work with.

Thank you for the list of things you changed, these were things I had completely missed!

Best regards,

Babak

[signer...@gmail.com]

Dear Babak

So, I've tried out your suggestion and the funny thing is that replacing the input of the fieldcalculator from the memory file to a shp file actually makes the error message not to appear. It seems to me that the memory file is in fact a geopackage file and having a gpkg as an input makes fieldcalculator crash in this case. I have no idea why though, as there are other instances in the code where it seems to me that fieldcalculator using gpkg input, works just fine.

But I guess, I can live with this and use this modified code to run the plugin now...^^

     y = processing.run("native:joinattributestable", {

                    'INPUT': hydro_dist, 'FIELD': 'id',

                    'INPUT_2': z['OUTPUT'],

                    'FIELD_2': 'id', 'FIELDS_TO_COPY': ['min'], 'METHOD': 1, 'DISCARD_NONMATCHING': False,

                    'PREFIX': '',

                    'OUTPUT': workspace + r'/Hydropower/Temp.shp'})

     y = processing.run("qgis:fieldcalculator", {

                    'INPUT': workspace + r'/Hydropower/Temp.shp',

                    'FIELD_NAME': 'HydroDist', 'FIELD_TYPE': 0, 'FIELD_LENGTH': 10, 'FIELD_PRECISION': 3,

                    'NEW_FIELD': False, 'FORMULA': 'if(\"HydroDist\" IS NULL,represent_value(\"min\"),0)',

                    'OUTPUT': 'memory:'})

Best, Mario

[Babak Khavari]

Dear Mario,

Happy that it worked for you. Can this perhaps be linked to the amount of memory on you computer then?

Do you get distances in your HydropowerDist column or only the value 0 with your approach?

Best, Babak

[signer...@gmail.com]

Dear Babak

I'm not sure.. After all, I tried to run the method in the console with a gpkg file as input (11 MB from my harddisk) and it also failed to execute.

You mean HydropowerDist in the final settlement file? Yeah, I get distances there. 

Best, Mario

[Babak Khavari]

Dear Mario,

So the tool crashes when you run it manually from the python console in QGIS? Unless you specified an actual output location the resulting file should have been saved to memory in this case as well. so if the memory is full it would still crash here. (This is just for me to understand the problem, nothing that you have to try). 

Great! Then you can feed that file into OnSSET. I will try to find a general solution that works for all.

Best, Babak

[signer...@gmail.com]

Dear Babak

That's correct, the tool also crashes in the python console. Even if I change the output to an actual file as well like this: 

y = processing.run("qgis:fieldcalculator", {'INPUT': r'C:/Users/Mario/Documents/OnSSET_GEP/Test1.gpkg', 'FIELD_NAME': 'HydroDist', 'FIELD_TYPE': 0, 'FIELD_LENGTH': 10, 'FIELD_PRECISION': 3, 'NEW_FIELD': False, 'FORMULA': 'if(\"HydroDist\" IS NULL,represent_value(\"min\"),0)', 'OUTPUT': r'C:/Users/Mario/Documents/OnSSET_GEP/Hydropower/test5.gpkg'})

What I could do, if nothing helps and you can't replicate it, is sharing the gpkg (in this case "Test1") with you. Just in case there is something wrong there.

Best, Mario

[Babak Khavari]

Dear Mario,

Yes, please, can you email that file to me or attach the file here?

Best, Babak

[signer...@gmail.com]

Hey Babak

As requested, I report the indexing bug plus my fix here.

For some reason, the indexes from my settlement file get mixed along the way. That creates an error message in onsset.py at line 1369 "unelectrified = np.where(filter_lcoe < min_code_lcoes)" as the indexes of the two arrays that should be compared, don't fit together.

I fixed it by adding the line below into the jupyter file to get the indexes into order:

onsseter.df = onsseter.df.sort_index()

That's basically it.

Best, Mario 

[signer...@gmail.com]

Dear Babak

There is another issue, I want you to ask about. In the results that I'm getting, the LCOE of diesel technologies is always 99. Even though I have diesel_tech = 1. So, I wanted to ask you whether you might have any idea what the problem could be.

Best, Mario

Am Freitag, 19. Oktober 2018 09:54:17 UTC+2 schrieb andreas.sahlberg:

[Babak Khavari]

Is this on the Jupyter Notebook here: https://github.com/OnSSET/onsset?

[signer...@gmail.com]

Yes, I run the code with the Jupyter file and diesel_techs are defined =1 there. I see the numbers for LCOEs in the results CSV.

[Babak Khavari]

Ok, that is not an expected result. I will have a look, thank you for pointing that out!

[Babak Khavari]

Dear Mario,

In the jupyter notebook. If you go to cell 7 " Start a scenario run, which calculate and compare technology costs for every settlement in the country" go to the where you call the  "calculate_off_grid_lcoes" and add a "diesel_techs=diesel_techs" at the end of it.

So you get:

sa_diesel_investment, sa_pv_investment, mg_diesel_investment, mg_pv_investment, mg_wind_investment, \ mg_hydro_investment = onsseter.calculate_off_grid_lcoes(mg_hydro_calc, mg_wind_calc, mg_pv_calc, sa_pv_calc, mg_diesel_calc, sa_diesel_calc, year, end_year, time_step, diesel_techs=diesel_techs)

instead of:

sa_diesel_investment, sa_pv_investment, mg_diesel_investment, mg_pv_investment, mg_wind_investment, \ mg_hydro_investment = onsseter.calculate_off_grid_lcoes(mg_hydro_calc, mg_wind_calc, mg_pv_calc, sa_pv_calc, mg_diesel_calc, sa_diesel_calc, year, end_year, time_step)

Best,

Babak

[Mario Signer]

Perfect, I've just tested it and now it works!

Best, Mario