DB istSOS loading speed

[junior muyumba]

Hello to all,

I hope you had a good time, for those of you who are back from vacations.

I'm here to ask for help from the istSOS community. My problem is related to the loading of data in the istSOS database. To do it, I use the python script : " csv2itsos.py ", which I call in a function that loops on all my 17 sensors (procedures). The 17 sensors record the precipitation, wind speed and sensor battery voltage data every minute (1440 lines recorded daily).

Previously, the sensor data collected every minute was stored and only sent every 6 hours to the server. As soon as they arrive, they are processed, put in ideal format and finally loaded into the istSOS database, (note that to finalize the loading of all the data from the 17 sensors usually takes more than two hours).

From last week the frequency of data reception has been changed to every 5 minutes instead of 6 hours. The goal is to see the events almost in real time!
And this is where things got complicated. A process is running when another one starts until it saturates the CPU.
While looking for a solution, I came across this article https://hakibenita.com/fast-load-data-python-postgresql ,where it is shown that it would be possible to significantly reduce the speed of data loading in PostgreSQL by using for example the copy module of python, which I even saw in the script "csv2itsos.py". But I didn't know how to do it.

If someone has a solution to reduce this loading time in seconds or less than 5 minutes, it would really help me.

I have attached the specifications of my machine if necessary.

Thanks in advance for your help.

Have a great day!

Junior