New import module

[Lechémia Théo]

Hi everyone,

As mentionned in a previous post, we start the develoment of a user interface for data import from csv file. In fact it was a strong need for our collegues to be autonomous on this task. 

After some weeks of work we have now a first beta version of this new tools.

We first start the tool using fully the existing API POST, but we had some issues, so we develop some new stuff... 

The soft use the python Flask Framework and has not JS Framework. It is not design to be very fast (in fact it use a the high level API of SQLAlchemy, python models etc...) but it work for large datasets. At this time we test with a 6 years hour per hour time serie and it takes few minutes...

The import follow this steps :

- select the procedure in which you want to import

- make a mapping between observed properties and the column of your csv file

- start the import. A backgroud task is running and you will receive ean email at the end of the process. You have a full report of the import in the email (errors logs, succed row etc...)

The home page is an history of previous imports.

The main issue we had developping this tool is the database multiple schema architecture (one database schema per SOS service). We do not find any way to make it work we SQLAlchemy (without be too "hacky"). So the tool is mono-service. You can configure the service in the config file (and also deploy multiple instance of the tool if you have multiple services...)

The tool is of course open source, so feel free to use, test, contribute, open issues or even contact us etc...

The code is here : https://github.com/PnEcrins/istSOS-import. You'll find (a poor) installation documentation ...

By the way, we also just start a little frontend project (only base on the existing API) to see procedure and plot data. We use plotly for ploting it and it's look like it solve the perfoming issue we had previously. We wiil keep you inform of the continuation of this project (code here : https://github.com/PnEcrins/istSOS-viewer)

best,

Théo