Data store, data model and server API

[Fernando Jiménez Moreno]

Hello folks,

as I have been reporting on the standup meetings doc during the last days, I have been doing some investigation about things related to the SensorWeb server side. Specifically, I have been looking at the data store and the data model and public API.

For the data store, as you know, Fabrice investigated the possibility of using InfluxDB to store the sensors observations, but he concluded that it did not look like a good fit, mostly because of the lack of spacial support. Which is a bummer, because in theory we are going to be doing some interesting work with maps. So instead of looking for other time series databases, I searched for spacial databases. And I found that PostgreSQL has an excellent extension for geographic data called PostGIS [1], which seems to be the choice for many products [2] that do or enable similar things to what we want to do here. I found Carto [3] specially interesting. Unless anyone disagrees, I would like us to start using PostgreSQL/PostGIS for our server(s).

For the public API and data model, I have been looking at how others are doing it as well. I found a few proprietary APIs (like [4] or [5]) from companies that you already heard before. These are good as a starting point or as a model for our own API. But I thought that instead of a proprietary and an ad-hoc API it was probably better to search for something more standard and more generic. If we want to extract a platform from this product, interoperability is something that we should be looking for. So I found this OGC [6] standard named SensorThings API [7]. After my finding, I learnt that Jim Porter already found and recommended this standard while he was doing his research for SensorWeb back in April [8]. He recently reiterated his findings on the mailing list. And I am glad to see that I share his very same conclusions. I believe the SensorThings API is a good candidate for us to implement. It is the evolution of an old OGC standard (Sensor Observation Service [9]) and it is under the OGC Sensor Web Enablement [10] standard suite. It has support for location and historical locations. It uses the SensorML standard for sensor description. It has a powerful query language based on the OData protocol [11]. It defines a way to batch requests or a extension for multiple datastreams. It has an extension for MQTT as well. Etcetera. I recommend you to watch this video for a short overview of the API in action [12].

I also read the W3C WoT [13] Web Things Model member submission paper [14]. I'll probably have to read the other writings from this group but for what I saw so far I think what this offers is not enough for our needs. It apparently has no built in support for geolocation and no query language, at least not one comparable to SensorThings' one. And in the end, it's just a proposal so far, so it's adoption is not comparable to OGC sensors related standards.

One cool thing about the WoT proposals is that in this model everything has an URL. So that fits perfectly well on the Web. In case that you are wondering, in the SensorThings API, every thing has an URL as well. From Things to Locations or Observations, everything gets an URL.

So if I have to choose, I would implement the SensorThings API.

What do you think?

Cheers,

/ Fernando

[1] http://postgis.net/
[2] https://en.wikipedia.org/wiki/PostGIS#Users
[3] https://carto.com/
[4] https://www.thethingsnetwork.org/docs/mqtt/#api-reference
[5] https://api.opensensors.io/index.html
[6] http://www.opengeospatial.org/
[7] http://www.opengeospatial.org/standards/sensorthings
[8] https://github.com/mozilla/connected-devices-experiments/issues/30
[9] https://en.wikipedia.org/wiki/Sensor_Observation_Service
[10] http://www.opengeospatial.org/ogc/markets-technologies/swe
[11] http://www.odata.org/
[12] https://www.youtube.com/watch?v=GMrnerjOqYs
[13] https://www.w3.org/WoT/
[14] http://model.webofthings.io

[Fabrice Desré]

Hey Fernando,

On 09/23/2016 08:24 AM, Fernando Jiménez Moreno wrote:
> enable similar things to what we want to do here. I found Carto [3]
> specially interesting. Unless anyone disagrees, I would like us to start
> using PostgreSQL/PostGIS for our server(s).

+1 from me. Mature, well supported and Postgresql should scale well
enough (btw, we need to estimate our volumetry).

>
> So if I have to choose, I would implement the SensorThings API.
>
> What do you think?

Disclaimer: I haven't read all the docs you referenced.
I looks like we have 2 options:
1- implement the SensorThings API.
2- go with a custom solution.

1- avoids the pitfalls of re-inventing the wheel, but some of this stuff
seems quite bloated. The json format is very verbose for instance. I
don't know if this is the result of a design by committee where they had
to please everyone, or if any complete enough solution would end up with
something similar.
2- would be likely a bit faster to come up with, with a risk of not
being as future proof.

So I'm not opposed to implement the SensorThings API, just not super
excited ;)

--
Fabrice Desré
Connected Devices
Mozilla Corporation

[steve...@sensorup.com]

Hi All,

My name is Steve, and I am the editor and working group chair of the OGC SensorThings API. I am also one of the authors of the OGC Sensor Web Enablement standard suite. I am a professor at the University of Calgary [1] and founder/CEO of SensorUp. Thanks to Fernando linking us here. It is great to see the work done here in this community.

If you have any question regarding OGC SensorThings API, please feel free to ask me. Will reply some of Fabrice's question in the next reply. This is just an introduction. :)

BTW, I see some group members are from TW. I grew up in TW and left Taiwan after my military service for graduate study. :)

Cheers,
Steve


[1] https://www.ucalgary.ca/utoday/issue/2016-05-11/using-internet-things-smart-devices-can-transform-society

[steve...@sensorup.com]

Hi Fabrice,

I am responding the "verbose" json question.

There are reasons for the "comprehensive" data model. It is based on the ISO 19156 Observation and Measurement standard. And there are some use cases requiring a more complicated data model, e.g., water quality. For water quality observations, the observation is normally collected at Location_A at Time_B. Then they are sent to a lab at Location_C at Time_D to get the result. Our data model supports that.

The other common example requiring a more complicated data model is for drones. Drones are flying at Location_A (sensor location) and its observing some targets at Location_B (feature of interest location). Our data model supports that too.

As you mentioned, adopting existing standards prevents reinventing the wheel. Our data model have been proven working for simple and complex applications. It's future proof and extensible.

OK. The elephant in the room is the verbose format. But in fact, our design considered that!! In our API, we have a query parameter called $select. It allows you to select ONLY the properties interest you. Check this example out: http://example.sensorup.com/v1.0/Observations?$select=result,phenomenonTime

By doing this, we cover both simplicity and comprehensiveness. :) I think this design is quite elegant.

Cheers,
Steve