Adding fallback support for UnWired Labs API

[Ben Withers]

Hey all!

After successfully setting up and running an instance of Ichnaea (which also has automatic daily updating to the latest MLS full snapshot via bash script!), I was asked to add support for falling back to UnWired Labs, our current service provider.

As far as I know, Ichnaea supports MLS and Combain (and Google?) APIs as fallback.

The UnWired Labs Location API docs can be found here: http://unwiredlabs.com/api#documentation

As you can see, it's not too different from other APIs, but the field names are identical to the Google API and the addition of a token field for the API token (rather than having token in the URL) and a status and balance field in the response (used for query status and token credit balance).

I think the best approach is to make a copy of the json constructors that build the json dict up as per the UnWired API when the api_key.fallback_name is "unwired" for example. Of course, schemas would be added to do the response checking and so on.

I would just like to know if anyone else has tried doing anything like this or if anyone has any advice for tackling this. I can imagine that once I start, I'll probably run into loads of issues regarding schema validation and so on.

I'll post progress here in case anyone else is interested.

Many thanks!

Ben

[Hanno Schlichting]

Hey.

> On 2. Aug 2017, at 18:22, Ben Withers <manic...@gmail.com> wrote:
> As far as I know, Ichnaea supports MLS and Combain (and Google?) APIs as fallback.

Yes. They all support almost the same API, with some very minor differences for special cases. So we end up with one implementation to support them all.

> The UnWired Labs Location API docs can be found here: http://unwiredlabs.com/api#documentation
>
> As you can see, it's not too different from other APIs, but the field names are identical to the Google API and the addition of a token field for the API token (rather than having token in the URL) and a status and balance field in the response (used for query status and token credit balance).
>
> I think the best approach is to make a copy of the json constructors that build the json dict up as per the UnWired API when the api_key.fallback_name is "unwired" for example. Of course, schemas would be added to do the response checking and so on.
>
> I would just like to know if anyone else has tried doing anything like this or if anyone has any advice for tackling this. I can imagine that once I start, I'll probably run into loads of issues regarding schema validation and so on.

I've added two bits to make this easier and created an issue over at https://github.com/mozilla/ichnaea/issues/508 <https://github.com/mozilla/ichnaea/issues/508>

There's a new fallback_schema column (varchar 64 )in the api_key table, which can be used as an explicit source for determining the fallback API schema. I've also changed our fallback provider code to look up the API schema based on this column. If it's null or ''ichnaea/v1'', it uses our own schema. It could act on something like 'unwiredlabs/v1' to use a different schema.

I think it should be enough to add new JSON colander schemas to https://github.com/mozilla/ichnaea/blob/master/ichnaea/api/locate/fallback.py <https://github.com/mozilla/ichnaea/blob/master/ichnaea/api/locate/fallback.py> and appropriate tests to https://github.com/mozilla/ichnaea/blob/master/ichnaea/api/locate/tests/test_fallback.py <https://github.com/mozilla/ichnaea/blob/master/ichnaea/api/locate/tests/test_fallback.py>. Nothing else in the codebase should require changes.

The only other problem I can see is the fallback cache. I think it's currently not separated by fallback name. It could make sense to separate it by fallback schema, so you don't end up mixing results from too very different providers. But that's only a concern in the unlikely case you are using multiple different fallback providers in one installation.

Hope this helps,
Hanno

[Ben Withers]

I have already integrated UnWired Labs as a fallback provider successfully, albeit in a very hacky way.

I modified fallback.py to detect when the api_key.fallback_name == 'unwired', then build the JSON manually to send to UnWired Labs. It works at the moment, but making the changes you suggested would be much better.

Due to the nature of my work, I can't spend too much time improving our company's instance, but I'd be happy to spend some of my time working on the UnWired Labs colander schema for the Ichnaea codebase. May take a while due to my inexperience with colander, but I'll do my best.

Kind regards,
Ben