Hosting web-compat-api: MDN vs. stand-alone

[Luke Crouch]

All,

John Whitlock is working on the new backend [1] for the Compat Data
project [2] that will power MDN's browser compatibility tables. [3]

In a pull request [4], and at yesterday's planning meeting, I proposed
we host the API under MDN at:

https://developer.mozilla.org/compat/api

Another option is to host the API as a micro-service app at a new
domain, e.g.,:

https://api.compat.mozilla.org/

MDN gives us some advantages:

* We won't need to provision any new servers from WebOps/IT
* We can re-use existing push (Chief) and monitoring (New Relic) services
* We can re-use MDN sessions for API authentication

Are these enough to convince us to host the API under MDN? What are the
advantages of hosting it as a micro-service outside of MDN?

Thanks,
-L

[1] https://github.com/jwhitlock/web-platform-compat
[2] https://wiki.mozilla.org/MDN/Development/CompatibilityTables
[3]
https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Compatibility_Tables
[4] https://github.com/jwhitlock/web-platform-compat/pull/1/files

[Justin Crawford]

> What are the advantages of hosting it as a micro-service outside of MDN?

* Deploying it standalone may make it more hackable by contributors:
there will be fewer prerequisites, fewer dependencies, and a smaller and
substantially less complex codebase to learn
* Deploying it standalone may give MDN developers an opportunity to work
with new/different technologies, if that is interesting to them
* Deploying it standalone maximizes optionality down the road: To hand
off ownership, or manage differently, or move into MongoDB, or whatever
* Deploying it standalone allows us to provide different SLAs for it

> Luke Crouch <mailto:lcr...@mozilla.com>
> July 29, 2014 8:17 AM

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn

--
Justin Crawford
Product Manager, Developer Relations | Mozilla
hoos...@mozilla.com

[Jeremie Patonnier]

Hi!

2014-07-29 16:17 GMT+02:00 Luke Crouch <lcr...@mozilla.com>:

> Are these enough to convince us to host the API under MDN?

For a v1 it seams easier to host it on MDN, but we should consider self
hosting at some point.

> What are the advantages of hosting it as a micro-service outside of MDN?
>

I see one advantage of doing this. If many people use the API to access the
data, any outage to that service will result on an outage on MDN and vis
versa. If the project is successful, it will possibly worth to move to self
hosting. But again, not a v1 concerne.

2014-07-29 16:24 GMT+02:00 Justin Crawford <hoos...@mozilla.com>:

>
> What are the advantages of hosting it as a micro-service outside of MDN?
>>
> * Deploying it standalone may make it more hackable by contributors: there
> will be fewer prerequisites, fewer dependencies, and a smaller and
> substantially less complex codebase to learn
> * Deploying it standalone may give MDN developers an opportunity to work
> with new/different technologies, if that is interesting to them
> * Deploying it standalone maximizes optionality down the road: To hand off
> ownership, or manage differently, or move into MongoDB, or whatever
> * Deploying it standalone allows us to provide different SLAs for it
>

+1

--
Jeremie
.............................
Web : http://jeremie.patonnier.net
Twitter : @JeremiePat <http://twitter.com/JeremiePat>

[John Whitlock]

I see six realistic options:

#1 The API is a separate Django project running in a 3rd-party PAAS under a
separate domain name, such as api.compat.mozilla.org.
#2 The API is a separate Django project running in a new Mozilla virtual
server, and appears under developer.mozilla.org by reverse proxy
#3 The API is a separate Django project running in MDN infrastructure, and
appears under developer.mozilla.org by reverse proxy (similar to
elasticsearch)
#4 The API is included as a Django app in the Kuma project
#5 The API is part of the Kuma project
#6 There is no API (additional kuma models, updated when the wiki changes,
with enhanced KumaScirpt).

#1 is definitely my preferred solution. It's simpler for me to develop,
and will maximize the chances that non-Mozillians will use the API and
contribute data and code. I agree with what others have said about the
benefits.

The first technical challenge to is easing authentication between kuma and
the new service. If we're going to allow data contributions from outside
MDN, then the API will need some user authentication flow, including login
pages, auth tokens, and click-through CC0 license agreements. However, the
first use case is data contributions from an MDN page. This can range from
no user interaction (requiring an auth handoff between kuma and the API,
and kuma to grow auth provider features) to just re-authenticating the user
(for example, with another Persona login).

The second challenge is selecting a PAAS and provisioning the server. I
can do the technical half. The other half is getting a Mozilla credit card
and signing up for a service, which will probably involve WebOps / IT.

The third challenge is procuring a DNS name and related SSL certificates,
which will definitely involve WebOps / IT.

Going with a different solution will reduce or eliminate these challenges,
but add their own. For example, solution #5 (part of the Kuma project)
means that we'll have to start with the standard release procedures for
this very beta code. Solution #4 means restricting the new code to the
kuma toolset - the Python version, Django version, caching solution, web
proxy, scaling solutions, etc.

My instinct is that it will be difficult to move between solutions in
future versions. If we start out with solution #4, we'll stay with
solution #4. Moving between solutions will be similar to the effort to add
another authentication solution to kuma, with the added challenge of little
benefit for MDN contributors - technically and politically hard.

If we're serious about outside contributors, v1 needs to be as stand-alone
as possible. The risk is there's no guarantee that if you build it, they
will come.

John

On Tue, Jul 29, 2014 at 9:17 AM, Luke Crouch <lcr...@mozilla.com> wrote:

> All,
>
> John Whitlock is working on the new backend [1] for the Compat Data
> project [2] that will power MDN's browser compatibility tables. [3]
>
> In a pull request [4], and at yesterday's planning meeting, I proposed we
> host the API under MDN at:
>
> https://developer.mozilla.org/compat/api
>
> Another option is to host the API as a micro-service app at a new domain,
> e.g.,:
>
> https://api.compat.mozilla.org/
>
> MDN gives us some advantages:
>
> * We won't need to provision any new servers from WebOps/IT
> * We can re-use existing push (Chief) and monitoring (New Relic) services
> * We can re-use MDN sessions for API authentication
>

> Are these enough to convince us to host the API under MDN? What are the

> advantages of hosting it as a micro-service outside of MDN?
>

[Jeremie Patonnier]

Hi!

2014-07-29 17:58 GMT+02:00 John Whitlock <jo...@factorialfive.com>:

> #1 is definitely my preferred solution. It's simpler for me to develop,
> and will maximize the chances that non-Mozillians will use the API and
> contribute data and code.
>

One of our long term goal is trying to convince other browser vendors to
directly contribute to that data base (no promises here a it is a highly
political subject). If we want to have a chance to achieve that, we need to
provide an entry point for the API which is 100% NOT Mozilla branded, like
said: api.arewecompatibleyet.org

> The first technical challenge to is easing authentication between kuma and
> the new service. If we're going to allow data contributions from outside

> MDN.
>

This is a mead range goal for us (meaning not for this year but this should
be possible in 2015)

> The second challenge is selecting a PAAS and provisioning the server. I
> can do the technical half. The other half is getting a Mozilla credit card
> and signing up for a service, which will probably involve WebOps / IT.
>
> The third challenge is procuring a DNS name and related SSL certificates,
> which will definitely involve WebOps / IT.
>

My guess is we could do a v1 on MDN in order to validate our most common
use cases then working on making the whole thing accessible by third party
out of MDN (solution #2 or #3 could ease a transition from embedded to
standalone with a quick early availability). But our goal are very clear
that third party contribution is a must have for the project.

> If we're serious about outside contributors, v1 needs to be as stand-alone
> as possible. The risk is there's no guarantee that if you build it, they
> will come.
>

I strongly support that point of view. And I say it again. Our main
objective for that project is doing it right (whatever it means) we are not
under time schedule pressure (yet). My personal wish would be to be able to
release a working v1 on MDN by the end of the year to ensure it's working,
but I will not jeopardize the future of that project to just have a
release. Of course, I'm not decision maker here (Ali and Justin are, as
content and product manager) but I will always support decision which turn
this project into something solid in the long run (actually it's so much
beneficial to the web platform and in line with the Mozilla top goals that
our worst case scenario should be a lack of interest from our management
rather than a blocker to do this clean and straight). I bet on Ali and
Justin to push the political side of that project in the right direction.
Let's focus on building a good platform for those data :D

Best,

[Les Orchard]

My drive-by $0.02: Microservices FTW!

I think MDN should be decomposed into more focused, simpler microservices wherever possible. The hosting details are worth working out with IT so that the service itself is simpler in the long run.

Further relevant reading: http://dev.otto.de/2014/07/29/scaling-with-microservices-and-vertical-decomposition/

> https://wiki.mozilla.org/MDN/Development/CompatibilityTables/Data_Requirements#Compatibility_Tables
> [4] https://github.com/jwhitlock/web-platform-compat/pull/1/files
> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn
>

--

Les Orchard <lorc...@mozilla.com>
{web,mad,computer} scientist