[BC] Micro services high level architecture

[Jeremie Patonnier]

Hi!

After the recent discussions about project milestones, l10n and some
others, I found necessary to clarify the high level vision of the various
"micro services" that are expected to live under the browsercompat.org
umbrella.

Rather than long discussion including terms that everybody will understand
in very different way I produced a schema that summarize my current
understanding of things:

- https://wiki.mozilla.org/images/c/c8/Micro-services-architecture.png


So I have two question for you:

- Do you think that schema actually represent our target for the project
(and why) ?
- Is it accurate enough or is there something missing or something that
need to be clarified ?

Thank you for your help :)

Best,
--
Jeremie
.............................
Web : http://jeremie.patonnier.net
Twitter : @JeremiePat <http://twitter.com/JeremiePat>

[John Whitlock]

Looks like the diagram I'd draw, with some changes around the importer and
the MDN / API linkage

The importer / DB relation is read/write, and there's a use relation to the
API.

The importer writes to the database, to cache MDN pages and keep track of
data issues on MDN. These importer tables are separate from the BC tables,
and could be in a different database, and will be dropped when the importer
is dropped. It could be modeled with an "importer database", left out as a
detail, or continue in the BrowserCompat DB bubble.

The importer reads existing compatibility data from the BC database.
Architecturally, it should read this from the API, but development time is
short, so I took some shortcuts. I've centralized access in mdn/data.py,
in case the importer needs to be split into a separate app in the future.

When it commits compatibility data from MDN, it uses the API, so that the
API's validators will run against the MDN data. All validation problems
should have been detected as data errors or issues, but it's an extra step
for data consistency and safety. That's the USE relationship.

As for MDN to API - there is (was?) a requirement that MDN user and IP bans
carried over to the BC API, that MDN admin permissions sync with BC admin
permissions, and that you can go from an MDN user's profile to their BC
contributions, and maybe the other way too. This will require a "use"
relationship back to MDN for authentication and authorization.

John

[Jeremie Patonnier]

Hi!

2016-01-26 18:11 GMT+01:00 John Whitlock <jwhi...@mozilla.com>:

> Looks like the diagram I'd draw, with some changes around the importer and
> the MDN / API linkage
>

Arg! Sorry, I miss that diagram. Where can I find it?
But that's good news that we are close, it means that I start understanding
the whole things :)

> The importer / DB relation is read/write, and there's a use relation to
> the API.
>
> The importer writes to the database, to cache MDN pages and keep track of
> data issues on MDN. These importer tables are separate from the BC tables,
> and could be in a different database, and will be dropped when the importer
> is dropped. It could be modeled with an "importer database", left out as a
> detail, or continue in the BrowserCompat DB bubble.
>

I will update the schema to reflect this. However, on a practical future
proof point, it could be a good idea to isolate that specific Importer DB
from the BC DB, that will make things easier to kill the importer when the
time will come and it will prevent the temptation to do some black magic
with those unreliable tables (which will increase the technical debt and we
should avoid that as much as possible).

> The importer reads existing compatibility data from the BC database.
> Architecturally, it should read this from the API, but development time is
> short, so I took some shortcuts. I've centralized access in mdn/data.py,
> in case the importer needs to be split into a separate app in the future.
>

As the plan is to get ride of the importer at some point, "shortcuts" are
okay (even if it doesn't exactly smell like roses).

> When it commits compatibility data from MDN, it uses the API, so that the
> API's validators will run against the MDN data. All validation problems
> should have been detected as data errors or issues, but it's an extra step
> for data consistency and safety. That's the USE relationship.
>

Ah, okay. As long as writing is safe, that's all good :)

> As for MDN to API - there is (was?) a requirement that MDN user and IP
> bans carried over to the BC API, that MDN admin permissions sync with BC
> admin permissions, and that you can go from an MDN user's profile to their
> BC contributions, and maybe the other way too. This will require a "use"
> relationship back to MDN for authentication and authorization.
>

I haven't add the auth logic here as the exact auth mechanism and
relationship with MDN remain fuzzy at that point (mainly due to the fact
that there are pending questions around MDN getting ride of Persona). I
know that's a point that need to be addressed ASAP and it will have its own
set of dedicated discussion. At that point I will tend to advocate making
things simple and get ride of the requirement of having MDN and BC
permissions sync (I postpone any SSO and other similar Auth concern up to
the Star Ship milestone).

[John Whitlock]

> 2016-01-26 18:11 GMT+01:00 John Whitlock <jwhi...@mozilla.com>:
>
>> Looks like the diagram I'd draw, with some changes around the importer
>> and the MDN / API linkage
>>
>
> Arg! Sorry, I miss that diagram. Where can I find it?
> But that's good news that we are close, it means that I start
> understanding the whole things :)
>

I haven't drawn this diagram, except on the back of a napkin. Sorry for the
unclear sentence.

I haven't add the auth logic here as the exact auth mechanism and
> relationship with MDN remain fuzzy at that point (mainly due to the fact
> that there are pending questions around MDN getting ride of Persona). I
> know that's a point that need to be addressed ASAP and it will have its own
> set of dedicated discussion. At that point I will tend to advocate making
> things simple and get ride of the requirement of having MDN and BC
> permissions sync (I postpone any SSO and other similar Auth concern up to
> the Star Ship milestone).
>

I also want to delay the auth discussion, except for the problem of emails.
There's a few needs for emails:

1) Password recovery (if the BC service has "native" username/password
accounts)
2) Account recovery (if the BC service has "native" accounts, "Someone
opened bug X asking to recover this account")
3) Licensing contact infö (if we need to contact an author about licensing
changes)
4) Other service notification (could be served by a mailing list)

So, I can think of ways to get rid of email entirely, if we do away with
"native" BC accounts. This is the only bit of PII in the database, which
means we get more freedom on where things get hosted.

At this point, I've already added native accounts, so the win of going
social-only would be removing some code. So, I think we can continue to
kick the auth can down the road, and leave it out of diagrams. Or decide in
Q1 and work toward that decision.

John

[Stephanie Hobson]

With John's suggestions I think this covers things well. Do we want to try
to represent the l10n stuff on the diagram?

S.

On Tue, Jan 26, 2016 at 11:09 AM, John Whitlock <jwhi...@mozilla.com>
wrote:

[Jeremie Patonnier]

Not before the next project meeting where I wish to bring the discussion
about l10n to make sure we have some agreement (I expect: yes, but who
knows ;).

[Jeremie Patonnier]

Oh, and one last thing just to be sure: is everybody okay to build that
HTML table renderer service? If so, I need to add it to the milestones, and
the pending question is which one (Bicycle, Scooter or Motorbike). My
personal guess would be Scooter as it doesn't seem needed for Bicycle
(which I would like to keep an light as possible) and Motorbike looks
already quite heavy.

[John Whitlock]

We may start work on that sooner, but scooter looks like a good milestone
to require it (as a display preview to go with the features & support C&M)

[Stephanie Hobson]

Yeah, scooter seems like the place we will be forced to sort out the
technical architecture for that.

On Wed, Jan 27, 2016 at 12:13 PM, John Whitlock <jwhi...@mozilla.com>
wrote:

[Jean-Yves Perrier]

Hi

I'm a RACI 'I' and not a 'C' in this area so take this as a naive question.

I wonder why we need an HTML table renderer service that early. Couldn't
the macro on Kuma do the job until later (like the Plane)?

On 27/01/2016 19:52, Jeremie Patonnier wrote:
> Oh, and one last thing just to be sure: is everybody okay to build that
> HTML table renderer service? If so, I need to add it to the milestones, and
> the pending question is which one (Bicycle, Scooter or Motorbike). My
> personal guess would be Scooter as it doesn't seem needed for Bicycle
> (which I would like to keep an light as possible) and Motorbike looks
> already quite heavy.
>

Jean-Yves Perrier
Senior St. Project Manager / Documentation Project / MDN

[Jeremie Patonnier]

2016-01-28 19:18 GMT+01:00 Jean-Yves Perrier <jype...@gmail.com>:

> I'm a RACI 'I' and not a 'C' in this area so take this as a naive question.
>

"I" people are always encourage to ask questions, that's the whole point of
being "I" :D

> I wonder why we need an HTML table renderer service that early. Couldn't
> the macro on Kuma do the job until later (like the Plane)?

Yes, as long as we are talking only about MDN. However, it appears that
many other micro services from BC could benefit from such a rendering
service. So rather than building another temporary solution and increase
our technical debt, it's better to build that service ASAP and make all the
others depend on it.

That said, yes, the current display solution on MDN can leave longer and we
can (and will) delay the switch to that new service for MDN display as long
as possible. however, it seems reasonable to not delay that switch longer
than the Car milestone (as it's the milestone targeted to display the new
tables to the whole MDN audience) again to keep the technical debt under
control.

Hope that makes things clearer.

Best,

[Sebastian Zartner]

On 28 January 2016 at 21:07, Jeremie Patonnier

<jeremie....@gmail.com> wrote:
> 2016-01-28 19:18 GMT+01:00 Jean-Yves Perrier <jype...@gmail.com>:
>
>> I'm a RACI 'I' and not a 'C' in this area so take this as a naive question.
>>
>
> "I" people are always encourage to ask questions, that's the whole point of
> being "I" :D
>
>
>> I wonder why we need an HTML table renderer service that early. Couldn't
>> the macro on Kuma do the job until later (like the Plane)?
>
>
> Yes, as long as we are talking only about MDN. However, it appears that
> many other micro services from BC could benefit from such a rendering
> service. So rather than building another temporary solution and increase
> our technical debt, it's better to build that service ASAP and make all the
> others depend on it.

The tables displayed on browsercompat.org may be different to the ones
shown at MDN due to a different use case. And other clients can build
their own UIs from the data. So I wonder, why do we actually need an
HTML renderer service shared between clients like MDN and the server
(i.e. browsercompat.org) at all? In my opinion this should be feature
only used by browsercompat.org.

Sebastian

[John Whitlock]

TL;DR - The KumaScript macro is a temporary solution until the rendering
service is ready. The primary use of the rendering service will be to
render the tables on MDN, to solve the problem of stale data. Use by other
services or third parties is secondary.

The long version...

Occasionally, we really want to refresh a large number of pages on MDN, but
we just can't. For example, when HTML5 switched from a Living Standard to a
W3C Recommendation, a lot of pages continued to show Living Standard for
months. This is because page refreshes are expensive, and a full refresh
of the site is measured in days. We currently refresh pages when they are
saved, or when someone manually force-refreshes the page. This is still
enough to cause strain on the site and require IT involvement. A better
refresh story for MDN is still a goal, but there are other goals we're
working toward first, like reducing page load time to an average of 2
seconds, and moving the infrastructure.

The refresh problem gets worse with BrowserCompat, because there are many
ways to update the compatibility data without updating the MDN page. Some
look like a current KumaScript update (update a specification from a
Candidate Recommendation to a Recommendation, add a localization for
"Firefox for Desktop") but even "traditional" edits like adding a supported
version for a feature are going to happen against the API, on the
contribution and moderation site, or even by a script directly against the
API. Updating a single bit of data, like promoting a Firefox version from
beta to the current released version, can impact hundreds to thousands of
pages on MDN. MDN refresh is not a viable solution at this time.

We thought of several solutions in Q3 2015 [1], but narrowed in on a
heavily cached site that serves up HTML fragments for MDN [2]. The workflow
will be:

1) On MDN page save, KumaScript will ask the render service for localized
HTML fragment for the current page, and inject this HTML into the rendered
MDN page. This looks a lot like the EmbedCompatTable macro [3], but the
table construction code is in the rendering service, and the KumaScript
portion is reduced to talking to the service, returning the results, and
error handling.
2) When a user views the MDN page, a bit of JS will ask the render service
for new localized HTML.
3) If nothing has changed since the page was saved, the render service will
return a "204 No Content" message, which, as implied, has no content. To be
more precise, a cache server in front of the render service will return the
204 message, since the JS will send headers it can use to confirm that the
MDN page is up-to-date.
4) When compatibility data changes, the caches will be invalidated, making
fresh content available. In the case of feature data, the cache may be
force invalidated, so that the MDN page will instantly get the updated
data. In the case of changes with a wide impact, like browser name
localizations, it may take 30-60 minutes for caches to expire and notice
the new value.
5) When the in-page JS asks the render service for new, localized HTML, the
cache server will recognize that new data is available, and ask the render
service to generate the new HTML. The MDN user briefly sees the "old" data,
which is then replaced with the latest compatibility data. The new rendered
page is stored in the renderer's cache server for future requests from the
same MDN page.

If something goes wrong (the renderer service is down, the user has
disabled JS, the user is a search engine crawler that doesn't execute JS),
then they will get the data at the time the page was last saved or
refreshed. Users that know the trick can still use force-refresh.

The table rendering service may be useful in the contribution and
moderation interface. The UI for adding and updating information will be
different from the table display, but we can potentially display a preview
of how the table will look with the new data. It also may be useful in
moderation, for displaying how the modification impacted the table
displayed on MDN. These may use the rendering service, but are more likely
to directly use the javascript used by the rendering service.

While I think we have a good infrastructure planned for wide, 3rd-party use
of the API, our first goal is to get the one site we control using it.
This includes the rendering service, which will provide MDN-specific HTML
fragments, which may or may not be usable on other websites. We'll
recommend that 3rd parties build their own presentations by working against
the API. We can plan a more general rendering service when there is a
demand.

John Whitlock
jwhitlock on #mdndev

[1] https://old.etherpad-mozilla.org/mdn-compat-rebuild-solutions
[2]
https://groups.google.com/forum/#!searchin/mozilla.dev.mdn/refresh/mozilla.dev.mdn/CkBIuYNwgU8/9gXLEeRaAQAJ
[3] https://developer.mozilla.org/en-US/docs/Template:EmbedCompatTable

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn
>

[Jeremie Patonnier]

Hi :)

+1 with John, and some clarification on a functional standpoint:

2016-01-29 9:45 GMT+01:00 Sebastian Zartner <sebastia...@gmail.com>:

> On 28 January 2016 at 21:07, Jeremie Patonnier
> <jeremie....@gmail.com> wrote:
> > 2016-01-28 19:18 GMT+01:00 Jean-Yves Perrier <jype...@gmail.com>:
> >> I wonder why we need an HTML table renderer service that early. Couldn't
> >> the macro on Kuma do the job until later (like the Plane)?
> >
> > Yes, as long as we are talking only about MDN. However, it appears that
> > many other micro services from BC could benefit from such a rendering
> > service. So rather than building another temporary solution and increase
> > our technical debt, it's better to build that service ASAP and make all
> the
> > others depend on it.
>
> The tables displayed on browsercompat.org may be different to the ones
> shown at MDN due to a different use case. And other clients can build
> their own UIs from the data. So I wonder, why do we actually need an
> HTML renderer service shared between clients like MDN and the server
> (i.e. browsercompat.org) at all? In my opinion this should be feature
> only used by browsercompat.org.

Absolutely, but such a rendering service is not necessarily meant to
produce just only one possible output.

If it is made as an independent service, it will be possible to improve it
over time to provide various rendering option with a nice GET REST API to
request custom HTML tables.

As soon as this service is ready to be autonomous it could even become a
sub project of the BC project in the long term for those who wish to make
it smarter.

Of course that remain a "basic" entry point for those who just want to
consume an HTML read only version of the data. For people who wish to build
more complex things, the data API is still there.

At the beginning it is intended to have this service made to accommodate
basic MDN needs and internal BC needs. But, as define in the Space Ship
milestone we want to build different displays. Having such a rendering
service will help us on many aspect.

[Sebastian Zartner]

Thank you for the detailed description and explanation!
I can now see the point for having this rendering service.

Sebastian

On 29 January 2016 at 16:52, Jeremie Patonnier <jeremie....@gmail.com>
wrote: