Proposal for rendering BrowserCompat tables on MDN

[Luke Crouch]

We've had a recurring choice on whether to render BrowserCompat tables on
MDN server-side with KumaScript, or client-side with JavaScript. The major
concerns are:

Server-Side with KumaScript:

1. Scaling KumaScript up to handle the extra load of communicating with
BrowserCompat API
2. Adding re-render trigger for all pages that include certain macros
3. Re-assessing #1 based on #2

Client-Side with JavaScript:

1. MDN loses compatibility content from search engine indices
2. Users without JS get no compatibility content

After some separate discussions, it seems that server-side KumaScript is
much more risky*, so we're coming together around a proposal to move
forward with a combined python & client-side implementation:

1. Publish 2 BrowserCompat API endpoints for features: JSON [1] and
(partial) HTML [2]
2. When a page on MDN is saved, we will re-fetch its latest compat HTML
content from BrowserCompat, so we can include it in the server-side
rendering (like we currently do with quick links, subnav, toc, and summary
HTML)
3. Add the client-side JS using the JSON endpoint to overlay the most
recent BrowserCompat table on top of the (possibly stale) table

I'm confident this is the best holistic approach for us:

* It mitigates the risks in using KumaScript
* It mitigates the risks in using client-side JavaScript
* It enables us to ship a quality implementation sooner

What do you all think?

Thanks,
-L

* Note: we *are* investing time and energy into making KumaScript more
reliable for the future [1][2], but it's currently too risky for the
BrowserCompat implementation

[1] https://github.com/mozilla/kumascript/pull/51
[2] https://github.com/mozilla/kumascript/pull/53

[Stephanie Hobson]

Yup, still like this approach.

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn
>

[Jamon Camisso]

Idea in terms of client-side JS: take a look at prerender.io
(https://github.com/prerender/prerender)

I'm certain it could be added to the Kuma stack fairly easily. That or
just rendering directly for SEO or javascript-less users with Phantomjs.

Just a thought.

Cheers, Jamon

[m...@renoirboulanger.com]

Hi Luke, List,

On Tuesday, September 1, 2015 at 11:29:30 AM UTC-4, Luke Crouch wrote:
[snip]

+1

Nothing's cheaper than what's already done.

Better having something than nothing.

>
> After some separate discussions, it seems that server-side KumaScript is
> much more risky*, so we're coming together around a proposal to move
> forward with a combined python & client-side implementation:
>
> 1. Publish 2 BrowserCompat API endpoints for features: JSON [1] and
> (partial) HTML [2]

+1

Let's make the API URL to keep about what you want, header to specify how.

... A browser would basically do:

curl -H 'Accept: text/html' browsercompat.herokuapp.com/api/v1/view_features/1138


... Getting raw JSON string:

curl -H 'Accept: application/vnd.api+json' browsercompat.herokuapp.com/api/v1/view_features/1138

... A partial could be:

curl -H 'Accept: application/vnd.api+partial' browsercompat.herokuapp.com/api/v1/view_features/1138



<fork rel="caching">
If I may make a tangent about HTTP caching;

1. Normalize what can be sent as accept to ensure we don't miss opportunities to serve cacheable content

For example, if we only have one version of a full page HTML. A request with `Accept: application/xhtml+xml`, should just rewrite that header to `Accept: text/html`

That way, the caching won't create another version.


Once we want our API to provide into a language, we could then make the same. Otherwise we miss caching opportunities.

e.g. we support english, french, greek.

A header

Accept-Language:fr-FR,fr;q=0.8,fr-CA;q=0.6,en-US;q=0.4,en;q=0.2

Would be simplified to

Accept-Language: fr



2. Add `Vary: Accept` header

So that we don't get the HTML version from cache when we asked JSON.
</fork>

> 2. When a page on MDN is saved, we will re-fetch its latest compat HTML
> content from BrowserCompat, so we can include it in the server-side
> rendering (like we currently do with quick links, subnav, toc, and summary
> HTML)
> 3. Add the client-side JS using the JSON endpoint to overlay the most
> recent BrowserCompat table on top of the (possibly stale) table
>
> I'm confident this is the best holistic approach for us:
>
> * It mitigates the risks in using KumaScript
> * It mitigates the risks in using client-side JavaScript
> * It enables us to ship a quality implementation sooner
>
> What do you all think?

Love it!

Renoir Boulanger

[John Whitlock]

I agree with the approach. It sounds like technical details are up for
discussion, so here's my mental implementation:

- Take David Walsh's Kumascript and convert to browser-based JavaScript
(it's 98% of the way there), so it can render like
https://browsercompat.herokuapp.com/view_feature/730
- Copy CSS / image assets from kuma to browsercompat. This is needed
because the HTML and JS will need to know the paths to assets after any
cache-busting name conversions.
- Add JS renderer of specifications section w/ assets
- Add a few more endpoints:

https://browsercompat.herokuapp.com/view_feature/web-css-display.mdn.html

Serves HTML fragment of specification and compatibility sections in MDN
format, with appropriate <h3> headers

https://browsercompat.herokuapp.com/view_feature/web-css-display.mdn.spec.html

Serves HTML fragment of specification table in MDN format, without the <h3>
header

https://browsercompat.herokuapp.com/view_feature/web-css-display.mdn.compat.html

Serves HTML fragment of the browser compatibility table in MDN format,
without the <h3> header

My preference is to render these HTML fragments using something like
PhantomJS from the Javascript. This way, the JavaScript is the canonical
way to generate the MDN representations, and the HTML fragments act like
pre-rendered JS. This would be similar to prerender.io but without the
third party service or the node dependency.

Technically, there is no reason this couldn't be done from the
kuma/kumascript codebase:

1) CSS/JS/image assets served from kuma CDN
2) KumaScript loads client-side JS and "pre-renders" HTML tables
3) Small client-side JS loads latest API data, and, if there is an update,
loads the full table-generating JS to replace in-page tables

However, in terms of implementation speed, taking kuma/kumascript out of
the equation gets us working code in a few weeks instead of months.

John

[Stephanie Hobson]

Talking with John W about this... if we're going to use JS and Phantom to
render the HTML it makes sense for this to be part of the contribution
interface app. The contribution display shares assets and design with the
table display so this way the templates can be managed in one place.

This is all theoretical for now ;)

S.




On Tue, Sep 8, 2015 at 11:48 AM, John Whitlock <jo...@factorialfive.com>
wrote:

[John Whitlock]

The MDN dev team met in Madison, WI for a work week, and made some progress
on MDN and BrowserCompat features, as well as the go-to-production plan.

We couldn't find any flaws in the idea of collecting the MDN pieces into
the the contribution app. These pieces include:
- A contribution interface for changing and adding contribution data
- JavaScript to convert raw API data into MDN-specific HTML tables
- Assets (images, fonts, etc) not already on MDN
- A headless JS renderer (PhantomJS) for pre-rendering content as HTML
- One source code repository to coordinate changes in MDN rendering and
workflow

This doesn't change the refresh strategy, just where the work occurs:
- The BrowserCompat API publishes JSON
- The Contribution App publishes HTML, rendered from JSON using the
headless JS renderer
- KumaScript injects the rendered HTML into the page
- On MDN page load, the page asks the API if there is newer JSON. If there
is, it loads the table-generating JS and replaces the source HTML.

The MDN Contribution site can use the headless JS renderer to compile JS
templates, and can use the table-generating JS for previews and other UI
elements.

Things may be adjusted on the way to deployment, but the strategy is the
same: inject HTML at page edit, rebuild as needed on page load, and move
scaling problems to the new services. We didn't make progress on this plan
(or write bugzilla tickets), but instead focused on API updates required
for the contribution interface.

The other discussion in Madison is the 'skateboard' plan, to get a working
version of the new compatibility displays into production. Stephanie will
present details of the plan, and you'll get to see for yourself soon on
staging. We have most of the pieces already, but we expect the project
priorities will shift to automating manual steps, fixing bugs, and
improving the skateboard. The two-phase rendering is not required for the
skateboard plan, so it may be a while before either part is implemented.

John

On Tue, Sep 15, 2015 at 2:32 PM, Stephanie Hobson <sho...@mozilla.com>
wrote: