Kuma Report, November 2016

[jwhi...@mozilla.com]

Here's what happened in November in Kuma, the engine of MDN:

- Moved KumaScript Macros to GitHub
- Added Kabyle to MDN
- Shipped the Developer Newsletter Sign-up
- Promoted the Mozilla Foundation Fundraiser
- Improved the Integration Test Suite
- Shipped a Beta Sample Database
- Tweaks and Fixes

Here's what is planned for December:

- Finish 2016, Plan for 2017

Done in November
================

Moved KumaScript Macros to GitHub
---------------------------------
In November 2015, Florian Scholz (Elchi3 on GitHub) put together a proposal for
moving KumaScript macros to GitHub:

https://groups.google.com/forum/#!msg/mozilla.dev.mdn/PEuH7WW2Xtk/3y7Tv9_rEgAJ

Florian immediately started the difficult work of cleaning up macros. On the
development side, we knew we'd need a few other pieces to make this work, such
as:

1. A local development environment that includes macro processing by default,
2. A database with enough data to develop macros locally, and
3. Updates and changes to the KumaScript engine.

Ryan Johnson (escatonne on GitHub) tackled this last part in November. He extracted the
macros from the Kuma database, adding them to the KumaScript repository along
with the names of the MDN contributors that developed them over the last
decade. He resurrected and improved the file-based macro loader, and turned it
on by default.

The change was shipped on November 28th, and has been running well. You can
now see all the macros in the KumaScript repository, and even submit a PR:

https://github.com/mozilla/kumascript/tree/master/macros

There is still work to do (and there will always be work to do), but this is
a huge step toward better macros on MDN.

Added Kabyle to MDN
-------------------
We've had a freeze on new languages on MDN, as we adjust to the team
transitions of 2016 and take a closer look at localization on MDN. We've
published a new process for adding languages to MDN:

https://developer.mozilla.org/en-US/docs/MDN/Contribute/Localize/Starting_a_localization

The Kabyle language team, led by Belkacem Mohammed, was the first to try out
the new process. Over the last few months, he and his team translated all of
the MDN interface to Kabyle:

https://pontoon.mozilla.org/projects/mdn/

Long-time MDN contributor Sphinx has volunteered to mentor the Kabyle team, so
we enabled it as an official MDN language:

https://developer.mozilla.org/kab/

Congratulations to the Kabyle team!

Shipped the Developer Newsletter Sign-up
----------------------------------------
Mozilla is starting a new developer-focused newsletter, and we're promoting it
on MDN. You can sign up from the MDN homepage, or from the bottom of any
article page.

Promoted the Mozilla Foundation Fundraiser
------------------------------------------
Did you know that MDN is part of Mozilla, a non-profit that works to keep the
Internet a global, public resource? Support the work of the Mozilla Foundation
by giving a tax-deductible donation during our December fundraiser. Or, just
admire the pixel art on the MDN homepage.

Improved the Integration Test Suite
-----------------------------------
Stephanie Hobson and Matt Brandt continue to refine the integration test suite,
and to define the scope of integrated testing.

The Site Reliability Engineering team doubled this quarter! Dave Parfitt
(metadave in IRC and GitHub) has joined Josh Mize in building our next-generation
infrastructure. The two of them took some time to fix issues in running the
integration tests in our Jenkins environment, as well as teach us more about
how this will work in the future.

Shipped a Beta Sample Database
------------------------------
The sample database is taking longer than planned, and we did not ship
everything we were hoping for November. We did ship a beta version of the
database to the #mdndev IRC channel and staff, and have gotten positive
feedback on it.

The beta database includes:
- All the pages linked from the homepage, their translations, 5 historical revisions, and stub accounts for those authors
- Settings to enable KumaScript rendering
- Waffle flags and switches used to enable features
- User groups and permissions used to control feature access
- Test accounts and pages used in integrated testing
- Tags and settings used to configure the search engine

Contact us in #mdndev if you want to try the beta database for yourself.

Tweaks and Fixes
----------------
Our staff and awesome contributors continue to fix bugs and add features. Here
are some of them, by bug number:

* 1239756 - Purge dead tags (safwanrahman)
* 1267197 - Fix locale redirects for zones (jbradberry)
* 1299227 - Update production database schema (jwhitlock)
* 1308168 - Always link to English on non-translated pages (safwanrahman)
* 1308222 - Fix error when moving a tree of pages (safwanrahman)
* 1315078 - Localize email placeholder (SphinxKnight)
* 1318821 - Fix syntaxbox in CKEditor toolbar (a2sheppy)

Planned for December
====================

Finish 2016, Plan for 2017
--------------------------
December has 31 days, but not a lot of them are appropriate for heads-down
software development. We take a week for our all-company meeting, where we
prioritize face-to-face discussions. Many of us are taking a week or more
of vacation this month as well.

When we have our in-person time, we'll be reflecting on the year, talking
about what went right, what went wrong, and what we'd like to do differently
in the next year. We'll think about our own plans, and how to best integrate
the energy and passion of our contributors.

When we are working on code, we'll continue with the work in progress, with
the goal of shipping software and documentation:

- The integration test suite
- 3rd-party software updates in Kuma and KumaScript
- Shipping the Sample Database
- Determine the workflow for KumaScript macro changes

Or, we'll just prepare some hot drinks and spend time with friends and family.
You'll have to read the December report to see how it worked out.

[jwhi...@mozilla.com]

On Friday, December 2, 2016 at 7:15:07 AM UTC-10, jwhi...@mozilla.com wrote:
> Here's what happened in November in Kuma, the engine of MDN:
>
> - Moved KumaScript Macros to GitHub

We've discovered that page rendering is 75% faster when loading KumaScript macros from disk:

https://www.dropbox.com/s/kh7zzdkbarjqeje/KumaScript%20Move%20to%20Github.png

However, MDN's viewers don't see any change:

https://www.dropbox.com/s/5v6yqawuxeko5v9/Kuma%20Performace%20after%20macro%20change.png

To understand why, you need to know more about page rendering on MDN.

Previously, KumaScript macros were stored in the Kuma database, like other MDN reference pages. They were also edited on MDN, restricted to a core team of macro editors. When an MDN page is saved, any KumaScript macros used on the page need to be processed into plain HTML. The KumaScript backend service renders these pages, and the resulting HTML is saved in the database for future viewers.

Here's what happens (or happened last month) when an MDN reference page is saved:

https://wiki.mozilla.org/images/thumb/2/2b/Kumascript.png/1000px-Kumascript.png

1. Kuma sends a "page render" request to the KumaScript server
2. The KumaScript server asks the Kuma API for the page source (using the ?raw URL parameter - see https://developer.mozilla.org/en-US/docs/MDN/Contribute/Tools/Document_parameters for more).
3. KumaScript macros are found in the raw page.
4. The KumaScript server requests the source of the macros, again using the ?raw parameter.
5. Macros may refer to other macros, and macros may require further API calls to construct sidebars, navigation, etc., resulting in many Kuma API requests.
6. The HTML version is fully rendered, and returned in the original "page render" request.

This process is slow, an average of 8 seconds last month. It is also infrequent, happening once or twice a minute during normal processing. 8 seconds is forever in web requests, so there are several layers of caching to prevent MDN viewers from seeing the effects of page rendering.

We didn't know this, but requesting the macro sources was a significant part of the page rendering process. In hindsight, it makes sense - macros include other macros, and then can request further data. This adds a serial chain of requests to the page rendering process. When the macros are loaded from files, the steps are faster, so the whole process is faster.

https://github.com/mozilla/kumascript/blob/master/overview.png

The page rendering process is now around 2 seconds, a 75% reduction. 2 seconds is still a long time for web requests, so caching is required. However, if we can further reduce this to 1/2 a second, we could potentially remove internal caches and rely on external caching and CDNs.

We only know about the reduction because Luke Crouch and David Walsh worked to add basic New Relic monitoring of KumaScript. The monitoring isn't deep enough to diagnose why the page rendering process is slow, but it did see this big performance improvement. This makes it more likely will add more timing hooks, so we can find the next 75% speed increase.