A/B test results: Sticky page buttons

[John Karahalis]

Last summer, Akshay added a feature where page buttons (*Languages*, *Edit*,
etc.) would stick to the top of the page when scrolled. Over the past few
weeks, we enabled the feature for some users to study how it affected
behavior. We ran the experiment with a total of 700,000 users, half of them
seeing sticky page buttons and half of them seeing normal page buttons.

The results are in. When these buttons are sticky, more users click the
edit button (+73%), more users arrive on the edit page (+41%), more users
submit completed edits (+38%). These are big improvements.

http://optimize.ly/~VlSVIb?token=ac6679f3bafcc4bfec70#view=2

The last statistic, completed page edits, is especially important. In the
past, we could only measure button clicks. We had to presume that more
clicks meant more completed edits, but we had no way of knowing for sure.
This time there's no doubt. When page buttons are sticky, more people edit
pages.

Moving forward, we should prioritize work to make buttons sticky on all
pages [1], including pages without tables of content [2].

Thanks to Akshay for making this a reality. As always, let me know if there
are any other A/B tests you think we should run.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1124199
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1124215

--
John Karahalis
Mozilla
openjck.com

[Justin Crawford]

Awesome! Nice work.

What is required to make this the default now? Just flipping a waffle flag?
Is there any reason to wait?



Justin Crawford
Product Manager, MDN
hoos...@mozilla.com

On Wed, Jan 21, 2015 at 9:25 AM, John Karahalis <jkara...@mozilla.com>
wrote:

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn
>

[John Karahalis]

Just some rework to the JavaScript. Currently, it checks the value of a
data-sticky attribute, which can be set to either *true* or *false*, to
determine if the buttons should be sticky. We just need to skip that check
and always make the buttons sticky.

Shouldn't take much time at all, so I'll submit a pull request as soon as
my local environment is working again. The TOC improvement would take more
work.

On Wed, Jan 21, 2015 at 11:42 AM, Justin Crawford <hoos...@mozilla.com>
wrote:

[Sebastian Zartner]

Really interesting data! Personally I wouldn't have expected the number of
edits to increase that much when people don't have to scroll to click the
buttons.

Note that I created the related bug 1108474
<https://bugzilla.mozilla.org/show_bug.cgi?id=1108474> a while ago. Will
that also be covered by the change?

Sebastian

[Akshay Aurora]

Wow, I had almost forgotten about this feature. I'm sure sheppy will
be the happiest person!

Let me know if I can be of more help on this, I'd love to catch up and
start contributing again.

On Thu, Jan 22, 2015 at 2:11 AM, Sebastian Zartner
<sebastia...@gmail.com> wrote:
> Really interesting data! Personally I wouldn't have expected the number of
> edits to increase that much when people don't have to scroll to click the
> buttons.
>

> Note that I created the related bug 1108474 a while ago. Will that also be

--
AA

[John Karahalis]

Hi Sebastian,

The "Compare Selected Revisions" button won't be covered by this change. It
shouldn't be too hard to add support for that button, though. I've also
recommended it to Akshay as something that might be fun to work on.

On Wed, Jan 21, 2015 at 3:41 PM, Sebastian Zartner <

sebastia...@gmail.com> wrote:

> Really interesting data! Personally I wouldn't have expected the number of
> edits to increase that much when people don't have to scroll to click the
> buttons.
>
> Note that I created the related bug 1108474

> <https://bugzilla.mozilla.org/show_bug.cgi?id=1108474> a while ago. Will

John Karahalis
Mozilla
openjck.com

[William Bamberg]

Is it OK that these buttons now overlap the content:
http://i.imgur.com/7EVxykc.png?

Will



On Wed, Jan 21, 2015 at 8:45 AM, John Karahalis <jkara...@mozilla.com>

[John Karahalis]

Thanks for the bug report. This happens when the table of contents is
collapsed. I opened bug 1126461
<https://bugzilla.mozilla.org/show_bug.cgi?id=1126461> and I'll take a look
today.

On Tue, Jan 27, 2015 at 2:55 PM, William Bamberg <wbam...@mozilla.com>
wrote:

[Stephanie Hobson]

There were a few bugs filed once we launched this to the wider audience,
unfortunately none of the front-end devs have time to work on them right
now so we (we = me and John) are going to roll back the change that made it
on-by-default until someone has time to address the bugs.

Thanks,
Stephanie.


On Tue, Jan 27, 2015 at 12:11 PM, John Karahalis <jkara...@mozilla.com>
wrote: