Clicks to edit button increased by over 40% with A/B testing!

[John Karahalis]

In our ongoing pursuit of quality, we recently tested textual variations to the edit button on all pages. Variations included EDIT PAGE (our baseline), EDIT , IMPROVE , and IMPROVE THIS DOC . Each variation was shown to a number of users (for a total (N) of 76,000 experiment participants) and we measured which variations were most effective at inspiring users to click.

The results are in, and we have a winner. Buttons labelled EDIT receive 40% more clicks than buttons labelled EDIT PAGE .

http://optimize.ly/~VoAkCt?token=20ca22118554616e752d

This experimentally validates the importance of user interface conventions. Many other platforms, including Wikipedia, also use the text EDIT . By using the same text on MDN, we eliminate the learning curve (however minor it may seem) and make the product more useful to a very significant number of people. User interface conventions are explained in detail by Donald Norman ( http://pages.citebite.com/e2b6r0q8s1oft ) and goodui.org ( http://goodui.org/#29 ). I strongly recommend that we consider conventions in all future interface design, so that we reap these same benefits elsewhere.

Next week, we will change the text to EDIT for all users. We also plan to experiment with other button properties (like color) in the future. If you have an idea for another experiment we can run, either on the edit button or elsewhere, please let us know.

--
John Karahalis
Mozilla
openjck.com

[David Bruant]

A
MA
ZING
!!!

Thanks John for this research!

David

Le 18/01/2014 01:43, John Karahalis a �crit :

[Gijs Kruitbosch]

Are there plans to allow editing by section? The current editing UI is
very cumbersome for editing large pages, especially in the source view.

~ Gijs

[Luke Crouch]

Section editing proves to be a difficult feature, but yes - we want to
re-enable it.

I think we could do another experiment where we put 'Edit' buttons on
each section of articles, and then open the full-page editor, but zoom
the editor window to the section that was clicked.

If we see a major up-tick in clicks, we'll know roughly how valuable the
section-editing feature could be.

-L

--
Q: Why is this email five sentences or less?
A: http://five.sentenc.es

[Jean-Yves Perrier]

The amount of click on edit is not the only metrics we should consider.
What is really interesting is how many "Save" it leads too. Another
metric is how much time we need to reach this save: people requesting a
section editing feature want it because it makes it easier to edit, not
because it allows them to edit.

Finally I don't think our page will look good with buttons scattered all
over the place.

On 19/01/2014 02:58, Luke Crouch wrote:
> Section editing proves to be a difficult feature, but yes - we want to
> re-enable it.
>
> I think we could do another experiment where we put 'Edit' buttons on
> each section of articles, and then open the full-page editor, but zoom
> the editor window to the section that was clicked.
>
> If we see a major up-tick in clicks, we'll know roughly how valuable
> the section-editing feature could be.
>
> -L
>


--

Jean-Yves Perrier
Technical Writer / Mozilla Developer Network

[John Karahalis]

I completely agree that we should also measure page saves.

We should remember that by proving more people click buttons labelled "EDIT", we are only proving that more people click buttons labelled "EDIT". We are not proving that more people actually edit, or that the buttons are easier to understand, or anything like that. These are additional claims, and they require additional evidence. (To be pedantic, we are not even proving that convention is the reason that more people click "EDIT" buttons, as imply in my original email, though I strongly suspect it is.)

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn

[Luke Crouch]

Are you suggesting we shouldn't change the button until we measure the
"Save" actions too?

How can we measure "our page looking good" ? :)

[John Karahalis]

We should feel comfortable changing the button on click data alone. We cannot be certain that more clicks mean more saves, but this seems much more likely than the alternative. We could run the numbers if we want to be sure, but it may not be worth the effort.

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

> From: "Luke Crouch" <lcr...@mozilla.com>
> To: "Jean-Yves Perrier" <jper...@mozilla.com>, "Gijs Kruitbosch"
> <gijskru...@gmail.com>, dev...@lists.mozilla.org
> Sent: Sunday, January 19, 2014 12:39:59 PM
> Subject: Re: Clicks to edit button increased by over 40% with A/B testing!

[John Karahalis]

(I should point out that my last email seemingly contradicts my email from fifteen minutes earlier. To clarify, we should measure page saves if we want to be certain that "EDIT" buttons lead to more saves, rather than just more clicks, but for the sake of practicality we can assume that it does without much risk.)

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

> From: "John Karahalis" <jkara...@mozilla.com>
> To: "Luke Crouch" <lcr...@mozilla.com>
> Cc: "Jean-Yves Perrier" <jper...@mozilla.com>, "Gijs Kruitbosch"
> <gijskru...@gmail.com>, dev...@lists.mozilla.org
> Sent: Sunday, January 19, 2014 12:49:58 PM
> Subject: Re: Clicks to edit button increased by over 40% with A/B testing!

> We should feel comfortable changing the button on click data alone. We cannot
> be certain that more clicks mean more saves, but this seems much more likely
> than the alternative. We could run the numbers if we want to be sure, but it
> may not be worth the effort.

> --
> John Karahalis
> Mozilla
> openjck.com

[Gijs Kruitbosch]

On 19/01/2014 14:50, Jean-Yves Perrier wrote:
> Finally I don't think our page will look good with buttons scattered all
> over the place.

Only show the button for one section at a time, when hovering over text
in that section? (this has obvious keyboard a11y issues, so you could
make it a user pref or make the section header focusable and then do the
same - bonus points for section navigation hotkeys)

~ Gijs

[Jean-Yves Perrier]

On 19/01/2014 17:49, John Karahalis wrote:
> We should feel comfortable changing the button on click data alone.

I do agree on this too.

[Jean-Yves Perrier]

Oh good idea! I have no objection about A/B testing in these conditions :-)

[Janet Swisher]

On 1/19/14 9:39 AM, Luke Crouch wrote:
>
> How can we measure "our page looking good" ? :)
>

I'm not sure if you're being serious or facetious. One way would be
through usability testing, looking for feedback like "it's ugly" or
"it's busy" or "why the hell are there all these edit buttons all over
the place?" Or we could just trust our design experts' professional
judgement.

--
Janet Swisher <mailto:jREMOVE...@mozilla.com>
Mozilla Developer Network <https://developer.mozilla.org>
Developer Engagement Community Organizer

[Luke Crouch]

Serious *and* facetious, as usual with me. I agree we should trust our
design & ux pro's on it.

[John Karahalis]

Agreed.

A/B tests measure what happens on screen, but visual appeal happens in the brain. The most reliable way to measure visual appeal is to ask users what they think (or use the expertise of our designers, who already know how users think).

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

> From: "Luke Crouch" <lcr...@mozilla.com>
> To: "Janet Swisher" <jswi...@mozilla.com>, dev...@lists.mozilla.org
> Sent: Monday, January 20, 2014 10:45:15 AM
> Subject: Re: Clicks to edit button increased by over 40% with A/B testing!

[Luke Crouch]

In that respect, we now have a company account with
http://www.usertesting.com/.

I think Holly and SUMO have used it to get those kinds of qualitative
answers about what users are thinking as they attempt to do certain
things on the site.