A/B test results: "Sign in with Persona" vs. "Sign in" and others

[John Karahalis]

Over the past two weeks, the MDN team tested four variations to its sign in button and measured the number of clicks each received. The variations were:


* Sign in with Persona (our baseline)
* Sign in
* Sign in / Register
* Sign in / Create account

We expected all three alternatives to outperform the baseline. We expected "Sign in" to outperform the baseline because it's more conventional, and because it does not use the word "with" (which may have earned a bad reputation on the Web). We expected the other two to outperform the baseline because, unlike the baseline, they do not incorrectly imply that the button is only useful to returning users.

Even after running the experiment with 100,000 users, none of the variations stood out as a statistically significant improvement over the others. This may sound like bad news--it isn't! These are not the results we wanted, but they are results! They demonstrate that using any of these alternatives would have been a mistake (at least as far as click rates are concerned). Doing so would mean dropping the Persona branding with no real benefit. We are no closer to understanding what changes we should make to the button to improve click rates, but we do know what changes we should not make.

http://optimize.ly/~Wsp1BB?token=12e64f0d019532318f6d

As for next steps, we can consider running this test again, measuring completed sign-ins instead. I would also like to see this test replicated on a totally new website---one where users are not already so familiar with the text "Sign in with Persona" that the potential drawbacks are overshadowed.

--
John Karahalis
Mozilla
openjck.com

[Luke Crouch]

This is so cool! Granted the MDN audience is usually more
"Mozilla-savvy" than the general populace, but it's cool that "with
Persona" appears to induce more trust and engagement with the site.

Nice work John!

-L

--
Q: Why is this email five sentences or less?
A: http://five.sentenc.es

[Ryan Feeley]

Really great!

Interesting, because this counters what we discovered during usability testing last year. This could be because we were not testing visitors of a Mozilla property but rather usertesting.com subjects testing a generic to-do list website (123done.org).

We found that many users had issues with the “Sign in with Persona” button – the logic we heard was “I don’t know what Persona is, just that I know I don’t have it, and therefore cannot sign in with it.”

When there is only one sign in option, it’s much easier to for a user to find their way. Unfortunately many sites have several sign in options (stackoverflow.com had 14 at my last count).

I wonder if the single sign-in option (not typical for Persona in the wild) for a captive Mozillian audience swayed these results?

Here was our original recommendations:
http://people.mozilla.org/~rfeeley/persona_variations/

And the PDF (now available on SUMO in HTML)
https://www.dropbox.com/s/tdjblhpwvhnqdyv/button-guidelines.pdf

Ryan Feeley
Product Designer, Identity
Mozilla UX
IRC: rfeeley

> _______________________________________________
> dev-identity mailing list
> dev-id...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-identity

[Luke Crouch]

I tend to agree that a captive Mozillian audience would strongly favor
the "with Persona."

So, I'd love to see other Mozilla Persona sites test it - i.e., Input,
Mozillians, and Bugzilla.

-L

> _______________________________________________
> dev-mdn mailing list
> dev...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-mdn

[Kadir Topal]

For those who are looking for the implementation on SUMO: it's on our
staging server: https://support.allizom.org

We are waiting for one last blocker to be resolved before we roll it out
on production: https://github.com/mozilla/browserid/issues/4003

--Kadir

Ryan Feeley schrieb:

> dev-webdev mailing list
> dev-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-webdev

[Luke Crouch]

Kadir,

Do you have plans to A/B test different "Sign in" text?

-L

[John Karahalis]

Very interesting. Thank you for sharing that history, Kadir.

We may both be right. Our users are much more technically proficient than most people, and they may have come to expect the "with Persona" version that we were using for the last two years. So we should be careful not to interpret the results too generally. Dropping "with Persona" would be a mistake for MDN right now, but every website is different.

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

[John Karahalis]

Ryan, the recommendations in that PDF are very interesting. We had not considered the copy "Sign in / Sign up". I will plan to re-run the experiment with that variation (and, if possible, also test completed sign-ins, which should give us a more complete picture of how the change affects users).

I will plan to update this thread with results as soon as we have them.

--
John Karahalis
Mozilla
openjck.com

----- Original Message -----

[Jean-Yves Perrier]

I'm curious.

If we do run the same A/B test from scratch, do we get the same results?

The cost of re-doing them would be minimal.

--
Jean-Yves

On 22/01/2014 22:11, John Karahalis wrote:
> Over the past two weeks, the MDN team tested four variations to its sign in button and measured the number of clicks each received. The variations were:
>
>
> * Sign in with Persona (our baseline)
> * Sign in
> * Sign in / Register
> * Sign in / Create account
>
> We expected all three alternatives to outperform the baseline. We expected "Sign in" to outperform the baseline because it's more conventional, and because it does not use the word "with" (which may have earned a bad reputation on the Web). We expected the other two to outperform the baseline because, unlike the baseline, they do not incorrectly imply that the button is only useful to returning users.
>
> Even after running the experiment with 100,000 users, none of the variations stood out as a statistically significant improvement over the others. This may sound like bad news--it isn't! These are not the results we wanted, but they are results! They demonstrate that using any of these alternatives would have been a mistake (at least as far as click rates are concerned). Doing so would mean dropping the Persona branding with no real benefit. We are no closer to understanding what changes we should make to the button to improve click rates, but we do know what changes we should not make.
>
> http://optimize.ly/~Wsp1BB?token=12e64f0d019532318f6d
>
> As for next steps, we can consider running this test again, measuring completed sign-ins instead. I would also like to see this test replicated on a totally new website---one where users are not already so familiar with the text "Sign in with Persona" that the potential drawbacks are overshadowed.
>


--

Jean-Yves Perrier
Technical Writer / Mozilla Developer Network

[Kadir Topal]

Not specifically, compared to our general user base very few people ever
log-in (less than 0.1%), so we'd need to run it with a huge amount of
users to get significant results, but we are permanently A/B testing via
Optimizely, and if there is interest in this, we could run it. SUMO
would be one of those places that have very few "Mozillians" logging in.

--Kadir

Luke Crouch schrieb: