A/B test experiment launch for webcompat.com/issues/new

[Mike Taylor]

Hi all,

Today I plan to enable the A/B test for the new bug reporting wizard
developed by the Open Innovation team[1]. The release plan is at [2],
but I'll repeat some of that here.

Note: if you want to experiment with the new form,
staging.webcompat.com/issues/new is set to serve the v2 form 100% of the
time.

The goal is to improve the number of valid bug reports we get by 30%. We
will run the experiment for 4 weeks and at that point decide to continue
with this approach or not.

Things still left to do:

1. Update our privacy policy

2. Set relevant environment variables on production servers and restart
the application.

The plan is to enable the new form for 3% of new users* for 1 week and
observe. If nothing terrible happens, we will bump that up to some
interestingly high value over the coming weeks.

3. Observe issues and file bugs. Any bug related to the new form can get
a "scope: new form QA" label added to it. There are some visual glitches
right now, but Open Innovation plans to iterate on these in the coming
week.

New bugs can be filed at
https://github.com/webcompat/webcompat.com/issues/, and feel free to
ping me directly for super bad stuff.

If we discover catastrophic bugs (users can't report issues...), the
plan is to stop serving the experiment to new users, and expire the
experiment cookie (see [4]).

* Note: we don't serve the experiment to people who have DNT enabled or
are logged in with GitHub. And if they block cookies, they obviously
won't get the experiment cookie.

[1] <https://github.com/webcompat/webcompat.com/issues/3031>
[2] <https://github.com/webcompat/webcompat.com/issues/2954>
[3] <https://webcompat-meet.herokuapp.com/wuEziE3SSASZ5ilB6KJsHw?both#>
[4] <https://github.com/webcompat/webcompat.com/issues/3030>

thanks,

--
Mike Taylor
Web Compat, Mozilla

[Mike Taylor]

On 10/24/19 11:24 AM, Mike Taylor wrote:
> Hi all,
>
> Today I plan to enable the A/B test for the new bug reporting wizard
> developed by the Open Innovation team[1]. The release plan is at [2],
> but I'll repeat some of that here.

Update: I discovered a blocker bug at
https://github.com/webcompat/webcompat.com/issues/3033 which took me
longer than I'd like to admit to figure out and fix.

Given the bugs we've found this late in the release plan, I think we
should pause for a day to let our SoftVision team test staging tomorrow
and pick up the deploy to production first thing on Monday (assuming we
found and resolved all the really bad ones).

I'd like to avoid asking Karl for support over the weekend and don't
feel comfortable enabling new things 15 minutes before I head to dinner. :)

[Christos Bacharakis]

Hi Mike,
well debugging over dinner sounds like fun right?

Joke aside sounds good from our end. Let's reconvene on Monday to press the
launch button together!
-Christos

--
Christos Bacharakis
Program Manager
Open Innovation Team

[Mike Taylor]

On 10/24/19 4:45 PM, Mike Taylor wrote> Given the bugs we've found this

late in the release plan, I think we
> should pause for a day to let our SoftVision team test staging tomorrow
> and pick up the deploy to production first thing on Monday (assuming we
> found and resolved all the really bad ones)

I've just deployed the v21.1.0[1] tag to staging and production, and
have set the relevant production environment variables to:

# A/B testing experiment ENV variables for production
# This expects 2 integers (important: not as a string)
# This configuration will serve form v2 to 3% of new users
FORM_V1_VARIATION=0 97
FORM_V2_VARIATION=97 100
# one week experimental cookie expiry
EXP_MAX_AGE=604800

And restarted the app. So we're live at 3%.

[1] <https://github.com/webcompat/webcompat.com/releases/tag/v21.1.0>

[Mike Taylor]

On 10/28/19 10:30 AM, Mike Taylor wrote:
> I've just deployed the v21.1.0[1] tag to staging and production, and
> have set the relevant production environment variables to:
>
> # A/B testing experiment ENV variables for production
> # This expects 2 integers (important: not as a string)
> # This configuration will serve form v2 to 3% of new users
> FORM_V1_VARIATION=0 97
> FORM_V2_VARIATION=97 100
> # one week experimental cookie expiry
> EXP_MAX_AGE=604800
>
> And restarted the app. So we're live at 3%.

Update:

I've deployed v21.1.1 to production and tweaked the experiment to serve
form v2 to 25% of new users.

env = FORM_V1_VARIATION=0 75
env = FORM_V2_VARIATION=75 100

[1] <https://github.com/webcompat/webcompat.com/releases/tag/v21.1.1>

[Mike Taylor]

On 11/5/19 5:37 PM, Mike Taylor wrote:
>
> I've deployed v21.1.1 to production and tweaked the experiment to serve
> form v2 to 25% of new users.
>
> env = FORM_V1_VARIATION=0 75
> env = FORM_V2_VARIATION=75 100
>
> [1] <https://github.com/webcompat/webcompat.com/releases/tag/v21.1.1>

I forgot to update this thread - last week on Nov 18th we bumped for v2
to 50% and deployed the v22.0.0 tag[1]

env = FORM_V1_VARIATION=0 50
env = FORM_V2_VARIATION=50 100

[1] https://github.com/webcompat/webcompat.com/releases/tag/v22.0.0