Feature Idea: Define your own option percentages...

[Ian Sefferman]

Hey guys,
Wondering what people think about / how feasible it'd be to do the following:

Say I have a fairly experimental feature for my site that I'd like to
quickly implement and test. But, it's really not ready for prime-time,
even 50% of my users would be too much. Maybe I want it to be closer
to 5% of my users, for various reasons (e.g. the software isn't ready
for scale or I expect it to perform _worse_ but I want to ensure
that's the case). I think it'd be great to have the definition of an
alternative be able to take a percentage of users who should see each
option and/or allow that to be changed via the dashboard.

As an added bonus, this could be a nice way to roll out new features
to your site quietly, testing for production/scale bugs more
gradually.

Thoughts?

Ian

--
Ian Sefferman | (406) IAN-1337
http://www.iseff.com

[Assaf Arkin]

As I understand it, progressive roll-out is part of EDD.  You put out a feature, measure how it performs, and progressively refine and roll it out to more people, making sure metrics don't get out of bounds.  A metric could be crashes/day, avg response time or anything else that could indicate problem.

Definitely something I'd like to see (and help implement).

Assaf

 

Ian

--
Ian Sefferman | (406) IAN-1337
http://www.iseff.com

--
To change your subscription:
http://groups.google.com/group/vanity-talk?hl=en

[Ian Sefferman]

That's a good way tothink about metrics. I'll start by just making the alternatives able to take a percentage of requests, and make the coin flip try to follow that. 

Ian

Sent from my iPhone

[Steve]

Ian, Assaf --

I also agree that it should be possible to define what percentage of
visitors will be given each option. In fact, I agree so much that I
actually thought that this was already possible with Vanity.

That said, wouldn't it be fairly easy to make Vanity do this (without
modifying Vanity itself)? It wouldn't be pretty, but I think this
would work:

ab_test "foo" do
alternatives :normal, :normal, :normal, :normal, :risky
metrics :foo_conversion
end

To make the test only give the :risky option to 20% of your users. Of
course the dashboard would look silly, but that shouldn't be a
dealbreaker.

Alternatively, I think that you could define a helper method that
would utilize vanity_identity and AbTest#chooses to divy the
participants up in any way you want. I wish identity management were
slightly better documented in the API docs. I had to go digging
through the code to find how identities are managed.

Since product management at my office has decreed that we are to test
a risky new layout on certain pages, and that the risky layout is only
to be shown to 10% of users, I think we will be attempting to do
this. I'll let you know how it goes.

Cheers

-Steve

[Assaf Arkin]

On Mon, Feb 1, 2010 at 11:17 AM, Steve <steve...@insiderpages.com> wrote:

Ian, Assaf --

I also agree that it should be possible to define what percentage of
visitors will be given each option.  In fact, I agree so much that I
actually thought that this was already possible with Vanity.

That said, wouldn't it be fairly easy to make Vanity do this (without
modifying Vanity itself)?  It wouldn't be pretty, but I think this
would work:

ab_test "foo" do
 alternatives :normal, :normal, :normal, :normal, :risky
 metrics :foo_conversion
end

To make the test only give the :risky option to 20% of your users.  Of
course the dashboard would look silly, but that shouldn't be a
dealbreaker.

This would obviously work.

The big issue is: right now A/B experiments statically divide the population.  If you want to go from 20% to 40%, you'll have to start a new experiment.  It's not a big issue, but a nicer way of doing it, maybe directly form the dashboard, would be nicer.

Alternatively, I think that you could define a helper method that
would utilize vanity_identity and AbTest#chooses to divy the
participants up in any way you want.  I wish identity management were
slightly better documented in the API docs.  I had to go digging
through the code to find how identities are managed.

Definitely need to fix identity management, and more flexibility when splitting up the audience: e.g you may want to only experiment on users who came through organic search.  Needs to be rethought.

Assaf

[Donald Piret]

Has any progress been made on this since it was asked a few years ago?

[Patrick Joyce]

There isn't anything built into Vanity to allow this, but you can do it by overriding the alternative_for method in your experiment.

I wrote up a quick blog post with code examples on it here: http://pragmati.st/2013/01/03/non-even-ab-testing-splits-with-vanity/

- Patrick Joyce

--