Using Vanity for complex experiments

[jonathankirst]

We've been using Vanity for a couple weeks now and really liking it
and want to dig a little deeper but the documentation does not seem to
be complete/up to date.

The use case I want to try it goes like this:

User clicks a link which takes them to a signup flow corresponding to
an split test experiment that will take them to one of two signup
flows. Currently, we track which signup flow has better conversion.
But we want to take it a step further and see which signup flow has
better user engagement down the line (they are significantly
different). We already measure that engagement in a metric. So
basically having two different conversion metrics for the same
experiment, one linearly following the other. Any ideas?

Also, does anyone know how to recalculate metrics (like conversion) as
a percentage instead of a total in the dashboard outside of
experiments? For example, signups may very according to the amount of
traffic we can generate with campaigns, but performance issues may
have been effecting conversion rates (again, outside of any
experiment). So instead of seeing the raw number of signups per day,
it would be awesome to also see percentage of conversions.

[Ian Young]

I'd be very interested in tracking multiple metrics against an experiment too. Has anyone used Vanity to do this?

Currently, Vanity lets you specify multiple metrics in the experiment definition, but it lumps them all together when tracking conversions (a user triggering any one of the given metrics counts as a converted user). I looked into the code a little and it seems we'd need to change how some of the data is stored to make this possible.

This article has a great example of what I have in mind: http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae-bingo. Sadly, his tool is actually a Python port of A/Bingo.


Ian