I'm looking into A/B testing tools for Rails. There are a number of good
options available, but to my surprise, none of the tools I've looked at
appear to support measuring multiple metrics for a single test. I know I'm
not the only one to think of this, because
http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae... describes
and screenshots exactly what I had in mind. Unfortunately, that's a Python
tool he's talking about.
It's easy enough to optimize your landing page based on a single metric
(conversions), but like Khan Academy, I want to test more nuanced questions
- are our users more engaged with the content? Do they participate more?
Can we increase click-through without harming long-term retention? It seems
somewhat constraining to test a new feature but only get an answer for a
single measurement.
So, anyone know of a tool I've overlooked that might do this? Am I really
the only one who wants this feature? Anyone have a persuasive argument in
favor of sticking to a single metric?
> I'm looking into A/B testing tools for Rails. There are a number of good options available, but to my surprise, none of the tools I've looked at appear to support measuring multiple metrics for a single test. I know I'm not the only one to think of this, because http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae... describes and screenshots exactly what I had in mind. Unfortunately, that's a Python tool he's talking about.
> It's easy enough to optimize your landing page based on a single metric (conversions), but like Khan Academy, I want to test more nuanced questions - are our users more engaged with the content? Do they participate more? Can we increase click-through without harming long-term retention? It seems somewhat constraining to test a new feature but only get an answer for a single measurement.
> So, anyone know of a tool I've overlooked that might do this? Am I really the only one who wants this feature? Anyone have a persuasive argument in favor of sticking to a single metric?
Thanks Rob. I did test Vanity and couldn't figure out how to get multiple
metrics working. Playing with it again, I think I now see how the feature
works and why it didn't match what I expected to see. Maybe you can pitch
in to tell me if I'm understanding it correctly.
In Vanity, if you define multiple metrics for an experiment, calling
track!() on any one of those metrics will mark the user as converted. But
it all goes into the same conversion count in the dashboard. So I could
track :account_created and :newsletter_signup as two different metrics, and
get conversion rates for an experiment based on how many users did *at
least one* of those things.
A nice enough feature, but unfortunately not quite the functionality I was
hoping for - I'd like to track and view each metric separately. Anyone used
Absurdity and know if it works the Vanity way, or my hypothetical way?
On Tue, Nov 6, 2012 at 12:02 PM, Robert Kaufman <rgkauf...@gmail.com> wrote:
> So you want to track multiple metrics for a given A/B test? That would be
> an Absurdity https://github.com/xing/absurdity! Oh, the Vanity
> http://vanity.labnotes.org/ of some people. ;-D
> On Nov 2, 2012, at 1:35 PM, Ian Young <ian.greenl...@gmail.com> wrote:
> Hey all,
> I'm looking into A/B testing tools for Rails. There are a number of good
> options available, but to my surprise, none of the tools I've looked at
> appear to support measuring multiple metrics for a single test. I know I'm
> not the only one to think of this, because
> http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae... describes
> and screenshots exactly what I had in mind. Unfortunately, that's a Python
> tool he's talking about.
> It's easy enough to optimize your landing page based on a single metric
> (conversions), but like Khan Academy, I want to test more nuanced questions
> - are our users more engaged with the content? Do they participate more?
> Can we increase click-through without harming long-term retention? It seems
> somewhat constraining to test a new feature but only get an answer for a
> single measurement.
> So, anyone know of a tool I've overlooked that might do this? Am I really
> the only one who wants this feature? Anyone have a persuasive argument in
> favor of sticking to a single metric?
It was some time ago, and I believe we had to modify the dashboard in order to show the metrics split out. I don't have access to that code any more, so I can't really be 100% sure how we did it.
Sorry,
Rob
On Nov 6, 2012, at 1:21 PM, Ian Young <ian.greenl...@gmail.com> wrote:
> Thanks Rob. I did test Vanity and couldn't figure out how to get multiple metrics working. Playing with it again, I think I now see how the feature works and why it didn't match what I expected to see. Maybe you can pitch in to tell me if I'm understanding it correctly.
> In Vanity, if you define multiple metrics for an experiment, calling track!() on any one of those metrics will mark the user as converted. But it all goes into the same conversion count in the dashboard. So I could track :account_created and :newsletter_signup as two different metrics, and get conversion rates for an experiment based on how many users did *at least one* of those things.
> A nice enough feature, but unfortunately not quite the functionality I was hoping for - I'd like to track and view each metric separately. Anyone used Absurdity and know if it works the Vanity way, or my hypothetical way?
> Ian
> On Tue, Nov 6, 2012 at 12:02 PM, Robert Kaufman <rgkauf...@gmail.com> wrote:
> So you want to track multiple metrics for a given A/B test? That would be an Absurdity https://github.com/xing/absurdity! Oh, the Vanity http://vanity.labnotes.org/ of some people. ;-D
> On Nov 2, 2012, at 1:35 PM, Ian Young <ian.greenl...@gmail.com> wrote:
>> Hey all,
>> I'm looking into A/B testing tools for Rails. There are a number of good options available, but to my surprise, none of the tools I've looked at appear to support measuring multiple metrics for a single test. I know I'm not the only one to think of this, because http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae... describes and screenshots exactly what I had in mind. Unfortunately, that's a Python tool he's talking about.
>> It's easy enough to optimize your landing page based on a single metric (conversions), but like Khan Academy, I want to test more nuanced questions - are our users more engaged with the content? Do they participate more? Can we increase click-through without harming long-term retention? It seems somewhat constraining to test a new feature but only get an answer for a single measurement.
>> So, anyone know of a tool I've overlooked that might do this? Am I really the only one who wants this feature? Anyone have a persuasive argument in favor of sticking to a single metric?
