A/B Testing

[Ian Young]

Hey all,

I'm looking into A/B testing tools for Rails. There are a number of good options available, but to my surprise, none of the tools I've looked at appear to support measuring multiple metrics for a single test. I know I'm not the only one to think of this, because http://bjk5.com/post/28269263789/lessons-learned-a-b-testing-with-gae-bingo describes and screenshots exactly what I had in mind. Unfortunately, that's a Python tool he's talking about.

It's easy enough to optimize your landing page based on a single metric (conversions), but like Khan Academy, I want to test more nuanced questions - are our users more engaged with the content? Do they participate more? Can we increase click-through without harming long-term retention? It seems somewhat constraining to test a new feature but only get an answer for a single measurement.

So, anyone know of a tool I've overlooked that might do this? Am I really the only one who wants this feature? Anyone have a persuasive argument in favor of sticking to a single metric?

Ian

[Robert Kaufman]

So you want to track multiple metrics for a given A/B test?  That would be an Absurdity https://github.com/xing/absurdity! Oh, the Vanity http://vanity.labnotes.org/ of some people. ;-D

Best,

Rob

PS: I know the Vanity docs don't really describe using more than one metric, but it has worked for me in the past and it does appear from http://vanity.labnotes.org/api/Vanity/Experiment/AbTest.html#metrics-instance_method that it should continue to work.

--
SD Ruby mailing list
sdr...@googlegroups.com
http://groups.google.com/group/sdruby

[Ian Young]

Thanks Rob. I did test Vanity and couldn't figure out how to get multiple metrics working. Playing with it again, I think I now see how the feature works and why it didn't match what I expected to see. Maybe you can pitch in to tell me if I'm understanding it correctly.

In Vanity, if you define multiple metrics for an experiment, calling track!() on any one of those metrics will mark the user as converted. But it all goes into the same conversion count in the dashboard. So I could track :account_created and :newsletter_signup as two different metrics, and get conversion rates for an experiment based on how many users did *at least one* of those things.

A nice enough feature, but unfortunately not quite the functionality I was hoping for - I'd like to track and view each metric separately. Anyone used Absurdity and know if it works the Vanity way, or my hypothetical way?

Ian

[Robert Kaufman]

It was some time ago, and I believe we had to modify the dashboard in order to show the metrics split out.  I don't have access to that code any more, so I can't really be 100% sure how we did it.  

Sorry,

Rob