Multiple goals and reports

[Stavros]

So I've been using django-lean to A/B test pages to some success,
until I tried using it with multiple goals a few days ago. I set up
two goals and two experiments, hoping that django-lean would show me
conversion rates for both goals, as there's nowhere to specify which
goal is which.

Instead, what I got was this, for one experiment:

http://i.imgur.com/CyNyY.png

and the other experiment showed me that there was a 14% reduction rate
in signups from the change with 90% confidence. The problem is that
the second experiment only shows *after* the user signs up!

There is nothing in the docs to explain how django-lean handles this,
or why I get duplicate columns in the table, or why the second
experiment has no data, or why some of the column headers are missing.
Could someone clarify?

Also, it this project alive? I submitted code that would remove the
jQuery dependency that is ready to merge, but nobody has so much as
touched the issue yet.

Thanks,
Stavros

[Erik Wright]

Hi Stavros,

I'm confused - I haven't seen any message about submitted changesets.
I'm going to look into that. If the tests pass, I'll be happy to merge
in your changes.

There appears to be a bug in the layout, hence the missing column
headers.

Each conversion report shows stats for all defined goals, as well as
for "any goal". "Any" considers users to have converted if they
achieve one or more of your goals.

So the 6 columns to the right, in your picture, are meant to be 3
columns each for each of your defined goals.

I think the issue you are coming up against is this: conversion goals
only apply to anonymous users. This is simply the use case that was
relevant to Akoha when we developed the tool. It could surely be
evolved to support conversion experiments on known users.

If I remember correctly, goals that occur after signup _are_ tracked
and reported for anonymous users, as long as they were enrolled in the
experiment before signup. That needs to be confirmed though.

As to whether the project is "alive", it is certainly being used at
Akoha, but I am no longer working there so I can't speak to their
future development plans for django-lean. I myself will likely
continue to review and shepherd patches although I'm not using the
project at the moment

If there is an active user out there who would like to take a lead in
the project, I encourage them to contact me.

Cheers,

Erik

[Stavros]

Hello and thank you for your reply.

The changeset is here:

http://bitbucket.org/akoha/django-lean/issue/14/remove-jquery-dependency

It's just a new js file that removes the need for jQuery. I've tested
it on the three main browsers and it works fine, but I encourage you
to test it as well.

About the columns, I looked into it and it's because it uses the first
column to get the goal names, but the first column has no data. I
think it's because the report generates for today() instead for
today()-1 when there's an end date set, but I wasn't sure so I left it
at that.

As for the anonymous user tracking, I find support for signed-up user
tracking indispensable. I'll look into extending it to scratch my itch
and submitting a patch, but I really hope you guys were using github,
as it makes it easier to see the latest version from the graph...

Thanks again!
Stavros

[er...@erikwright.com]

Hi Stavros,

Thanks for the patch. I'll try to merge it in shortly. It turned out
that my bitbucket email was going to my old akoha.com address. I've
fixed that now so I should remain abreast of future issues and merge
requests.

There is support for measuring _engagement_ of signed-up users, but
this is more of a single quantitative value per user rather than
goals, which are more categorical.

BitBucket (and Mercurial) are very similar to Git, so I'm sure if you
are familiar with the latter you will find Mercurial to be pretty easy
to pick up. It's true that some of the visualization features of BB
are not yet as strong as GH. That being said, a lot of Django and
Python development is done at bitbucket.org.

Cheers,

Erik

[Stavros]

Thanks for your help. I saw in another post that you mention that the
"end date" is not supposed to be set. Until now, I always assumed that
I was supposed to set the end date and django-lean would pick the
appropriate action depending on the results on that date. Could you
please, please specify on the homepage that this is not the case, and
that we should *NOT* set the ending date manually?

Thanks!
Stavros