What happens when running the same AB test experiment with different variants when stopping and starting again?
56 views
Skip to first unread message
Ronak Badheka
May 31, 2018, 7:56:53 PM5/31/18
to Firebase Google Group
Hey everyone,
A quick question:
A quick question:
I have an A/B testing experiment running with 3 variants: Control, Var A and Var B at 33.3% distribution for each.
Now I want to stop that experiment and run it again with just 2 variants Control and Var B and have 50% distribution for each.
My question is: will the 33.3% of users who got variant B during the first run of the same experiment get a different variant when I run this experiment again (by keeping the same remote config parameter name) with just 2 variants (Control and Var B)?
I'll end the existing experiment that is running with 3 variants for all users and then duplicate it with just two variants. It will have the same remote config parameter name and the values as before, just one less variant.
Now I want to stop that experiment and run it again with just 2 variants Control and Var B and have 50% distribution for each.
My question is: will the 33.3% of users who got variant B during the first run of the same experiment get a different variant when I run this experiment again (by keeping the same remote config parameter name) with just 2 variants (Control and Var B)?
I'll end the existing experiment that is running with 3 variants for all users and then duplicate it with just two variants. It will have the same remote config parameter name and the values as before, just one less variant.
Can anyone answer please? Thank you.
Kato Richardson
Jun 1, 2018, 11:53:21 AM6/1/18
to Firebase Google Group
Hi Ronak,
Not an expert here, I can't think of any reason why a separate experiment would re-use allocations from another experiment. No obvious value to linking those together behind the scenes.
However, I'll caveat that by saying I don't really know how they get bucketed into the variants. It's possible the algorithm could work in some way that causes them to end up in the same place; I doubt it.
☼, Kato
--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/7bdf8530-38da-483d-9728-5117e618751c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ronak Badheka
Jun 2, 2018, 11:33:32 AM6/2/18
to fireba...@googlegroups.com
Hi Kato,
As I mentioned it is the same experiment that is being stopped and duplicated and started again. So in a way it is the same experiment. The reason we want to do this is because we have eliminated one of the variants from the experiment as it was performing negatively so we want those users to get a different variant just so we could stop the negative conversion. However, at the same time we want to keep experiment going between two other variants to determine a leader.
As I mentioned it is the same experiment that is being stopped and duplicated and started again. So in a way it is the same experiment. The reason we want to do this is because we have eliminated one of the variants from the experiment as it was performing negatively so we want those users to get a different variant just so we could stop the negative conversion. However, at the same time we want to keep experiment going between two other variants to determine a leader.
On Fri, Jun 1, 2018 at 8:53 AM, 'Kato Richardson' via Firebase Google Group <fireba...@googlegroups.com> wrote:
Hi Ronak,Not an expert here, I can't think of any reason why a separate experiment would re-use allocations from another experiment. No obvious value to linking those together behind the scenes.However, I'll caveat that by saying I don't really know how they get bucketed into the variants. It's possible the algorithm could work in some way that causes them to end up in the same place; I doubt it.☼, Kato
On Thu, May 31, 2018 at 4:56 PM Ronak Badheka <ronak....@move.com> wrote:
--Hey everyone,
A quick question:I have an A/B testing experiment running with 3 variants: Control, Var A and Var B at 33.3% distribution for each.
Now I want to stop that experiment and run it again with just 2 variants Control and Var B and have 50% distribution for each.
My question is: will the 33.3% of users who got variant B during the first run of the same experiment get a different variant when I run this experiment again (by keeping the same remote config parameter name) with just 2 variants (Control and Var B)?
I'll end the existing experiment that is running with 3 variants for all users and then duplicate it with just two variants. It will have the same remote config parameter name and the values as before, just one less variant.
Can anyone answer please? Thank you.
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/7bdf8530-38da-483d-9728-5117e618751c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
--
You received this message because you are subscribed to a topic in the Google Groups "Firebase Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/firebase-talk/Jtx0bvpKsVo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to firebase-talk+unsubscribe@googlegroups.com.
To post to this group, send email to fireba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/CADypTEZtCw5G2-AgN%3DgH%2B%3DsYf82oNFmQ3mkR08-1%2BxhCV9KEbQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages