Variant users increase during A/B testing

[Singapore News App]

I have 8 variants, each with 50 users assigned. After running the experiment for 2 months, the users in each variant increased to more than 100. Why is this happening and will it make the results inaccurate?

[Jen Harvey Hugg]

Hello!

If you're seeing more users in the experiment because you have more users and they match your targeting conditions, that's expected and the results should still be accurate. However, let's dig in a little bit:

- Are you seeing this in the Firebase console, via inspecting BigQuery results, or both?

- What targeting conditions are you using/what percentile/how are you assigning the users to experiment groups? 

Also note that "users" are assigned to experiment variants according to a hash of the experiment ID and the user’s Firebase installation ID, so you could have one user (person), but three experiment members (installation IDs) if the same user accesses your app using three different devices. If you're not doing so already, you might want to look at user audience or user properties to control experiment membership (note that there's some latency for new audiences and for user audience enrollment).

If these extra users don't match the targeting conditions you set and aren't the result of different installation IDs, can you send more detail, including project, experiment ID, targeting conditions, and any other relevant detail that would help the team troubleshoot to Firebase Support (https://firebase.google.com/support/troubleshooter/ab_testing/help) so that our team can take a closer look?

Thanks a bunch in advance!

[Singapore News App]

- Seeing it in Firebase. Not using BigQuery yet.

- Targeting and distribution is decided by version, OS, and 10% of the total userbase. Due to the increase in users in each variant, the total users experiment membership increased to 33% by itself.

Same user with multiple memberships are marginal and not an issue here.