A/B Testing, low number of exposed users

[Leonardo Lobato]

Hello,

Is the A/B testing report on Firebase broken?

When I setup an A/B testing experiment, the dashboard says ~5k users will be targeted with the filters I specify (1% of the user base, a specific iOS app version with the "Build number", filtering by a user property). I do have an activation event enabled. 

But after this experiment has been running for a week, it only shows 15 users have been exposed.

If I filter the app's analytics events by the same filters I used for the experiment, I can see both the activation event and the event I'm measuring for have been triggered around 3k times. So I know the experiment itself is running, I'm just not getting the report.

The only way the app could trigger those events is if the user has been selected to be on a variant, as the feature is enabled only by a specific flag on remote configuration.

Is there anything else that could be wrong?

Thank you!

[Leonardo Lobato]

I was able to investigate this a bit further by exporting all the a/b test events from Firebase to BigQuery.

Turns out the issue is that the user property Firebase says will be present for the experiment (for example, `firebase_exp_1`) is simply not present on most events. From what I understand, it would be created by Firebase automatically as remote config is activated and the user is selected for a variant.

Again: the only way the user could trigger these events is through a remote config value that is attached to an experiment variant, so I know remote config is delivering the values.

Anyone have an idea on why the `firebase_exp_` user property could be missing?