Firebase Analytics to run A/B testing (experiment) and Statistical Significance

[Patrick Ng]

Hi,

I was planning to use Google Content Experiment to run AB test on mobile, but the doc indicates that this option is supported only for legacy purpose, and instead we should use Firebase Remote Config to run experiment.

The Firebase doc talks about using Remote Config to run experiment, but I couldn't see any mentioning about sample size and statistical significance.  Without them, we couldn't tell when we can declare that the difference we see has reached a certain level of statistical significance, and may easily run into a false positive situation.

Wonder if anyone has extra info or advice on that?

- Patrick

[Safa Alai]

Patrick,

Remote Config at this point allows you to set up the experiment but it does not have a way of reporting it.

To do so you really need to do two things:

1.  Record the experiment in a User Property of Firebase Analytics.  You can detect an experiment variant by using a Config parameter.

2.  Export the results into Big Query and do the queries to figure out the results.

We are actually actively working on A/B test suite for Remote Config but do not expect it to be ready until at least the end of this quarter.

[Jake]

Hi,

Just wondering if there's any progress/status update on A/B testing?

Jake

[Steven Hsieh]

Hi Safa,

We are also expecting the A/B testing suite! Is there any status update? Can't wait for it!

Jake於 2016年9月7日星期三 UTC+8上午7時50分07秒寫道：

[Hamoud AQ]

+1