Noticable difference in number of requests when running A/B test

[jude....@greedygame.com]

We are trying to run  A/B test between direct Admob integration and  an in house optimization tool for displaying ads. Through the tool we use a different admob unit id and we evaluate the performance based on the numbers on Admob console. We ran the experiments thrice, but in all experiments, direct admob integration seems to have higher number of requests recorded. The format we are using is standard banner with a refresh time of 30 seconds configured on Admob console. The floor is set to Google Optimized. The following are the numbers that we have seen so far for the latest experiment.

Unit - Requests

Direct Admob - 4760

Through inhouse tool - 3988

​We are running a remote config based A/B experiment on Firebase with  50-50 split of users between the two units.

Noticing the difference we also ran the experiment on test device for 5 mins on each variant and recorded the number of ad requests being sent. Both were sending equal number of ad requests and were also recording the impressions.

We also checked the session time between the two groups, and other device related info and everything seemed to be a 50/50 split. Both groups had an average session time of 14 mins and devices were split across different brands in almost same percentages. 

Can somebody help us with some insights on to why the numbers might be different and any source through which I can more information on the difference in numbers?

NB:-  The in-house tool for the sake of this experiment can be summarized as a proxy which requests the ad to AdMob on behalf of the app. There is a back-end involved in getting the corresponding AdMob unit id for the placement.

[Mobile Ads SDK Forum Advisor]

Hi Jude,

Thank you for raising this to us. Would you able to provide the following via Reply to author option, so we could further check and analyze your use case?

• App ID
• Ad Units being used for A/B testing
• Any document that shows the detailed test of your use case 

Regards,

Teejay Wennie Mobile Ads SDK Team

ref:_00D1U1174p._5004Q2dJbp8:ref

[jude....@greedygame.com]

Hi Team,

Is there any update on this issue?

[Mobile Ads SDK Forum Advisor]

Hi Jude,

Thank you for bringing this to our attention. If you run this internal tool with a small sample size (< 10 requests if possible) could you please capture and send us a Charles log to see why AdMob has more requests than your internal network? Could you also highlight in that log what the requests to your optimization service look like?

Thanks,
Justin

 

ref:_00D1U1174p._5004Q2dJbp8:ref

[jude....@greedygame.com]

Hi Team,

I have shared the requested information via Reply Private option. Please check and let me know your thoughts. Meanwhile, has there been any other A/B testing done between Admob and any other tool? Is there any report that you can share. I tried Googling for any such info but did not find anything useful. Please share any docs that you have.