REGRESSION: Install Analytics from Chrome Webstore are inaccurate

[David Emelianov]

With the recent migration to GA4, the install data reported by the Chrome Web Store has become wildly inconsistent with the programmatically recorded install data that we are tracking from our Chrome Extension. Previously, (before the move to GA4) the install data was consistent across both sources, so this appears to be a severe regression in Webstore functionality.

Programmatically calculated install analytics data:

We collect our own install analytics data by listening to the chrome.runtime.onInstalled event. If the event reason is "install", then we record an Install event in our analytics, like so:

chrome.runtime.onInstalled.addListener(async (details) => {

if (details.reason === "install") {

Analytics.instance.trackEvent({

category: "ExtensionManagement",

action: "Install",

});

Our implementation of this programmatic tracking solution has not changed in a long time, so the data we are seeing is fairly consistent. Here is the data for installs, collected per day since April:

Web Store reported install analytics data (pre-GA4):

Prior to the migration to GA4, the install data that we were seeing from the Web Store was mostly consistent with our programmatic install data. Here is a screenshot demonstrating how we would view this data in Google Analytics, using Goals linked to the /install Goal URL: 

Note that this data is fairly consistent with the programmatic data shown above. A few data points to demonstrate:

April 10: 

Webstore reported 722 installs. 

Our programmatic tracking reported 819 installs from 760 unique users.

May 1:

Webstore reported 534 installs.

Our programmatic tracking reported 637 installs from 599 unique users.

June 6:

Webstore reported 456 installs.

Our programmatic tracking reported 618 installs from 578 unique users.

I understand that there is somewhat of a discrepancy in the data (20%), mostly because a single user could install from CWS on 1 device and then Chrome Sync will install the extension on N devices. 

Web Store reported install analytics data (GA4):

After migrating our Webstore setup to GA4 in mid-July, we are seeing extremely different results for our install data. Here is a screenshot, showing that our average installs per day are being reported as roughly 150-200:

  

 

Comparing this to our programmatic data, the discrepancy is significantly larger in relative terms (50-100%):

July 19:

Webstore reported 184 installs. 

Our programmatic tracking reported 360 installs from 349 unique users.

July 31:
Webstore reported 200 installs.

Our programmatic tracking reported 343 installs from 322 unique users.

Aug 7:

Webstore reported 217 installs.

Our programmatic tracking reported 356 installs from 332 unique users.

This appears to be a significantly larger discrepancy from the old tracking method on the Web Store. It seems that tracking has gotten less accurate with the new setup.

PS - another massively impactful consequence of the GA4 migration is the inability to link our install data to our Ads account. Since the Webstore does not give us admin access to our GA4 entity, we are unable to link the entity to our Ads account. This means that our ad spending is no longer optimized for install performance, which is why our install volume has been dropping ever since the GA4 migration.

[Zach Warneke]

Hey Dave,

Chrome Web Store eng here - thanks for reaching out!

I took a quick look at your install stats using our developer dashboard (non-GA) metrics. You can find these yourself if you go to our developer dashboard, then navigate to Item -> Analytics -> Installs & uninstalls. Here's the difference between these metrics and the GA metrics:

• CWS dev dashboard install metrics: come from Chromium, i.e. when your extension gets installed, Chromium sends an install ping that ultimately increments the install metric we show on our developer dashboard
• CWS GA (pre- and post-GA4) install metrics: come from CWS' listing page for your extension, i.e. whenever a user clicks the "Add to Chrome" button and the install succeeds, our page sends an event to GA

Here's a comparison of the numbers from both of the sources we provide (Chromium, CWS GA). You can verify these numbers yourself using the developer dashboard, as mentioned above.

Date: Chromium, CWS GA

April 10: 717, 722
May 1: 543, 534
June 6: 489, 456

July 19: 201, 184
July 31: 215, 200
Aug 7: 233, 217

The metrics from both sources are fairly close both pre- and post-GA4 migration, so it seems unlikely that "tracking has gotten less accurate with the new setup". Moreover, I took a look at the chart you provided, and I noticed that some of the dates pre-GA4 (not listed in your message) had similar discrepancies in relative terms. E.g. on April 28, the Chromium stats showed 59 installs, whereas your chart showed ~190 installs (I assume the horizontal lines are increments of 200, although the axes were missing). That's a difference of >100%, which is what you reported seeing only after the GA4 migration. I only have access to the Chromium metrics and not the GA metrics, but I assume the numbers are similar.

In general, it seems like there's consistently been a discrepancy of ~100-150 both pre- and post-GA4. I think this thread covers possible reasons for that discrepancy well, and I unfortunately don't have any more insight to provide though.

Let me know if you have any questions or would like to discuss further!

Zach Warneke