UseCounters available in HTTP Archive BigQuery

147 views
Skip to first unread message

Rick Byers

unread,
Jan 31, 2017, 9:40:23 AM1/31/17
to platform-predictability, blink-api-owners-discuss, Patrick Meenan, Ilya Grigorik
Thanks to a ton of work by pmeenan@, UseCounter data is now captured by HTTP Archive and stored in the BigQuery tables.  That means, to find the URLs of the sites from the top 500k which trigger (for example) the 'TimeElement' UseCounter during page load, you can just issue a query like this:

SELECT 
  url,
  JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.TimeElement') AS featureTime
FROM [httparchive:har.2017_01_15_chrome_pages]
HAVING featureTime IS NOT NULL
LIMIT 500

This will also return the time (since the start of load) that the counter was first triggered in case that is useful for some reason :-).

I've added this example to our HTTP Archive for web compat decision making doc.

The CSS counters will be available after M57 hits stable (due to this bug).

Mike West

unread,
Jan 31, 2017, 10:02:17 AM1/31/17
to Rick Byers, platform-predictability, blink-api-owners-discuss, Patrick Meenan, Ilya Grigorik
On Tue, Jan 31, 2017 at 6:40 AM, 'Rick Byers' via platform-predictability <platform-pr...@chromium.org> wrote:
Thanks to a ton of work by pmeenan@, UseCounter data is now captured by HTTP Archive and stored in the BigQuery tables.  That means, to find the URLs of the sites from the top 500k which trigger (for example) the 'TimeElement' UseCounter during page load, you can just issue a query like this:

This is great, thanks Rick! It will really help track down usage data in a real-world way that should help us make more informed decisions.

Is the data collected from the new histogram, or the old histogram?

-mike

Patrick Meenan

unread,
Jan 31, 2017, 10:45:13 AM1/31/17
to Mike West, Rick Byers, platform-predictability, blink-api-owners-discuss, Patrick Meenan, Ilya Grigorik
Sorry for my ignorance, what are the old and new histograms?  The code is collected through a trace event generated at the same point where the histogram counter is ticked.

On Tue, Jan 31, 2017 at 7:01 AM, Mike West <mk...@google.com> wrote:

Ilya Grigorik

unread,
Jan 31, 2017, 11:19:59 AM1/31/17
to Patrick Meenan, Mike West, Rick Byers, platform-predictability, blink-api-owners-discuss, dev-web-...@google.com, dev-web
+dev-web fyi! Handy resource.

Mike West

unread,
Jan 31, 2017, 12:30:52 PM1/31/17
to Ilya Grigorik, Patrick Meenan, Rick Byers, platform-predictability, blink-api-owners-discuss, dev-web-...@google.com, dev-web
On Tue, Jan 31, 2017 at 8:19 AM, Ilya Grigorik <igri...@google.com> wrote:
+dev-web fyi! Handy resource.

On Tue, Jan 31, 2017 at 7:45 AM, Patrick Meenan <pme...@chromium.org> wrote:
Sorry for my ignorance, what are the old and new histograms?  The code is collected through a trace event generated at the same point where the histogram counter is ticked.

`Blink.UseCounter.Features` vs `WebCore.FeatureObserver`. It's not at all clear to me if the distinction matters at the layer where you're collecting data. Rick? :)

Rick Byers

unread,
Feb 1, 2017, 10:10:40 AM2/1/17
to Mike West, Ilya Grigorik, Patrick Meenan, platform-predictability, blink-api-owners-discuss, dev-web-...@google.com, dev-web
Yeah, technically the code is injected at the point of the new histogram (not in the LegacyUseCounter code).  But the distinction shouldn't mean too much (eg. the notion of what exactly is and isn't a PageVisit doesn't particularly matter in the HTTP Archive case).

fal...@chromium.org

unread,
Feb 8, 2017, 9:34:03 PM2/8/17
to platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, rby...@google.com
This is great. Is it possible to sort the results to see the top sites using the feature?

Philip Jägenstedt

unread,
Feb 8, 2017, 10:29:11 PM2/8/17
to fal...@chromium.org, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, rby...@google.com
This could be very inefficient, but I figured out how to find the top sites using marquee and it looks reasonable:


--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/abc2383a-59ca-4c96-b3fe-4828160a652c%40chromium.org.

Matt Falkenhagen

unread,
Feb 8, 2017, 10:42:56 PM2/8/17
to Philip Jägenstedt, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
Thanks! This doesn't look right though for ServiceWorkerControlledPage, as I'd expect facebook, youtube, etc to be at the top.

2017-02-09 12:28 GMT+09:00 Philip Jägenstedt <foo...@chromium.org>:
This could be very inefficient, but I figured out how to find the top sites using marquee and it looks reasonable:


On Wed, Feb 8, 2017 at 9:34 PM <fal...@chromium.org> wrote:
This is great. Is it possible to sort the results to see the top sites using the feature?


On Tuesday, January 31, 2017 at 11:40:23 PM UTC+9, Rick Byers wrote:
Thanks to a ton of work by pmeenan@, UseCounter data is now captured by HTTP Archive and stored in the BigQuery tables.  That means, to find the URLs of the sites from the top 500k which trigger (for example) the 'TimeElement' UseCounter during page load, you can just issue a query like this:

SELECT 
  url,
  JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.TimeElement') AS featureTime
FROM [httparchive:har.2017_01_15_chrome_pages]
HAVING featureTime IS NOT NULL
LIMIT 500

This will also return the time (since the start of load) that the counter was first triggered in case that is useful for some reason :-).

I've added this example to our HTTP Archive for web compat decision making doc.

The CSS counters will be available after M57 hits stable (due to this bug).

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYeVM0XMY6eA-yg107jJMnqsh6W6Zj-ZZqkLsvLNdS%3DPYw%40mail.gmail.com.

Philip Jägenstedt

unread,
Feb 8, 2017, 10:47:24 PM2/8/17
to Matt Falkenhagen, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
Would the use counter be hit on first page load when navigating to those sites with a fresh profile, i.e. while logged out? The httparchive use counter data will be quite limited in this way, being probably useful for things that appear in markup, but anything that happens after user interaction would be missed.

On Wed, Feb 8, 2017 at 10:42 PM Matt Falkenhagen <fal...@chromium.org> wrote:
Thanks! This doesn't look right though for ServiceWorkerControlledPage, as I'd expect facebook, youtube, etc to be at the top.

2017-02-09 12:28 GMT+09:00 Philip Jägenstedt <foo...@chromium.org>:
This could be very inefficient, but I figured out how to find the top sites using marquee and it looks reasonable:


On Wed, Feb 8, 2017 at 9:34 PM <fal...@chromium.org> wrote:
This is great. Is it possible to sort the results to see the top sites using the feature?


On Tuesday, January 31, 2017 at 11:40:23 PM UTC+9, Rick Byers wrote:
Thanks to a ton of work by pmeenan@, UseCounter data is now captured by HTTP Archive and stored in the BigQuery tables.  That means, to find the URLs of the sites from the top 500k which trigger (for example) the 'TimeElement' UseCounter during page load, you can just issue a query like this:

SELECT 
  url,
  JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.TimeElement') AS featureTime
FROM [httparchive:har.2017_01_15_chrome_pages]
HAVING featureTime IS NOT NULL
LIMIT 500

This will also return the time (since the start of load) that the counter was first triggered in case that is useful for some reason :-).

I've added this example to our HTTP Archive for web compat decision making doc.

The CSS counters will be available after M57 hits stable (due to this bug).

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predicta...@chromium.org.

To post to this group, send email to platform-pr...@chromium.org.
--
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To post to this group, send email to blink-api-ow...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-api-owners-discuss/CAJ_xCimt%3D7ehvQMWVZrb%3DoYiyrMkqkXvBbxS6N_HB83dg9cipg%40mail.gmail.com.

Matt Falkenhagen

unread,
Feb 8, 2017, 10:52:03 PM2/8/17
to Philip Jägenstedt, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
Ah, that makes sense. The service worker counter requires the page to be navigated to when the profile already has a SW already registered for that site.

2017-02-09 12:47 GMT+09:00 Philip Jägenstedt <foo...@chromium.org>:
Would the use counter be hit on first page load when navigating to those sites with a fresh profile, i.e. while logged out? The httparchive use counter data will be quite limited in this way, being probably useful for things that appear in markup, but anything that happens after user interaction would be missed.

On Wed, Feb 8, 2017 at 10:42 PM Matt Falkenhagen <fal...@chromium.org> wrote:
Thanks! This doesn't look right though for ServiceWorkerControlledPage, as I'd expect facebook, youtube, etc to be at the top.

2017-02-09 12:28 GMT+09:00 Philip Jägenstedt <foo...@chromium.org>:
This could be very inefficient, but I figured out how to find the top sites using marquee and it looks reasonable:


On Wed, Feb 8, 2017 at 9:34 PM <fal...@chromium.org> wrote:
This is great. Is it possible to sort the results to see the top sites using the feature?


On Tuesday, January 31, 2017 at 11:40:23 PM UTC+9, Rick Byers wrote:
Thanks to a ton of work by pmeenan@, UseCounter data is now captured by HTTP Archive and stored in the BigQuery tables.  That means, to find the URLs of the sites from the top 500k which trigger (for example) the 'TimeElement' UseCounter during page load, you can just issue a query like this:

SELECT 
  url,
  JSON_EXTRACT(payload, '$._blinkFeatureFirstUsed.Features.TimeElement') AS featureTime
FROM [httparchive:har.2017_01_15_chrome_pages]
HAVING featureTime IS NOT NULL
LIMIT 500

This will also return the time (since the start of load) that the counter was first triggered in case that is useful for some reason :-).

I've added this example to our HTTP Archive for web compat decision making doc.

The CSS counters will be available after M57 hits stable (due to this bug).

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predictability+unsub...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.
To post to this group, send email to blink-api-owners-discuss@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYe22ABJZeyPmCST8gyJUs%2BGNQv5-qyTHbTh%3DX3OY8CKow%40mail.gmail.com.

Philip Jägenstedt

unread,
Feb 8, 2017, 11:34:01 PM2/8/17
to Matt Falkenhagen, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
Is there perhaps another counter for SW registration that would get hit on the first load?

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predicta...@chromium.org.

To post to this group, send email to platform-pr...@chromium.org.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To post to this group, send email to blink-api-ow...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

Matt Falkenhagen

unread,
Feb 8, 2017, 11:45:47 PM2/8/17
to Philip Jägenstedt, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
I don't think so. We could add one.

SW usage is a bit tricky due to the persistence aspect. We could add a UseCounter when the page calls register(), but pages might not bother calling register() when there is already a registration.

ServiceWorkerControlledPage is counted whenever a page is controlled by a service worker, which typically happens only on subsequent loads.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predictability+unsub...@chromium.org.

To post to this group, send email to platform-predictability@chromium.org.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-discuss+unsub...@chromium.org.
To post to this group, send email to blink-api-owners-discuss@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predictability+unsub...@chromium.org.
To post to this group, send email to platform-predictability@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-predictability/CAARdPYdApOj3rkD%3DhA3_KHQZRACL5g3RAtdKyBxXXx5JMwLvPw%40mail.gmail.com.

Philip Jägenstedt

unread,
Feb 9, 2017, 3:56:49 PM2/9/17
to Matt Falkenhagen, platform-predictability, blink-api-ow...@chromium.org, pme...@chromium.org, igri...@chromium.org, Rick Byers
In an httparchive crawl, there won't already be a SW registered, but it could of course still be that the registration doesn't happen during the early page loading in which case it won't show in in httparchive.

It's really only for things like the HTML parser where one could be fairly confident that httparchive has good coverage, so it won't work as a tool to find all big users of an API for the purpose of making a breaking change. However, I hope it will be very useful for finding representative usage of an API. You might also get lucky and find a single big user which could totally swing the numbers, if lucky :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

--
You received this message because you are subscribed to a topic in the Google Groups "platform-predictability" group.
To unsubscribe from this topic, visit https://groups.google.com/a/chromium.org/d/topic/platform-predictability/Gcp4bdcvoKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to platform-predicta...@chromium.org.

To post to this group, send email to platform-pr...@chromium.org.
You received this message because you are subscribed to the Google Groups "blink-api-owners-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-api-owners-d...@chromium.org.
To post to this group, send email to blink-api-ow...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-predictability" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-predicta...@chromium.org.
To post to this group, send email to platform-pr...@chromium.org.
Reply all
Reply to author
Forward
0 new messages