Intent to Ship: MediaStreamTrack Stats (Audio)

[Henrik Boström]

Contact emails

hb...@chromium.org, ol...@chromium.org, h...@chromium.org

Specification

https://w3c.github.io/mediacapture-extensions/#the-mediastreamtrackaudiostats-interface

Summary

The MediaStreamTrack Statistics API, or `track.stats`, has already shipped for video tracks: see previous I2S here.

This is the same API but for audio tracks, also motivated by the app's desire to measure capture quality. This is very important for conferencing websites such as Google Meet, Microsoft Teams, Zoom, Goto Meetings, etc. All of which has expressed an interest in the audio portion of this API.

The API is only available for getUserMedia() sourced audio tracks, i.e. microphone, so the API is behind a user prompt and only available during capture.

The new interface we want to ship is MediaStreamTrackAudioStats which allow measuring two things from the audio capture pipeline:

1. The number of audio frames, including if any audio frames are dropped by the device, OS or User Agent. This allows measuring glitches in captured audio.

2. The input latency, such as due to buffering or other delays in delivering the audio frames to the track's sinks.

Blink component

Blink>GetUserMedia

Risks

Interoperability and Compatibility

Because the API provides statistics about capture quality, rather than providing capture functionality, the interop/compatibility risk is small.

Gecko: Standards position issue
WebKit: Standards position issue

Standardization

While the audio stats API is written by the W3C WebRTC Working Group and track statistics overall is not controversial, there is an ongoing disagreement with Mozilla about whether or not dropped frames (= totalFrames - deliveredFrames) should be exposed to the web in the audio case. The disagreement is tracked by this issue. Our need for this metric has been discussed at Virtual Interims, see recording with 10:39 timestamp but no consensus could be reached (rough consensus was reached between everyone except Mozilla). Youenn (Apple) has shown that frames can be dropped due to Bluetooth connection and is not just "measuring browser bugs".

Web developers: Positive

- The I2P shows support from Teams, Zoom and GoTo meetings.

- In the spec issue regarding the disagreement, more developer support is expressed (e.g. alfredh from Nvidia and steely-glint).

WebView application risks

None

Will this feature be supported on all six Blink platforms (Windows, Mac, Linux, ChromeOS, Android, and Android WebView)?

Yes

Is this feature fully tested by web-platform-tests?

Yes and WPTs will be written as part of implementing this, however unit tests will also be needed to verify accuracy of metrics on a lower level. WPTs will be more general like "frame counters increase over time during capture" and that run-to-completion semantics are preserved.

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5141112910249984

Links to previous Intent discussions

Intent to prototype:

https://groups.google.com/a/chromium.org/d/msgid/blink-dev/bb6c1af3-9eb3-4c6f-a136-dee709b7f906n%40chromium.org

[Rick Byers]

I looked into the details of the standards debate on this issue. It sounds like it's still unclear whether the spec for this has WG support or not, right? I certainly wouldn't want to mislead anyone as to API maturity / likely interoperability by shipping based on a WebRTC WG specification if there is an unresolved process concern.

That said, I think Chromium's position on the technical debate here is pretty clear - we do believe there's value in such stats APIs, even IF they can only represent browser bugs (it's why we ship a crash reporting API, which has been similarly controversial with Mozilla and Apple). We disagree that "there's nothing web developers can do about it". Maybe that's true for Apple and Mozilla, but for Google we rely critically on developer feedback through both open (crbug.com) and private (Google partnerships) channels and we want to make it as easy as possible for developers to report an issue they're seeing to us in a way that's actionable. Our privacy policy limits Chrome's visibility into what's going on in the wild. So usually we find this requires a mix of both site-acquired telemetry and browser-required telemetry, and find the two can often complement each other nicely.  

Henrik, my advice if the WG doesn't have consensus for this API is to move it to some incubation venue (like a WICG group). You clearly have a community of web developers who want it, so it's probably more productive to focus standards energy among allies who share an interest for the use case, right? If you're willing to promptly move this to a WICG spec in the event the WG asks to remove it from their spec, then I don't think this debate changes anything from a blink API owner's perspective so I'm OK treating it as non-blocking. A subset of API owners met today (Daniel, Mike Taylor, Philip and I), and agreed with this stance. WDYT?

In terms of testing, normally we ask to see the tests land before approving an I2S. Any reason we shouldn't wait for that here?

Thanks,

   Rick

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/a594fc55-8030-4848-9fe6-549eccfdd8a8n%40chromium.org.

[Rick Byers]

On Wed, Dec 13, 2023 at 12:39 PM Rick Byers <rby...@chromium.org> wrote:

I looked into the details of the standards debate on this issue. It sounds like it's still unclear whether the spec for this has WG support or not, right? I certainly wouldn't want to mislead anyone as to API maturity / likely interoperability by shipping based on a WebRTC WG specification if there is an unresolved process concern.

That said, I think Chromium's position on the technical debate here is pretty clear - we do believe there's value in such stats APIs, even IF they can only represent browser bugs (it's why we ship a crash reporting API, which has been similarly controversial with Mozilla and Apple). We disagree that "there's nothing web developers can do about it". Maybe that's true for Apple and Mozilla, but for Google we rely critically on developer feedback through both open (crbug.com) and private (Google partnerships) channels and we want to make it as easy as possible for developers to report an issue they're seeing to us in a way that's actionable. Our privacy policy limits Chrome's visibility into what's going on in the wild. So usually we find this requires a mix of both site-acquired telemetry and browser-required telemetry, and find the two can often complement each other nicely.  

Sorry, unfortunate typo: "browser-acquired telemetry". Of course all of Chrome's telemetry is under user control, not "required" 😂

[Henrik Boström]

Thanks, Rick. Responses inline.

On Wednesday, December 13, 2023 at 6:39:48 PM UTC+1 Rick Byers wrote:

I looked into the details of the standards debate on this issue. It sounds like it's still unclear whether the spec for this has WG support or not, right? I certainly wouldn't want to mislead anyone as to API maturity / likely interoperability by shipping based on a WebRTC WG specification if there is an unresolved process concern.

I think it's safe to say we don't have consensus on the frame counters (exposing dropped/glitches), while capture latency hopefully be less contentious.

That said, I think Chromium's position on the technical debate here is pretty clear - we do believe there's value in such stats APIs, even IF they can only represent browser bugs (it's why we ship a crash reporting API, which has been similarly controversial with Mozilla and Apple). We disagree that "there's nothing web developers can do about it". Maybe that's true for Apple and Mozilla, but for Google we rely critically on developer feedback through both open (crbug.com) and private (Google partnerships) channels and we want to make it as easy as possible for developers to report an issue they're seeing to us in a way that's actionable. Our privacy policy limits Chrome's visibility into what's going on in the wild. So usually we find this requires a mix of both site-acquired telemetry and browser-required telemetry, and find the two can often complement each other nicely.  

Henrik, my advice if the WG doesn't have consensus for this API is to move it to some incubation venue (like a WICG group). You clearly have a community of web developers who want it, so it's probably more productive to focus standards energy among allies who share an interest for the use case, right? If you're willing to promptly move this to a WICG spec in the event the WG asks to remove it from their spec, then I don't think this debate changes anything from a blink API owner's perspective so I'm OK treating it as non-blocking. A subset of API owners met today (Daniel, Mike Taylor, Philip and I), and agreed with this stance. WDYT?

Me moving this to a WICG spec in the event that the WG asks to remove them from mediacapture-extensions sounds good to me.

If me and/or the WebRTC Audio Team has access to a WICG spec for these things, that may also give us a venue for, in the future, exploring playout glitch metrics in a more enthusiastic setting, which is in the early stages of discussions internally.

In terms of testing, normally we ask to see the tests land before approving an I2S. Any reason we shouldn't wait for that here?

Given the "controversy" around the glitch metrics, the WebRTC Audio Team wanted to get some Blink owners signals before they spend the engineering efforts to implement this (including WPTs).

But if Blink owners also see the value in these metrics and we have a plan (= move to WICG in the event that they are removed) I see no reason not to ask them to start implementation today.

For reference, see the WPTs we added for the video track stats here. The audio track stats WPTs should be similar and also complemented by C++ unit tests in lower layers.

Thanks,

   Rick

On Fri, Dec 8, 2023 at 11:53 AM Henrik Boström <hb...@chromium.org> wrote:

Contact emails

hb...@chromium.org, olka@chromium.org, h...@chromium.org

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

[Rick Byers]

Sounds good Henrik! Yes, from our brief discussion in the API owners meeting I believe you have support from at least 3 API owners to proceed in this direction. It's important to us that we engage constructively in the standards process even in areas of differing opinion, but it's also crucial to our process that we not let such differences in opinion and priority be an effective veto power over what we choose to ship in Chromium. I'd encourage you and the WebRTC group to formalize processes for amicably agreeing to disagree on the importance of different use cases, while still being open to technical feedback and doing what we reasonably can to maximize the chance that the APIs we ship can eventually be interoperable if priorities of the other engines someday change [*]. API owners would also be interested to hear any other arguments for why Chromium shipping these APIs would be bad for the web (on this list, or anywhere else). I know there's a messy history with WebRTC in particular and services coming to depend on Chromium-only APIs when suitable standards-track alternatives are available in other engines. That's IMHO definitely not a pattern we want to risk repeating.

Of course you also need Chrome privacy and security reviews, since it's important that features like this don't create a hole in our careful balance of side channel attack mitigations. But I see you already have privacy approval so hopefully security isn't too far off. You might want to wait for a signal there before starting implementation.

Personally I'm also less concerned about interoperability risks when it comes to metrics API. It's already the case that our top performance metrics (Core Web Vitals) have APIs exposed only in Chromium. There's certainly some interop risk there, eg. of sites optimizing in engine-specific ways. But in practice we've seen developers use these APIs mostly to make their sites faster in ways that generally apply to all engines. So in that case, Safari and Firefox are getting most of the benefit of these APIs existing without having to incur most of the cost, which seems like a fine outcome to me. Also, I'm confident that if we eventually agree with other engines on some better way to expose the same information, then we can deprecate and remove any API shape we ship today and customers can migrate over without causing user-visible breakage (worst-case we just return dummy values on the deprecated APIs). 

Rick

[*] My favorite example of this is Pointer Events where Apple was opposed to the use case, but also had good technical critiques of the API. We eventually (after a lot of research, open debate, and some flip-flopping by me) shipped a version of the API that addressed the legitimate technical concerns without addressing Apple's objections around the use case. Years later when a use case materialized for Apple (the Apple Pen), they just shipped the API in a fully interoperable fashion. That (as well as cases of the inverse where we realize we were wrong and unship) is what we mean by the blink process being designed for "eventual interoperability". 

[Alex Russell]

LGTM1

[Chris Harrelson]

LGTM2 to ship, with a commitment to move this part of the spec to WICG if it gets removed from the mediacapture-extensions spec.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org.

[Mike Taylor]

LGTM3, with same condition.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-HbWWNeZ4jyRr38WR75FtxOA2DUhkwrHH0-OZnjkq9cg%40mail.gmail.com.

[Henrik Boström]

Excellent, we have a plan!

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/2e451af1-5e0a-4dd9-a12c-fc30c7dff7dbn%40chromium.org.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+unsubscribe@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAOMQ%2Bw-HbWWNeZ4jyRr38WR75FtxOA2DUhkwrHH0-OZnjkq9cg%40mail.gmail.com.