Intent to Ship: AudioContext.outputLatency

[Ajay Rahatekar]

Contact emails

hong...@chromium.org, ajayra...@google.com

Explainer

N/A - the specification (W3C Recommendation) is already published.

Specification

https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency

Summary

AudioContext.outputLatency property is the estimation in seconds of audio output latency.

Technically, this is the interval between the time the UA requests the host system to play a buffer and the time at which the first sample in the buffer is actually processed by the audio output device. For devices such as speakers or headphones that produce an acoustic signal, this latter time refers to the time when a sample’s sound is produced.

Blink component

Blink>WebAudio

TAG review

Completed: Web Audio API specification is W3C Recommendation.

TAG review status

Completed: Web Audio API specification is W3C Recommendation.

Risks

There is a risk of the feature being used for fingerprinting. However outputLatency is the buffer size of the platform-provided audio callback, so the value is inherently platform-specific. That said, the majority of the platform audio buffer size is widely known. (MacOS = 128 frames, Windows = 10ms, Android = 96 frames, etc)

This feature  does not expose more than what you can query/infer from the UA string.

Interoperability and Compatibility

Gecko: Shipped/Shipping

WebKit: No signal

Web developers: Positive (https://github.com/WebAudio/web-audio-api/issues/12)

Other signals: A WebCodecs demo application needs this feature.

Is this feature fully tested by web-platform-tests?

No - this property represents a platform-provided output latency value of a selected audio device. It does not work properly without an actual audio device.

Flag name

N/A

Requires code in //chrome?

False

Tracking bug

http://crbug.com/785409

Launch bug

http://crbug.com/1258109

Estimated milestones

98

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5682265146261504

This intent message was generated by Chrome Platform Status.

--

Ajay Rahatekar | Technical Program Manager | ajayra...@google.com | 650-797-1279

[Yoav Weiss]

On Fri, Nov 12, 2021 at 5:33 PM 'Ajay Rahatekar' via blink-dev <blin...@chromium.org> wrote:

Contact emails

hong...@chromium.org, ajayra...@google.com

Explainer

N/A - the specification (W3C Recommendation) is already published.

Regardless, could you add a few sentences explaining what use cases the feature targets and how developers are expected to use it?

 

Specification

https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency

Summary

AudioContext.outputLatency property is the estimation in seconds of audio output latency.

Technically, this is the interval between the time the UA requests the host system to play a buffer and the time at which the first sample in the buffer is actually processed by the audio output device. For devices such as speakers or headphones that produce an acoustic signal, this latter time refers to the time when a sample’s sound is produced.

Blink component

Blink>WebAudio

TAG review

Completed: Web Audio API specification is W3C Recommendation.

Link to the TAG review?

 

TAG review status

Completed: Web Audio API specification is W3C Recommendation.

Risks

There is a risk of the feature being used for fingerprinting. However outputLatency is the buffer size of the platform-provided audio callback, so the value is inherently platform-specific. That said, the majority of the platform audio buffer size is widely known. (MacOS = 128 frames, Windows = 10ms, Android = 96 frames, etc)

This feature  does not expose more than what you can query/infer from the UA string.

Would that exposure map cleanly to UA-Platform, which is considered low-entropy and exposed by default? Or would it add more than that?

/cc +Mike Taylor   

Interoperability and Compatibility

Gecko: Shipped/Shipping

Link?

 

WebKit: No signal

Have we reached out? https://bit.ly/blink-signals

 

Web developers: Positive (https://github.com/WebAudio/web-audio-api/issues/12)

Other signals: A WebCodecs demo application needs this feature.

Is this feature fully tested by web-platform-tests?

No - this property represents a platform-provided output latency value of a selected audio device. It does not work properly without an actual audio device.

I'm guessing it is tested using browser tests or unit tests without an audio device, right?

Have y'all filed bugs against WPT to enable testing it similarly? 

Flag name

N/A

Requires code in //chrome?

False

Tracking bug

http://crbug.com/785409

Launch bug

http://crbug.com/1258109

Estimated milestones

98

Link to entry on the Chrome Platform Status

https://chromestatus.com/feature/5682265146261504

This intent message was generated by Chrome Platform Status.

--

Ajay Rahatekar | Technical Program Manager | ajayra...@google.com | 650-797-1279

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAHB%2BDAgeEo86rN4TrL4n8DutRczgXAogTwYLTo%2BPZco84aMq%3Dg%40mail.gmail.com.

[Mike Taylor]

On 11/15/21 1:47 AM, Yoav Weiss wrote:

On Fri, Nov 12, 2021 at 5:33 PM 'Ajay Rahatekar' via blink-dev <blin...@chromium.org> wrote:

TAG review status

Completed: Web Audio API specification is W3C Recommendation.

Risks

There is a risk of the feature being used for fingerprinting. However outputLatency is the buffer size of the platform-provided audio callback, so the value is inherently platform-specific. That said, the majority of the platform audio buffer size is widely known. (MacOS = 128 frames, Windows = 10ms, Android = 96 frames, etc)

This feature  does not expose more than what you can query/infer from the UA string.

Would that exposure map cleanly to UA-Platform, which is considered low-entropy and exposed by default? Or would it add more than that?

/cc +Mike Taylor   

Looking at https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency, it states that it depends on the platform _and_ the hardware output device. If I use an app using outputLatency with speaker A, then switch to speaker B, will the outputLatency remain the same?

[Hongchan Choi]

Thanks for the review, Yoav. My responses are inline:

On Sun, Nov 14, 2021 at 10:48 PM Yoav Weiss <yoav...@chromium.org> wrote:

On Fri, Nov 12, 2021 at 5:33 PM 'Ajay Rahatekar' via blink-dev <blin...@chromium.org> wrote:

Contact emails

hong...@chromium.org, ajayra...@google.com

Explainer

N/A - the specification (W3C Recommendation) is already published.

Regardless, could you add a few sentences explaining what use cases the feature targets and how developers are expected to use it?

This feature can be useful when you need to synchronize a video stream (or any kind of visual generated by the system) and an audio stream produced by Web Audio API. For example, WebCodecs can be used to decode a MediaStream into raw video and audio data, and then an app can apply processing only on audio data via Web Audio API. This will break the original synchronization, but this can be fixed by taking the hardware audio latency into account.

 

 

Specification

https://www.w3.org/TR/webaudio/#dom-audiocontext-outputlatency

Summary

AudioContext.outputLatency property is the estimation in seconds of audio output latency.

Technically, this is the interval between the time the UA requests the host system to play a buffer and the time at which the first sample in the buffer is actually processed by the audio output device. For devices such as speakers or headphones that produce an acoustic signal, this latter time refers to the time when a sample’s sound is produced.

Blink component

Blink>WebAudio

TAG review

Completed: Web Audio API specification is W3C Recommendation.

Link to the TAG review?

https://github.com/w3ctag/design-reviews/issues/212

This was a blanket review for the entire API surface. No issues were raised against the `outputLatency` property.

 

 

TAG review status

Completed: Web Audio API specification is W3C Recommendation.

Risks

There is a risk of the feature being used for fingerprinting. However outputLatency is the buffer size of the platform-provided audio callback, so the value is inherently platform-specific. That said, the majority of the platform audio buffer size is widely known. (MacOS = 128 frames, Windows = 10ms, Android = 96 frames, etc)

This feature  does not expose more than what you can query/infer from the UA string.

Would that exposure map cleanly to UA-Platform, which is considered low-entropy and exposed by default? Or would it add more than that?

/cc +Mike Taylor   

I'll do some more investigation and follow up.

 

Interoperability and Compatibility

Gecko: Shipped/Shipping

Link?

https://bugzilla.mozilla.org/show_bug.cgi?id=1324552

 

 

WebKit: No signal

Have we reached out? https://bit.ly/blink-signals

We have not reached out, and did not see any comments in the spec discussion.

 

 

Web developers: Positive (https://github.com/WebAudio/web-audio-api/issues/12)

Other signals: A WebCodecs demo application needs this feature.

Is this feature fully tested by web-platform-tests?

No - this property represents a platform-provided output latency value of a selected audio device. It does not work properly without an actual audio device.

I'm guessing it is tested using browser tests or unit tests without an audio device, right?

Have y'all filed bugs against WPT to enable testing it similarly? 

Currently we only have an IDL test:

https://wpt.fyi/results/webaudio/idlharness.https.window.html?label=experimental&label=master&aligned

No. We have not filed a bug for the new type of testing. I am not aware if we have that kind of testing infrastructure on WPT. Please correct me if I am mistaken.

[Hongchan Choi]

The specification says: If the audio output device is changed the outputLatency attribute value will be updated accordingly.

So the answer is no. The value will change accordingly.

[Mike Taylor]

If that's the case, then outputLatency can reveal more entropy than just platform alone, right? It would be useful to know what this looks like in practice, or what mitigations we might be able to apply depending on the size of these latency differences.

[Ajay Rahatekar]

The small number of platform + audio output device combination i.e. a large number of users with the same platform and output device  (users on macOS using airpods) would likely be a natural obfuscator. Not sure if we have data about the distribution.

[Mike Taylor]

I suppose it would be nice to do some testing to verify the idea that there's a small number of platform + audio device pairs. I was hoping Mozilla's hardware survey might have something on that, but it doesn't track speakers.

Just from some simple testing with the following script (thanks dmcardle@ for the snippet) in Firefox on macOS by 3 people, we have very different values that change dramatically when you unplug a headset:

```

(function getNextSample() {
    console.log((new AudioContext).outputLatency);
    setTimeout(getNextSample, 10);
})();

```

Here's the values on my macbook, using Firefox Nightly:

bose headset plugged in: 0.005895833

take headset out: 0.013083333

plug bose headset back in: 0.005875 (different, but stable)

https://www.w3.org/TR/webaudio/#priv-sec mentions some mitigation strategies - do we have a plan to apply any of them?

[Hongchan Choi]

Thanks for the feedback, Mike. This is really helpful!

Mitigation strategies include adding jitter (dithering) and quantization so that the exact skew is incorrectly reported

This bit refers to the randomization mechanism deployed by other browsers, but this approach defeats the purpose of this specific API and it is also argued in the spec:

Note however that most audio systems aim for low latency, to synchronise the audio generated by WebAudio to other audio or video sources or to visual cues (for example in a game, or an audio recording or music making environment). Excessive latency decreases usability and may be an accessibility issue.

 

If the reported value is incorrect, the A/V synchronization won't be possible. (Perhaps chcunningham@ can say more about this use case.)

One mitigation idea: we can do some research to collect values reported by major platforms/devices and devise a reasonable "bucket system" for them.

[Chris Harrelson]

Ok. Please file a signals request on webkit-dev.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/CAGJqXNv9r4hew4pknp-7iLHjP5hUY2mh1EkrHLFaeQaBzQf5VA%40mail.gmail.com.

[Yoav Weiss]

Hey folks!

It seems like there are some things to figure out here before shipping. Could y'all file a spec issue to discuss this with the broader community?

[Hongchan Choi]

Hi Yoav,

Could you clarify what needs to be discussed from the spec side? The feature (and its behavior) is properly specified, and its privacy aspect is already reviewed by W3C:

https://github.com/WebAudio/web-audio-api/issues/1498 (outputLatency)

https://github.com/WebAudio/web-audio-api/issues/1495 (general security/privacy consideration)

I thought the request from this thread is to devise a Chrome-specific mitigation, not reopening the spec discussion. Please correct me if I'm mistaken.

Cheers,

Hongchan

[Chris Cunningham]

> If the reported value is incorrect, the A/V synchronization won't be possible. (Perhaps chcunningham@ can say more about this use case.)

The typical video player design drives the clock with audio. In short, you send buffers of audio to the OS, use the latency value to know when those buffers actually reach the user's ears, and choose a video frame accordingly. Apps using WebCodecs will design apps this way (example here).

As Mike notes, the latency value can vary by quite a lot depending on the device (huge differences for bluetooth vs wired), so it's critical to have the platform tell you when this changes. In terms of precision, we can _probably_ do some rounding like to the nearest millisecond. Higher values may lead to issues; user studies have shown sensitivity to fairly low values of drift. 

Aside: we may eventually desire a display latency value for similar reasons. 

[Yoav Weiss]

Apologies, I misread the discussion. A spec issue is indeed not needed, as this is already covered.

From my perspective, since this exposes active entropy, the fact that it exposes more data than the (passively exposed) platform is not prohibitive on its own.

At the same time, it'd be good to work with privacy-focused folks to make sure we expose as little information as needed to maintain the API's functionality, but not more.

[Hongchan Choi]

Thanks for the clarification, Yoav!

I hope the mitigation approach in the previous comment makes sense from the privacy perspective, but it will hurt the usability/value of this API significantly.

For the privacy-focused discussion, should I use other venues rather than this I2S thread?

Cheers,

Hongchan

[Chris Cunningham]

Re: privacy discussion, Hongchan and I scheduled to meet with Paul Jensen next week.

Also, I recently determined that this new WebAudio API (outputLatency) is largely redundant with an existing API (getOutputTimestamp()). 

https://github.com/WebAudio/web-audio-api/issues/2461

Chrome has already shipped the latter API, so I believe this new API doesn't add potential for additional fingerprinting. Shipping both APIs is still a good idea for the sake of inter-op.

[Hongchan Choi]

I agree that outputLatency is better in terms of both usability and interoperability.

FYI, the counter shows getOutputTimestamp is about 0.001% and there are no partners who are using this API:

https://chromestatus.com/metrics/feature/popularity#AudioContextGetOutputTimestamp

[Yoav Weiss]

So are y'all planning to deprecate and remove `getOutputTimestamp` once this ships? If so, that sounds reasonable.

I note Chris asked y'all to file for a webkit signal upthread. Did that happen?

[Hongchan Choi]

So far we have not discussed the deprecation plan, but I believe that's reasonable as well.

I registered myself to the webkit-dev list to post a question, but somehow I am getting 403 forbidden. The email is sent to the list, I am not seeing the message is getting posted on the list. Not sure what to do there.

[Mike Taylor]

To circle back - last week we (myself + some other privacy/anti-covert tracking reviewers) met with some folks working on this feature. We ended up with an AI for Hongchan to land metrics on the values that getOutputTimestamp would produce (if called, when a new AudioContext is created, IIRC) as a way to get some data on the utility for using either of these APIs for device fingerprinting.

[Yoav Weiss]

LGTM1 conditional on those metrics landing (but no need to wait for data from the metrics to come in).

[Mike Taylor]

LGTM2 w/ same condition.

[Chris Harrelson]

LGTM3.

--
You received this message because you are subscribed to the Google Groups "blink-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blink-dev+...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/blink-dev/936f3e2f-2001-0449-13fd-d3a0548b673c%40chromium.org.

[Hongchan Choi]

Thank you all!

1. I'll land the metrics CL and follow up here.

2. The discussion in the WebKit mailing list is ongoing, so I'll report back when it's done.

Cheers,

Hongchan

[Joe Medley]

Is this shipping in 98?

[Ajay Rahatekar]

The metrics are landing in Jan. So likely shipping in 99.

-Ajay

[Joe Medley]

Please update the Chrome Status entry as soon as you know. (That will inform me as well.)

Joe Medley | Technical Writer, Chrome DevRel | jme...@google.com | 816-678-7195

If an API's not documented it doesn't exist.

[Ajay Rahatekar]

Will do. Ty Joe.

[Hongchan Choi]

Hello all,

We landed the metric collection CL (https://crrev.com/c/3413794) and have gathered some data:

# Notable latency range
- Android Chrome: Mean 40.96, Peak 3~5ms
- Chrome OS: Mean 46.37, Peak at 15ms
- Mac OS X: Mean 935.36, Peak at 17ms
- Linux (dev): Mean 45.32, Peak at 33ms
- Windows: Mean 31.77, Peak at 31ms

More details can be found here: go/webaudio-outputlatency-metric-collection (sorry, googler-only)

# Thoughts
- Notable peaks roughly match the platform.
- Mac OS X has a diverse distribution and there is a small peak at 200ms on Mac OSX: it may indicate that the client is using the “AirPods”. (~1%)

We believe the value exposed by this property does not pose a significant issue on the user privacy, but any feedback would be appreciated!

Best,

Hongchan