Changing the thread model of Audio Worklet

[Hongchan Choi]

Hi platform-architecture-dev,

I am planning a (potentially sizable) change in the threading model of Audio Worklet. To recap:

• After the previous discussion on using RT thread for Audio Worklet, we have done the experimentation behind the flag and collected some positive feedback from close partners.
• palmer@ suggested limiting the scope to a "top-level" document and a consensus has been made in the security review.

Changes being planned are:

Using real-time priority thread for AudioWorkletThread

The AudioWorkletThread will be using an RT thread for the JS execution, but there is a restriction: an RT thread will only be allowed when an AudioContext is 1) spawned from a top-level document and 2) a real-time one.

I also suggest that we cap the number of RT threads, but I have not thought about the details yet.

And this leads to the next point: -

Per-context instantiation of AudioWorkletThread 

Currently, multiple AudioWorklet instances share a per-frame singleton AudioWorklerThread just like other worklet variants (Paint, Animation). If a frame has multiple contexts (realtime or offline) they all share one worker (DISPLAY priority) thread for JS audio processing.

However, this model has a problem. An OfflineAudioContext does not require a high priority thread because it doesn't have urgency. On the other hand, an AudioContext with AudioWorklet needs an RT thread. So if we have two or more contexts with different modes (real-time or offline) running in a frame, there will be a conflict in thread priority. (See https://crbug.com/1051992)

This is why I believe we need per-context thread instantiation. As nhiroki@ mentioned, the proposal basically makes an AudioContext with AudioWorklet an "AudioWorker", so this could potentially be a concern of performance isolation.

Safety devices

These are rough ideas to protect this feature from abuse:

• Cap the number of concurrent RT threads per frame.
  
• Dynamically adjust the thread priority when a tab goes background.
• A kill switch for the feature.

We are not considering:

• Penalizing an AudioWorklet if JS script execution causes too many audio buffer underruns. (This seems to be Mozilla's implementation)

Any feedback would be appreciated!

Best,

Hongchan

[Kentaro Hara]

Thanks Hongchan for driving all the work!

Using real-time priority thread for AudioWorkletThread
The AudioWorkletThread will be using an RT thread for the JS execution, but there is a restriction: an RT thread will only be allowed when an AudioContext is 1) spawned from a top-level document and 2) a real-time one.

+1.

Per-context instantiation of AudioWorkletThread 
Currently, multiple AudioWorklet instances share a per-frame singleton AudioWorklerThread just like other worklet variants (Paint, Animation). If a frame has multiple contexts (realtime or offline) they all share one worker (DISPLAY priority) thread for JS audio processing.
However, this model has a problem. An OfflineAudioContext does not require a high priority thread because it doesn't have urgency. On the other hand, an AudioContext with AudioWorklet needs an RT thread. So if we have two or more contexts with different modes (real-time or offline) running in a frame, there will be a conflict in thread priority. (See https://crbug.com/1051992)
This is why I believe we need per-context thread instantiation. As nhiroki@ mentioned, the proposal basically makes an AudioContext with AudioWorklet an "AudioWorker", so this could potentially be a concern of performance isolation.

If you go with the per-context thread instantiation, how many realtime physical threads will you create per process?

My understanding is 1 because all AudioWorklets will share the one realtime physical thread. But then I don't understand what you mean by "Cap the number of concurrent RT threads per frame." Won't it be 1...?

I'm fine with creating separate physical threads for AudioWorklets, OfflineAudioContext etc. My question is: can we share one realtime physical thread for all AudioWorklets on the process?

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNvD%3Dc_UXN_B0cZok2VNJc9z3cCHh%2BHyCPQynnz2A%2BzGHw%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Hiroki Nakagawa]

Thank you for starting the thread.

On Fri, Aug 7, 2020 at 10:43 AM Kentaro Hara <har...@chromium.org> wrote:

Thanks Hongchan for driving all the work!

Using real-time priority thread for AudioWorkletThread
The AudioWorkletThread will be using an RT thread for the JS execution, but there is a restriction: an RT thread will only be allowed when an AudioContext is 1) spawned from a top-level document and 2) a real-time one.

+1.

+1.

 

Per-context instantiation of AudioWorkletThread 
Currently, multiple AudioWorklet instances share a per-frame singleton AudioWorklerThread just like other worklet variants (Paint, Animation). If a frame has multiple contexts (realtime or offline) they all share one worker (DISPLAY priority) thread for JS audio processing.
However, this model has a problem. An OfflineAudioContext does not require a high priority thread because it doesn't have urgency. On the other hand, an AudioContext with AudioWorklet needs an RT thread. So if we have two or more contexts with different modes (real-time or offline) running in a frame, there will be a conflict in thread priority. (See https://crbug.com/1051992)
This is why I believe we need per-context thread instantiation. As nhiroki@ mentioned, the proposal basically makes an AudioContext with AudioWorklet an "AudioWorker", so this could potentially be a concern of performance isolation.

If you go with the per-context thread instantiation, how many realtime physical threads will you create per process?

My understanding is 1 because all AudioWorklets will share the one realtime physical thread. But then I don't understand what you mean by "Cap the number of concurrent RT threads per frame." Won't it be 1...?

I'm fine with creating separate physical threads for AudioWorklets, OfflineAudioContext etc. My question is: can we share one realtime physical thread for all AudioWorklets on the process?

I wonder if the issue could be somehow mitigated by just allocating separate threads to the set of OfflineAudioContext and the set of AudioContext with AudioWorklet. That requires only 2 threads, and we could lower the priority of the thread for OfflineAudioContext?

[Hiroki Nakagawa]

On Fri, Aug 7, 2020 at 5:48 PM Hiroki Nakagawa <nhi...@chromium.org> wrote:

Thank you for starting the thread.

On Fri, Aug 7, 2020 at 10:43 AM Kentaro Hara <har...@chromium.org> wrote:

Thanks Hongchan for driving all the work!

Using real-time priority thread for AudioWorkletThread
The AudioWorkletThread will be using an RT thread for the JS execution, but there is a restriction: an RT thread will only be allowed when an AudioContext is 1) spawned from a top-level document and 2) a real-time one.

+1.

+1.

 

Per-context instantiation of AudioWorkletThread 
Currently, multiple AudioWorklet instances share a per-frame singleton AudioWorklerThread just like other worklet variants (Paint, Animation). If a frame has multiple contexts (realtime or offline) they all share one worker (DISPLAY priority) thread for JS audio processing.
However, this model has a problem. An OfflineAudioContext does not require a high priority thread because it doesn't have urgency. On the other hand, an AudioContext with AudioWorklet needs an RT thread. So if we have two or more contexts with different modes (real-time or offline) running in a frame, there will be a conflict in thread priority. (See https://crbug.com/1051992)
This is why I believe we need per-context thread instantiation. As nhiroki@ mentioned, the proposal basically makes an AudioContext with AudioWorklet an "AudioWorker", so this could potentially be a concern of performance isolation.

If you go with the per-context thread instantiation, how many realtime physical threads will you create per process?

My understanding is 1 because all AudioWorklets will share the one realtime physical thread. But then I don't understand what you mean by "Cap the number of concurrent RT threads per frame." Won't it be 1...?

I'm fine with creating separate physical threads for AudioWorklets, OfflineAudioContext etc. My question is: can we share one realtime physical thread for all AudioWorklets on the process?

I wonder if the issue could be somehow mitigated by just allocating separate threads to the set of OfflineAudioContext and the set of AudioContext with AudioWorklet. That requires only 2 threads, and we could lower the priority of the thread for OfflineAudioContext?

s/somehow mitigated/somewhat mitigated/

[Hongchan Choi]

Thanks for the responses!

To haraken@:

> If you go with the per-context thread instantiation, how many realtime physical threads will you create per process?

I am still exploring different ideas. As a counter example, when you create an AudioContext, it creates a real-time priority thread. So having multiple RT threads per process is already possible. When you have 4 contexts running concurrently, you'll have 4 RT threads. So if we want to match such behavior, one possibility is to spawn a real-time thread per each AudioWorklet object.

> My understanding is 1 because all AudioWorklets will share the one realtime physical thread. But then I don't understand what you mean by "Cap the number of concurrent RT threads per frame." Won't it be 1...?

I also thought allowing unlimited RT thread spawning seems too much, and that's why I suggested a thread pool (with max cap) for Audio Worklet RT threads.

> I'm fine with creating separate physical threads for AudioWorklets, OfflineAudioContext etc. My question is: can we share one realtime physical thread for all AudioWorklets on the process?

Could you clarify a bit more? Creating a separate thread per AudioWorklet is fine, but we also need to share a physical thread for all AudioWorklets? (Perhaps typo here?)

To nhiroki@

> I wonder if the issue could be somehow mitigated by just allocating separate threads to the set of OfflineAudioContext and the set of AudioContext with AudioWorklet. That requires only 2 threads, and we could lower the priority of the thread for OfflineAudioContext?

What a coincidence. I thought about this solution a few days ago. This design works because:

1. For real-time contexts, they (should) never completely block the backing thread. If that happens, it means the author's code is trying to do too much, so it's fair to glitch. The application with a reasonable design should use one RT thread per application.

2. For OfflineAudioContext, there's no hard requirement for the deadline or render budget, so it could just use a background priority thread when it becomes available.

So the new structure would be:

WorkletThreadHolder

  : AudioWorkletThread

    : RealtimeAudioWorkletThread (backed by an RT worker thread) - this is shared by all AudioContexts.

    : OfflineAudioWorkletThread (backed by a background pri thread) - this is shared by all OfflineAudioContexts.

I guess I need to abandon the current prototype, but this design looks more sensible. WDYT?

Best,

Hongchan

You received this message because you are subscribed to the Google Groups "web-audio-team" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web-audio-tea...@google.com.
To view this discussion on the web visit https://groups.google.com/a/google.com/d/msgid/web-audio-team/CABj5diPWdWaDSoX4fP-8nO6cJ0O6FFKJvcmMjT%2B_D5j3yP0dfQ%40mail.gmail.com.

[Raymond Toy]

FWIW, I like the separation into AudioContext and OfflineAudioContext.  We don't create too many extra threads.  (Recall that each convolvernode can create up to 3 threads to do the convolution.) 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNvcKrcCgyJi-Dcp9vs04_zdNc8nyhY2c%2BE-ypudVDCKoA%40mail.gmail.com.

[Hiroki Nakagawa]

On Tue, Aug 11, 2020 at 8:14 AM Raymond Toy <rt...@chromium.org> wrote:

FWIW, I like the separation into AudioContext and OfflineAudioContext.  We don't create too many extra threads.  (Recall that each convolvernode can create up to 3 threads to do the convolution.) 

On Mon, Aug 10, 2020 at 3:02 PM Hongchan Choi <hong...@chromium.org> wrote:

Thanks for the responses!

To haraken@:

> If you go with the per-context thread instantiation, how many realtime physical threads will you create per process?

I am still exploring different ideas. As a counter example, when you create an AudioContext, it creates a real-time priority thread. So having multiple RT threads per process is already possible. When you have 4 contexts running concurrently, you'll have 4 RT threads. So if we want to match such behavior, one possibility is to spawn a real-time thread per each AudioWorklet object.

> My understanding is 1 because all AudioWorklets will share the one realtime physical thread. But then I don't understand what you mean by "Cap the number of concurrent RT threads per frame." Won't it be 1...?

I also thought allowing unlimited RT thread spawning seems too much, and that's why I suggested a thread pool (with max cap) for Audio Worklet RT threads.

> I'm fine with creating separate physical threads for AudioWorklets, OfflineAudioContext etc. My question is: can we share one realtime physical thread for all AudioWorklets on the process?

Could you clarify a bit more? Creating a separate thread per AudioWorklet is fine, but we also need to share a physical thread for all AudioWorklets? (Perhaps typo here?)

To nhiroki@

> I wonder if the issue could be somehow mitigated by just allocating separate threads to the set of OfflineAudioContext and the set of AudioContext with AudioWorklet. That requires only 2 threads, and we could lower the priority of the thread for OfflineAudioContext?

What a coincidence. I thought about this solution a few days ago. This design works because:

1. For real-time contexts, they (should) never completely block the backing thread. If that happens, it means the author's code is trying to do too much, so it's fair to glitch. The application with a reasonable design should use one RT thread per application.

2. For OfflineAudioContext, there's no hard requirement for the deadline or render budget, so it could just use a background priority thread when it becomes available.

So the new structure would be:

WorkletThreadHolder

  : AudioWorkletThread

    : RealtimeAudioWorkletThread (backed by an RT worker thread) - this is shared by all AudioContexts.

    : OfflineAudioWorkletThread (backed by a background pri thread) - this is shared by all OfflineAudioContexts.

I guess I need to abandon the current prototype, but this design looks more sensible. WDYT?

This design looks good to me :)

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAE3TgXH%3D-QYFiUE8GXSRDMTB3kAQ9UpcZ7UsO-CRwPKtf%2BBnYw%40mail.gmail.com.

[Kentaro Hara]

Could you clarify a bit more? Creating a separate thread per AudioWorklet is fine, but we also need to share a physical thread for all AudioWorklets? (Perhaps typo here?)

I tried to mean what nhiroki@ proposed. One physical thread for all AudioContexts and one physical thread for all OfflineAudioContexts.

I'm happy everyone is on the same page :)

[Hongchan Choi]

Thanks for building a consensus so quickly! :D I guess I can start working on a new patch now.

-Hongchan

[Hongchan Choi]

To follow-up, the patch is landed and available on 87.0.4247.0. 

The partners are quite thrilled about this change, and I appreciate all your help and support for this work!

Cheers,

Hongchan