Introducing a flag setting for realtime priority thread for AudioWorklet

646 views
Skip to first unread message

Hongchan Choi

unread,
May 13, 2019, 1:20:33 PM5/13/19
to platform-architecture-dev, Raymond Toy, Chris Palmer
crbug issue: https://bugs.chromium.org/p/chromium/issues/detail?id=813825

After numerous discussions with pro-audio and game audio partners, WebAudio team would like to do some experiments with realtime priority thread for Audio Worklet.

This idea was previously discussed in this group before the launch of Audio Worklet. We decided to use the "graphics priority thread" for Audio Worklet. Apparently, it opened up the huge opportunities for audio devs, but also it was somewhat disappointing that the audio start glitching when the visual change dominate the system. Many audio devs are claiming this is a show-stopper for serious audio applications, and they are willing to experiment with the realtime priority thread.

So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

Any feedback would be appreciated.

-Hongchan

Kentaro Hara

unread,
May 13, 2019, 2:26:44 PM5/13/19
to Hongchan Choi, platform-architecture-dev, Raymond Toy, Chris Palmer
So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

+1 to experimenting with the flag and getting feedback from developers.

However, I don't think we can ship it as is, as we discussed in the old thread. We'll need to implement some way to prevent any arbitrary JS from consuming 100% CPU time (e.g., interrupt V8 periodically to yield execution).


--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/84e439b5-d611-41b2-a327-3ca142ad8617%40chromium.org.


--
Kentaro Hara, Tokyo, Japan

Hongchan Choi

unread,
May 13, 2019, 2:31:40 PM5/13/19
to platform-architecture-dev, hong...@chromium.org, rt...@chromium.org, pal...@chromium.org
Thanks, haraken! :)

On Monday, May 13, 2019 at 11:26:44 AM UTC-7, Kentaro wrote:
So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

+1 to experimenting with the flag and getting feedback from developers.

However, I don't think we can ship it as is, as we discussed in the old thread. We'll need to implement some way to prevent any arbitrary JS from consuming 100% CPU time (e.g., interrupt V8 periodically to yield execution).

How/where can I kick off this discussion? Any suggestion?
 

From: Hongchan Choi <hong...@chromium.org>
Date: Mon, May 13, 2019 at 6:20 PM
To: platform-architecture-dev
Cc: Raymond Toy, Chris Palmer

crbug issue: https://bugs.chromium.org/p/chromium/issues/detail?id=813825

After numerous discussions with pro-audio and game audio partners, WebAudio team would like to do some experiments with realtime priority thread for Audio Worklet.

This idea was previously discussed in this group before the launch of Audio Worklet. We decided to use the "graphics priority thread" for Audio Worklet. Apparently, it opened up the huge opportunities for audio devs, but also it was somewhat disappointing that the audio start glitching when the visual change dominate the system. Many audio devs are claiming this is a show-stopper for serious audio applications, and they are willing to experiment with the realtime priority thread.

So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

Any feedback would be appreciated.

-Hongchan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

Kentaro Hara

unread,
May 13, 2019, 2:45:44 PM5/13/19
to Hongchan Choi, platform-architecture-dev, rtoy, Chris Palmer
How/where can I kick off this discussion? Any suggestion?

I thought about this a bit more (to see if there's any simpler way to achieve it without interrupting V8).

Is it possible to change the thread priority dynamically? If yes, one approach would be 1) to observe an execution time of each worklet script and 2) if the time exceeds some threshold (e.g., 50 ms), bump down the thread priority.

This is not a perfect solution in the sense that the first script execution may consume 100% CPU time and that the thread priority may be bumped down by an unexpected factor (e.g., by heavy pages that happened to be hosted in the same process). However, it would be okay to not worry about these edge cases too much because there are already many ways to evilly eat CPU time...



From: Hongchan Choi <hong...@chromium.org>
Date: Mon, May 13, 2019 at 7:31 PM
To: platform-architecture-dev 
Cc: <hong...@chromium.org>, <rt...@chromium.org>, <pal...@chromium.org>

Thanks, haraken! :)

On Monday, May 13, 2019 at 11:26:44 AM UTC-7, Kentaro wrote:
So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

+1 to experimenting with the flag and getting feedback from developers.

However, I don't think we can ship it as is, as we discussed in the old thread. We'll need to implement some way to prevent any arbitrary JS from consuming 100% CPU time (e.g., interrupt V8 periodically to yield execution).

How/where can I kick off this discussion? Any suggestion?
 
From: Hongchan Choi <hong...@chromium.org>
Date: Mon, May 13, 2019 at 6:20 PM
To: platform-architecture-dev
Cc: Raymond Toy, Chris Palmer

crbug issue: https://bugs.chromium.org/p/chromium/issues/detail?id=813825

After numerous discussions with pro-audio and game audio partners, WebAudio team would like to do some experiments with realtime priority thread for Audio Worklet.

This idea was previously discussed in this group before the launch of Audio Worklet. We decided to use the "graphics priority thread" for Audio Worklet. Apparently, it opened up the huge opportunities for audio devs, but also it was somewhat disappointing that the audio start glitching when the visual change dominate the system. Many audio devs are claiming this is a show-stopper for serious audio applications, and they are willing to experiment with the realtime priority thread.

So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

Any feedback would be appreciated.

-Hongchan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/43f0378f-edbd-4e91-b38a-2cb48c71fd57%40chromium.org.

Dave Tapuska

unread,
May 13, 2019, 2:55:46 PM5/13/19
to Kentaro Hara, Hongchan Choi, platform-architecture-dev, rtoy, Chris Palmer
Kentaro, I was thinking this as well. Basically it could be done if you need it to ask for a thread priority bump and some class limited how much that was done, ie. ensuring the main thread was run suffciiently enough or something like that.

I thought we didn't generally do dynamic thread priority adjustment in Chrome because it was expensive on some platform. But I'd defer to someone on the scheduling team might have an idea.

Also with my WorkerThread pause/resume code it is also is easier to interrupt a worker thread now. You could pause; send something to the main thread ensure it executes and then resume.  But generally I prefer the approach for a "temporary upgrade" of this thread....

dave.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jyD8vvXshPcTxh0RAJ6fhnrJGi_t2Lx4KJ3t-LpgQRWQQ%40mail.gmail.com.

Kentaro Hara

unread,
May 13, 2019, 3:06:53 PM5/13/19
to Dave Tapuska, Hongchan Choi, platform-architecture-dev, rtoy, Chris Palmer
Also with my WorkerThread pause/resume code it is also is easier to interrupt a worker thread now. You could pause; send something to the main thread ensure it executes and then resume.  But generally I prefer the approach for a "temporary upgrade" of this thread....

Yeah. The reason I want to avoid using V8 interruption is that it requires another thread to keep interrupting the audio rendering thread periodically (e.g., 50 ms). That interruption logic will add a lot of overhead... :/

Hongchan Choi

unread,
May 13, 2019, 4:05:36 PM5/13/19
to platform-architecture-dev, hong...@chromium.org, rt...@chromium.org, pal...@chromium.org


On Monday, May 13, 2019 at 11:45:44 AM UTC-7, Kentaro wrote:
How/where can I kick off this discussion? Any suggestion?

I thought about this a bit more (to see if there's any simpler way to achieve it without interrupting V8).

Is it possible to change the thread priority dynamically? If yes, one approach would be 1) to observe an execution time of each worklet script and 2) if the time exceeds some threshold (e.g., 50 ms), bump down the thread priority.

Actually, FireFox WebAudio devs are planning a similar approach. However, I don't think this approach is quite helpful for the audio processing purpose. It's because this dynamic change is invisible from dev's point of view. It just happens and starts glitching in a unpredictable way. Currently, Audio Worklet does not have a way of notifying the user code when the buffer underflow happens.

Having dynamic thread priority might be beneficial to other purposes, but I prefer not to use for the Audio Worklet case. Especially for this experiment behind the flag - because I want to establish some ground with audio devs by running the RT thread in various scenarios. Many people believe using the RT thread will solve all the problems, but I kind of doubt that it is an absolute magic.

 

This is not a perfect solution in the sense that the first script execution may consume 100% CPU time and that the thread priority may be bumped down by an unexpected factor (e.g., by heavy pages that happened to be hosted in the same process). However, it would be okay to not worry about these edge cases too much because there are already many ways to evilly eat CPU time...



From: Hongchan Choi <hong...@chromium.org>
Date: Mon, May 13, 2019 at 7:31 PM
To: platform-architecture-dev 
Cc: <hong...@chromium.org>, <rt...@chromium.org>, <pal...@chromium.org>

Thanks, haraken! :)

On Monday, May 13, 2019 at 11:26:44 AM UTC-7, Kentaro wrote:
So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

+1 to experimenting with the flag and getting feedback from developers.

However, I don't think we can ship it as is, as we discussed in the old thread. We'll need to implement some way to prevent any arbitrary JS from consuming 100% CPU time (e.g., interrupt V8 periodically to yield execution).

How/where can I kick off this discussion? Any suggestion?
 
From: Hongchan Choi <hong...@chromium.org>
Date: Mon, May 13, 2019 at 6:20 PM
To: platform-architecture-dev
Cc: Raymond Toy, Chris Palmer

crbug issue: https://bugs.chromium.org/p/chromium/issues/detail?id=813825

After numerous discussions with pro-audio and game audio partners, WebAudio team would like to do some experiments with realtime priority thread for Audio Worklet.

This idea was previously discussed in this group before the launch of Audio Worklet. We decided to use the "graphics priority thread" for Audio Worklet. Apparently, it opened up the huge opportunities for audio devs, but also it was somewhat disappointing that the audio start glitching when the visual change dominate the system. Many audio devs are claiming this is a show-stopper for serious audio applications, and they are willing to experiment with the realtime priority thread.

So I would like to float this idea one more time. If this requires more extensive justification with threat eval and etc, I'll start a doc. However, I think adding a flag so devs can test/experiment would not be so controversial.

Any feedback would be appreciated.

-Hongchan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

Hongchan Choi

unread,
May 13, 2019, 4:11:27 PM5/13/19
to platform-architecture-dev, dtap...@chromium.org, hong...@chromium.org, rt...@chromium.org, pal...@chromium.org


On Monday, May 13, 2019 at 12:06:53 PM UTC-7, Kentaro wrote:
Also with my WorkerThread pause/resume code it is also is easier to interrupt a worker thread now. You could pause; send something to the main thread ensure it executes and then resume.  But generally I prefer the approach for a "temporary upgrade" of this thread....

Yeah. The reason I want to avoid using V8 interruption is that it requires another thread to keep interrupting the audio rendering thread periodically (e.g., 50 ms). That interruption logic will add a lot of overhead... :/

This won't be pretty with audio, but I just don't have any idea how bad it will be. Just for the reference, our render quantum is ~3ms. If the render request is not fulfilled in this window, the underflow happens.

Another question: when this periodic interruption detects the delay in the Audio Worklet Thread, how does it intervene? Does it stop the current script being executed? That also leads to the other issue because the next render frame might expect some states from the previous render frame, but this intervention will render the previous quantum in a premature state.

 


Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

Raymond Toy

unread,
May 13, 2019, 4:32:32 PM5/13/19
to Hongchan Choi, platform-architecture-dev, pal...@chromium.org
I agree with Hongchan on this.  A dynamically changing priority will be super-confusing to devs because sometimes it works and sometimes it doesn't because we momentarily reduced priority and then increased it back.  I am perfectly happy to let devs lock up an AudioWorklet if they're not careful.  Well, assuming this doesn't mess up other v8 isolates (which I hear aren't truly isolated from everything else?).

Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

Kentaro Hara

unread,
May 13, 2019, 5:28:32 PM5/13/19
to Raymond Toy, Hongchan Choi, platform-architecture-dev, Chris Palmer
Thanks, I now understand that dynamically changing the thread priority is not an option.

Another question: when this periodic interruption detects the delay in the Audio Worklet Thread, how does it intervene? Does it stop the current script being executed?

No. The V8 interruption just interrupts the script execution and temporarily gets control back to a C++ callback in Blink. Once the C++ callback returns, the script execution continues. I was assuming that we call pthread_yield() in the C++ callback.

I am perfectly happy to let devs lock up an AudioWorklet if they're not careful.  Well, assuming this doesn't mess up other v8 isolates (which I hear aren't truly isolated from everything else?).

The concern is not the AudioWorklet itself but the fact that the AudioWorklet occupies 100% of the CPU resources and disturbs other tabs and more important work...

(As I mentioned in the original thread, it's not a good design to run worklet (i.e., any arbitrary script) on a realtime thread...)


Hongchan Choi

unread,
May 14, 2019, 5:58:38 PM5/14/19
to platform-architecture-dev, rt...@chromium.org, hong...@chromium.org, pal...@chromium.org
> Yeah. The reason I want to avoid using V8 interruption is that it requires another thread to keep interrupting the audio rendering thread periodically (e.g., 50 ms). That interruption logic will add a lot of overhead... :/

Can it be something like 1 second or beyond? Or 50ms is something common for this kind of mechanism? I am not sure what kind of overheads are expected to happen. Can you elaborate a little so I can understand how it works?

Also this might be a slight digression, but who's the best person to talk to when it comes to the thread priorities?
I file this issue a while ago, but no responses yet: https://bugs.chromium.org/p/chromium/issues/detail?id=955744

-Hongchan
Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

Chris Palmer

unread,
May 14, 2019, 9:34:39 PM5/14/19
to Hongchan Choi, platform-architecture-dev, rt...@chromium.org
Hongchan and Raymond talked to me a few weeks ago about the security aspect of all this. I had some tiny thoughts:

* Shipping any experiment behind a flag is pretty much OK by me. So I think they can at least get some developer feedback on a pure RT AudioWorklet, even if that's not what we ultimately ship.

* Another idea was enforcing that the RT AudioWorklet is spawned only by the top-level document. The idea there is that, even if the thing is using 100% of a CPU core, it's more likely to actually be audio-app.example.com, and in line with the user's expectations. There are going to be legitimate applications that really do need to use a whole core for audio processing, so we need to find a way to make that OK. We probably can agree, at least for the first version, that iframes are less likely to be the app that the user is intentionally engaging with. (We wouldn't want to let an ad eat a whole core, for example. Or even to have RT at all.)

Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/2af84745-fb05-4bbb-b097-186bb1665dda%40chromium.org.

Kentaro Hara

unread,
May 15, 2019, 3:06:14 AM5/15/19
to Chris Palmer, Hongchan Choi, platform-architecture-dev, rtoy
* Shipping any experiment behind a flag is pretty much OK by me. So I think they can at least get some developer feedback on a pure RT AudioWorklet, even if that's not what we ultimately ship.

+1 on this!

* Another idea was enforcing that the RT AudioWorklet is spawned only by the top-level document. The idea there is that, even if the thing is using 100% of a CPU core, it's more likely to actually be audio-app.example.com, and in line with the user's expectations. There are going to be legitimate applications that really do need to use a whole core for audio processing, so we need to find a way to make that OK. We probably can agree, at least for the first version, that iframes are less likely to be the app that the user is intentionally engaging with. (We wouldn't want to let an ad eat a whole core, for example. Or even to have RT at all.)

Yeah, this sounds like a reasonable approach to me. I'm fine with going with it :)

My unhappiness is about introducing new web APIs that have a risk of slowing down the web (AudioWorklet allows developers to run any arbitrary script on a realtime thread) but it will be a question to the standardization discussion rather than the implementation. Given that we have decided to ship it after considering all the risks, we need to think about a reasonable way to implement it. From that perspective, the approach Chris suggested sounds reasonable to me.


Kentaro Hara

unread,
May 15, 2019, 3:13:17 AM5/15/19
to Chris Palmer, Hongchan Choi, platform-architecture-dev, rtoy
Can it be something like 1 second or beyond? Or 50ms is something common for this kind of mechanism? I am not sure what kind of overheads are expected to happen. Can you elaborate a little so I can understand how it works?

1) When the AudioWorklet starts, ask the main thread to call v8::Isolate::RequestInterrupt() against the audio rendering thread's isolate every X ms.
2) The AudioWorklet will be interrupted mostly every X ms. Check if AudioWorklet has been running script for more than X ms. If no, just ignore the interruption. If yes, call sleep(Y ms).

This will add a bunch of cross-thread interactions, so is not really nice. I'd personally prefer the approach Chris proposed.


Also this might be a slight digression, but who's the best person to talk to when it comes to the thread priorities?
I file this issue a while ago, but no responses yet: https://bugs.chromium.org/p/chromium/issues/detail?id=955744

Sorry for missing it -- I replied on the bug :)

Hongchan Choi

unread,
May 15, 2019, 12:08:46 PM5/15/19
to platform-architecture-dev, pal...@chromium.org, hong...@chromium.org, rt...@chromium.org
Great. Thanks everyone for feedback and guidance!

It sounds like we have two different tasks: 1) experimenting the RT thread with RT thread flag setting and 2) investigating the solution that Chris suggested above. (i.e. using RT thread only for top-level document)

I already have a CL for 1) so we can start the review. For 2), I'll start a design doc so we can kick of the in-depth discussion.


On Wednesday, May 15, 2019 at 12:13:17 AM UTC-7, Kentaro wrote:
Can it be something like 1 second or beyond? Or 50ms is something common for this kind of mechanism? I am not sure what kind of overheads are expected to happen. Can you elaborate a little so I can understand how it works?

1) When the AudioWorklet starts, ask the main thread to call v8::Isolate::RequestInterrupt() against the audio rendering thread's isolate every X ms.
2) The AudioWorklet will be interrupted mostly every X ms. Check if AudioWorklet has been running script for more than X ms. If no, just ignore the interruption. If yes, call sleep(Y ms).

This will add a bunch of cross-thread interactions, so is not really nice. I'd personally prefer the approach Chris proposed.

Agreed. Any blocking operation hurts the audio performance.
 


Also this might be a slight digression, but who's the best person to talk to when it comes to the thread priorities?
I file this issue a while ago, but no responses yet: https://bugs.chromium.org/p/chromium/issues/detail?id=955744

Sorry for missing it -- I replied on the bug :)

Thanks! I hope we can fix this soon.

 


Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

Alexander Timin

unread,
May 17, 2019, 12:14:06 PM5/17/19
to Hongchan Choi, platform-architecture-dev, pal...@chromium.org, Raymond Toy, scheduler-dev
+scheduler-dev 

It would be interesting to monitor the changes in the thread runtime of the WebAudioThread (we capture that in RendererScheduler.TaskCPUDurationPerThreadType2 histogram). 

There are bunch of things that we can do here from the scheduling perspective, but it seems that limiting it to top-level document only is the best of them all.


Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/bfa7283d-0f26-479e-af13-af2868e21dc2%40chromium.org.

Hongchan Choi

unread,
May 17, 2019, 5:04:45 PM5/17/19
to platform-architecture-dev, hong...@chromium.org, pal...@chromium.org, rt...@chromium.org, schedu...@chromium.org, alt...@chromium.org
Thanks for your feedback (and the code review), Alexander!

The CL for the option 1) is up here and ready to land:

For the option 2), I'll start a doc so we can have a venue for discussion. Would love to hear your thoughts from the scheduler's perspective. 

Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

Chris Palmer

unread,
May 17, 2019, 6:22:46 PM5/17/19
to Hongchan Choi, platform-architecture-dev, rt...@chromium.org, schedu...@chromium.org, alt...@chromium.org
I should add: for the top-level-document-only solution, it also seems OK for iframes in the same origin as the top-level document to have access to RT AudioWorklets.

Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/05e7813d-0b38-43aa-ab9d-2530ceab7fcb%40chromium.org.

Hongchan Choi

unread,
May 21, 2019, 12:50:43 PM5/21/19
to platform-architecture-dev, hong...@chromium.org, rt...@chromium.org, schedu...@chromium.org, alt...@chromium.org
FWIW, the flag setting for RT thread has landed:
Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.


--
Kentaro Hara, Tokyo, Japan


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.

Hongchan Choi

unread,
Jun 18, 2020, 4:18:55 PM6/18/20
to platform-architecture-dev, Raymond Toy, schedu...@chromium.org, alt...@chromium.org, Chris Palmer
Hi all,

After almost 1 year, finally I am revisiting this issue again. From where we left off, the general consensus was "limiting it to top-level document only". Does this still hold for everyone here?

Considering these 4 cases:
1. abc.com => abc.com/worklet.js (main frame, same origin)
2. abc.com => [iframe] abc.com/worklet.js (in iframe, same origin)
3. abc.com => def.com/worklet.js (main frame, script from different origin)
4. abc.com => [iframe] def.com/worklet.js (in iframe, script from different origin)

I have no knowledge on this topic yet, but my guess is 1. will be the only case that can run a RT thread.

Secondly, for the formal security review I will follow these steps. Any advice would be appreciated!

What do you all think?

Best,
Hongchan


Thanks, haraken! :)

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.


--
Kentaro Hara, Tokyo, Japan


--
Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

Kentaro Hara

unread,
Jun 18, 2020, 9:49:35 PM6/18/20
to Hongchan Choi, platform-architecture-dev, Raymond Toy, scheduler-dev, Alexander Timin, Chris Palmer
+1 to experiment and get developer feedback. I have no concern about doing the experiment.

Beyond the experiment (i.e., shipping), I'm not fully convinced if it's a good idea to expose a web primitive that enables developers to run any arbitrary JS on a real-time thread though. Even if it's top-level document only, the script can cause starvation on other tabs. I'm not intending to be a blocker but want to hear thoughts from the security team as well. So, +1 to going through the security review.


You received this message because you are subscribed to the Google Groups "scheduler-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheduler-de...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/scheduler-dev/CAGJqXNsjvH%3D8Qjo1VUVGV%3DgV5cLRjkTdC5ipOrUe5ckY_H2u2Q%40mail.gmail.com.

Hongchan Choi

unread,
Jun 19, 2020, 1:35:03 PM6/19/20
to Kentaro Hara, platform-architecture-dev, Raymond Toy, scheduler-dev, Alexander Timin, Chris Palmer
The experiment has been on-going and I need to collect comments from developers to share with everyone here. Anecdotally, some Electron Apps are using this experimental RT thread flags to get the stable performance of Audio Worklet.

Through multiple tracing data analysis, I found that the weakest link is a thread-hop from AudioDeviceThread (RT) to AudioWorkletThread (DISPLAY). When a system goes under the pressure, posted tasks from AudioDeviceThread gets delayed significantly on AudioWorkletThread. It especially gets worse on the system with low CPU counts. Even after numerous improvements over GC and memory allocation, glitches from delayed tasks were unavoidable. The good news is recent improvements on the area made this issue very rare.

There also have been several reports of glitches when there is an intensive graphic/visual change. This is understandable because AudioWorkletThread shares its task queue with other DISPLAY thread tasks.

-Hongchan


To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jyvwYc3K%3DrEM4ASmwuAdb5%3DYKv5Md4KBcc0C%3DC1-skH%2BQ%40mail.gmail.com.

Kentaro Hara

unread,
Jun 21, 2020, 10:41:44 PM6/21/20
to Hongchan Choi, platform-architecture-dev, Raymond Toy, scheduler-dev, Alexander Timin, Chris Palmer
May I ask a probably stupid question? :)

Through multiple tracing data analysis, I found that the weakest link is a thread-hop from AudioDeviceThread (RT) to AudioWorkletThread (DISPLAY).

I think you are trying to achieve the following goals:

1) AudioWorkletThread runs with the REALTIME priority
2) The user script and GCs on AudioWorkletThread doesn't cause any glitches

Assuming 1) and 2) are doable, does it mean that we can run Audio Worklets on the AudioDeviceThread (RT) directly?

If you say it will cause many glitches on AudioDeviceThread, it sounds like you're saying 1) and 2) are hard to achieve...?

Hongchan Choi

unread,
Jun 22, 2020, 2:00:19 PM6/22/20
to Kentaro Hara, platform-architecture-dev, Raymond Toy, scheduler-dev, Alexander Timin, Chris Palmer
That's not a stupid question at all. Actually we had a similar discussion in the past. (It was ~3 years ago and I can't find the link anymore)

> Assuming 1) and 2) are doable, does it mean that we can run Audio Worklets on the AudioDeviceThread (RT) directly?

That would be ideal, but the architecture team ruled that idea out because AudioDeviceThread is not equipped with a task runner and other stuff (i.e. Blink thread infra). So it will essentially be reinventing a Blink worker thread by wrapping a pthread from the outside of Blink. Reducing thread hop is always great because we can shave off a few milliseconds of latency that way, but it would be a non-trivial project IMO.
-Hongchjan

Carl Younger

unread,
Jan 28, 2022, 8:20:17 PM1/28/22
to platform-architecture-dev, hong...@chromium.org, platform-architecture-dev, rt...@chromium.org, scheduler-dev, Alexander Timin, Chris Palmer, Kentaro

Just a thought on this: Realtime audio threads are generally a "pro-audio" thing. In that context, users routinely enable and disable system defaults to aggressively optimize for latency, stability etc. Optimizing their browser would be natural.

Given that, maybe this feature should always being aggressively optimized, and always being behind a flag.

Regular Web users would be completely isolated, pro-audio devs and musicians would opt into fully optimal performance, and it's the simplest option to maintain.

I doubt you could standardize a flag as a proper web standard, but this feature could get by as a de facto standard (other browsers know what to implement). Developers really just need confidence that it'll be maintained.

Carl Younger

unread,
Jan 28, 2022, 10:01:54 PM1/28/22
to platform-architecture-dev, Carl Younger, hong...@chromium.org, platform-architecture-dev, rt...@chromium.org, scheduler-dev, Alexander Timin, Chris Palmer, Kentaro

You removed the flag?!? Seriously.

You cannot do realtime audio in the browser without this. I was already using handwritten WebAssembly in audio worklets to try to maintain decent performance, and you crippled the entire Chromium platform?? What about Electron?

I'm done with the Web. It really sucks for audio.

Carl Younger

unread,
Jan 30, 2022, 2:21:11 PM1/30/22
to platform-architecture-dev, Carl Younger, hong...@chromium.org, platform-architecture-dev, rt...@chromium.org, scheduler-dev, Alexander Timin, Chris Palmer, Kentaro
I'd like to say sorry for the outburst I posted a couple of days ago. I misunderstood the situation, and it was one of those days. Either way, I shouldn't have started ranting in this thread. Sorry for that.
Reply all
Reply to author
Forward
0 new messages