Thread priority in Blink and WebAudio rendering

[hong...@chromium.org]

Hi Blink architecture team,

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread. This change was the prerequisite of AudioWorklet project(mainly to run V8 to process JS code on the WebAudio's rendering thread). However, it created some issues due to the lower thread priority of WebThread causing glitches in the audio stream rendered by WebAudio.

The problem has been investigated in this issue (googler-only), and I created a patch to gather more opinions from the Blink architecture team. The initial comments from rtoy@ and haraken@ were to avoid the RT thread in Blink. If that's the case, how do we prevent the WebThread from descheduled due to the lower priority - particularly on the low powered devices?

Thanks!

[Kentaro Hara]

Yeah, I'd hesitate to set a higher priority to the audio thread than the main thread.

 

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread.

Why was the renderer's audio device thread not causing the performance issue? What priority was set to the audio device thread?

--

Kentaro Hara, Tokyo, Japan

[hong...@chromium.org]

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":

https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

The AudioDeviceThread is the real-time thread (I believe it is higher than the platform main thread), but the WebThread is the equivalent of "worker thread". Right?

We all knew that exposing the real-time thread to the worklet surface is potentially dangerous. To summarize my questions:

1) How to escalate the WebAudio's thread priority as we can in Blink.

2) What could be the alternative to avoid glitches? (missed render deadline due to the descheduled task) 

For 2), olka@ suggested to increase FIFO capacity to buffer the more data, but I think that it increases the latency and it should be the last resort for the WebAudio.

[Kentaro Hara]

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":
https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

Oh, we've already set the real-time priority on the audio device thread.

If that is already happening (and if it is an intentional design), I'm okay with raising the priority of the WebThread used for WebAudio. In other words, I'm okay with landing this CL.

On Tue, Jul 11, 2017 at 1:04 AM, <hong...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":

https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

The AudioDeviceThread is the real-time thread (I believe it is higher than the platform main thread), but the WebThread is the equivalent of "worker thread". Right?

We all knew that exposing the real-time thread to the worklet surface is potentially dangerous. To summarize my questions:

1) How to escalate the WebAudio's thread priority as we can in Blink.

I'm a bit confused. Won't your CL work to escalate the WebAudio's thread priority?

Or are you concerned about running worklet code on the audio thread (because it will end up with running arbitrary scripts on the real-time thread)?

 

2) What could be the alternative to avoid glitches? (missed render deadline due to the descheduled task) 

For 2), olka@ suggested to increase FIFO capacity to buffer the more data, but I think that it increases the latency and it should be the last resort for the WebAudio.

On Sunday, July 9, 2017 at 5:57:14 PM UTC-7, Kentaro wrote:

Yeah, I'd hesitate to set a higher priority to the audio thread than the main thread.

 

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread.

Why was the renderer's audio device thread not causing the performance issue? What priority was set to the audio device thread?

On Sat, Jul 8, 2017 at 4:57 AM, <hong...@chromium.org> wrote:

Hi Blink architecture team,

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread. This change was the prerequisite of AudioWorklet project(mainly to run V8 to process JS code on the WebAudio's rendering thread). However, it created some issues due to the lower thread priority of WebThread causing glitches in the audio stream rendered by WebAudio.

The problem has been investigated in this issue (googler-only), and I created a patch to gather more opinions from the Blink architecture team. The initial comments from rtoy@ and haraken@ were to avoid the RT thread in Blink. If that's the case, how do we prevent the WebThread from descheduled due to the lower priority - particularly on the low powered devices?

Thanks!

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org.

[Olga Sharonova]

On Tue, Jul 11, 2017 at 9:52 AM, Kentaro Hara <har...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":
https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

Oh, we've already set the real-time priority on the audio device thread.

If that is already happening (and if it is an intentional design), I'm okay with raising the priority of the WebThread used for WebAudio. In other words, I'm okay with landing this CL.

On Tue, Jul 11, 2017 at 1:04 AM, <hong...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":

https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

The AudioDeviceThread is the real-time thread (I believe it is higher than the platform main thread), but the WebThread is the equivalent of "worker thread". Right?

We all knew that exposing the real-time thread to the worklet surface is potentially dangerous. To summarize my questions:

1) How to escalate the WebAudio's thread priority as we can in Blink.

I'm a bit confused. Won't your CL work to escalate the WebAudio's thread priority?

Or are you concerned about running worklet code on the audio thread (because it will end up with running arbitrary scripts on the real-time thread)?

It will allow any arbitrary site to eat up all the CPU and make Chrome useless (and potentially hang the whole system), won't it? Should we have a security review for this change as well?

 

 

2) What could be the alternative to avoid glitches? (missed render deadline due to the descheduled task) 

For 2), olka@ suggested to increase FIFO capacity to buffer the more data, but I think that it increases the latency and it should be the last resort for the WebAudio.

On Sunday, July 9, 2017 at 5:57:14 PM UTC-7, Kentaro wrote:

Yeah, I'd hesitate to set a higher priority to the audio thread than the main thread.

 

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread.

Why was the renderer's audio device thread not causing the performance issue? What priority was set to the audio device thread?

On Sat, Jul 8, 2017 at 4:57 AM, <hong...@chromium.org> wrote:

Hi Blink architecture team,

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread. This change was the prerequisite of AudioWorklet project(mainly to run V8 to process JS code on the WebAudio's rendering thread). However, it created some issues due to the lower thread priority of WebThread causing glitches in the audio stream rendered by WebAudio.

The problem has been investigated in this issue (googler-only), and I created a patch to gather more opinions from the Blink architecture team. The initial comments from rtoy@ and haraken@ were to avoid the RT thread in Blink. If that's the case, how do we prevent the WebThread from descheduled due to the lower priority - particularly on the low powered devices?

Thanks!

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org.

[Kentaro Hara]

On Tue, Jul 11, 2017 at 5:38 PM, Olga Sharonova <ol...@chromium.org> wrote:

On Tue, Jul 11, 2017 at 9:52 AM, Kentaro Hara <har...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":
https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

Oh, we've already set the real-time priority on the audio device thread.

If that is already happening (and if it is an intentional design), I'm okay with raising the priority of the WebThread used for WebAudio. In other words, I'm okay with landing this CL.

On Tue, Jul 11, 2017 at 1:04 AM, <hong...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":

https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

The AudioDeviceThread is the real-time thread (I believe it is higher than the platform main thread), but the WebThread is the equivalent of "worker thread". Right?

We all knew that exposing the real-time thread to the worklet surface is potentially dangerous. To summarize my questions:

1) How to escalate the WebAudio's thread priority as we can in Blink.

I'm a bit confused. Won't your CL work to escalate the WebAudio's thread priority?

Or are you concerned about running worklet code on the audio thread (because it will end up with running arbitrary scripts on the real-time thread)?

It will allow any arbitrary site to eat up all the CPU and make Chrome useless (and potentially hang the whole system), won't it? Should we have a security review for this change as well?

Probably.

I'm confused in many ways. In the first place, if we worry about the realtime-ness of the audio thread, why do we want to allow worklet code to run on the audio thread? Running an arbitrary script on the audio thread will break the realtime-ness.

 

 

 

2) What could be the alternative to avoid glitches? (missed render deadline due to the descheduled task) 

For 2), olka@ suggested to increase FIFO capacity to buffer the more data, but I think that it increases the latency and it should be the last resort for the WebAudio.

On Sunday, July 9, 2017 at 5:57:14 PM UTC-7, Kentaro wrote:

Yeah, I'd hesitate to set a higher priority to the audio thread than the main thread.

 

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread.

Why was the renderer's audio device thread not causing the performance issue? What priority was set to the audio device thread?

On Sat, Jul 8, 2017 at 4:57 AM, <hong...@chromium.org> wrote:

Hi Blink architecture team,

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread. This change was the prerequisite of AudioWorklet project(mainly to run V8 to process JS code on the WebAudio's rendering thread). However, it created some issues due to the lower thread priority of WebThread causing glitches in the audio stream rendered by WebAudio.

The problem has been investigated in this issue (googler-only), and I created a patch to gather more opinions from the Blink architecture team. The initial comments from rtoy@ and haraken@ were to avoid the RT thread in Blink. If that's the case, how do we prevent the WebThread from descheduled due to the lower priority - particularly on the low powered devices?

Thanks!

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com.

[Olga Sharonova]

On Tue, Jul 11, 2017 at 10:51 AM, Kentaro Hara <har...@chromium.org> wrote:

On Tue, Jul 11, 2017 at 5:38 PM, Olga Sharonova <ol...@chromium.org> wrote:

On Tue, Jul 11, 2017 at 9:52 AM, Kentaro Hara <har...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":
https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

Oh, we've already set the real-time priority on the audio device thread.

If that is already happening (and if it is an intentional design), I'm okay with raising the priority of the WebThread used for WebAudio. In other words, I'm okay with landing this CL.

On Tue, Jul 11, 2017 at 1:04 AM, <hong...@chromium.org> wrote:

I believe it's happening here and it is "base::ThreadPriority::REALTIME_AUDIO":

https://cs.chromium.org/chromium/src/media/audio/audio_device_thread.cc?sq=package:chromium&l=14

The AudioDeviceThread is the real-time thread (I believe it is higher than the platform main thread), but the WebThread is the equivalent of "worker thread". Right?

We all knew that exposing the real-time thread to the worklet surface is potentially dangerous. To summarize my questions:

1) How to escalate the WebAudio's thread priority as we can in Blink.

I'm a bit confused. Won't your CL work to escalate the WebAudio's thread priority?

Or are you concerned about running worklet code on the audio thread (because it will end up with running arbitrary scripts on the real-time thread)?

It will allow any arbitrary site to eat up all the CPU and make Chrome useless (and potentially hang the whole system), won't it? Should we have a security review for this change as well?

Probably.

I'm confused in many ways. In the first place, if we worry about the realtime-ness of the audio thread, why do we want to allow worklet code to run on the audio thread? Running an arbitrary script on the audio thread will break the realtime-ness.

 

Worklet code runs on a separate web thread, doesn't it? And it provides audio to the audio thread via FIFO. FIFO underruns because Worklet thread does not get scheduled in time to produce audio data => real-time audio thread renders zeroes. Or am I misunderstanding the problem?

 

 

 

 

2) What could be the alternative to avoid glitches? (missed render deadline due to the descheduled task) 

For 2), olka@ suggested to increase FIFO capacity to buffer the more data, but I think that it increases the latency and it should be the last resort for the WebAudio.

On Sunday, July 9, 2017 at 5:57:14 PM UTC-7, Kentaro wrote:

Yeah, I'd hesitate to set a higher priority to the audio thread than the main thread.

 

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread.

Why was the renderer's audio device thread not causing the performance issue? What priority was set to the audio device thread?

On Sat, Jul 8, 2017 at 4:57 AM, <hong...@chromium.org> wrote:

Hi Blink architecture team,

Recently, we changed the thread that drives the audio rendering in WebAudio from the renderer's audio device thread to blink::WebThread. This change was the prerequisite of AudioWorklet project(mainly to run V8 to process JS code on the WebAudio's rendering thread). However, it created some issues due to the lower thread priority of WebThread causing glitches in the audio stream rendered by WebAudio.

The problem has been investigated in this issue (googler-only), and I created a patch to gather more opinions from the Blink architecture team. The initial comments from rtoy@ and haraken@ were to avoid the RT thread in Blink. If that's the case, how do we prevent the WebThread from descheduled due to the lower priority - particularly on the low powered devices?

Thanks!

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org.

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com.

[Kentaro Hara]

Ah, sorry, I was misunderstanding.

What is your proposal? a) or b)?

a) Set the realtime priority to the audio device thread & Set the realtime priority to the worklet thread.

b) Set the realtime priority to the audio device thread & Do not set the realtime priority to the worklet thread.

c) Do not set the realtime priority to the audio device thread & Do not set the realtime priority to the worklet thread.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com.

[Olga Sharonova]

b) and c) are the current state of things (depending on the platform).

Real-time priority of audio thread on ChromeOS is a trade-off ChromeOS team made to mitigate performance issues on low-power devices (+dgreid@).

My proposals are

e) Prefetched some audio (increases latency)

f) Dynamically resample the audio to be able to serve each pull request (see meda::AudioShifter)

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com.

[Dylan Reid]

On Tue, Jul 11, 2017 at 11:29:37AM +0200, Olga Sharonova wrote:
> b) and c) are the current state of things (depending on the platform).
> Real-time priority of audio thread on ChromeOS is a trade-off ChromeOS team
> made to mitigate performance issues on low-power devices (+dgreid@).

I don't see real time audio as a trade off. If you want low latency
audio on a linux based system, every audio thread has to be RT scheduled
or you will glitch often.

>
> My proposals are
>
> e) Prefetched some audio (increases latency)
>
> f) Dynamically resample the audio to be able to serve each pull request
> (see meda::AudioShifter

> <https://cs.chromium.org/chromium/src/media/base/audio_shifter.h?q=AudioShifter&sq=package:chromium&l=39>
> )

> >>>>> <https://chromium-review.googlesource.com/c/546736/>.

> >>>>>
> >>>>>
> >>>>> On Tue, Jul 11, 2017 at 1:04 AM, <hong...@chromium.org> wrote:
> >>>>>
> >>>>>> I believe it's happening here and it is "base::ThreadPriority::REALTIM
> >>>>>> E_AUDIO":
> >>>>>> https://cs.chromium.org/chromium/src/media/audio/audio_devic
> >>>>>> e_thread.cc?sq=package:chromium&l=14
> >>>>>>
> >>>>>> The AudioDeviceThread is the real-time thread (I believe it is higher
> >>>>>> than the platform main thread), but the WebThread is the equivalent of
> >>>>>> "worker thread". Right?
> >>>>>> We all knew that exposing the real-time thread to the worklet surface
> >>>>>> is potentially dangerous. To summarize my questions:
> >>>>>>
> >>>>>> 1) How to escalate the WebAudio's thread priority as we can in Blink.
> >>>>>>
> >>>>>
> >>>>> I'm a bit confused. Won't your CL

> >>>>> <https://chromium-review.googlesource.com/c/546736/> work to escalate

> >>>>>>>> <https://bugs.chromium.org/p/chromium/issues/detail?id=734539> (googler-only),

> >>>>>>>> and I created a patch

> >>>>>>>> <https://chromium-review.googlesource.com/c/548239> to gather more

> >>>>>>>> opinions from the Blink architecture team. The initial comments from rtoy@
> >>>>>>>> and haraken@ were to avoid the RT thread in Blink. If that's the
> >>>>>>>> case, how do we prevent the WebThread from descheduled due to the lower
> >>>>>>>> priority - particularly on the low powered devices?
> >>>>>>>>
> >>>>>>>> Thanks!
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Kentaro Hara, Tokyo, Japan
> >>>>>>>
> >>>>>> --
> >>>>>> You received this message because you are subscribed to the Google
> >>>>>> Groups "platform-architecture-dev" group.
> >>>>>> To unsubscribe from this group and stop receiving emails from it,

> >>>>>> send an email to platform-architect...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org

> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .

> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architect...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch

> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .

> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architect...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch

> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .

[Hongchan Choi]

As Hiroki suggested in the issue entry, I believe the issue is the lower thread priority of the thread offered by the platform.

> Most CPU1 cycles were occupied by CrBrowserMain thread and Chrome_ChildIOThread.

> Most CPU2 cycles were occupied by Chrome_IOThread. They all have priority 112.

> WebAudio Render only has priority 120. It could not get enough cycles to finish its work.

I believe we have 4 levels of priority and the platform thread is using NORMAL. If this is the case, all other platform threads has the same issue but the audio is much more susceptible to the delay or the missed deadline.

My question is: can the WebAudio rendering be DISPLAY if REALTIME_AUDIO is potentially dangerous? FWIW, the compositor thread in Blink uses the DISPLAY priority and this is where AnimationWorklet JS code will run.

Olga@ Dynamic resampling will alter the audio content somehow and that's not what we want for WebAudio. Also I am not sure what prefetching means (since WebAudio generates data on the fly), but I assuming you meant increasing FIFO and pull size from the device thread. That certainly can be done, but will be a disappointment for many WebAudio app developers because of the latency.

From my observation so far, this glitch becomes severe in low-end devices (e.g. 2 cores) and linux-based system. The device in the bug entry is actually both.

To post to this group, send email to platform-arc...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

[Dylan Reid]

On Tue, Jul 11, 2017 at 04:53:27PM +0000, Hongchan Choi wrote:
> As Hiroki suggested in the issue entry, I believe the issue is the lower
> thread priority of the thread offered by the platform.
>
> > Most CPU1 cycles were occupied by CrBrowserMain thread and
> Chrome_ChildIOThread.
> > Most CPU2 cycles were occupied by Chrome_IOThread. They all have priority
> 112.
> > WebAudio Render only has priority 120. It could not get enough cycles to
> finish its work.
>
> I believe we have 4 levels of priority

> <https://cs.chromium.org/chromium/src/base/threading/platform_thread.h?gsn=Options&l=108>

> and the platform thread is using NORMAL

> <https://cs.chromium.org/chromium/src/base/threading/thread.h?gsn=Options&l=88>.

> If this is the case, all other platform threads has the same issue but the
> audio is much more susceptible to the delay or the missed deadline.
>
> My question is: can the WebAudio rendering be DISPLAY if REALTIME_AUDIO is
> potentially dangerous? FWIW, the compositor thread in Blink uses the DISPLAY
> priority

> <https://cs.chromium.org/chromium/src/content/renderer/render_thread_impl.cc?type=cs&q=CompositorThread&l=1156>

> and this
> is where AnimationWorklet JS code will run.

For general Linux realtime audio is probably risky to give a renderer.
For cros, we should let the renderers have RT prio up to some % of the
CPU. On Android apps are allowed up to 10% before they are throttled.
The kernel can do all of this throttling if we configure the
chrome_renderers cpu cgroup to apply the limits.

[Kentaro Hara]

On Wed, Jul 12, 2017 at 1:53 AM, Hongchan Choi <hong...@chromium.org> wrote:

As Hiroki suggested in the issue entry, I believe the issue is the lower thread priority of the thread offered by the platform.

> Most CPU1 cycles were occupied by CrBrowserMain thread and Chrome_ChildIOThread.

> Most CPU2 cycles were occupied by Chrome_IOThread. They all have priority 112.

> WebAudio Render only has priority 120. It could not get enough cycles to finish its work.

I believe we have 4 levels of priority and the platform thread is using NORMAL. If this is the case, all other platform threads has the same issue but the audio is much more susceptible to the delay or the missed deadline.

My question is: can the WebAudio rendering be DISPLAY if REALTIME_AUDIO is potentially dangerous? FWIW, the compositor thread in Blink uses the DISPLAY priority and this is where AnimationWorklet JS code will run.

Your proposal is:

- Set the DISPLAY priority to the audio device thread % CrOS

- Do not set the DISPLAY priority to the thread on which the audio worklet runs

right?

If yes, I think that is fine (because we had been setting the RELATIME_AUDIO to the audio device thread % CrOS).

Olga@ Dynamic resampling will alter the audio content somehow and that's not what we want for WebAudio. Also I am not sure what prefetching means (since WebAudio generates data on the fly), but I assuming you meant increasing FIFO and pull size from the device thread. That certainly can be done, but will be a disappointment for many WebAudio app developers because of the latency.

From my observation so far, this glitch becomes severe in low-end devices (e.g. 2 cores) and linux-based system. The device in the bug entry is actually both.

> >>>>>> send an email to platform-architecture-dev+unsub...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architecture-dev+unsub...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architecture-dev+unsub...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--

You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

[Hongchan Choi]

Now I see where the confusion is.

> - Set the DISPLAY priority to the audio device thread % CrOS

My concern is the lower thread priority in offered by the Blink platform. It has nothing to do with either CrOS or AudioDeviceThread. Whatever the renderer has been doing, I do not have any intent to change that. I am specifically talking about the Blink platform.

> - Do not set the DISPLAY priority to the thread on which the audio worklet runs

Unfortunately, you got it reversed. I want the DISPLAY priority for the WebAudio rendering thread, which also runs AudioWorklet. This is a sort of compromise because exposing a RT thread to JS code is potentially dangerous, but the compositor thread in Blink is already using this priority.

* A little bit of background:

(1) WebAudio renderer used to run on the AudioDeviceThread. (RT priority)

(2) AudioWorklet needed V8, so now WebAudio renderer is detached from AudioDeviceThread and runs on the Blink WebThread.

(3) WebAudio renderer renders the data and AudioDeviceThread pulls out of WebAudio's FIFO.

* Problem: WebThread has low priority so we can't get the same performance that we had in (1).

Sorry for the confusion and please let me know if there is anything unclear.

On Tue, Jul 11, 2017 at 10:31 AM Kentaro Hara <har...@chromium.org> wrote:

On Wed, Jul 12, 2017 at 1:53 AM, Hongchan Choi <hong...@chromium.org> wrote:

As Hiroki suggested in the issue entry, I believe the issue is the lower thread priority of the thread offered by the platform.

> Most CPU1 cycles were occupied by CrBrowserMain thread and Chrome_ChildIOThread.

> Most CPU2 cycles were occupied by Chrome_IOThread. They all have priority 112.

> WebAudio Render only has priority 120. It could not get enough cycles to finish its work.

I believe we have 4 levels of priority and the platform thread is using NORMAL. If this is the case, all other platform threads has the same issue but the audio is much more susceptible to the delay or the missed deadline.

My question is: can the WebAudio rendering be DISPLAY if REALTIME_AUDIO is potentially dangerous? FWIW, the compositor thread in Blink uses the DISPLAY priority and this is where AnimationWorklet JS code will run.

Your proposal is:

- Set the DISPLAY priority to the audio device thread % CrOS

- Do not set the DISPLAY priority to the thread on which the audio worklet runs

right?

If yes, I think that is fine (because we had been setting the RELATIME_AUDIO to the audio device thread % CrOS).

Olga@ Dynamic resampling will alter the audio content somehow and that's not what we want for WebAudio. Also I am not sure what prefetching means (since WebAudio generates data on the fly), but I assuming you meant increasing FIFO and pull size from the device thread. That certainly can be done, but will be a disappointment for many WebAudio app developers because of the latency.

From my observation so far, this glitch becomes severe in low-end devices (e.g. 2 cores) and linux-based system. The device in the bug entry is actually both.

> >>>>>> send an email to platform-architect...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architect...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architect...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

[Kentaro Hara]

Sorry about the confusion. I now understand the situation.

On Wed, Jul 12, 2017 at 2:51 AM, Hongchan Choi <hong...@chromium.org> wrote:

Now I see where the confusion is.

> - Set the DISPLAY priority to the audio device thread % CrOS

My concern is the lower thread priority in offered by the Blink platform. It has nothing to do with either CrOS or AudioDeviceThread. Whatever the renderer has been doing, I do not have any intent to change that. I am specifically talking about the Blink platform.

> - Do not set the DISPLAY priority to the thread on which the audio worklet runs

Unfortunately, you got it reversed. I want the DISPLAY priority for the WebAudio rendering thread, which also runs AudioWorklet. This is a sort of compromise because exposing a RT thread to JS code is potentially dangerous, but the compositor thread in Blink is already using this priority.

* A little bit of background:

(1) WebAudio renderer used to run on the AudioDeviceThread. (RT priority)

(2) AudioWorklet needed V8, so now WebAudio renderer is detached from AudioDeviceThread and runs on the Blink WebThread.

(3) WebAudio renderer renders the data and AudioDeviceThread pulls out of WebAudio's FIFO.

* Problem: WebThread has low priority so we can't get the same performance that we had in (1).

Yeah, this is a hard issue.

I think that the real problem is that worklet is assumed to run on a realtime thread but it is not designed that way. Worklet may run a long-running script on the realtime thread. Worklet may allocate a lot of objects and trigger GCs on the realtime thread. There's no mechanism to prevent Web developers from doing this.

I think it's reasonable to set the RT priority on the compositor thread and the Blink audio thread but it's problematic that worklet is not designed to run on the realtime thread. Is there any discussion on the spec side around it?

 

Sorry for the confusion and please let me know if there is anything unclear. 

 

On Tue, Jul 11, 2017 at 10:31 AM Kentaro Hara <har...@chromium.org> wrote:

On Wed, Jul 12, 2017 at 1:53 AM, Hongchan Choi <hong...@chromium.org> wrote:

As Hiroki suggested in the issue entry, I believe the issue is the lower thread priority of the thread offered by the platform.

> Most CPU1 cycles were occupied by CrBrowserMain thread and Chrome_ChildIOThread.

> Most CPU2 cycles were occupied by Chrome_IOThread. They all have priority 112.

> WebAudio Render only has priority 120. It could not get enough cycles to finish its work.

I believe we have 4 levels of priority and the platform thread is using NORMAL. If this is the case, all other platform threads has the same issue but the audio is much more susceptible to the delay or the missed deadline.

My question is: can the WebAudio rendering be DISPLAY if REALTIME_AUDIO is potentially dangerous? FWIW, the compositor thread in Blink uses the DISPLAY priority and this is where AnimationWorklet JS code will run.

Your proposal is:

- Set the DISPLAY priority to the audio device thread % CrOS

- Do not set the DISPLAY priority to the thread on which the audio worklet runs

right?

If yes, I think that is fine (because we had been setting the RELATIME_AUDIO to the audio device thread % CrOS).

Olga@ Dynamic resampling will alter the audio content somehow and that's not what we want for WebAudio. Also I am not sure what prefetching means (since WebAudio generates data on the fly), but I assuming you meant increasing FIFO and pull size from the device thread. That certainly can be done, but will be a disappointment for many WebAudio app developers because of the latency.

From my observation so far, this glitch becomes severe in low-end devices (e.g. 2 cores) and linux-based system. The device in the bug entry is actually both.

> >>>>>> send an email to platform-architecture-dev+unsub...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architecture-dev+unsub...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architecture-dev+unsub...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Hiroki Nakagawa]

FYI: AnimationWorklet plans to migrate its running thread from the compositor thread to Blink WebThread and may experience the same issue (see issue 731727, +majidvp@) 

> >>>>>> send an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architecture-dev+unsubsc...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Kinuko Yasuda]

Would it make sense to set some timer and deadline for each task that runs on the RT priority thread, and either let it block (in v8) and yield CPU for some time (or terminate it if its CPU usage starts to look really abusive), so that we could allow well-behaving Audio Worklets to use RT-priority thread with reasonable allowance but also can throttle the usage when necessary?

> >>>>>> send an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architecture-dev+unsubsc...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

--

Kentaro Hara, Tokyo, Japan

--

You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsub...@chromium.org.
To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jxNVNYu93w%3D210HPpp0Z4jZPFh65hmsWanhgcOLQuJwhA%40mail.gmail.com.

[Hongchan Choi]

(Oops. I forgot to reply to all in this thread. Repeating my response to Kinuko's suggestion.)

I am very interested in this idea, but honestly have no idea where to start. Could you point me some examples of such design pattern in Blink?

One issue with this design is if an AudioWorkletNode cannot be finish its task because the timer abruptly pauses the execution it will be in a weird state. The node will not recover from that state in subsequent render quanta. However, this means the developer is already doing too much for the given render quantum, so it is sort of expected behavior. We can simply give up the node, flag it and skip the process for the future operation. (This needs a spec change, but I believe it is reasonable and trivial.)

> >>>>>> send an email to platform-architect...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architect...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architect...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

[Raymond Toy]

On Tue, Jul 11, 2017 at 7:29 PM, Kinuko Yasuda <kin...@chromium.org> wrote:

Would it make sense to set some timer and deadline for each task that runs on the RT priority thread, and either let it block (in v8) and yield CPU for some time (or terminate it if its CPU usage starts to look really abusive), so that we could allow well-behaving Audio Worklets to use RT-priority thread with reasonable allowance but also can throttle the usage when necessary?

At the maximum sample rate (384 kHz), the timer would be set to 333 microsec (the time needed for each render quantum of 128 samples).  Would that be ok to have a timer fire every 300 microsec?

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jxNVNYu93w%3D210HPpp0Z4jZPFh65hmsWanhgcOLQuJwhA%40mail.gmail.com.

[Dale Curtis]

Err, I don't think that's accurate, we don't allow 384kHz 1 sample buffers. At our minimum buffer size of 128 it's 3000Hz; or .33ms.

- dale

[Raymond Toy]

On Thu, Jul 13, 2017 at 1:41 PM, Dale Curtis <dalec...@chromium.org> wrote:

Err, I don't think that's accurate, we don't allow 384kHz 1 sample buffers. At our minimum buffer size of 128 it's 3000Hz; or .33ms.

I'm confused.  Isn't 0.33 ms = 300 microsec? 

[Dale Curtis]

Err, sorry I misread your comment; you're right! I'm not sure I'd expect 384kHz @ 128 samples to actually work FWIW.

- dale

[Raymond Toy]

On Thu, Jul 13, 2017 at 1:46 PM, Dale Curtis <dalec...@chromium.org> wrote:

Err, sorry I misread your comment; you're right! I'm not sure I'd expect 384kHz @ 128 samples to actually work FWIW.

Yeah, I agree that 384kHz @ 128 samples will work, but we do buffer in the FIFO to process more than 128 samples, so I guess the timers should be set to that size instead of 128 samples.  So maybe timers will get fired every 1 ms or so at most.

Perhaps that's doable without introducing lots of additional overhead.

I'm actually ok with letting audio glitch if JS is too slow.  I don't think it's a good idea to let people abuse the higher priority though.

[Kentaro Hara]

Would it make sense to set some timer and deadline for each task that runs on the RT priority thread, and either let it block (in v8) and yield CPU for some time (or terminate it if its CPU usage starts to look really abusive), so that we could allow well-behaving Audio Worklets to use RT-priority thread with reasonable allowance but also can throttle the usage when necessary?

Can we probably use V8Interruptor? We can request V8 to interrupt JavaScript periodically and yield CPU for some time for other threads.

That said, I feel like that it's just a work-around of the problem. The real problem is that worklet is mis-designed -- worklet provides a way to run an arbitrary script on a thread that is expected to be the RT priority. (I don't have a good answer for this unfortunately :/)

[Adam Klein]

On Thu, Jul 13, 2017 at 4:59 PM, Kentaro Hara <har...@chromium.org> wrote:

Would it make sense to set some timer and deadline for each task that runs on the RT priority thread, and either let it block (in v8) and yield CPU for some time (or terminate it if its CPU usage starts to look really abusive), so that we could allow well-behaving Audio Worklets to use RT-priority thread with reasonable allowance but also can throttle the usage when necessary?

Can we probably use V8Interruptor? We can request V8 to interrupt JavaScript periodically and yield CPU for some time for other threads.

That said, I feel like that it's just a work-around of the problem. The real problem is that worklet is mis-designed -- worklet provides a way to run an arbitrary script on a thread that is expected to be the RT priority. (I don't have a good answer for this unfortunately :/)

I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

[Kinuko Yasuda]

On Fri, Jul 14, 2017 at 9:12 AM, Adam Klein <ad...@chromium.org> wrote:

On Thu, Jul 13, 2017 at 4:59 PM, Kentaro Hara <har...@chromium.org> wrote:

Would it make sense to set some timer and deadline for each task that runs on the RT priority thread, and either let it block (in v8) and yield CPU for some time (or terminate it if its CPU usage starts to look really abusive), so that we could allow well-behaving Audio Worklets to use RT-priority thread with reasonable allowance but also can throttle the usage when necessary?

Can we probably use V8Interruptor? We can request V8 to interrupt JavaScript periodically and yield CPU for some time for other threads.

Yes, I think we can possibly use RequestInterrupt / Interruptor for that.

 

That said, I feel like that it's just a work-around of the problem. The real problem is that worklet is mis-designed -- worklet provides a way to run an arbitrary script on a thread that is expected to be the RT priority. (I don't have a good answer for this unfortunately :/)

I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

From runtime pov I agree, but I could also imagine some real-time multi-media applications might still want to do that.

From app-developer pov I'd feel it could still be a valid design, but provided that it's possible to properly control the CPU usage and user experience (e.g. if it's possible to enforce only a fixed slice of RT-priority execution only on foreground tabs).

 

(I agree that it'd be good to review the worklet behavior from spec side as well, so interested in the WebAudio side's answers to Adam's question)

[Olga Sharonova]

BTW, we are discussing only the worklet behavior, but isn't the whole WebAudio rendering now switched to 2-thread model?

[Hongchan Choi]

+Ian Kilpatrick 

> I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

Yes, that's correct. It was the prerequisite of AudioWorklet because we wanted to run the user-suppiled JS code along with other native AudioNodes. For this reason, all of WebAudio rendering now runs on the WebThread, which has the lower priority than AudioDeviceThread and the main browser thread.

Also Raymond and I had discussed the timer mechanism, we don't think it really works for WebAudio's dynamic graph processing model because of the reason that I mentioned earlier. The premature drop-off due to the timer will put AudioNodes in a weird state and the graph will never be able to recover from it. At that point, dropping and recreating the whole context might be better.

In terms of the priority, the same level with the main thread might be sufficient for our purpose.

 

- dale

- dale

> >>>>>> send an email to platform-architect...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architect...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architect...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jxNVNYu93w%3D210HPpp0Z4jZPFh65hmsWanhgcOLQuJwhA%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Raymond Toy]

On Fri, Jul 14, 2017 at 9:24 AM, Hongchan Choi <hong...@chromium.org> wrote:

+Ian Kilpatrick 

> I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

Yes, that's correct. It was the prerequisite of AudioWorklet because we wanted to run the user-suppiled JS code along with other native AudioNodes. For this reason, all of WebAudio rendering now runs on the WebThread, which has the lower priority than AudioDeviceThread and the main browser thread.

Also Raymond and I had discussed the timer mechanism, we don't think it really works for WebAudio's dynamic graph processing model because of the reason that I mentioned earlier. The premature drop-off due to the timer will put AudioNodes in a weird state and the graph will never be able to recover from it. At that point, dropping and recreating the whole context might be better.

In terms of the priority, the same level with the main thread might be sufficient for our purpose.

I think the same priority as the main thread is a good starting point.  You can't abuse the AudioWorklet thread to get higher priority,  And it's a somewhat reasonable priority for something that has strict real-time constraints that are very noticeable when the you miss the constraints.

 

- dale

- dale

> >>>>>> send an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architecture-dev+unsubsc...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architecture-dev+unsubsc...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architecture-dev+unsubsc...@chromium.org.

To post to this group, send email to platform-architecture-dev@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jxNVNYu93w%3D210HPpp0Z4jZPFh65hmsWanhgcOLQuJwhA%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Kinuko Yasuda]

On Sat, Jul 15, 2017 at 1:56 AM, Raymond Toy <rt...@chromium.org> wrote:

On Fri, Jul 14, 2017 at 9:24 AM, Hongchan Choi <hong...@chromium.org> wrote:

+Ian Kilpatrick 

> I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

Yes, that's correct. It was the prerequisite of AudioWorklet because we wanted to run the user-suppiled JS code along with other native AudioNodes. For this reason, all of WebAudio rendering now runs on the WebThread, which has the lower priority than AudioDeviceThread and the main browser thread.

Also Raymond and I had discussed the timer mechanism, we don't think it really works for WebAudio's dynamic graph processing model because of the reason that I mentioned earlier. The premature drop-off due to the timer will put AudioNodes in a weird state and the graph will never be able to recover from it. At that point, dropping and recreating the whole context might be better.

In terms of the priority, the same level with the main thread might be sufficient for our purpose.

I think the same priority as the main thread is a good starting point.  You can't abuse the AudioWorklet thread to get higher priority,  And it's a somewhat reasonable priority for something that has strict real-time constraints that are very noticeable when the you miss the constraints.

Yeah if we could start with the same priority as the main thread that sounds a lot more feasible to me.

[Kentaro Hara]

On Sat, Jul 15, 2017 at 5:31 AM, Kinuko Yasuda <kin...@chromium.org> wrote:

On Sat, Jul 15, 2017 at 1:56 AM, Raymond Toy <rt...@chromium.org> wrote:

On Fri, Jul 14, 2017 at 9:24 AM, Hongchan Choi <hong...@chromium.org> wrote:

+Ian Kilpatrick 

> I'd tend to agree that running JS on a RT thread sounds like the core problem here. Just skimmed a bit of the "intent to implement", though, and it sounds like the design of AudioWorklet might require that it runs on the same thread as the rest of WebAudio. Is that correct?

Yes, that's correct. It was the prerequisite of AudioWorklet because we wanted to run the user-suppiled JS code along with other native AudioNodes. For this reason, all of WebAudio rendering now runs on the WebThread, which has the lower priority than AudioDeviceThread and the main browser thread.

Also Raymond and I had discussed the timer mechanism, we don't think it really works for WebAudio's dynamic graph processing model because of the reason that I mentioned earlier. The premature drop-off due to the timer will put AudioNodes in a weird state and the graph will never be able to recover from it. At that point, dropping and recreating the whole context might be better.

In terms of the priority, the same level with the main thread might be sufficient for our purpose.

I think the same priority as the main thread is a good starting point.  You can't abuse the AudioWorklet thread to get higher priority,  And it's a somewhat reasonable priority for something that has strict real-time constraints that are very noticeable when the you miss the constraints.

Yeah if we could start with the same priority as the main thread that sounds a lot more feasible to me. 

Sounds reasonable to me (assuming that the performance with that priority is acceptable for WebAudio).

[Hongchan Choi]

Thanks kinuko@ and haraken@.

From the discussion, I learned WebThread has the priority of 120 by default and I am assuming it is what you get with ThreadPriority::NORMAL. (code) Although the granularity is somewhat simple, I can experiment with DISPLAY first, which is one level higher than NORMAL but slower than REALTIME_AUDIO.

The problem is I do not know how to inspect the thread priority of the Chrome processes. Can someone give me some clue?

Thanks.

 

- dale

- dale

> >>>>>> send an email to platform-architect...@chromium.org.

> >>>>>> To post to this group, send email to platform-architecture-dev@chro
> >>>>>> mium.org.
> >>>>>> To view this discussion on the web visit
> >>>>>> https://groups.google.com/a/chromium.org/d/msgid/platform-ar
> >>>>>> chitecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org
> >>>>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/d59b348f-79fd-47e9-9704-607b64a1b7db%40chromium.org?utm_medium=email&utm_source=footer>
> >>>>>> .
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Kentaro Hara, Tokyo, Japan
> >>>>>
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "platform-architecture-dev" group.
> >>>> To unsubscribe from this group and stop receiving emails from it, send

> >>>> an email to platform-architect...@chromium.org.

> >>>> To post to this group, send email to platform-architecture-dev@chro
> >>>> mium.org.
> >>>> To view this discussion on the web visit https://groups.google.com/a/ch
> >>>> romium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3
> >>>> AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com
> >>>> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_T1s6TAP3AYNFEHz4mN%2BTOUO_XhiuGojyin2mgLZqzEsQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >>>> .
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Kentaro Hara, Tokyo, Japan
> >>>
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "platform-architecture-dev" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an

> >> email to platform-architect...@chromium.org.

> >> To post to this group, send email to platform-architecture-dev@chro
> >> mium.org.
> >> To view this discussion on the web visit https://groups.google.com/a/ch
> >> romium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_
> >> GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com
> >> <https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAHri2_TnESttQ_GaMmnUP-e1Xz1KqK27rCmyYK1%3DPLiFfX-y3Q%40mail.gmail.com?utm_medium=email&utm_source=footer>
> >> .
> >>
> >
> >
> >
> > --
> > Kentaro Hara, Tokyo, Japan
> >

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/20170711153336.GB188394%40dgreid.mtv.corp.google.com.

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CAGJqXNsZ7PbMQLuyP4aAPOggyOEJ91n9JSDZ9oj4rKGKVWWFmQ%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

--

Kentaro Hara, Tokyo, Japan

--
You received this message because you are subscribed to the Google Groups "platform-architecture-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to platform-architect...@chromium.org.
To post to this group, send email to platform-arc...@chromium.org.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/platform-architecture-dev/CABg10jxNVNYu93w%3D210HPpp0Z4jZPFh65hmsWanhgcOLQuJwhA%40mail.gmail.com.

--

Kentaro Hara, Tokyo, Japan

[Hongchan Choi]

Following up from my last email with several questions:

1. What is the thread priority of the main thread across the platforms? It seems like all platforms have its own scheme.

Android (code)

{ThreadPriority::BACKGROUND, 10},

{ThreadPriority::NORMAL, 0},

{ThreadPriority::DISPLAY, -4},

{ThreadPriority::REALTIME_AUDIO, -16},

Linux (code)

{ThreadPriority::BACKGROUND, 10},

{ThreadPriority::NORMAL, 0},

{ThreadPriority::DISPLAY, -8},

{ThreadPriority::REALTIME_AUDIO, -10},

Mac (code)

ThreadPriority::BACKGROUND: 0

ThreadPriority::NORMAL: 0.5

ThreadPriority::DISPLAY: 0.5

ThreadPriority::REALTIME_AUDIO: 1.0

Win (code)

ThreadPriority::BACKGROUND: THREAD_PRIORITY_LOWEST;

ThreadPriority::NORMAL: THREAD_PRIORITY_NORMAL;

ThreadPriority::DISPLAY: THREAD_PRIORITY_ABOVE_NORMAL;

ThreadPriority::REALTIME_AUDIO: THREAD_PRIORITY_TIME_CRITICAL;

2. By looking at the code, I thought the obvious choice is the "Normal" priority but that already has been the priority of Blink's platform thread (WebThread). In some platforms, the "Normal" priority will give you something lower that the main browser thread (the original issue) and I couldn't figure out how this works.

Any ideas or suggestions?

-Hongchan

[hong...@google.com]

FYI, this is the (speculative) list of regressions from the lower thread priority of WebAudio renderer:

crbug.com/749344

crbug.com/755173

crbug.com/734539

M60 is now out there, so I am expecting to see more issues around this regression. I'll be working on the fix by raising the thread priority, but please feel free to add more suggestions if you have any.

 

- dale

- dale

> >&gt

[hong...@google.com]

Adding maxmorin@ to the discussion.

[Kentaro Hara]

+1 to experimenting with the same thread priority as the main thread.

[maxm...@chromium.org]

I think it makes sense to give it a priority higher than normal, but lower than realtime. If I understand correctly, I also think the AudioDeviceThread can have a similar priority, as long as we are very careful to never ever reintroduce crbug.com/710245 :). Also, if we start running arbitrary javascript on a high-priority thread, we have to make sure a malicious page cannot destabilize the browser or the OS (e.g. by busy-looping).

[Olga Sharonova]

Adjusting the priorities should reduce probability of the glitches, but still the new FIFO-based 2-thread model is bound to have them: there is not design guarantee for a glitch-free rendering.

So another question: should we really be using the 2-thread model in all the cases? How often worklet nodes are (expected to be) present in the graph?

[Hongchan Choi]

maxmorin@

I am thinking of using base::ThreadPriority::DISPLAY. This is what the browser main thread uses, and also will be used by AnimationWorklet (compositor thread). The name itself does not match our case, (perhaps base::ThreadPriority::INTERACTIVE is more appropriate if I am allowed to introduce a new one) but that's another issue.

olka@

Yes, that's a good question that I don't have an answer. From my perspective, a glitch-free rendering is not possible with WebAudio because it is all about dynamic graph. Even the single-thread model does not guarantee that. rtoy@ has been working on the rendering optimization in WebAudio and it significantly boosts performance without touching the audio infra (FIFO, thread). I rather have that optimization instead of switching the threading model for WebAudio on the fly. (User can call audioWorklet.addModule() anytime) Dropping the original thread and picking up a new rendering thread in the middle of rendering sound already problematic.

[Hongchan Choi]

haraken@

Thanks for the confirmation. I am working on a POC CL, and raising the priority was helpful in eliminating glitches on the low-end device. (issue comment)

[Olga Sharonova]

On Tue, Aug 15, 2017 at 5:02 PM, Hongchan Choi <hong...@google.com> wrote:

maxmorin@

I am thinking of using base::ThreadPriority::DISPLAY. This is what the browser main thread uses, and also will be used by AnimationWorklet (compositor thread). The name itself does not match our case, (perhaps base::ThreadPriority::INTERACTIVE is more appropriate if I am allowed to introduce a new one) but that's another issue.

olka@

Yes, that's a good question that I don't have an answer. From my perspective, a glitch-free rendering is not possible with WebAudio because it is all about dynamic graph. Even the single-thread model does not guarantee that. rtoy@ has been working on the rendering optimization in WebAudio and it significantly boosts performance without touching the audio infra (FIFO, thread). I rather have that optimization instead of switching the threading model for WebAudio on the fly. (User can call audioWorklet.addModule() anytime) Dropping the original thread and picking up a new rendering thread in the middle of rendering sound already problematic.

 

What are specific problems you anticipate?

We experimented with switching audio rendering thread when working on renderer-side mixing of audio. grunell@ (cc-ed) cleaned up most of the code dealing with thread affinity AFAIK.

In a single thread model a glitch is about missing a deadline on browser/renderer edge. In the new two-thread model a glitch is about an empty FIFO and how pulls and pushes are scheduled. My concern is that call stack optimization could again reduce a probability of glitches, but we can't overcome scheduling issues by optimizing performance of one thread in a two-thread model.

So how often do you expect a worklet node to be present in the graph? Which nodes does it replace and how often are they used now? Because looks like for the rest of the cases the new model decreases quality without bringing value. If it's a large subset then it would make sense to consider switching the model on the fly when adding a worklet node.

[hong...@chromium.org]

I mistakenly left my last response to Olga. Here is the copy of it.

On Tuesday, August 15, 2017 at 2:37:12 PM UTC-7, Olga Sharonova wrote:

On Tue, Aug 15, 2017 at 5:02 PM, Hongchan Choi <hong...@google.com> wrote:

maxmorin@

I am thinking of using base::ThreadPriority::DISPLAY. This is what the browser main thread uses, and also will be used by AnimationWorklet (compositor thread). The name itself does not match our case, (perhaps base::ThreadPriority::INTERACTIVE is more appropriate if I am allowed to introduce a new one) but that's another issue.

olka@

Yes, that's a good question that I don't have an answer. From my perspective, a glitch-free rendering is not possible with WebAudio because it is all about dynamic graph. Even the single-thread model does not guarantee that. rtoy@ has been working on the rendering optimization in WebAudio and it significantly boosts performance without touching the audio infra (FIFO, thread). I rather have that optimization instead of switching the threading model for WebAudio on the fly. (User can call audioWorklet.addModule() anytime) Dropping the original thread and picking up a new rendering thread in the middle of rendering sound already problematic.

 

What are specific problems you anticipate?

If we make audioWorklet.addModule() as a one-way call to change the thread, it might be doable. Once AudioWorklet-related objects get created and becomes a part of the audio graph, there is no elegant way to change the thread. By default, there will be a significant pause from suspending/resuming the context and rebuilding the graph from scratch. So no worklet (single thread) -> worklet (dual thread) might work, but not the other way around.

 

We experimented with switching audio rendering thread when working on renderer-side mixing of audio. grunell@ (cc-ed) cleaned up most of the code dealing with thread affinity AFAIK.

In a single thread model a glitch is about missing a deadline on browser/renderer edge. In the new two-thread model a glitch is about an empty FIFO and how pulls and pushes are scheduled. My concern is that call stack optimization could again reduce a probability of glitches, but we can't overcome scheduling issues by optimizing performance of one thread in a two-thread model.

I agree. My guess is when the system is under the pressure, the throughput of TaskScheduler fluctuates wildly. I believe this is why we did not observe noticeable regression in the test.

 

So how often do you expect a worklet node to be present in the graph? Which nodes does it replace and how often are they used now? Because looks like for the rest of the cases the new model decreases quality without bringing value. If it's a large subset then it would make sense to consider switching the model on the fly when adding a worklet node.

As I mentioned before, I cannot give you a concrete answer. However, the Worklet is the most anticipated feature in WebAudio from the developer community. With the rising WebAssembly, now the game and pro audio industry are interested in this system. That's why we wanted to push the thread change to the stable build.