GPU/compositor thread priorities on Windows

[Sunny Sachanandani]

Hi all,

I've written up some investigation of GPU/compositor thread priority and scheduling issues on Windows (using ETW traces).  The doc as WIP as I'll keep adding in benchmark/UMA results in the coming weeks, but early feedback is much appreciated.

https://docs.google.com/document/d/1XqXkbq8uH20iD4zN0GnEjYpor2wG7D8VLI0FZjOXBDc/edit?usp=sharing

Thanks,

Sunny

[Sunny Sachanandani]

Hey Geoff,

I wanted to kick off a discussion about ANGLE shader translation worker thread priorities.  The "Priority Inversions" section in the doc describes the problem in detail, but the gist of it is that we need to increase ANGLE worker thread priorities so that they're not frequently preempted by renderer threads.

There are a few unanswered questions:

1) Should the embedder provide a thread pool for ANGLE to use?  How would ANGLE expose this?

2) Unfortunately, standard library provides no way for setting thread priorities in a cross-platform way so we need some platform specific bits.   Using #ifdef is the simplest approach, but I'm sure we'll need some abstraction here.

3) Even if ANGLE implements the thread pool, should we provide a way for the client to specify thread priority?  Various options here: a) inherit thread priority from client thread b) have client provide an opaque integer thread priority that is platform specific c) have client pass a high priority flag that ANGLE maps to an actual thread priority

https://chromium-review.googlesource.com/c/angle/angle/+/1679632  is a PoC for a thread pool implementation that uses std::thread (instead of current std::async) and uses #ifdef for increasing thread priority on Windows.

- Sunny

[Bruce Dawson]

I just want to give a +1 to Sunny's ETW trace analysis. I haven't looked at the trace or replicated his results but his analysis and interpretation looks spot on.

[Sushanth Rajasankar]

Thanks Sunny for the analysis.

 

Reading through the document, I think lowering the priority of OOPIF renderer process and bumping up GPU process priority is something we should experiment with.

 

Bumping up GPU process priority is justifiable, since it is at the end of the pipeline and would not starve other process because it would be throttled by inputs to it being generated by lower priority process. This is also similar in mental model to DWM running at high priority 13.

Bumping priority of the process vs individual threads – it appears almost all threads in the GPU process are valuable “GPU IO thread, GPU main thread, viz display compositor thread, and the vsync thread" any other threads in the GPU process are going to be workers like the shader compilation thread that we would like to have at higher priority for the same reasons.

 

Rather than bumping up individual renderer process threads, which has the potential to starve something else in the system that the user cares about – bumping down the priority of OOPIF renderers especially ones classified as Ads is appealing to me.

For example with the current approach for a main page with ticking CSS animations, will giving the compositor thread higher priority prioritize this animation over everything else in the system even when the renderer is not the active tab or if the window is minimized.

 

If we are pursuing bumping up Renderer process threads, I think it should be tied to input. The mental model here would be similar to how windows bumps up the priority of the thread corresponding to the window receiving input messages.

 

Perhaps it makes sense to stage these changes, especially if just prioritizing GPU process offers wins in some cases.

--
To unsubscribe from this group and stop receiving emails from it, send an email to graphics-dev...@chromium.org.

[Gabriel Charette]

Awesome analysis Sunny!

Priority boosting has bitten us in the past too and I've been meaning to investigate disabling it : crbug.com/879097.

Increasing priority of the GPU process isn't crazy either. Assuming it uses base::ThreadPool: it's BEST_EFFORT tasks, if any, would still run at low thread priority.

Making ThreadPriority::DISPLAY to THREAD_PRIORITY_HIGHEST seems slightly more dangerous. Especially since I'd like us to unify our usage of ThreadPriority::DISPLAY on all platforms (no reason to only use it on Android/CrOS IMO -- it's either right on all platforms or wrong on all platforms, platform-specific policies just make priority inversion bugs harder to spot).

PS: I don't have comment rights on the doc. Typical recommendation is to give comment access to chromi...@chromium.org (untick "send message"!) and view access to the world.

--
You received this message because you are subscribed to the Google Groups "scheduler-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheduler-de...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/scheduler-dev/DM6PR00MB04730B3A4857AFE451667A77FDFC0%40DM6PR00MB0473.namprd00.prod.outlook.com.

[Gabriel Charette]

Also see +Chris Hamilton 's https://docs.google.com/document/d/1BYLWbxPreoK2Ml6qhWqThnRDjDQjfCL_DmeB1qtzBjY/edit?usp=drivesdk

[Sunny Sachanandani]

On Fri, Jun 28, 2019 at 12:27 PM Gabriel Charette <g...@chromium.org> wrote:

Also see +Chris Hamilton 's https://docs.google.com/document/d/1BYLWbxPreoK2Ml6qhWqThnRDjDQjfCL_DmeB1qtzBjY/edit?usp=drivesdk

Le ven. 28 juin 2019 15 h 16, Gabriel Charette <g...@chromium.org> a écrit :

Awesome analysis Sunny!

Priority boosting has bitten us in the past too and I've been meaning to investigate disabling it : crbug.com/879097.

Increasing priority of the GPU process isn't crazy either. Assuming it uses base::ThreadPool: it's BEST_EFFORT tasks, if any, would still run at low thread priority.

GPU main thread uses a base::SingleThreadTaskExecutor: https://cs.chromium.org/chromium/src/content/gpu/gpu_main.cc?l=248

But it has its own cooperative scheduler with a notion of priorities so we could probably do something similar.  I'm not too worried about lowering thread priorities in GPU process because we don't have (any?) BEST_EFFORT tasks.  We do have some idle time (by GPU main thread activity - not system idle) polling tasks (queries) but those also unblock the pipeline in some way.

 

Making ThreadPriority::DISPLAY to THREAD_PRIORITY_HIGHEST seems slightly more dangerous. Especially since I'd like us to unify our usage of ThreadPriority::DISPLAY on all platforms (no reason to only use it on Android/CrOS IMO -- it's either right on all platforms or wrong on all platforms, platform-specific policies just make priority inversion bugs harder to spot).

Note that on Windows THREAD_PRIORITY_HIGHEST doesn't really mean a high priority unless the process priority class is also high.  See this table for the actual numerical value based on thread priority and process priority class (for reference browser usually runs at priority 12 , DWM at priority 15, renderer at priority 9): https://docs.microsoft.com/en-us/windows/desktop/procthread/scheduling-priorities

 

PS: I don't have comment rights on the doc. Typical recommendation is to give comment access to chromi...@chromium.org (untick "send message"!) and view access to the world.

Folks subscribed to chromium-dev should now have permissions to comment on the doc.

 

On Fri, Jun 28, 2019 at 1:41 PM 'Sushanth Rajasankar' via scheduler-dev <schedu...@chromium.org> wrote:

Thanks Sunny for the analysis.

 

Reading through the document, I think lowering the priority of OOPIF renderer process and bumping up GPU process priority is something we should experiment with.

 

Bumping up GPU process priority is justifiable, since it is at the end of the pipeline and would not starve other process because it would be throttled by inputs to it being generated by lower priority process. This is also similar in mental model to DWM running at high priority 13.

Bumping priority of the process vs individual threads – it appears almost all threads in the GPU process are valuable “GPU IO thread, GPU main thread, viz display compositor thread, and the vsync thread" any other threads in the GPU process are going to be workers like the shader compilation thread that we would like to have at higher priority for the same reasons.

I agree that bumping process priority is the more scalable approach here given that we have worker threads in drivers too.  We probably won't have to change thread priorities at all in that case (even for ANGLE).

 

 

Rather than bumping up individual renderer process threads, which has the potential to starve something else in the system that the user cares about – bumping down the priority of OOPIF renderers especially ones classified as Ads is appealing to me.

For example with the current approach for a main page with ticking CSS animations, will giving the compositor thread higher priority prioritize this animation over everything else in the system even when the renderer is not the active tab or if the window is minimized.

There are folks already working on lowering process priority for ad renderer processes: https://chromium-review.googlesource.com/c/chromium/src/+/1645978

 

 

If we are pursuing bumping up Renderer process threads, I think it should be tied to input. The mental model here would be similar to how windows bumps up the priority of the thread corresponding to the window receiving input messages.

I agree, and this could be orthogonal to both GPU process and ad renderer process priority changes.

[Gabriel Charette]

Okay if the GPU process doesn't really have BEST_EFFORT tasks then it could make sense to bump up the priority of the entire process.