Submit() is occasionally taking much longer.

[Jean-Colas Prunier]

Hi everyone,

While having a chance to play more animations with Dawn I realized that on some occasions the animation wasn't smooth. I narrowed it down to the Submit call that on occasion takes a much longer time than normal. It goes from just a few milliseconds to say 20 or 40 or even 70... these numbers don't mean much as they seem to depend on the scene complexity. The more the scene the higher the jumps. The scene I am describing here has about 100k triangles. The time I give you includes all the time it takes to prepare the passes. If I just look at the time it takes for the Submit call on that scene it varies to less than 1 ms to 2 ms on average. With, as mentioned, jumps to 60+. As I said these jumps are rare but visible (as they result is delays in the animation - breaking real-time playback).

I am not using the latest version of Dawn so will update this asap, but was wondering if you had seen this before in the past. And if not what I can do to debug this further ... (and help you look into it).

Looking forward to your feedback. 

-jc

[Corentin Wallez]

Hey Jean-Colas,

Do you have more information on which OS you use, how often you submit commands (at 60fps, or as fast as possible?), how Dawn is used (through JS, in native, ...). It could be for example that Metal runs against its limit of in-flight command buffers and needs to wait for the completion of some of them before proceeding.

Cheers,

Corentin

--
You received this message because you are subscribed to the Google Groups "Dawn Graphics" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dawn-graphic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/0fc65e66-1cb3-4b4a-b740-8eef1faa0e40n%40googlegroups.com.

[Jean-Colas Prunier]

Update: If I write the vertex and index data for the 3D scene on each render iteration, this behavior shows up. If I call WriteBuffer only once when I load the scene (obviously not playing the animation just writing the data for the first frame), and rendering the same content over and over on each render iteration, then this seems to go away. I will keep digging.

[Jean-Colas Prunier]

I am on Windows for this one test. I have a MacOS build but broken now( sans can't test on this for now. So I am using Vulkan in this case. Yes at this point I don't do anything fancy, I just render as often as possible, just calculating what should be the frame for the animation at the current time so that I can play the animation at the correct framerate. This is not optimized by far, but even the, the jumps don't make a lot of sense since in 99% of the time, Submit takes few ms.

[Jean-Colas Prunier]

Ah sorry Corentin, it's using Dawn native.

[Jean-Colas Prunier]

In addition to the info I have you Corentin.

The app I have right now has both a UI and a viewport. All using Dawn native. There's some animation on the UI that resizes the viewport etc. Now if I play the animation with no animation on the UI, the viewport content renders fine without any peak. No stutter. When the UI is ongoing, that's when I see some problems: it causes a stutter. In the viewport pass I copy the data for the geometry into the index and vertex buffer like so:

queue.WriteBuffer(render_pipeline_3dview_->index_buffer_, 0,
static_cast<uint8_t*>(scene_render_data_.indices.data()),
scene_render_data_.indices.size() * sizeof(uint32_t));

queue.WriteBuffer(render_pipeline_3dview_->vertex_buffer_, 0,
static_cast<uint8_t*>(  scene_render_data_.vertex_data.data()),
scene_render_data_.vertex_data.size() * sizeof(Imath::V3f));

Nothing fancy here. But if skip this (say fill the buffer at startup) then there's no stutter when the UI gets animated?

Now the render pass is really basic:
- fetch data for the UI
- fill buffer for the UI
- set the command for the UI pass

- potentially do the same thing for a stencil pass before

- dothe same thing for the viewport

- composite the UI and the Viewport in a texture

- render to quad

(the quad rendering thing is temp, it's supposed to be replaced by a rectangle textured with the viewport texture in the UI pass - but I don't think this would be an issue). So I stack this as a set of commands and submit.

I tried at 60fps (mode Fifo/Mailbox) so rendering is constrained by the refresh rate. Doesn't make any difference. The data I am copying is really small (not even 4MB). The render times is around 2 to 3ms. I am really confused (. 

I realize copying the data on every frame is not ideal, and that's only temp, but with such small data.

If you guys have any ideas I would really appreciate it. I will keep searching but I am starting to run out of ideas for now. Thanks a lot.

On Thursday, October 26, 2023 at 8:28:54 PM UTC+2 cwa...@google.com wrote:

[Dominic Cerisano]

You might want to spawn/fork an asynchronous thread to do the 'submit' which is apparently blocking/synchronous.
In a browser this is what workers are for, a thread pool to keep the main rendering (DOM) thread non-blocking and smooth during blocking IO calls.

Dominic Cerisano

dcer...@gmail.com

NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. 
AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. 

To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/afcd0200-4817-4776-a19e-fd7128bf4148n%40googlegroups.com.

[Jean-Colas Prunier]

Thanks Dominic.
I understand that both WriteBuffer and (hopefully so) Submit are non-blocking. The issue must lie somewhere else, which, at this point I suspect to be an attempt to read data in a cpu to gpu write operation that's no longer there. I am still trying to figure out why and how to debug that.

[Dominic Cerisano]

NP. The principal is the same, though. You might be able to use a std::promse to async wait on the submit().
I try to keep UI and graphics on separate threads, although your issue might be another use case for multi-queue.

Dominic Cerisano

dcer...@gmail.com

NOTICE: Confidential message which may be privileged. Unauthorized use/disclosure prohibited. 
AVIS : Message confidentiel dont le contenu peut être privilégié. Utilisation/divulgation interdites sans permission. 

To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/3b16080b-7625-4eae-90e4-e2673e8ea07dn%40googlegroups.com.

[Jean-Colas Prunier]

Hi Corentin

Your in-fly idea could lead to somewhere. Rather than cumulating the commands and submitting them as a batch, I did a test in which I submit the commands in a more granular fashion, and it seems to have eliminated the problem. I am not sure how this impacts performances ... I assume you have experience with the limit beyond which the number of commands submitted in one batch is causing issues? In my case I'd say I'd submit 5 in one go, + all the WriteBuffers in the middle. Which doesn't seem to be a lot, but I lack experience here to really know. 

- So would submitting commands as soon as possible the best practice?
- Does it impact performance vs sending them in batch

- Is there a "soft" limit beyond which the batch size can cause issue like the one I described?

Thanks

On Thursday, October 26, 2023 at 8:28:54 PM UTC+2 cwa...@google.com wrote:

[Corentin Wallez]

Hey Jean-Colas,

In some very-low latency use-cases, submitting some commands early can be useful, but these cases are somewhat rare (and usually VR/AR) so I don't think yours is one of them. There is medium fixed cost to each submit command so minimizing the number of them is a good idea, but ~5 is definitely fine.

Cheers,

Corentin

To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/2cf9872c-c15a-4079-853c-8a161c2b54dbn%40googlegroups.com.

[Jean-Colas Pro]

Thanks Corentin.

At least I have a « dirty » workaround for now. I want to rebuild this pipeline anyway running the UI and the 3D content in separate threads and texture the viewport port in the UI pass with the 3D content render so this will change the process quite a lot. But to according to your reply, with ~5 commands it should be fine, and yet I really have these weird jumps which seems to be some odd synchronization issues. I have a deadline now, but once this is past I will definitely spend time looking into it. 

I didn’t bother posting on Matrix for this one, as it touched on the Dawn API native. I am trying to stick to this rule of thumb. 

Thanks again for your input and suggestion. Without the in-flight suggestion, I would have not tried to make submit more granular, even if, for now I don’t understand why the previous approach causes that issue.

-jc

To view this discussion on the web visit https://groups.google.com/d/msgid/dawn-graphics/CAGdfWNPNR360SEg3uTPDj4qpOOKbR6hv6-y3VHv686EgY0r1Ow%40mail.gmail.com.