IPC latency

[Andrew Moylan]

Hi, expected Mojo IPC latency in practice has been discussed previously eg this thread about 2 years ago

groups.google.com/a/chromium.org/g/chromium-mojo/c/E2h4sGuvrlg/m/afpNBaD8AQAJ

I have been using a rule of thumb "up to 1ms ping via Mojo is normal, rare spikes to 10ms" that I garnered from some some hacky experiment.

In that thread bgeffon mentioned a variety of improvements along the way.

So I am wondering:

What's the latest on Mojo IPC latency? Have there been major changes/improvements in the last couple of years?

CC +ji...@google.com, nap...@google.com who were recently interested in this question.

[Ken Rockot]

1 ms would be extremely high for general IPC latency. It's nowhere near that in isolation, but it's also probably not useful to talk about Mojo IPC latency in isolation. In practice our latency in Chromium is dominated by task scheduling latency at the receiver.

In the best cases the kernel may context-switch to the receiving IO-thread immediately on send; and then if the targeted endpoint lives on that IO thread, we can dispatch immediately and might see an end-to-end IPC latency as low as 10-20 microseconds.

More typically the context switch isn't immediate, so assume some microseconds go by before we hit the IO thread. Also more typically we don't bind Mojo receivers on the IO thread, so the IO thread will only wake for minimal routing work before posting a dispatch task to the appropriate thread. From there it depends on what the target thread is doing.

For a busy renderer's main thread, dispatch task latency can range from a few microseconds to a few seconds.

 --
You received this message because you are subscribed to the Google Groups "chromium-mojo" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-moj...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/286d4a37-349d-4b98-a1b6-384255ad32cfn%40chromium.org.

[Ken Rockot]

On Tue, Jan 17, 2023 at 10:40 AM Ken Rockot <roc...@chromium.org> wrote:

On Mon, Jan 16, 2023 at 7:38 PM 'Andrew Moylan' via chromium-mojo <chromi...@chromium.org> wrote:

Hi, expected Mojo IPC latency in practice has been discussed previously eg this thread about 2 years ago

groups.google.com/a/chromium.org/g/chromium-mojo/c/E2h4sGuvrlg/m/afpNBaD8AQAJ

I have been using a rule of thumb "up to 1ms ping via Mojo is normal, rare spikes to 10ms" that I garnered from some some hacky experiment.

In that thread bgeffon mentioned a variety of improvements along the way.

So I am wondering:

What's the latest on Mojo IPC latency? Have there been major changes/improvements in the last couple of years? 

CC +ji...@google.com, nap...@google.com who were recently interested in this question.

1 ms would be extremely high for general IPC latency. It's nowhere near that in isolation, but it's also probably not useful to talk about Mojo IPC latency in isolation. In practice our latency in Chromium is dominated by task scheduling latency at the receiver.

In the best cases the kernel may context-switch to the receiving IO-thread immediately on send; and then if the targeted endpoint lives on that IO thread, we can dispatch immediately and might see an end-to-end IPC latency as low as 10-20 microseconds.

More typically the context switch isn't immediate, so assume some microseconds go by before we hit the IO thread. Also more typically we don't bind Mojo receivers on the IO thread, so the IO thread will only wake for minimal routing work before posting a dispatch task to the appropriate thread. From there it depends on what the target thread is doing.

For a busy renderer's main thread, dispatch task latency can range from a few microseconds to a few seconds.

Also to answer the other question (re: updates) the ipcz-based Mojo implementation is rolling out to Stable (not on Chrome OS yet) and it does bring down the minimum latency in the best case. On a local device I consistently observed a minimum latency of ~12us without ipcz and ~10us with.

It's not clear whether this will translate to a meaningful change in Chromium though. Input events are likely the most latency-sensitive use of IPC we have, but there are still other factors which might eclipse any fundamental IPC latency.

[K. Moon]

I think another thing to keep in mind is that IPC latency isn't strictly additive: Sending 10 IPCs simultaneously does not necessarily have 10x the overall latency of sending 10 IPCs one after another. (In fact, I'd be quite surprised if it was anywhere close to 2x, let alone 10x.) It's a hard thing to reason about without more specifics about the scenario you're trying to measure.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/chromium-mojo/CA%2BapAgF086Qgjsrzdrzz0EpA_ipaG%3DvSrSVgehi2e-5Ff16HiA%40mail.gmail.com.

[Andrew Moylan]

On Wed, 18 Jan 2023, 5:40 am Ken Rockot, <roc...@chromium.org> wrote:

On Mon, Jan 16, 2023 at 7:38 PM 'Andrew Moylan' via chromium-mojo <chromi...@chromium.org> wrote:

Hi, expected Mojo IPC latency in practice has been discussed previously eg this thread about 2 years ago

groups.google.com/a/chromium.org/g/chromium-mojo/c/E2h4sGuvrlg/m/afpNBaD8AQAJ

I have been using a rule of thumb "up to 1ms ping via Mojo is normal, rare spikes to 10ms" that I garnered from some some hacky experiment.

In that thread bgeffon mentioned a variety of improvements along the way.

So I am wondering:

What's the latest on Mojo IPC latency? Have there been major changes/improvements in the last couple of years? 

CC +ji...@google.com, nap...@google.com who were recently interested in this question.

1 ms would be extremely high for general IPC latency. It's nowhere near that in isolation, but it's also probably not useful to talk about Mojo IPC latency in isolation. In practice our latency in Chromium is dominated by task scheduling latency at the receiver.

Thanks for the very useful reply. +Jing Wang you might be ok then if your receiving Task Scheduler is high priority for processing your touch screen events? Presumably it already may be. 

[Jing Wang]

Thanks for all the information, very helpful!

+Andrew Moylan How do I set my receiving task scheduler to high priority? Do you have an example?

[Andrew Moylan]

I assume we are talking about the priority on the task runner: crsrc.org/c/base/task/task_traits.h;l=33

@Ken Rockot Correct me if I am wrong!

@Jing Wang I am guessing the existing code that handles touchscreen events is already running on a task runner priority higher than "USER_VISIBLE"?

My adhoc experiments a couple of years ago (the ones that left me expecting 100-1000us normal latency) were for ping/pong messages on an eve device between chromium UI sequence and ML Service daemon.

I didn't experiment with higher taskrunner priorities, would be interesting.

[Ken Rockot]

On Tue, Jan 17, 2023 at 3:41 PM Andrew Moylan <amo...@google.com> wrote:

I assume we are talking about the priority on the task runner: crsrc.org/c/base/task/task_traits.h;l=33

@Ken Rockot Correct me if I am wrong!

I don't actually know what difference you can expect to see by tweaking task priority. Maybe big, maybe small. It depends on a number of other factors, like how the task environment is configured, how busy the task queue is, how slow the other tasks tend to be, etc.

If latency is critical and your implementation is very lightweight (or you can at least defer its more expensive work to another thread), your best bet could be to bind your receiver on the IO thread. This is not something I recommend lightly though, as we really want to keep as much work as possible off of that thread.

@Jing Wang I am guessing the existing code that handles touchscreen events is already running on a task runner priority higher than "USER_VISIBLE"?

It might help to understand a bit more about what you're actually doing. Touch event IPCs already get routed directly to the renderer IO thread, but maybe you're measuring some other source of latency on another thread?

 

My adhoc experiments a couple of years ago (the ones that left me expecting 100-1000us normal latency) were for ping/pong messages on an eve device between chromium UI sequence and ML Service daemon.

I didn't experiment with higher taskrunner priorities, would be interesting.

Yeah if you measure a ping-pong from browser UI thread to some non-busy service and back, I imagine the total latency is utterly dominated by the scheduling of the reply task on the perpetually overworked browser UI thread.

[Jing Wang]

Hi Ken,

A bit more context here: we are considering adding a new module/task in this class (https://source.chromium.org/chromium/chromium/src/+/main:ui/events/ozone/evdev/touch_event_converter_evdev.h;drc=71630a0e336a703e21de9ebeb98a5abf84e8c96c;l=43), it will receive a new data stream from the touch screen and we want to run some neural network model on the data we received to improve palm rejection. To run the NN model, one of the approaches (might be the easiest to implement) is to use the existing ML service via mojo ipc. Since it's about touch experience, we care about the latency, according to your previous emails, it seems reasonable to expect the total extra latency for calling ML service mojo (excluding the actual model inference time) to be under 100us in most cases and under 1ms in extreme case, which we think satisfies our needs. Please correct me if I'm wrong.

Thanks,

Jing

[Ken Rockot]

On Tue, Jan 17, 2023 at 4:22 PM Jing Wang <ji...@google.com> wrote:

Hi Ken,

A bit more context here: we are considering adding a new module/task in this class (https://source.chromium.org/chromium/chromium/src/+/main:ui/events/ozone/evdev/touch_event_converter_evdev.h;drc=71630a0e336a703e21de9ebeb98a5abf84e8c96c;l=43), it will receive a new data stream from the touch screen and we want to run some neural network model on the data we received to improve palm rejection. To run the NN model, one of the approaches (might be the easiest to implement) is to use the existing ML service via mojo ipc. Since it's about touch experience, we care about the latency, according to your previous emails, it seems reasonable to expect the total extra latency for calling ML service mojo (excluding the actual model inference time) to be under 100us in most cases and under 1ms in extreme case, which we think satisfies our needs. Please correct me if I'm wrong.

Under 100us seems very reasonable to expect for browser-to-ML-service latency.

Is palm rejection an async/best-effort sort of thing? If instead it will require a round-trip back to the browser UI thread before the processed touch events can be used as inputs to Chrome UI or web contents, that seems likely to incur a significant penalty on input latency.

[Jing Wang]

Thanks for confirmation.

Yes, it's kind of a best-effort thing. The UI response will not be waiting for the ML service result, the ML service will mark touch strokes as palms so if we receive touch events for the same touch stroke (they all have unique ids), we can just cancel that touch. It's always better to cancel early than late, but 1ms delay seems fine to me.