Video Streaming latency is oddly high

[Sebastian Kunz]

Hello I am developing an application that streams the desktop to the remote peer in the browser. The remote peer can then control the streamers computer remotely. The latency is around 100ms in a local network, which is great. But you can feel the input lag.

To capture the frames I use Windows Desktop Duplication Api, which takes less than 10ms to provide me with the latest desktop frame. I am then using a hardware encoder to encode the frame on the gpu. Encoding takes about 4-5ms. Capturing is done at native resolution which is 4k for me. So where is all the latency coming from? 100ms seems a little high. Does the browser introduce any latency? How long does Decoding take? Is there any way to meassure webrtc latency effectively?

Any pointers are appreciated :) 

[Robert Ayrapetyan]

Hello, have you managed to address this problem? I'm encountering a similar issue, which is detailed here: https://groups.google.com/g/discuss-webrtc/c/wuGQ0GpxMz4. While my timing differs slightly (encoding takes 13 ms), the final lag is approximately the same as yours, at around 100 ms. In the Firefox browser, it's somewhat less, perhaps around 60 ms, but the issue still persists.

I see things like:

constexpr int kStartDelayMs = 80;

defined in the webrtc source code. However, I'm uncertain if the latency we're encountering is solely determined by this hardcoded value.

[Muhammad Usman Bashir]

@Robert Ayrapetyan, buffering delay can be adjusted, but be cautious as reducing this value can make the stream more susceptible to network jitter. The default kStartDelayMs 80ms is quite conservative. Reducing kStartDelayMs to lower values like 30-40ms can lower the initial latency and speed up ramp up time. However, going too low increases the risk of underruns.

The jitter buffer delay adaptively changes during a call based on measured network jitter. So kStartDelayMs only affects the very start.
Other key buffer parameters are min_playout_delay_ms and max_playout_delay_ms. Reducing min_playout_delay_ms from e.g. 200ms to 100ms allows faster buffer reduce.
The render_delay_ms in the video engine sets how far rendering lags behind the real-time frame arrival. Lower values here reduce latency directly but increase the chance of rendering glitches if frame arrival fluctuates.

I'd recommend starting by reducing kStartDelayMs, min_playout_delay_ms and render_delay_ms in small increments and testing the effect on latency and stability. Please share console logs with "webrtc_video_engine.cc" search records.

[Robert Ayrapetyan]

Hi Muhammad!

I believe your statement holds true for a typical WebRTC session (e.g.
a client connected to a remote server with a good network connection).
However, in my specific scenario, I'm running all processes locally on
the same machine, and the round-trip time (RTT) is consistently below
2 milliseconds,
I have abundant encoding and decoding resources at my disposal, and
there are no congestion points anywhere in the data flow.
I have disabled an audio stream to avoid all kinds of sync issues.
minPlayoutDelay stays on 0. Jitter is 0.001. PacketsLost is 0.
nackCount stays on 0. I've set kStartDelayMs and kMax to 0.
As per logs, from the time the event was triggered to the
"OnFrameBufferUpdated" delay is about 17ms.
But yet an additional rendering lag of 100 ms comes from nowhere in my setup.

Here is a log produced by webrtc_video_engine as you've requested (not
sure how it can be useful to resolve my issue):

[425765:12:0922/063713.757668:INFO:webrtc_video_engine.cc(753)]
WebRtcVideoEngine::WebRtcVideoEngine()
[425765:13:0922/063713.778952:VERBOSE1:webrtc_video_engine.cc(1584)]
OnReadyToSend: Not ready.
[425765:13:0922/063713.779930:INFO:webrtc_video_engine.cc(1119)]
SetSenderParameters: {codecs: [VideoCodec[96:H264],
VideoCodec[97:red], VideoCodec[98:ulpfec], VideoCodec[99:rtx],
VideoCodec[100:rtx]], conference_mode: no, extensions: [{uri:
http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01,
id: 1}], extmap-allow-mixed: false, max_bandwidth_bps: -1, mid:
video0}
[425765:13:0922/063713.780259:INFO:webrtc_video_engine.cc(1127)]
Negotiated codec: VideoCodec[96:H264]
[425765:13:0922/063713.780404:INFO:webrtc_video_engine.cc(2815)]
AddRecvStream: {id:webrtctransceiver0;ssrcs:[726735724,1405591231];ssrc_groups:{semantics:FID;ssrcs:[726735724,1405591231]};cname:user4072096014@host-fcf5e148;stream_ids:user4072096014@host-fcf5e148;}
[425765:13:0922/063713.782051:VERBOSE1:webrtc_video_engine.cc(2777)]
SetReceive: false
[425765:13:0922/063713.783512:VERBOSE1:webrtc_video_engine.cc(1370)]
SetSend: false
[425765:13:0922/063713.784416:INFO:webrtc_video_engine.cc(2972)]
SetSink: ssrc:726735724 (ptr)
[425765:13:0922/063713.824854:VERBOSE1:webrtc_video_engine.cc(2777)]
SetReceive: false
[425765:13:0922/063713.826594:VERBOSE1:webrtc_video_engine.cc(1370)]
SetSend: false
[425765:13:0922/063713.827270:INFO:webrtc_video_engine.cc(2717)]
SetReceiverParameters: {codecs: [VideoCodec[96:H264],
VideoCodec[97:red], VideoCodec[98:ulpfec], VideoCodec[99:rtx],
VideoCodec[100:rtx]], extensions: [{uri:
http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01,
id: 1}]}
[425765:13:0922/063713.828291:INFO:webrtc_video_engine.cc(2723)]
Changing FlexFEC payload type (recv) from 49 to -1
[425765:13:0922/063713.828448:INFO:webrtc_video_engine.cc(2734)]
Changing recv codecs from {VideoCodec[96:VP8], VideoCodec[98:VP9],
VideoCodec[100:VP9], VideoCodec[35:VP9], VideoCodec[37:VP9],
VideoCodec[102:H264], VideoCodec[104:H264], VideoCodec[106:H264],
VideoCodec[108:H264], VideoCodec[110:H264], VideoCodec[39:H264],
VideoCodec[41:H264], VideoCodec[43:H264], VideoCodec[45:AV1],
VideoCodec[47:AV1]} to {VideoCodec[96:H264]}
[425765:13:0922/063713.830277:VERBOSE1:webrtc_video_engine.cc(2777)]
SetReceive: true
[425765:13:0922/063713.830454:VERBOSE1:webrtc_video_engine.cc(1370)]
SetSend: false
[425765:13:0922/063713.918954:VERBOSE1:webrtc_video_engine.cc(1584)]
OnReadyToSend: Ready.
[425765:13:0922/063713.926355:VERBOSE1:webrtc_video_engine.cc(2777)]
SetReceive: true
[425765:13:0922/063713.926410:VERBOSE1:webrtc_video_engine.cc(1370)]
SetSend: false

> --
> This list falls under the WebRTC Code of Conduct - https://webrtc.org/support/code-of-conduct.
> ---
> You received this message because you are subscribed to a topic in the Google Groups "discuss-webrtc" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/discuss-webrtc/glPKevi1jok/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to discuss-webrt...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/b1a52a0f-bc91-4a63-afe7-d504e59d2951n%40googlegroups.com.

[Muhammad Usman Bashir]

@Robert, logs show codec negotiation is happening properly and frames are being received. 

Some other key points:

• Upgrade WebRTC stack to m112 to onwards.
• If you can compile ' webrtc.lib ', try instrumentation to measure time at each stage from capture -> encode -> decode -> render.
• Verify all rendering optimizations are enabled in the client (e.g, zero-copy).
• Test with simpler stream settings as a baseline (lower resolution, different codecs).
• Additionally, please share debug logs from the video_receive_stream.cc and video_send_stream.cc.  The VideoReceiveStream and VideoSendStream implementations will help identify where the extra latency is introduced.
• Additionally, @Robert you can implement a datachannel-based NTP. This will help me to better understand the latency over other channels of webrtc.

@Robert, Last year, I had the opportunity to add support for and develop FFMPEG + WebRTC as well as OBS + WebRTC Streaming. I am achieving latency below 20ms at 4K resolution over LAN. Last month, I upgraded to the WebRTC m115 stack. I would be happy to assist you with your video lag issue.

[Robert Ayrapetyan]

Thanks Muhammad!
Just curious - how did you measure latency (20ms) in your setup?
I mean as per measurements latency in my setup is around 20ms as well,
but visual lag (e.g. mouse cursor movements) is much higher.
Have you had any user navigation support in your webrtc session?


On Sat, Sep 23, 2023 at 4:31 AM Muhammad Usman Bashir

> To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/9bdee096-b1c2-43f8-a508-bd6cc4638088n%40googlegroups.com.

[V I]

There are different types of latency - there are some absurdly low values you get reported from webrtc, and then there's glass-to-glass latency which is a different beast entirely. That said, reliable 30-40ms glass-to-glass latency is achievable on localhost or wired LAN.

One of the important things which is often overlooked is libwebrtc's bandwidth estimator and pacer, the trick is to crank up the bitrate to make libwebrtc send data faster. Basically, if you plan to send 10mbps, set it to 30-50mbps with PeerConnectionInterface::SetBitrate

You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAAboi9vvj5xhPKigvedCgsf59KuerBCTn6RFeOHPveYwzELajQ%40mail.gmail.com.

[Robert Ayrapetyan]

In my case libwebrtc is used only on the client side (Chromium
browser), while the server side (stream generation) relies solely on
the GStreamer framework.
Is it possible to invoke the "PeerConnectionInterface::SetBitrate"
method from JavaScript to test this?
A client-side of my application is intended to be a standard Chrome or
Firefox browser, rather than customized one.
It's worth noting that my issue doesn't appear to be related to
bitrate problems. I consistently maintain 60 frames per second with no
frame drops, NACKs, a minimum playout delay of 0, and so on. The
anomaly seems to occur after a frame reaches the
"OnFrameBufferUpdated" function on the client side, where an
unexpected additional delay of approximately 80 milliseconds appears
to be introduced from an unknown source...

> To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAFa3_%3D-vDaqKsWf6NCwX-VsqogWfiiEDt_H8q0rZtaFGHph1-g%40mail.gmail.com.