Consistent FPS for game streaming, don't wait for keyframes

[Florian Grill]

Hi,

I found a post back from 2017 and maybe someone has already an answer to this question 

https://groups.google.com/g/discuss-webrtc/c/dAlWHmqkhNk

In specific, I have a WebRTC connection from my Android app to a game console which is streaming in 60fps and full hd/ hd. Everything runs fine but if I simulate bad network conditions, especially frame drops this will significantly decrease fps and is introducing a lot of stuttering. I use my own compiled webrtc lib and I already adapted some things for my needs. I basically want consistent fps and I don't want to wait for a key frame when there are dropped packages. I want to send every frame to the decoder and let it handle the error concealment. I know this will cause artefacts but it's better than delays and huge frame jumps especially if you are using WebRTC for game streaming.

I also implemented a native game streaming protocol before and there I did exactly that, if there was a dropped frame I sent a key frame request but I continued to further send frames to the decoder. This caused some small artefacts but it was fixed as soon as the key frame arrived. So my question is that somehow easily possible with the WebRTC lib (compiled for Android)? If not what must be adjusted in the native code to achieve this? I don't expect a complete guide but maybe just a hint of what I should look into.

[V I]

Chrome's WebRTC Encoded Transform behaves exactly this way - video frames are delivered as if nothing happened causing video artefacts. What you can do is try to plug into the video pipeline via this method: https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/api/rtp_receiver_interface.h;l=114?q=SetDepacketizerToDecoderFrameTransformer&ss=chromium%2Fchromium%2Fsrc - I suspect this is how it's implemented in Chrome, I might be wrong though

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/c31d8ed7-77d1-4cdd-8fe5-a80f5ba0c91cn%40googlegroups.com.

[Florian Grill]

Okay interesting I have to check that. Like I said I'm in an Android native environment and it definitely doesn't behave that way. Hope this will work out as for game streaming this would really be a benefit.

[Philipp Hancke]

That is because encoded transform operates before the decoder which is the entity that typically detects a "frame loss" and waits for a decodeable frame.

libwebrtc is essentially still a VPx decoding pipeline and in VPx decoding with loss is a terribly bad idea and leads to garbage.

H264 behavior matches the VPx even though H264 as a codec deals with this scenario *much* better but for video conferencing the consensus (or rather libwebrtc's dominance in expectations?) is that the smearing this causes is not acceptable while sub-second video freezes are ok perception-wise.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/02cf22d1-5205-4486-8582-0a298e275231n%40googlegroups.com.

[Florian Grill]

Yeah I get that and it makes sense but for game streaming it's not at all acceptable to wait for a keyframe several seconds and yes that can happen during some strange network interferences. The gamestreaming service only streams h264 and I'm familiar with h264 as I used it in another native streaming project, the decoders on Android can deal perfectly fine with skipped frames. It will cause some artefacts but this will resolve itself upon the next received keyframe. This is usually how low latency game streaming works and this is the behavior I want :) I just have to check which native transformation code is performed on Android and I guess I have to modify this part to not wait for a keyframe, if I have understood that correctly? Not sure if this is easily achievable.

[Florian Grill]

I think I figured it out finally, in the frame_buffer2.cc you have to disable this check

https://source.chromium.org/chromium/chromium/src/+/main:third_party/webrtc/modules/video_coding/frame_buffer2.cc;l=154

It works much much better now, I can even play somehow with 25% package loss (of course not a realistic scenario) wherase previously this was already almost impossible at 5-10% package loss because of these many frame skips/ drops. Maybe this information is also useful for someone else. Not sure if this is the only part one can fine tune for a gamestreaming use case though?