Unexpected video corruption on packet losses when depayloading RTP VP8 simulcast

[mparis]

Hello,

as some of you know, Chrome sometimes does not retransmit some lost RTP VP8 packets when asked by NACKs.

This is typically done because, at receiver side, Chrome is able to skip unneeded RTP packets / VP8 frames to decode in order to reduce latency.

The sender obviously knows which packets are not needed and it decides to not retransmit them on some network conditions: latency, bandwidth available, etc.

The packets not retransmitted are typically: padding RTP packets used to probe bandwidth and RTP packets of higher temporal layers (enabled on simulcast).

IMO, the sender should not apply this logic and should retransmit whatever packet it is asked in order to ease other receiver implementations different from Chrome/libwebrtc.

Unfortunately, it does not work as I wanted so I am implementing an RTP VP8 depayloader being able to properly handle cases where some lost RTP packets are not retransmitted.

I did not found any problem handling missed padding RTP packets, but yes missed packets belonging to Temporal Layer 2.

The implementation is based on [1] and other docs telling the same info, related to the next image (except red arrows) where the black arrows are the dependency between frames to be respected in order to not having problems on decoding.

My goal is to achieve a MVP in order to have something working ASAP, so I did the simplest solution based on the info I say.
Assuming that the packets are previously ordered by RTP seqnum, the solution just discards any TempLayer 1 or 2 frame after  any missed RTP packet belonging to a to TempLayer 1 or 2 frame until a TempLayer 0 frame.
With this, I came across a case where the video is corrupted. Using the image example:

1. An RTP packet of Frame1 (P2) is missed.
2. Frame1 (P2), Frame2 (P1) and Frame3 (P2) are skipped.
3. Frame4 (P0) is properly decoded (no video corruption).
4. Frame5 (P2), Frame6 (P1) and Frame7 (P2) are corrupted.
5. Frame8 (P0) and following are properly decoded (no video corruption).

After performing several tests I realized that RTP packets belonging TempLayer 1 frames are always retransmitted, so I inferred that (continuing with the same example) Frame5 (P2) and Frame6 (P1) depend on Frame2 (P1) to be properly decoded. It is what I want to highlight with red arrows.

Taking this into account in a second implementation, I  have not seen any video corruption, but I would like to know if:

• q1: I am right with what I inferred?
• q2: Is Chrome's JitterBuffer also taking this detail into account?
  
• q2: Have I missed any details regarding how temporal scalability works?
• q3: Is there any doc, spec or info telling how specifically VP8 temporal scalability works in Chrome?
  

Thanks in advance!!

Refs
[1] https://draft.ortc.org/#rtcrtpencodingtemporal-example*

[Sergio Garcia Murillo]

Obvious question, are you rewriting the picId and tl0picidx?

Best regards
Sergio

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/66c8e6c2-ba7d-42bf-9d0f-6a4fb3a5c124%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mparis]

Hello Sergio,
thanks for the feedback!!

This is a flow just received from Chrome to be recorded/decoded, where we can ignore switchings (the issue is happening without them).
So picId and tl0picidx rewriting is not needed at all in this specific case.

Cheers!!

[Sergio Garcia Murillo]

How are you playing it?

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e2450da5-3cf4-4203-8fe8-ffcf4bab77bf%40googlegroups.com.

[mparis]

I am using ffplay and ffmpeg for detailed processing.