M58 H.264 decoding compatability issue

[foryou...@gmail.com]

Looks like in 58.0.3029.81 when the remote the RTC client receives h.264 stream that is with NALU type of SEI within it, the SW ffmpeg-based h.264 decoder is not able to decode it. This does not happen in M57. 

Is this behavior expected? why this change?

[Philip Eliasson]

I assume you are talking about SEI recovery points. The SEI NALU can be used for many things, and in the case of a recovery point it could be considered to be a keyframe, but the problem is that WebRTC does not have a deep understanding of H264 so we can't distinguish between different types of SEIs. Earlier all SEIs were (incorrectly) interpreted as keyframes which could cause decoding artifacts, so we changed it (https://codereview.chromium.org/2769863007/) to always be interpreted as a delta frame, so this behavior is expected.

Does this cause a problem for you, and if so, in what case?

/Philip

[Hitesh Chouhan]

Hi Philip,

In our inter-op testing between Chrome and native soft clients running on Windows, we have observed that Chrome is not able to decode remote H.264 video stream (while vice-versa video is working fine). The soft client is using Intel MSDK hardware encoder which is generating SEI packets in the H.264 video stream. As per above reply, SEI packets are getting treated as delta frame. In our observation from Chromium logs, SEI packet when treated as delta frame is causing Keyframe to be discarded right from the beginning and hence resulting in no video being displayed on the Chrome side. We are looking in to fixing soft clients so that they don’t send SEI packets. However, we believe that the same issue could occur during inter-op with any other third party client that our customers use and whose source code we cannot modify. Hence, we are working on understanding Chromium code to see how we could skip the SEI frame, instead of adding it to frame buffer as delta frame. Could you please provide any pointers on how to fix this issue in Chrome?

Also, please find attached Wireshark trace with RTP packets for the H264 stream.

Thanks,

Hitesh

[andy424]

Not exactly the same issue but related in that it is another H264 decoding issue that I observed since chrome58. chrome57 was able to decode H264 high profile 4.1 even though the claim is that only constrained baseline profile is supported. However, in chrome58, this capability has disappeared. Why taketh away higher level profile decoding functionality? Anyone else can confirm or shed light on this on future plans to support higher profiles on H264 in chrome and firefox?

https://groups.google.com/forum/#!topic/meetecho-janus/uWz90cbEzb0

[andy424]

I noticed there is an active chromium bug report in assigned state relating to H264 decoding issues in chrome58

https://bugs.chromium.org/p/chromium/issues/detail?id=717884

[Hitesh Chouhan]

Our issue is specific to H264 video stream which has SEI NALUs, we are looking into Chromium logs(with added logs around H264 NALUs handling) to understand this behavior. We observe no video issue on Chrome if sequence of NALUs is as follows:

SPS, PPS, SEI, IDR or Keyframe, SEI, Delta-Frame, SEI, Delta-Frame....

As per Philip's reply above the fix https://codereview.chromium.org/2769863007/ was done to treat SEI packets as Delta frames which were earlier getting treated as Keyframes. So we are looking around this code to understand how above sequence of NALUs are getting handled. 

[Alexandre GOUAILLARD]

andy,

high profile is supported in the decoder (ffmpeg-based), but not in the encoder (openH264 based).

On Fri, May 5, 2017 at 6:58 AM, andy424 <sake...@gmail.com> wrote:

I noticed there is an active chromium bug report in assigned state relating to H264 decoding issues in chrome58

https://bugs.chromium.org/p/chromium/issues/detail?id=717884

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/53879115-1183-485d-8f76-0fa5fefa6d7a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Hitesh Chouhan]

Hi Philip,

We have tried few changes in Chromium to handle SEI packets coming in H264 stream and found it to be working. We have created bug https://bugs.chromium.org/p/chromium/issues/detail?id=719095 to track this issue of blank video on Chrome. We have attached a diff file with fix on this ticket and also attaching to this post. Could you please advise if our analysis (as updated in the ticket) is correct and suggest the right way to fix it?

Thanks,

Hitesh

[foryou...@gmail.com]

Looks like after merging this patch, Chrome 59 stable release on most Intel platform on Windows will not work if the received WebRTC h.264 stream starts an AU with nal_type=9(AU delimiter). No frame is decoded per watching the trace from DXVA decoder or webrtc.

This does not happen on M58.

We observed on Chrome 59, on Windows + Intel platform, each AU starts with AU delimeter, while for Chrome 58, there's no AU delimeter. Which change is causing this bitstream difference?