Issues sending custom h264 bitstream through WebRTC's h264 encoder implementation.

[mike ads]

I've modified the software h264 encoding implementation in WebRTC (h264_encode_impl.cc). I pass dummy frames from my video capturer to the WebRTC worker thread using capturer->OnFrameCaptured(), which triggers the Encode() in order to make OpenH264 whatever frame was sent.

Since I'm sending a dummy frame, there is nothing to actually encode. Instead, I've cut out the OpenH264 calls, and I'm passing it my own encoded bitstream that I'm getting from a hardware encoder. I'm creating the EncodedImage object around this bitstream, and I'm looping through the bitstream to find the {0, 0, 0, 1} NAL unit markers and putting their offsets and lengths into RTP fragment header. Once it's all done, call the provided callback function.

Chrome renders the first large keyframe which i can see in my browser, as well as maybe a few following frames, until the video feed freezes after about a second. I check chrome//webrtc-internals and i see that chrome is still receiving video data and decoding it, but it's not drawing anything. After some time, the WebRTC worker thread stops responding to my OnFrameCaptured() calls, Encode() stops being called, and the video stream stops.

I know my h264 bistream is valid because right before calling the callback i can append the bitstream to a file, which i can then use ffmpeg to turn into a playable mp4, which plays normally for the entire duration that i was sending video data. ffmpeg does complain however that there are no timestamps set on the bitstream, could this be the issue?

In chrome I only see the initial keyframe that's sent in my browser (and maybe a few delta frames, hard to tell). On firefox, i see nothing at all.

I'm confident that my fragmentation header offsets and lengths for the NAL units is correct since I've checked multiple times. Are there any special properties the h264 bitstream might need to have to be playable in chrome? Timestamps? Some sort of internal metadata or format?

I'm using Chrome Canary with the h264 command line set. And I'm modifying my SDP to only allow the H264 codec.

Any ideas? Thanks

[mike ads]

Forgot to mention, the attached file contains a few seconds of video footage that can ffmpeg'd to a valid .mp4 file and played. I just appended all the encoded bitstreams together to a single fine.

Maybe someone can tell if something inside is missing.

[mike ads]

Whoops, was trying to edit but deleted my post.

Small update: I'm reading the WebRTC codec RFC (https://tools.ietf.org/html/draft-ietf-rtcweb-video-06#section-6.2) and i noticed it mentioned that browsers (and "non browsers") must implement the "constrained baseline" h264 encoding profile. OpenH264 is strictly only constrained baseline, so I'm thinking maybe Chrome is only able to decode constrained baseline as well. Although this is likely not true, since chrome uses ffmpeg for decoding, so i can't imagine why such a restriction would be in place.

Nevertheless, I've set my encoder to constrained baseline, which it supports, and the video stream still behaves as before. If anyone can correct me about which h264 profiles Chrome is capable of decoding (hopefully all of them), please do. Thanks.

I turned on debugging in Chrome and took a look at the log file and I'm seeing the errors corresponding to the frames:

[13436:1036:0326/010056:ERROR:h264_decoder_impl.cc(320)] avcodec_decode_video2 error: -1094995529

[13436:1036:0326/010056:WARNING:generic_decoder.cc(158)] Failed to decode frame with timestamp 3287187385, error code: -1

[13436:1036:0326/010056:ERROR:h264_decoder_impl.cc(320)] avcodec_decode_video2 error: -1094995529

[13436:1036:0326/010056:WARNING:generic_decoder.cc(158)] Failed to decode frame with timestamp 3287188645, error code: -1

[13436:1036:0326/010056:ERROR:h264_decoder_impl.cc(320)] avcodec_decode_video2 error: -1094995529

[13436:1036:0326/010056:WARNING:generic_decoder.cc(158)] Failed to decode frame with timestamp 3287191615, error code: -1

[13436:1036:0326/010056:ERROR:h264_decoder_impl.cc(320)] avcodec_decode_video2 error: -1094995529

[13436:1036:0326/010056:WARNING:generic_decoder.cc(158)] Failed to decode frame with timestamp 3287194945, error code: -1

Quick googling shows that the -1094995529 error is AVERROR_INVALIDDATA.

[Alexandre GOUAILLARD]

Hi mike,

You're right, constrained baseline, mode 0, mode 1 and only some profile Ids (RFC6184).

The support of H.264 in chrome is very new, and not working in itself. You can see the bug below for problems. In short, come and firefox cannot interoperate today, even though they support the same codec (on paper).

https://bugs.chromium.org/p/chromium/issues/detail?id=591971

Now, you can wait a little bit (google is working hard right now on addressing this) (ETA: 6~12 weeks), and other non google are there helping testing and reporting bugs), or you could try to use firefox code.

While firefox and google share the same webrtc.org code in theory, in practice, the codes differ a little bit. First the network part is completely different (but you don't care so much in your case), and also , the media engines differ a little bit. Mozilla imports the webrtc.org code in their tree from time to time, and maintain some modifications there. Those modification may or may not be back-ported upstream, and when they do, they may or may not be accepted by google. Example at hand, mozilla implements support for H.264 more than a year ago (based on contributed code by BlackBerry), through openH264 (including leveraging cisco pre-compiled plugin) for both encoding and decoding, and proposed an updated patch back to webrtc.org. Did not happen.

In your case, the main classes being the same, you shouldn't have a problem porting your modifications. Mozilla code having been in production for longer than chrome's, you're less likely to hit problems not caused by your code. You could test your modified-firefox against vanilla firefox, waiting for chrome H264 implementation to mature (E.G. Cr 50 to reach stable at least).

HTH.

Alex.

PS: beyond this, it's very likely that not having consistent timestamps is causing problems.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/eea95e1a-4f64-452b-b782-47bed25676bd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

Principal Architect - Citrix, San Francisco

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Brian Baldino]

Hey Mike,

This is a bit of a hunch, but your symptoms sound similar to what we've seen in the past as being related to feedback.  We've had lots of instances where a receiver is receiving data and it's not rendering it, and the reason is it can't correctly decode because it's lost a frame or something.  If you look in webrtc-internals on the receiver, do you see any of the feedback stats (nacks sent, plis sent, etc.) that are constantly increasing?  If so, maybe your encoder isn't properly wired-in to the webrtc stack to handle feedback and send IDRs when necessary.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f6915b0f-569b-4d56-bf98-d2d4c6bde2ec%40googlegroups.com.

[mike ads]

@Brian: Thanks for the suggestions, you are right that I have both Nacks and Plis that are constantly going up. Sometimes only one, sometimes the other, sometimes both.

The "feedback" from WebRTC seems pretty simple overall, since there are only 3 things. Bitrate/Framerate set by SetRates(), and the frame_type vector that's passed as an Encode() param, but only ever has a single element that's always either kVideoFrameDelta or kVideoFrameKey. So if i ever get kVideoFrameKey, i tell my encoder to produce a IDR frame for that frame, which i have confirmed it does every time I ask it to.

As for timestamps mentioned earlier, from what i read NAL units don't actually contain any timestamps. There is a timestamp field in the fragment header, but the existing WebRTC code doesn't set it, I'm assuming it's set somewhere further down the line, or just ignored.

One issue I'm suspecting is that while the OpenH264 implementation mentions that the encoding process produces multiple layers each containing multiple NAL's. The bitstreams I receive from my encoder are single NAL units (besides the initial one, which is 3 NALs). Maybe this is somehow throwing it off, I'm not sure.

[Sergio Garcia Murillo]

quick checks: are you converting it to annex b bitstream before doing the rtp packetization? Are you using single NAL mode or fragmentation?

Best regards
Sergio

--

[Brian Baldino]

NAL/bitstream (like you and Sergio have mentioned) could definitely be a problem, but it might be nice to confirm that your encoder is getting asked for IDRs and is producing them when requested.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f7c25541-2907-4679-be30-a0ca2afcbfc8%40googlegroups.com.

[mike ads]

@Sergio: I believe that the encoder I'm using emits a bitstream which is already in Annex B format. I'm still learning about h264 bitstreams, but based on what I'm reading the fact that i can find the NAL markers {0,0,0,1} in the bitstream suggests to me that it's already in Annex B. These markers are not in random positions either, there is always a {0,0,0,1} marker at the start of the emitted bitstream for instance. So I am not doing any conversion, I am copying the bitstream directly from video memory into the EncodedImage buffer, and setting the buffer's _length accordingly, although the buffer itself has a size of [1280*720*4], just in case. This is parallel to what the original WebRTC code is doing.

As for single NAL mode or fragmentation, I'm not sure. I believe that when my encoder notifies me that encoding is complete for the frame i gave it, the output buffer contains a single full NAL unit which i pass to WebRTC in its entirety.

@Brian, I can confirm that I'm receiving requests from WebRTC to force an IDR frame through the frame_types param passed to Encode(). frame_types is set to either kVideoFrameDelta for a normal frame, or kVideoFrameKey to force an IDR frame. When I observe a kVideoFrameKey, I set the setting on my encoder to force an IDR frame for that next frame, the setting is called "NVIFR_H264_ENC_PARAM_FLAG_FORCEIDR".

Unfortunately, the frame-information that my encoder gives me after encoding is complete seems limited to a boolean "IsIFrame" property of the output statistics, and does not specify what type of I-frame it is, IDR or other. According to the original WebRTC code, I am required to set the "frame_type" property of the EncodedImage to either kVideoFrameDelta, or kVideoFrameKey. This becomes a problem since the WebRTC code does not consider all I-frames as kVideoFrameKey frames. Only IDR frames. This is the code they use:

static FrameType EVideoFrameType_to_FrameType(EVideoFrameType type) {
  switch (type) {
    case videoFrameTypeInvalid:
      return kEmptyFrame;
    case videoFrameTypeIDR:
      return kVideoFrameKey;
    case videoFrameTypeSkip:
    case videoFrameTypeI:
    case videoFrameTypeP:
    case videoFrameTypeIPMixed:
      return kVideoFrameDelta;
    default:
      LOG(LS_WARNING) << "Unknown EVideoFrameType: " << type;
      return kVideoFrameDelta;
  }
}

Since i can't mark all "IsIFrame == true" frames as kVideoFrameKey (because WebRTC only considers IDR frames as key frames), and my encoder doesn't tell me the I-frame type, i have to manually check the NALU code following the {0x0, 0x0, 0x0, 0x1} start code for the 0x65 value, which, as I understand, indicates an IDR frame, and i can ignore the IsIFrame property from my encoder entirely. I actually bitshift the code by 3 and compare it to uint8_t(5) << 3, because it's only the 3-7th bits that indicate the type, according to: http://stackoverflow.com/questions/24884827/possible-locations-for-sequence-picture-parameter-sets-for-h-264-stream

Doing this i can confirm that each time the frame_type paramater of Encode() contains a kVideoFrameKey value, i set the NVIFR_H264_ENC_PARAM_FLAG_FORCEIDR param on my encoder for that frame, and the output always has a {0x0, 0x0, 0x0, 0x1, 0x65} as the first 5 bytes. Confirming, as far as I'm aware, that the encoded frame is an IDR frame, as it should be.

I've put together a debug output for each encoded frame immediately before the callback. The output also contains the hex bitstream, which can be used to verify that the proper IDR's are being encoded and sent. If someone could please take a look and ensure that everything is as it should be, i would very much appreciate it.

[mike ads]

Thanks for all the help everyone, keeping me focused on the IDR frames helped me finally figure it out.

What i noticed was that after about frame number ~80, the first force IDR request was made. And pretty much every frame after that was also a force IDR request, forever. So it seems like the initial IDR (frame #0) was good, but all the next IDRs that were being generated were not good.

The difference seems to be that the initial IDR actually has 3 NALS. The SPS, PPS, and then the IDR frame data itself. The following IDR frames skip the SPS and PPS, and just send the IDR frame alone. Apparently, this is unacceptable. I solved the issue by preserving the SPS and PPS NALUs, and appending them to the buffer each time an IDR frame is generated, and of course including them in the frag header. They seem to be static throughout the session, so hopefully I'm not breaking anything.

After doing this, the stream plays normally.

Again, thanks for all the help.

[Alexandre GOUAILLARD]

Great !

and thanks for documenting your effort on the mailing list.

Alex.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/c9b8bbbd-b925-4d9a-a20e-8aeb3dfce995%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Oren Shir]

@mike, thanks for the updates.

Keep in mind that the PPS and SPS are not guaranteed to never change in all cases. Maybe reusing them is okay in your specific situation, but if you are not sure of it I'd try to change the scene or change encoder settings to see what happens. Sometimes using the wrong PPS and SPS can have subtle impact, for example on colors, so unless you  pay close attention you won't know you are using old PPS and SPS.  

Oren

[mike ads]

@Oren. Thanks for the info! I was worried that the SPS/PPS could indeed change over time, so I did some poking around through the many encoder settings available to me until i found the one responsible for setting SPS/PPS header repeat on IDR frames, aptly called "bRepeatSPSPPSHeader". I set this to true and removed my own code for preserving the header, and everything works perfectly now.

Thanks for the help!

[Andriy Buchynskyy]

Hello, could you tell how did you send EncodedImage to browser?