Insertable Stream Recognised as NALU in H264

[Aidan Howie]

I am using insertable streams via the TransformableFrameInterface to append additional data to the frame which I am sending. Within this data I have a series of bytes "00 00 00 01" which is being picked up by RtpPacketizerH264 as a NALU. This is then being put into it's own packet and when received at the receiver, a warning "Failed to parse PPS id and SPS id from PPS slice" is logged. Is there anyway to tell WebRTC that the insertable stream data should not be parsed as part of a NALU?

[V I]

You just need to insert emulation prevention bytes in your data prior to pushing it into libwebrtc

On Fri, 29 Sep 2023 at 14:57 Aidan Howie <ahow...@gmail.com> wrote:

I am using insertable streams via the TransformableFrameInterface to append additional data to the frame which I am sending. Within this data I have a series of bytes "00 00 00 01" which is being picked up by RtpPacketizerH264 as a NALU. This is then being put into it's own packet and when received at the receiver, a warning "Failed to parse PPS id and SPS id from PPS slice" is logged. Is there anyway to tell WebRTC that the insertable stream data should not be parsed as part of a NALU?

--
This list falls under the WebRTC Code of Conduct - https://webrtc.org/support/code-of-conduct.
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/fcd468dd-2e8a-4235-963a-00ca2b3f1bb2n%40googlegroups.com.

[Aidan Howie]

Thanks for the reply. Would this not still clash with potential data within the Insertable Stream? Say I wanted to send a big endian uint32 of value 769 (00 00 03 01), how would I distinguish this from emulation prevention bytes? 

[Harald Alvestrand]

Sending your modified frames through the H.264 packetizer is a Bad Idea - the H.264 packetizer is much too complex for comfort.

Unfortunately we haven't added all the APIs needed to do this righter in the spec yet - see https://github.com/w3c/webrtc-encoded-transform/pull/186 and in particular the use of setMetadata to see what is currently being proposed to address this issue.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e1adb644-2a72-4259-b541-cdf29139e6e1n%40googlegroups.com.

[Aidan Howie]

Thanks for the additional information. I will implement into a quick fix for now and wait until the API has been agreed and libwebrtc updated to change the implementation.

[V I]

It's relatively easy to trick libwebrtc's H.264 parser if you know where to inject your arbitrary data. Appending data the way you do is pretty robust as long as there are no Annex B startcodes in it.

In the case you described 00 00 03 01 has to be transformed into 00 00 03 03 01 on the sender, and then you just replace all the 00 00 03 with 00 00 on the receiver. This process is as old as H.264 itself, just Google for Annex B and emulation prevention, or read ISO/IEC 14496-10, section 7.4.1.1 Encapsulation of an SODB within an RBSP

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/59ffbe3c-d8dc-4ca8-b2fe-ddc6d0a8473dn%40googlegroups.com.

[Rajneesh Soni]

custom data can be added to H.264 stream as SEI message.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAFa3_%3D9Dajw%3DnMsACLdGNVLG5Xiun%3DQSKKTh%3DdvrbgvRfydZ%3DQ%40mail.gmail.com.

--

regards,

Rajneesh

[Aidan Howie]

Thanks for the information, I was unaware this existed. This is something I should look into in the future.

[V I]

Right, a custom SEI message is probably the best way to do it - but again, you need to run emulation prevention on it

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f8550cec-42fe-45a2-ad22-b4a5f54ec724n%40googlegroups.com.