How to properly use Insertable Streams from C++
1,039 views
Skip to first unread message
Tom Leavy
May 13, 2020, 2:34:39 PM5/13/20
to discuss-webrtc
I am working on testing out the new Insertable Streams functionality from within in a native C++ app. I'm having an interesting issue where (I think) I'm setting up a FrameTransformerInterface properly and assigning it to RTPSender / RTPReceiver via the exposed SetEncoderToPacketizerFrameTransformer and SetDepacketizerToDecoderFrameTransformer functions . The problem I'm having is that currently it seems that even when I simply make a passthrough transformer data is encoded by but never received by the other party via RTPReceiver. I think this is due to a fundamental misunderstanding of the API on my side around how to properly implement a FrameTransformerInterface. I can see the transform function is called, but it seems to result in the other side never getting any packets? I know this is sort of bleeding edge right now, but pointing me in the right direction would be highly appreciated. See below code example.
class RTCPassthroughTransformer : public FrameTransformerInterface {
public:
cricket::MediaType m_mediaType;
RTCPassthroughTransformer(cricket::MediaType mediaType)
{
m_mediaType = mediaType;
}
void Transform(std::unique_ptr<TransformableFrameInterface> transformable_frame)
{
rtc::ArrayView<const uint8_t> transformableFrameData = transformable_frame->GetData();
transformable_frame->SetData(transformableFrameData);
}
};
Philipp Hancke
May 14, 2020, 10:31:22 AM5/14/20
to discuss...@googlegroups.com
if you're just interested in encryption,
might be easier to use (its sync; this was no longer true for the JS variant)
--
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/f8c9d241-bda7-4afd-bf41-1f215d379bfa%40googlegroups.com.
Tom Leavy
May 14, 2020, 11:46:22 AM5/14/20
to discuss...@googlegroups.com
Thanks for the heads up. I did see that API and wasn’t sure if it was the same outcome as the new callbacks as far as how the data is handled. If it is the same I will try to go that route instead since the encryption being sync isn’t an issue for me.
I was able to figure out how to use the proper callbacks to get my original passthrough encoder working, and I found a crash in rtp_sender_video_frame_transformer_delegate in the M83 branch.
This function:
void RTPSenderVideoFrameTransformerDelegate::OnTransformedFrame(
std::unique_ptr<TransformableFrameInterface> frame) {
rtc::CritScope lock(&sender_lock_);
// The encoder queue gets destroyed after the sender; as long as the sender is
// alive, it's safe to post.
if (!sender_)
return;
rtc::scoped_refptr<RTPSenderVideoFrameTransformerDelegate> delegate = this;
encoder_queue_->PostTask(ToQueuedTask(
[delegate = std::move(delegate), frame = std::move(frame)]() mutable {
delegate->SendVideo(std::move(frame));
}));
}
std::unique_ptr<TransformableFrameInterface> frame) {
rtc::CritScope lock(&sender_lock_);
// The encoder queue gets destroyed after the sender; as long as the sender is
// alive, it's safe to post.
if (!sender_)
return;
rtc::scoped_refptr<RTPSenderVideoFrameTransformerDelegate> delegate = this;
encoder_queue_->PostTask(ToQueuedTask(
[delegate = std::move(delegate), frame = std::move(frame)]() mutable {
delegate->SendVideo(std::move(frame));
}));
}
I get a null dereference crash on encoder_queue_->PostTask … the sender is not null but the encoder_queue is. This is happening on the first frame of a new video, I think there is a race condition where the sender is made, and I received the registration callback, but the encoder_queue is not yet initialized. I believe this is happening because even though these callbacks are designed to support async processing, I’m calling things synchronously in in my RTCPassthroughTransformer::Transform function
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CADxkKiL%3DOKviEFbTb9b0JT2mbxDEDEfrEoBfd3K4hQ%3DtbP9dFw%40mail.gmail.com.
Jason Thomas
May 11, 2021, 5:43:53 PM5/11/21
to discuss-webrtc
Hi Tom,
Did you get any further with understanding how to use the C++ insertable streams API?
As far as I can tell a large part of the implementation is in the Chromium source instead of in libwebrtc.
After tracing through chains of delegates and various levels of misdirection, I've managed to implement my own class that implements webrtc::FrameTransformerInterface, and which can register callbacks for an SSRC in RTCRtpReceiver, and RTCRTPSender.
I can mutate the frame through the TransformableFrameInterface, passed into 'Transform', and then pass it into the callback for the SSRC for the frame to get it to forward it to the remote end.
On the remote side I use 'Transform' to transform it back, and it appears to work.
Did you get any further with understanding how to use the C++ insertable streams API?
As far as I can tell a large part of the implementation is in the Chromium source instead of in libwebrtc.
After tracing through chains of delegates and various levels of misdirection, I've managed to implement my own class that implements webrtc::FrameTransformerInterface, and which can register callbacks for an SSRC in RTCRtpReceiver, and RTCRTPSender.
I can mutate the frame through the TransformableFrameInterface, passed into 'Transform', and then pass it into the callback for the SSRC for the frame to get it to forward it to the remote end.
On the remote side I use 'Transform' to transform it back, and it appears to work.
The issue I am having is in most of the insertable stream examples on the JS level it appears the full frame is passed up as a DOMArrayBuffer (In Chromium: modules/peerconnection/rtc_encoded_video_frame_delegate.cc: `DOMArrayBuffer* RTCEncodedVideoFrameDelegate::CreateDataBuffer() const`), which I expect then is mutated in the JS space, and then inserted back into TransformableVideoFrameInterface via 'SetData`:
modules/peerconnection/rtc_encoded_video_frame_delegate.cc:51:
`
`
void RTCEncodedVideoFrameDelegate::SetData(const DOMArrayBuffer* data) {
MutexLocker lock(mutex_);
if (webrtc_frame_ && data) {
webrtc_frame_->SetData(rtc::ArrayView<const uint8_t>(
static_cast<const uint8_t*>(data->Data()), data->ByteLength()));
}
}
`
Most of the examples I have seen using insertable streams just append additional data to the end of the payload, and then extract it, restoring the frame to its original.
Unfortunately in my example, while I can mutate the frame using SetData, and basically break it, or modify fields inside of it, if I try using SetData with a larger buffer, I don't see that data when I set up an FrameTransformerInterface and attach it to the receiving RTCRtpReceiver.
The frame seems to only ever be the same size as the frame I sent. I expect it's because it's reconstructing everything including the size from RTP. I just can't see anywhere in Chromium where the metadata attached from the DOM gets attached into the frame being sent... Nothing like, adding a metadata field to the RTP headers... It does have a lot of its internals exposed through RTCEncodedVideoFrame in Chromium: modules/peerconnection/rtc_encoded_video_frame.cc, and I suppose it's possible the metadata and the frame and everything are being constructed somewhere into the data payload that is being placed back via SetData into the TransformableVideoFrameInterface?
It would be great if there was documentation anywhere on this. Mostly I've been scouting through commit messages when this was being built.
Why not make an entire Insertable Streams compatible implementation in libwebrtc, rather than making a very different interface, and leaving most of it in Chromium?
Any help you have, or anything you discovered with be greatly helpful.
My ultimate goal is to just have a per-frame metadata transmission mechanism. It doesn't necessarily have to meet the same interface as insertable streams.
Most of the examples I have seen using insertable streams just append additional data to the end of the payload, and then extract it, restoring the frame to its original.
Unfortunately in my example, while I can mutate the frame using SetData, and basically break it, or modify fields inside of it, if I try using SetData with a larger buffer, I don't see that data when I set up an FrameTransformerInterface and attach it to the receiving RTCRtpReceiver.
The frame seems to only ever be the same size as the frame I sent. I expect it's because it's reconstructing everything including the size from RTP. I just can't see anywhere in Chromium where the metadata attached from the DOM gets attached into the frame being sent... Nothing like, adding a metadata field to the RTP headers... It does have a lot of its internals exposed through RTCEncodedVideoFrame in Chromium: modules/peerconnection/rtc_encoded_video_frame.cc, and I suppose it's possible the metadata and the frame and everything are being constructed somewhere into the data payload that is being placed back via SetData into the TransformableVideoFrameInterface?
It would be great if there was documentation anywhere on this. Mostly I've been scouting through commit messages when this was being built.
Why not make an entire Insertable Streams compatible implementation in libwebrtc, rather than making a very different interface, and leaving most of it in Chromium?
Any help you have, or anything you discovered with be greatly helpful.
My ultimate goal is to just have a per-frame metadata transmission mechanism. It doesn't necessarily have to meet the same interface as insertable streams.
Jason Thomas
May 11, 2021, 6:19:24 PM5/11/21
to discuss-webrtc
I think perhaps the answer is here:
https://github.com/w3c/webrtc-encoded-transform/issues/37
I suppose I need the GFD extension, or to handle the specific packetization for H.264/VP8/VP9...
https://github.com/w3c/webrtc-encoded-transform/issues/37
I suppose I need the GFD extension, or to handle the specific packetization for H.264/VP8/VP9...
On Thursday, May 14, 2020 at 9:46:22 AM UTC-6 Tom Leavy wrote:
Jason Thomas
May 12, 2021, 6:40:58 PM5/12/21
to discuss-webrtc
Just for anyone interested, I did manage to append data to the payload of my stream and send it to the receiving side/unpack it.
Maybe it was just specific to my setup, because I'm using H.264, but the GFD extension makes little difference.
The issue was, extending the size of my payload didn't affect what was transmitted because the fragmentation_headers are pre-calculated.
TransformableVideoSenderFrame is anonymous and implements the TransformableVideoFrameInterface in modules/rtp_rtcp/source/rtp_sender_video_frame_transformer_delegate.cc, and GetFragmentationHeader() isn't accessible through TransformableVideoFrameInterface.
The fragmentation headers need to be adjusted to include the additional space for your extended data, or it won't be transmitted by the packetizer (at least for the case of H.264).
To do this, I modified the header for TransformableVideoFrameInterface in src/api/frame_transformer_interface.h to include:
virtual RTPFragmentationHeader* GetFragmentationHeader() const {return nullptr;}
And set it to 'override' this interface in TransformableVideoSenderFrame.
Then I just adjust the Fragmentation header to include the additional space for my payload, and it gets transferred, and I can unpack it on the opposite end.
Maybe I could put together a gist or something for anyone who is interested in the future in doing this.
- Jason.
Maybe it was just specific to my setup, because I'm using H.264, but the GFD extension makes little difference.
The issue was, extending the size of my payload didn't affect what was transmitted because the fragmentation_headers are pre-calculated.
TransformableVideoSenderFrame is anonymous and implements the TransformableVideoFrameInterface in modules/rtp_rtcp/source/rtp_sender_video_frame_transformer_delegate.cc, and GetFragmentationHeader() isn't accessible through TransformableVideoFrameInterface.
The fragmentation headers need to be adjusted to include the additional space for your extended data, or it won't be transmitted by the packetizer (at least for the case of H.264).
To do this, I modified the header for TransformableVideoFrameInterface in src/api/frame_transformer_interface.h to include:
virtual RTPFragmentationHeader* GetFragmentationHeader() const {return nullptr;}
And set it to 'override' this interface in TransformableVideoSenderFrame.
Then I just adjust the Fragmentation header to include the additional space for my payload, and it gets transferred, and I can unpack it on the opposite end.
Maybe I could put together a gist or something for anyone who is interested in the future in doing this.
- Jason.
Message has been deleted
Ivan Hutomo
Nov 4, 2022, 2:20:27 AM11/4/22
to discuss-webrtc
Hello everyone, I would to add a custom set frame encryptor inside a self-build electron. I already tried insertable streams but it was not working together with my modified webrtc for android. So here I already tried this custom encryptor in low-level webrtc and its working. My problem now is how I could set my custom frame encryptor in higher-level code, so I could decide to set it from JavaScript. I tried to add something here in the third_party/blink
new RTCPeerConnection()
According to Standard WebRTC API
Including lib-jitsi-meet.js from Jitsi-Meet
They're using the following API
RTCPeerConnection
RTCRtpSender
RTCRtpReceiver
So,
Encrypt frames could be here in RTCRtpSender.
Decrypt frames could be here in RTCRtpReceiver.
third_party\blink\renderer\modules\peerconnection
rtc_rtp_receiver.cc
rtc_rtp_receiver.h
rtc_rtp_sender.cc
rtc_rtp_sender.h
rtc_rtp_transceiver.cc
rtc_rtp_transceiver.h
Now I tried to add my custom frame encryptor and decryptor to the corresponding webrtc's sender and corresponding webrtc's receiver. I am adding the code inside rtc_rtp_sender.cc something like below:
void RTCRtpSender::setFrameEncryptor(std::vector<uint8_t> new_keys) {
rtc::scoped_refptr<webrtc::OctonFrameEncryptor> encryptor_(
new webrtc::OctonFrameEncryptor(new_keys));
sender_.get()->SetFrameEncryptor(encryptor_);
// std::unique_ptr<RTCRtpSenderPlatform> sender_ -> SetFrameEncryptor(encryptor_);
}
But I could not get the correct corresponding webrtc's sender, always compile an error with something like this:
But I could not get the correct corresponding webrtc's sender, always compile an error with something like this:
../../third_party/blink/renderer/modules/peerconnection/rtc_rtp_sender.cc(577,18): error: no member named 'SetFrameEncryptor' in 'blink::RTCRtpSenderPlatform'
sender_.get()->SetFrameEncryptor(encryptor_);
Thank you everyone for helping
sender_.get()->SetFrameEncryptor(encryptor_);
Thank you everyone for helping
Reply all
Reply to author
Forward
0 new messages