Integration of external video encoder with internal source

[Andriy Tylychko]

How can I integrate external video encoder with internal source? Is there any open-source sample I can check?

I created `MyVideoEncoderFactory: public webrtc::VideoEncoderFactory`. Its `QueryVideoEncoder` method returns `CodecInfo` with `has_internal_source == true`. Its `CreateVideoEncoder` method returns `MyVideoEncoder: public webrtc::VideoEncoder`.

I pass `MyVideoEncoderFactory` instance to `webrtc::CreatePeerConnectionFactory` as `video_encoder_factory` parameter. 

Now, looking at peer connection client sample, I need to create a video track. There're 2 overloads of `PeerConnectionFactory::CreateVideoTrack` both requiring `VideoTrackSourceInterface`. It's where I'm lost.

`MyVideoEncoder` is (technically) also my video track source. Does this mean it should implement something else in addition to `webrtc::VideoEncoder` interface so I can use its instance in `CreateVideoTrack`? Not sure as it seems it's the task of `MyVideoEncoderFactory` to create `MyVideoEncoder` instance. Or is there another method to create a video track?

[Andriy Tylychko]

Sorry, correction: there's only one `PeerConnectionFactory::CreateVideoTrack()`

I've checked "H264 HW encode using VideoToolbox" changelist  but I don't understand it, most probably I'm missing other parts. I don't see how video track is constructed.

Also I found `FakeVideoTrackSource` (used in unit-tests) that doesn't produce any frames. Sounds like it can be what I need. Should I use something similar to create a video track?

[Niels Moller]

On Wed, Feb 21, 2018 at 3:57 PM, Andriy Tylychko
<andriy....@gmail.com> wrote:
> How can I integrate external video encoder with internal source? Is there
> any open-source sample I can check?
>
> I created `MyVideoEncoderFactory: public webrtc::VideoEncoderFactory`. Its
> `QueryVideoEncoder` method returns `CodecInfo` with `has_internal_source ==
> true`. Its `CreateVideoEncoder` method returns `MyVideoEncoder: public
> webrtc::VideoEncoder`.

Using internal_source is deprecated, and it will hopefully be deleted
reasonably soon.

The way I'd recommend is to implement a VideoTrackSourceInterface
which wraps encoded frames as as VideoFrameBuffer with type kNative,
pass them through the normal pipeline, and inject an encoder which can
recognize this and unwrap the encoded data.

Regards,
/Niels

[Andriy Tylychko]

Thanks Niels.

I'm still quite confused. WebRTC notifies encoder when it needs key-frame produced or bitrate changed. In case of fake encoder with custom VideoTrackSourceInterface implementation that produces encoded frames, the encoder has no control over encoding process. 

Should fake encoder communicate with VideoTrackSourceInterface behind the scene to pass this info (frame type/bitrate/etc.)?  

VideoTrackSourceInterface and Encoder normally communicate asynchronously (passing frames thru pipeline), can this lead to a case when a couple of frames are already in pipeline (already encoded) and bitrate change won't affect them? Can this lead to network congestion and packet drops?

Is it possible to derive a single class from VideoTrackSourceInterface and VideoEncoder so such "behind the scene" communication is not required? Would this lead to a shorter (better) pipeline that starts from source which is also an encoder? I guess it would be similar to "encoder with internal source". Any info why it was deprecated?

Thanks,

Andriy

[Niels Moller]

On Wed, Mar 21, 2018 at 11:50 AM, Andriy Tylychko
<andriy....@gmail.com> wrote:
> Thanks Niels.
>
> I'm still quite confused.

That's understandable.

> WebRTC notifies encoder when it needs key-frame
> produced or bitrate changed. In case of fake encoder with custom
> VideoTrackSourceInterface implementation that produces encoded frames, the
> encoder has no control over encoding process.
>
> Should fake encoder communicate with VideoTrackSourceInterface behind the
> scene to pass this info (frame type/bitrate/etc.)?

I think so. Since you are in control of both objects, I hope that
should work fine.

Another more complicated option might be to have the source produce
encoded frames with multiple spatial and/or temporal layers, so that
some of the data can be discarded later in the pipeline to match
desired bit rate.

> VideoTrackSourceInterface and Encoder normally communicate asynchronously
> (passing frames thru pipeline), can this lead to a case when a couple of
> frames are already in pipeline (already encoded) and bitrate change won't
> affect them? Can this lead to network congestion and packet drops?

I don't think there's any expectation that changes to bitrate
allocation take effect instantly, and there's also buffering *after*
the encoder. But you should aim to minimize the number of pending
frames.

> Is it possible to derive a single class from VideoTrackSourceInterface and
> VideoEncoder so such "behind the scene" communication is not required?

I'm not aware of anything ruling out such an implementation, but there
might be some obstacles.

> I guess it would be similar to "encoder with internal
> source". Any info why it was deprecated?

I'm not familiar with the usecases for which it was originally
introduced. It's deprecated because supporting it requires a few ugly
hacks spread out over the pipeline.

[Andriy Tylychko]

Thanks a lot!

I tried to implement a custom `VideoTrackSourceInterface` but seems it's not enough, as I didn't have any notification when it should start capturing and a couple of other incompletenesses. So I implemented a custom video capturer and created is as

rtc::scoped_refptr<webrtc::VideoTrackInterface> video_track(

peer_connection_factory_->CreateVideoTrack(

kVideoLabel,

peer_connection_factory_->CreateVideoSource(std::move(custom_capturer_))));

Please let me know if I'm doing something wrong or if there's a simpler/better solution.

It kind of works but I have an issue with dropped frames. 

First of all, when capturer is started encoder is not ready yet, so capturer already sent several frames (including SPS/PPS and key-frame) that are dropped all except that last one by `VideoStreamEncoder::EncodeTask::Run()`. While this state can be recovered by requesting a new key-frame with SPS/PPS, but still is unfortunate and I'd rather avoid this. I kind of worked this around but it's an ugly hack. Please let me know if there's a clear indication that the pipeline is ready so capturing can be started w/o dropping frames.

Much more subtle issue I'm faced with is dropping frames in the middle of streaming by the same `VideoStreamEncoder::EncodeTask::Run()`. The capturer produces pre-encoded frames with 30FPS. The encoder is a fake one that just do packing and so should be extremely fast. That's why I don't understand while it happens that the capturer manages to produce another frame when the prev one haven't been consumed yet by the encoder.

Is there any way to disable frame dropping? if there're multiple places where frames can be dropped and no single solution for all of them, is there a way to disable frame dropping by `VideoStreamEncoder::EncodeTask::Run()`? Reading the code I see nothing like this, `VideoStreamEncoder::EncodeTask::Run()` drops frames when more than one frame is queued for encoding. Also I don't see a possibility to provide a customize `VideoStreamEncoder` implementation (at least a simple one).

[Niels Moller]

On Fri, Mar 23, 2018 at 1:34 PM, Andriy Tylychko
<andriy....@gmail.com> wrote:
> I tried to implement a custom `VideoTrackSourceInterface` but seems it's not
> enough, as I didn't have any notification when it should start capturing and
> a couple of other incompletenesses.

You could start capturing when you get the first AddOrUpdateSink
(before that, you have nowhere to send captured frames).

> rtc::scoped_refptr<webrtc::VideoTrackInterface> video_track(
> peer_connection_factory_->CreateVideoTrack(
> kVideoLabel,
> peer_connection_factory_->CreateVideoSource(std::move(custom_capturer_))));
>
> Please let me know if I'm doing something wrong or if there's a
> simpler/better solution.

As I said, cricket::VideoCapturer is deprecated. The above
CreateVideoSource method will be deleted at some point.

> First of all, when capturer is started encoder is not ready yet, so capturer
> already sent several frames (including SPS/PPS and key-frame) that are
> dropped all except that last one by `VideoStreamEncoder::EncodeTask::Run()`.
> While this state can be recovered by requesting a new key-frame with
> SPS/PPS, but still is unfortunate and I'd rather avoid this. I kind of
> worked this around but it's an ugly hack. Please let me know if there's a
> clear indication that the pipeline is ready so capturing can be started w/o
> dropping frames.

As far as I'm aware, the pipeline should be ready by the time the
video source gets the first AddOrUpdateSink.

> Much more subtle issue I'm faced with is dropping frames in the middle of
> streaming by the same `VideoStreamEncoder::EncodeTask::Run()`. The capturer
> produces pre-encoded frames with 30FPS. The encoder is a fake one that just
> do packing and so should be extremely fast. That's why I don't understand
> while it happens that the capturer manages to produce another frame when the
> prev one haven't been consumed yet by the encoder.

I guess you could signal "encoder ready" to the source behind the
scenes, but that's not very pretty.

> Is there any way to disable frame dropping? if there're multiple places
> where frames can be dropped and no single solution for all of them, is there
> a way to disable frame dropping by `VideoStreamEncoder::EncodeTask::Run()`?

It might make sense to make the limit on number of queued frames configurable.

And there might be other places on the pipeline that can drop frames.

> Also I don't see a possibility to provide a
> customize `VideoStreamEncoder` implementation (at least a simple one).

No, it's not easily replaceable.

Your usecase is a bit unusual, but I'd expect there are quite a few
applications that want to do this. Please file bugs for any serious
obstacles you encounter.