Question on How to Use Augmented Video Frame Buffer

[Pavan Jakhu]

I’m having some trouble setting up the encoder to support Augmented Frames. I’m using the Unity Plugin as the base so this is how I’m initializing the Peer Connection Factory and the Multiplex Encoding/Decoding Factory:

1. g_peer_connection_factory = webrtc::CreatePeerConnectionFactory(
2.     g_worker_thread.get(), g_worker_thread.get(), g_signaling_thread.get(),
3.     nullptr, webrtc::CreateBuiltinAudioEncoderFactory(),
4.     webrtc::CreateBuiltinAudioDecoderFactory(),
5.     std::unique_ptr<webrtc::MultiplexEncoderFactory>(
6.         new webrtc::MultiplexEncoderFactory(
7.             absl::make_unique<webrtc::InternalEncoderFactory>(), true)),
8.     std::unique_ptr<webrtc::MultiplexDecoderFactory>(
9.         new webrtc::MultiplexDecoderFactory(
10.             absl::make_unique<webrtc::InternalDecoderFactory>(), true)),
11.     nullptr, nullptr);

As you can see I’m passing true in the second parameter to use Augmented Frames. Also I have a dedicated class to spit out Video Frames that have Augmented Video Frame Buffers: 

1. class CustomFrameSource {
   
2.  public:
3.   CustomFrameSource(int width, int height, int interval_us);
4.  
5.   webrtc::VideoRotation GetRotation() const;
6.   void SetRotation(webrtc::VideoRotation rotation);
7.  
8.   // Override configuration.
9.   webrtc::VideoFrame GetFrame(const uint8_t* dataRGBA, int strideRGBA, int width, int height,
10.                               std::unique_ptr<uint8_t[]> augmentedData, uint16_t augmentedDataSize,
11.                               webrtc::VideoRotation rotation,
12.                               int interval_us);
13.  
14.  private:
15.   const int width_;
16.   const int height_;
17.   const int interval_us_;
18.  
19.   webrtc::VideoRotation rotation_ = webrtc::kVideoRotation_0;
20.   int64_t next_timestamp_us_ = rtc::kNumMicrosecsPerMillisec;
21. };
22.  
23. CustomFrameSource::CustomFrameSource(int width, int height, int interval_us)
24.     : width_(width), height_(height), interval_us_(interval_us) {
25.   RTC_CHECK_GT(width_, 0);
26.   RTC_CHECK_GT(height_, 0);
27.   RTC_CHECK_GT(interval_us_, 0);
28. }
29.  
30. webrtc::VideoRotation CustomFrameSource::GetRotation() const {
31.   return rotation_;
32. }
33.  
34. void CustomFrameSource::SetRotation(webrtc::VideoRotation rotation) {
35.   rotation_ = rotation;
36. }
37.  
38. webrtc::VideoFrame CustomFrameSource::GetFrame(const uint8_t* dataRGBA, int strideRGBA, int width, int height,
39.                               std::unique_ptr<uint8_t[]> augmentedData, uint16_t augmentedDataSize,
40.                               webrtc::VideoRotation rotation,
41.                               int interval_us) {
42.  
43.   RTC_CHECK_GT(width, 0);
44.   RTC_CHECK_GT(height, 0);
45.   RTC_CHECK_GT(interval_us, 0);
46.  
47.   rtc::scoped_refptr<webrtc::AugmentedVideoFrameBuffer> augmentedVideoFrameBuffer =
48.     new rtc::RefCountedObject<webrtc::AugmentedVideoFrameBuffer>(
49.       RGBABuffer::Copy(width, height, dataRGBA, strideRGBA),
50.       std::move(augmentedData), augmentedDataSize);
51.  
52.   webrtc::VideoFrame::Builder builder;
53.   builder.set_video_frame_buffer(augmentedVideoFrameBuffer)
54.          .set_timestamp_us(next_timestamp_us_)
55.          .set_rotation(rotation);
56.  
57.   next_timestamp_us_ += interval_us;
58.   return builder.build();
59. }

I’m just passing those Video Frames to a Video Track Source that I’m setting as the source to a Video Track for my Peer Connection on one side of the call:

1. class CustomVideoSource : public rtc::AdaptedVideoTrackSource {
   
2. public:
3.     CustomVideoSource();
4.     ~CustomVideoSource() override;
5.  
6.     void OnFrameCaptured(const webrtc::VideoFrame& frame);
7.  
8.     virtual void AddRef() const override { return; }
9.     virtual rtc::RefCountReleaseStatus Release() const override { return rtc::RefCountReleaseStatus::kDroppedLastRef; }
10.     virtual SourceState state() const override { return source_state_; }
11.     virtual bool remote() const override { return false; }
12.     virtual bool is_screencast() const override { return false; }
13.     virtual absl::optional<bool> needs_denoising() const override { return false; }
14.  
15. protected:
16.     SourceState source_state_ = kInitializing;
17. };
18.  
19. CustomVideoSource::CustomVideoSource() : AdaptedVideoTrackSource(60) {
20.     source_state_ = kInitializing;
21. }
22.  
23. CustomVideoSource::~CustomVideoSource() {
24.     source_state_ = kEnded;
25. }
26.  
27. void CustomVideoSource::OnFrameCaptured(const webrtc::VideoFrame& frame) {
28.     source_state_ = kLive;
29.     OnFrame(frame);
30. }

This is where I realized I’m not actually sending the augmented data incorrectly. I’m using a Video Sink Interface to get the Video Frames and sending it to Unity:

1. class VideoObserver : public rtc::VideoSinkInterface<webrtc::VideoFrame> {
2.  public:
3.   VideoObserver() {}
4.   ~VideoObserver() {}
5.   void SetVideoCallback(I420FRAMEREADY_CALLBACK callback);
6.  
7.  protected:
8.   // VideoSinkInterface implementation
9.   void OnFrame(const webrtc::VideoFrame& frame) override;
10.  
11.  private:
12.   I420FRAMEREADY_CALLBACK OnI420FrameReady = nullptr;
13.   std::mutex mutex;
14. };
15.  
16. void VideoObserver::SetVideoCallback(I420FRAMEREADY_CALLBACK callback) {
17.   std::lock_guard<std::mutex> lock(mutex);
18.   OnI420FrameReady = callback;
19. }
20.  
21. void VideoObserver::OnFrame(const webrtc::VideoFrame& frame) {
22.   std::unique_lock<std::mutex> lock(mutex);
23.   if (!OnI420FrameReady)
24.     return;
25.    
26.   webrtc::VideoFrameBuffer* buffer = frame.video_frame_buffer().get();
27.  
28.   webrtc::AugmentedVideoFrameBuffer* augmented_data_buffer =
29.       static_cast<webrtc::AugmentedVideoFrameBuffer*>(
30.           frame.video_frame_buffer().get());
31.   uint8_t* augmented_data = augmented_data_buffer->GetAugmentingData();
32.   uint16_t augmented_data_size = augmented_data_buffer->GetAugmentingDataSize();
33.  
34.   if (buffer->type() != webrtc::VideoFrameBuffer::Type::kI420A) {
35.     rtc::scoped_refptr<webrtc::I420BufferInterface> i420_buffer =
36.         buffer->ToI420();
37.  
38.     OnI420FrameReady(i420_buffer->DataY(), i420_buffer->DataU(),
39.                      i420_buffer->DataV(), nullptr,
40.                      i420_buffer->StrideY(), i420_buffer->StrideU(),
41.                      i420_buffer->StrideV(), 0,
42.                      frame.width(), frame.height(),
43.                      augmented_data, augmented_data_size);
44.  
45.   } else {
46.     // The buffer has alpha channel.
47.     webrtc::I420ABufferInterface* i420a_buffer = buffer->GetI420A();
48.  
49.     OnI420FrameReady(i420a_buffer->DataY(), i420a_buffer->DataU(),
50.                      i420a_buffer->DataV(), i420a_buffer->DataA(),
51.                      i420a_buffer->StrideY(), i420a_buffer->StrideU(),
52.                      i420a_buffer->StrideV(), i420a_buffer->StrideA(),
53.                      frame.width(), frame.height(),
54.                      augmented_data, augmented_data_size);
55.   }
56. }

The value I’m getting on the other side of the call for augmented_data_size is 1920 instead of 24 (6 floats in bytes).

 

My guess is I’m not setting up the Multiplex Encoding/Deciding Factory correctly. I thought about calling CreateVideoEncoder from either the Internal Encoding Factory or Multiplex Encoding Factory to get the Video Encoder object, then calling InitEncode and pass a Video Codec object with the codec_type variable set to Multiplex. Same thing with Internal Decoding Factory or Multiplex Decoding Factory. It doesn’t seem to work since I think I might incorrectly setting the second and third parameter of InitEncode/InitDecode. 

[Pavan Jakhu]

I forgot to mention that I'm on release 70.

[Nihal Ar]

Hi, Did you fix it?