scaling the video stream with webrtc dotnet

[Amrit Kohli]

hello everyone,

 

i am trying very hard to understand how to scale the video stream with webrtc dotnet.

 

we are trying to scale it natively, from the WebRTCNative .csproj.  i’ve tried a number of different things, including trying to use ScaleFrom and CropAndScaleFrom, and the most promising code I found was this:

 

  rtc::scoped_refptr<I420Buffer> yuv_buffer =

      I420Buffer::Create(target_width, target_height);

 

  yuv_buffer->ScaleFrom(buffer);

 

  rtc::scoped_refptr<I420Buffer> axx_buffer =

      I420Buffer::Create(target_width, target_height);

 

  libyuv::ScalePlane(buffer.DataA(), buffer.StrideA(), buffer.width(),

                     buffer.height(), axx_buffer->MutableDataY(),

                     axx_buffer->StrideY(), target_width, target_height,

                     libyuv::kFilterBox);

 

  rtc::scoped_refptr<I420ABufferInterface> merged_buffer = WrapI420ABuffer(

      yuv_buffer->width(), yuv_buffer->height(), yuv_buffer->DataY(),

      yuv_buffer->StrideY(), yuv_buffer->DataU(), yuv_buffer->StrideU(),

      yuv_buffer->DataV(), yuv_buffer->StrideV(), axx_buffer->DataY(),

      axx_buffer->StrideY(), rtc::Bind(&KeepBufferRefs, yuv_buffer, axx_buffer));

 

  return merged_buffer;

 

 

this comes directly from the method      

 

  rtc::scoped_refptr<I420ABufferInterface> ScaleI420ABuffer(

    const I420ABufferInterface& buffer,

    int target_width,

    int target_height)

 

from the libyuv library:

 

chromium / external / webrtc / lkgr / . / common_video / libyuv / webrtc_libyuv.cc

 

so, in the code above, the following classes and interfaces and methods are not defined (see attached image).

 

...and given buffer as the source buffer holding the image we want to scale.

 

I420Buffer, I420ABufferInterface, and WrapI420ABuffer, and rtc::Bind

 

but at this point, I’m not even sure if I’m using the right method.  i’m also not even sure if i’m putting the calculation to scale in the proper place.

 

i’ve also seen folks trying to scale the stream from one of the Peers.  but wouldn’t this raise the problem of each peer having to check to see if the image has been scaled already, considering the Peers would be running the same client code?

 

and besides, the kurento-utils package hides the getUserMedia call, which makes it difficult to take advantage of the utilities and simultaneously work with the WebRTC api.

 

i have the following includes in conductor.cc which is where the above code snippet comes from:

 

#include "defaults.h"

#include "conductor.h"

 

#include "webrtc/base/checks.h"

#include "webrtc/api/test/fakeconstraints.h"

#include "webrtc/api/video_codecs/video_encoder.h"

#include "webrtc/modules/video_coding/codecs/vp8/simulcast_encoder_adapter.h"

#include "webrtc/modules/video_coding/codecs/vp8/include/vp8.h"

#include "webrtc/modules/video_capture/video_capture_factory.h"

#include "webrtc/media/engine/webrtcvideocapturerfactory.h"

#include "webrtc/common_video/libyuv/include/webrtc_libyuv.h"

#include "libyuv/scale.h"    // NOLINT

#include "libyuv/convert.h"  // NOLINT

#include "libyuv/planar_functions.h"

#include "libyuv/scale_argb.h"

#include "libyuv/basic_types.h"

#include "libyuv/scale_row.h"

#include "webrtc/api/video/i420_buffer.h"

#include "webrtc/common_video/include/video_frame_buffer.h"

#include "webrtc/base/bind.h"

#include "webrtc/base/checks.h"

#include "webrtc/base/keep_ref_until_done.h"

#include "webrtc/base/random.h"

#include "webrtc/system_wrappers/include/clock.h"

 

// for servers

#include "webrtc/p2p/base/relayserver.h"

#include "webrtc/p2p/base/stunserver.h"

#include "webrtc/p2p/base/basicpacketsocketfactory.h"

#include "webrtc/p2p/base/turnserver.h"

#include "webrtc/base/asyncudpsocket.h"

#include "webrtc/base/optionsfile.h"

#include "webrtc/base/stringencode.h"

#include "webrtc/base/thread.h"

 

//#define L_LITTLE_ENDIAN

//#include "leptonica/allheaders.h"

#include "turbojpeg/turbojpeg.h"

#include "atlsafe.h"

 

what am i doing wrong?  i know i’m missing something fundamental about the includes, or i’m trying to go after a method that’s deprecated, or I’m simply using the wrong strategy here.  the above code is placed right at the Conductor::PushFrame(uint8_t * img, int pxFormat) method.  i’m literally trying to scale the image inside the PushFrame method here before the frame gets pushed.  why wouldn’t this work?

 

is there anything anyone can do to help me with this problem?  please?

 

thanks,

 

-amrit