scaling the video stream with webrtc dotnet

[Amrit Kohli]

hello everyone,

 

i am trying very hard to understand how to scale the video stream with webrtc dotnet.

 

we are trying to scale it natively, from the WebRTCNative .csproj.  i’ve tried a number of different things, including trying to use ScaleFrom and CropAndScaleFrom, and the most promising code I found was this:

 

  rtc::scoped_refptr<I420Buffer> yuv_buffer =

      I420Buffer::Create(target_width, target_height);

 

  yuv_buffer->ScaleFrom(buffer);

 

  rtc::scoped_refptr<I420Buffer> axx_buffer =

      I420Buffer::Create(target_width, target_height);

 

  libyuv::ScalePlane(buffer.DataA(), buffer.StrideA(), buffer.width(),

                     buffer.height(), axx_buffer->MutableDataY(),

                     axx_buffer->StrideY(), target_width, target_height,

                     libyuv::kFilterBox);

 

  rtc::scoped_refptr<I420ABufferInterface> merged_buffer = WrapI420ABuffer(

      yuv_buffer->width(), yuv_buffer->height(), yuv_buffer->DataY(),

      yuv_buffer->StrideY(), yuv_buffer->DataU(), yuv_buffer->StrideU(),

      yuv_buffer->DataV(), yuv_buffer->StrideV(), axx_buffer->DataY(),

      axx_buffer->StrideY(), rtc::Bind(&KeepBufferRefs, yuv_buffer, axx_buffer));

 

  return merged_buffer;

 

 

this comes directly from the method      

 

  rtc::scoped_refptr<I420ABufferInterface> ScaleI420ABuffer(

    const I420ABufferInterface& buffer,

    int target_width,

    int target_height)

 

from the libyuv library:

 

chromium / external / webrtc / lkgr / . / common_video / libyuv / webrtc_libyuv.cc

 

so, in the code above, the following classes and interfaces and methods are not defined:

 

 

I420Buffer, I420ABufferInterface, and WrapI420ABuffer.

 

but at this point, I’m not even sure if I’m using the right method.  i’m also not even sure if i’m putting the calculation to scale in the proper place.

 

i’ve also seen folks trying to scale the stream from one of the Peers.  but wouldn’t this raise the problem of each peer having to check to see if the image has been scaled already, considering the Peers would be running the same client code?

 

and besides, the kurento-utils package hides the getUserMedia call, which makes it difficult to take advantage of the utilities and simultaneously work with the WebRTC api.

 

i have the following includes in conductor.cc which is where the above code snippet comes from:

 

#include "defaults.h"

#include "conductor.h"

 

#include "webrtc/base/checks.h"

#include "webrtc/api/test/fakeconstraints.h"

#include "webrtc/api/video_codecs/video_encoder.h"

#include "webrtc/modules/video_coding/codecs/vp8/simulcast_encoder_adapter.h"

#include "webrtc/modules/video_coding/codecs/vp8/include/vp8.h"

#include "webrtc/modules/video_capture/video_capture_factory.h"

#include "webrtc/media/engine/webrtcvideocapturerfactory.h"

#include "webrtc/common_video/libyuv/include/webrtc_libyuv.h"

#include "libyuv/scale.h"    // NOLINT

#include "libyuv/convert.h"  // NOLINT

#include "libyuv/planar_functions.h"

#include "libyuv/scale_argb.h"

#include "libyuv/basic_types.h"

#include "libyuv/scale_row.h"

#include "webrtc/api/video/i420_buffer.h"

#include "webrtc/common_video/include/video_frame_buffer.h"

#include "webrtc/base/bind.h"

#include "webrtc/base/checks.h"

#include "webrtc/base/keep_ref_until_done.h"

#include "webrtc/base/random.h"

#include "webrtc/system_wrappers/include/clock.h"

 

// for servers

#include "webrtc/p2p/base/relayserver.h"

#include "webrtc/p2p/base/stunserver.h"

#include "webrtc/p2p/base/basicpacketsocketfactory.h"

#include "webrtc/p2p/base/turnserver.h"

#include "webrtc/base/asyncudpsocket.h"

#include "webrtc/base/optionsfile.h"

#include "webrtc/base/stringencode.h"

#include "webrtc/base/thread.h"

 

//#define L_LITTLE_ENDIAN

//#include "leptonica/allheaders.h"

#include "turbojpeg/turbojpeg.h"

#include "atlsafe.h"

 

what am i doing wrong?  i know i’m missing something fundamental about the includes, or i’m trying to go after a method that’s deprecated, or I’m simply using the wrong strategy here.  the above code is placed right at the Conductor::PushFrame(uint8_t * img, int pxFormat) method.  i’m literally trying to scale the image inside the PushFrame method here before the frame gets pushed.  why wouldn’t this work?

 

is there anything anyone can do to help me with this problem?  please?

 

thanks,

 

-amrit

[Niels Moller]

On Sun, Jun 17, 2018 at 2:33 AM, Amrit Kohli <am...@screenmeet.com> wrote:

hello everyone,

 

i am trying very hard to understand how to scale the video stream with webrtc dotnet.

I'm afraid I don't know anything about "webrtc dotnet", so I'm assuming you're somehow using the native C++ api with some to me unknown dotnet glue on top.

  rtc::scoped_refptr<I420Buffer> yuv_buffer =

      I420Buffer::Create(target_width, target_height);

 

  yuv_buffer->ScaleFrom(buffer);

The above is the easiest way to do it (there are also variants doing cropping before scaling). The methods are declared in api/video/i420_buffer.h.

Regards,

/Niels

[Amrit Kohli]

hello Niels,

first of all, thank you for your response.

“webrtc dotnet” is otherwise known as WebRTC.NET, which is a Microsoft .NET Framework wrapper around the native C++ api. the wrapper simply exports the public methods of the api so they are available to .NET-based projects/assemblies/executables.

the wrapper i am using comes from here:

https://github.com/radioman/WebRtc.NET

so, the api was built to compile and build on Microsoft Visual Studio. the “WebRtcNative” project in that repo is what i believe to be the C++ api.

here’s the most recent iteration of what i am trying:

        void Conductor::PushFrame(uint8_t * img, int pxFormat)

{

if (capturer)

{

                        // output to the Debug Console

rtc::LogMessage::LogToDebug(rtc::LoggingSeverity::INFO);

LOG(INFO) << "Scaling buffer now...AMRIT";

rtc::scoped_refptr<webrtc::I420Buffer> yuv_buffer =

webrtc::I420Buffer::Create(1024, 768);

width_ = 1024;

height_ = 768;

auto videoBuffer = capturer->video_buffer;// .get()->NativeToI420Buffer().get();

//yuv_buffer->ScaleFrom(*capturer->video_frame->video_frame_buffer);

yuv_buffer.get()->ScaleFrom(*videoBuffer);

auto yuv = (uint8_t*) yuv_buffer->DataY();

//auto yuv = (uint8_t*) capturer->video_buffer->DataY();

if (yuv != nullptr)

{

if (img != nullptr)

{

const int pad = 4;

int pitch = TJPAD(tjPixelSize[pxFormat] * width_);

if (jpegc == nullptr)

jpegc = tjInitCompress();

int r = 0;

if (jpegc)

r = tjEncodeYUV3(jpegc, img, width_, pitch, height_, pxFormat, yuv, pad, TJSAMP_420, true ? TJFLAG_FASTDCT : TJFLAG_ACCURATEDCT);

LOG(INFO) << "::::::::::::::::::::::::::::Pushing Scaled Frame:::::::::::::::::::::::::::::::::::::::::::";

if (r == 0)

{

capturer->PushFrame(false);

}

else

{

LOG(LS_ERROR) << tjGetErrorStr();

}

}

else

{

capturer->PushFrame(true);

}

}

}

}

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAAO0x17dqkxhwiQLyPrKtGVr6jMUAm3PEwtWqTS-BYrzZH8NUg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Amrit Kohli]

hello Niels,

first of all, thank you for your response.

secondly, i accidentally sent my message before i had finished writing it.

“webrtc dotnet” is otherwise known as WebRTC.NET, which is a Microsoft .NET Framework wrapper around the native C++ api. the wrapper simply exports the public methods of the api so they are available to .NET-based projects/assemblies/executables.

the wrapper i am using comes from here:

https://github.com/radioman/WebRtc.NET

so, the api was built to compile and build on Microsoft Visual Studio. the “WebRtcNative” project in that repo is what i believe to be the C++ api.

here’s the most recent iteration of what i am trying:

        void Conductor::PushFrame(uint8_t * img, int pxFormat)

{

if (capturer)

{

                        // output to the Debug Console

rtc::LogMessage::LogToDebug(rtc::LoggingSeverity::INFO);

LOG(INFO) << "Scaling buffer now...AMRIT";

                        // create the target buffer:

rtc::scoped_refptr<webrtc::I420Buffer> yuv_buffer =

webrtc::I420Buffer::Create(1024, 768);

                        // set instance variables (used by the Conductor to track the size of the video buffer for the frame that is being pushed)

width_ = 1024;

height_ = 768;

                        // convert the video_buffer (which is going to receive the `img` parameter) to a webrtc::VideoFrameBuffer

auto videoBuffer = capturer->video_buffer.get()->NativeToI420Buffer().get();

// commented out intentionally

//yuv_buffer->ScaleFrom(*capturer->video_frame->video_frame_buffer);

                        // scale the capturer's video buffer to the target yuv_buffer created above.

yuv_buffer.get()->ScaleFrom(*videoBuffer);

                        // get the pointer to the Y data segment in the resized buffer

auto yuv = (uint8_t*) yuv_buffer->DataY();

                        // commented out intentionally

//auto yuv = (uint8_t*) capturer->video_buffer->DataY();

                        // push the frame in the scaled buffer

if (yuv != nullptr)

{

if (img != nullptr)

{

const int pad = 4;

int pitch = TJPAD(tjPixelSize[pxFormat] * width_);

if (jpegc == nullptr)

jpegc = tjInitCompress();

int r = 0;

if (jpegc)

r = tjEncodeYUV3(jpegc, img, width_, pitch, height_, pxFormat, yuv, pad, TJSAMP_420, true ? TJFLAG_FASTDCT : TJFLAG_ACCURATEDCT);

LOG(INFO) << "::::::::::::::::::::::::::::Pushing Scaled Frame:::::::::::::::::::::::::::::::::::::::::::";

if (r == 0)

{

capturer->PushFrame(false);

}

else

{

LOG(LS_ERROR) << tjGetErrorStr();

}

}

else

{

capturer->PushFrame(true);

}

}

}

}

when i do this, i am getting a green image at the client.  what i noticed is that the physical image at the original desktop scale is in the `img` pointer.  i've also seen now that i'm passing the `img` pointer to the full sized image into the tjEncodeYUV3 method.  it seems that the `img` pointer should point to an image buffer that is the same size as the scaled buffer, not the original image.  but then, i'm not sure how to transform the `img` pointer to a pointer to a scaled image the same size as the scaled buffer.

based on what you see in the code above, is there any obvious or maybe not so obvious problem with what i'm trying to do.  does any of this code look familiar to you for the PushFrame method of the webrtc api?

i appreciate any insight you can provide.  

if this is not the correct place to ask this question, i'm open to suggestions as to where the appropriate place for this as of yet unsupported api by Microsoft browsers post should go.  in other words, if you're writing a native WebRTC client for Windows, where might this question be posed?

thank you for indulging my question.

sincerely,

-amrit

[Amrit Kohli]