audio/video stream Video Encoder VP9 and H264

[giovanni...@gmail.com]

Hi guys

I am developing a pepper api to stream audio/video upon network.

till now, I managed to integrate the libfdk_aac, profile AAC_LD, for the audio encoder and the X264 for the video encoder. Regarding the video encoder  I have tried to encode frames of 640x480 pixels. The encoder takes an average of 300 msec to encode a frame, normally the same code outside the "pepper API" takes an average of 2 msec. Since this is unacceptable for my purposes, I thought the problem was due to the fact that inside the "pepper API", the X264 code, can not use the assembly. Then I decide to switch to VP9 encoder and using the internal implementation of the  encoder, in this case it takes an average of 50 msec, and this is still to slow.

For the moment the work around was to throw away two frames on three and encode just this one. In this way, in my trace I have one frame and one encoded frame. At the end of the post, I put the trace before and after the work around.

Since is unbelievable that the internal implementation of the browser, takes so much time to encode a frame, I am wondering what am i doing wrong.

any suggestion?

Giovanni

Bye

Before the work around

encode-time=0.0251698;result=0;main-thread=0

track-get-frame-time=0.03757;result=0;main-thread=1

track-get-frame-time=0.00806999;result=0;main-thread=1

encode-time=0.0648999;result=0;main-thread=0

track-get-frame-time=0.03795;result=0;main-thread=1

track-get-frame-time=0.04443;result=0;main-thread=1

track-get-frame-time=0.0124302;result=0;main-thread=1

track-get-frame-time=0.0465901;result=0;main-thread=1

encode-time=0.11976;result=0;main-thread=0

track-get-frame-time=0.0398898;result=0;main-thread=1

encode-time=0.03618;result=0;main-thread=0

track-get-frame-time=0.06565;result=0;main-thread=1

track-get-frame-time=5.00679e-05;result=0;main-thread=1

encode-time=0.0392499;result=0;main-thread=0 

After  the work around

track-get-frame-time=0.14198;result=0;main-thread=1

encode-time=0.0337899;result=0;main-thread=0

track-get-frame-time=0.1703;result=0;main-thread=1

encode-time=0.0372999;result=0;main-thread=0

track-get-frame-time=0.0873299;result=0;main-thread=1

encode-time=0.0405202;result=0;main-thread=0

track-get-frame-time=0.14212;result=0;main-thread=1

encode-time=0.0253599;result=0;main-thread=0

track-get-frame-time=0.12925;result=0;main-thread=1

encode-time=0.0218101;result=0;main-thread=0

track-get-frame-time=0.13498;result=0;main-thread=1

encode-time=0.0368099;result=0;main-thread=0

track-get-frame-time=0.13621;result=0;main-thread=1

encode-time=0.0271699;result=0;main-thread=0

track-get-frame-time=0.12736;result=0;main-thread=1

[Bill Budge]

From your description, it sounds like you're using the Pepper MediaStream APIs to grab audio and video frames, then encoding and sending the frames using a NaCl plugin, and finally sending the encoded stream using Pepper Network APIs.

If this is correct, I wouldn't expect the encoding to take that much time. Without seeing your code, it's hard to say what might be causing a slowdown. My first question would be whether you are measuring throughput or latency. MediaStream and Network APIs often introduce latency (delays from request to completion) but the throughput might be high (actual amount of data being processed.)

[giovanni...@gmail.com]

Hi

here is the whole chain.

Our code grabs the audio and video by using the  MediaStreamVideoTrack and MediaStreamAudioTrack classes.

To encode the Audio we integrated the Fraunhofer source code (libfdk_aac, profile AAC_LD) and to encode the video we tried both  the encoder X264 and

Vp9. To send the streaming upon the network we use the live555 source code.

all the sources that I mention, have been compiled by using pepper_47 and the TOOLCHAIN pnacl.

When we stream only the audio the latency is below of 100 msec, when we stream audio and video the latency grow up to 2 sec.

We put some traces to undestand why the latency in so high, and we realized that to encode a video frame by using the x264  it takes an average of 300 msec.

As said in my previous mail, we have switched to the VP9 encoder implemented inside the browser and in this case to encode a frame (640x480 pixel) it takes an average of 80-90 msec. This is to much to substain 20 fps. By the way, why I cannot set the framerate of the video source? 

Anyway, to remove from the field, any possible trouble about the stuff that you mentioned i attached the video_encoded.cc, where we add a thread for the video encoder and some new traces to the console related to the encode time, the OnGetBitstreamBuffer  time, the track frame.

If you compile and run "make TOOLCHAIN=pnacl CONFIG=Debug run" in the console you could see the performance of the VP9 encoder.

here is the whole chain

Our code grabs the audio and video by using the  MediaStreamVideoTrack and MediaStreamAudioTrack classes.

To encode the Audio we integrated the fraunhofer source code (libfdk_aac, profile AAC_LD) and to encode the video we tried both  the encoder X264 and

Vp9. To send the streaming upon the network we use the live555 source code.

all the sources that I mention, have been compiled by using pepper_47 and the TOOLCHAIN pnacl.

When we stream only the audio the latency is below of 100 msec, when we stream audio and video the latency grow up to 2 sec.

We put some traces to undestand why the latency in so high, and we realized that to encode a video frame by using the x264  it takes an average of 300 msec.

As said in my previous mail, we have switched to the VP9 encoder implemented inside the browser and in this case to encode a frame (640x480 pixel) it takes an average of 100 msec.

Anyway, to remove from the field, any possible trouble about the stuff that you mentioned i attached the video_encoded.cc, where we add a thread for the video encoder and some new traces to the console related to the encode time, the OnGetBitstreamBuffer  time, the track frame.

If you compile and run "make TOOLCHAIN=pnacl CONFIG=Debug run" in the console you could see the performance of the VP9 encoder.

Giovanni

Bye

[Bill Budge]

One thing that stands out immediately is how you throttle your calls to the encoder with 'is_encode_ticking_'.

You would get higher throughput and lower latency if you call Encode repeatedly so as to have as many VideoFrames in flight as possible. The way you do this is to call Encode when you receive the callback for a prior call to Encode. The example should demonstrate this technique.

On Thursday, October 20, 2016 at 4:51:42 AM UTC-7, giovanni...@gmail.com wrote:

[giovanni...@gmail.com]

I am not understanding  what you are saying, could you please clarify the concept? In the meantime I tried to follow your suggestion and  starting from the example video_encode.cc (pepper_47 examples) I have added some traces to log the encode time into the console, if you compile and run it (make TOOLCHAIN=pnacl CONFIG=Debug run), you will see that the encode time is too long about 30 msec, and the cpu (core 2 Duo) goes to 94 %.

I have attached the original video_encode.cc in which i added the encode time log and some logs.

Do you have any suggestion? why the encode is so long and the cpu consuming is so expensive?

Bye