Issues sending frames from native WebRTC code.

[mike ads]

I wrote some native webRTC code and as per some sample code I've found online, I'm using openCV to capture some frames from a stream and send them over webRTC.

This is the sample code that I'm using: http://pragmaticjoe.blogspot.com/2015/09/sending-custom-video-stream-through.html

On my native code side, i create my custom video capturer, create a separate thread to grab frames from OpenCV, convert those frames to an appropriate format, and Invoke them on the worker thread. This is the code I'm using:

cricket::CaptureState CustomVideoCapturer::Start(const cricket::VideoFormat& capture_format){

if (capture_state() == cricket::CS_RUNNING) return cricket::CS_RUNNING;

//Loop trying to open stream.

while (!cvCapture.isOpened()){

printf("Capturer is not open -> will try to reopen");

cvCapture.open(STREAM);

}

//Start frame-grabbing thread for cvCapture. Pass it "this" Video Capture Instance since 

//it's a static method.

grabberThread = CreateThread(NULL, NULL, (LPTHREAD_START_ROUTINE)&grabCapture, this, NULL, NULL);

SetCaptureFormat(&capture_format);

    return cricket::CS_RUNNING;

}

void* CustomVideoCapturer::grabCapture(CustomVideoCapturer* arg)

{

CustomVideoCapturer* customVideoCapturer = (CustomVideoCapturer*)(arg);

cv::Mat frame;

while (cvCapture.read(frame) && customVideoCapturer->IsRunning()){

cv::Mat bgra(frame.rows, frame.cols, CV_8UC4);

cv::cvtColor(frame, bgra, CV_BGR2BGRA); //opencv reads the stream in BGR format by default

webrtc::VideoFrame vframe;

vframe.CreateEmptyFrame(bgra.cols, bgra.rows, bgra.cols, (bgra.cols + 1) / 2, (bgra.cols + 1) / 2); //convert the frame to I420, which is the supported format for webrtc transport

webrtc::ConvertToI420(webrtc::kBGRA, bgra.ptr(), 0, 0, bgra.cols, bgra.rows, 0, webrtc::kVideoRotation_0, &vframe);

vector<uint8_t> capture_buffer_;

size_t length = webrtc::CalcBufferSize(webrtc::kI420, vframe.width(), vframe.height());

capture_buffer_.resize(length);

webrtc::ExtractBuffer(vframe, length, &capture_buffer_[0]);

cricket::WebRtcCapturedFrame* webrtc_frame = new cricket::WebRtcCapturedFrame(vframe, &capture_buffer_[0], length);

//Signal frame capture on worker thread

customVideoCapturer->worker->Invoke<void>(rtc::Bind(&CustomVideoCapturer::SignalFrameCapturedOnStartThread, customVideoCapturer, webrtc_frame));

}

return 0;

}

void CustomVideoCapturer::SignalFrameCapturedOnStartThread(const cricket::CapturedFrame *frame){

SignalFrameCaptured(this, frame);

}

Unfortunately, no video actually shows up in my browser. I'm seeing the following errors.

1) As mentioned in the guide, for each sent frame i get the following error in WebRTC: webrtc: (video_capture_input.cc:108): Same/old NTP timestamp for incoming frame. Dropping.
 It seems like each frame is being dropped due to some timestamp issues. The guide recommended modifying video_capture_input.cc to set the timestamp in there, I did so but the error did not go away, so together with setting the timestamp on each frame, i just commented out the code that throws that error.

2) After that, in chrome logs via Sawbuck, i see errors for each frame "Received packet with a sequence number which is out of frameboundaries". or "301 consecutive old packets received. Flushing the jitter buffer".

Passing frames on the worker thread via SignalFrameCaptured seems to be the standard correct way of doing this, so I'm really not sure what I'm doing wrong here.

Any suggestions would be very appreciated. Thank you!

[mike ads]

I was able to get it working. I was not setting the time stamp properly (I was using GetLastTickCount(), which gives you milliseconds instead of nano seconds. I set the timestamp properly on each frame and everything started working properly.

 frame->time_stamp = chrono::duration_cast<chrono::nanoseconds>(chrono::system_clock::now().time_since_epoch()).count();