WebRtc drops frames sometimes, looks like a bug very much

[Aleksey Telyshev]

Hello,

I noticed interesting behavior on one computer. I thought it is by design But after looking at code I believe it is bug. Computer should be quite new and fast. Here the problem:

libjingle/trunk/webrtc/modules/video_capture/video_capture_impl.cc

int32_t VideoCaptureImpl::DeliverCapturedFrame(I420VideoFrame& captureFrame,

                                               int64_t capture_time) {

  UpdateFrameCount();  // frame count used for local frame rate callback.

  const bool callOnCaptureDelayChanged = _setCaptureDelay != _captureDelay;

  // Capture delay changed

  if (_setCaptureDelay != _captureDelay) {

      _setCaptureDelay = _captureDelay;

  }

  // Set the capture time

  if (capture_time != 0) {

    captureFrame.set_render_time_ms(capture_time - delta_ntp_internal_ms_);

  } else {

    captureFrame.set_render_time_ms(TickTime::MillisecondTimestamp());

  }

  if (captureFrame.render_time_ms() == last_capture_time_) {

    // We don't allow the same capture time for two frames, drop this one.

    return -1;

  }

  last_capture_time_ = captureFrame.render_time_ms();

  if (_dataCallBack) {

    if (callOnCaptureDelayChanged) {

      _dataCallBack->OnCaptureDelayChanged(_id, _captureDelay);

    }

    _dataCallBack->OnIncomingCapturedFrame(_id, captureFrame);

  }

  return 0;

}

Generally speaking nothing is wrong with the code but this piece:

  if (captureFrame.render_time_ms() == last_capture_time_) {

    // We don't allow the same capture time for two frames, drop this one.

    return -1;

  }

that drops the frame get executed quite often on fast computers. Once I looked at stack I found that this method executed 2 times. 

And second call happens quite soon, sometime less then 1 microsecond interval. That time both time stamps are the same due this code

  last_capture_time_ = captureFrame.render_time_ms();

And on second execution somebody expected that this call 

captureFrame.set_render_time_ms(TickTime::MillisecondTimestamp());

will get something different.

But WebRTC uses time stamp precision in milliseconds 

(which is bad idea, for instance Media Foundation uses 100 nanosecond interval). And new timestamp still the same (again it could be different if timestamp was in microseconds). 

So it cause to drop quite legal and useful frame. 

It is very tricky problem and happens on computer which able to run that piece of code fast. I have only one such machine, and it drops frames quite intensively. 

Actually it can make video quite shaky on fast computer (where you would expect better result) 

Regards

Aleksey

[SProgrammer]

I think this is your input - source issue. When you capture camera sometimes the CAPS do not match properly as a result it start dropping frames from input source to the capture in most of that case, its a HARDWARE issue not the SOFTWARE (WebRTC).

I advise with same system you use random web-cam specially Logitech C920 for linux seems to be getting better,  try HP camera they are very stable and respecting our frameworks, or else try the new camera from Cisco USB.

I used them and have never found drop frame issue. But i also tested with other brands like Trust and many other camera's they produce drop frame problem as default. So i would say its very much hardware issue. Also this problem happens with Sony, LifeSize, Tandberg systems when Hardware camera CAPS are not matched with software CAPS

[Aleksey Telyshev]

Hello SProgrammer 

Not sure that I understand relation. Camera made image, passed to media pipeline. Done.

What WebRTC doing with tat frame belongs to WebRTC. If they call DeliverCapturedFrame recursively in 1 millisecond and check for that timestamp that WebRTC problem, if timestamp would be in nano 

the problem wouldn't be occurred, right ? 

Regards

Aleksey

[SProgrammer]

Not sure that I understand relation. Camera made image, passed to media pipeline. Done.

> WRONG - i did this test with Sony hardwares and random cameras like CCTV, Pen tilt zoom, Doom camera, Bosh camera, Axis camera lot of cameras, what i have seen is that Sony rejects random camera because the CAPS not matching with there software CAPS logic, as a result in the debug of Sony capture, i can see they issued DROP frame. But manufacturer of camera ignores it (here WebRTC should be considered as Sony and manufacturer of camera should study how WebRTC CAPS works and add thsoe patch in my mind)

What WebRTC doing with tat frame belongs to WebRTC. If they call DeliverCapturedFrame recursively in 1 millisecond and check for that timestamp that WebRTC problem, if timestamp would be in nano 

the problem wouldn't be occurred, right ? 

> Sony doing same i believe, Polycom doing same i believe, LifeSize doing same i believe. WebRTC if does why its wrong?

I hope it helps to understand, i had same issues with random equipments.

[BobK]

The browsers have limited buffers for arriving packets.  The number of packets and rate at which they are transmitted are likely affected by the camera being used.  If sent too fast the packets will be missed at the browser.

Bob Kirnum,

Dialogic

[Aleksey Telyshev]

This is correct and for this purposes they have special object called Dropper.

Actually if you check WebRTC code, you will see that they have several places where video frame could be dropped.

But what I pointed is absolutely different case... it is just bug, If call gets back less then one millisecond they will drop it, hmmm, 

Aleksey Telyshev 

Vos Video 

[André Susano Pinto]

Aleksey note that capture time plays a role on video rendering as well.

Requesting for sub-ms precision would only make sense if you are sending above 1000fps or your refresh rate is above 1000hz. (actually if capture jitter is 10ms.. there would be problems at 100fps, but bear with me to the next paragraph).

If you have two frames being delivered to fast and that makes webrtc timestamp them with same millisecond most likely you are suffering from jitter on the path from your camera sensor till video_capture_impl.cc, which is why that class allows the capturer to provide a timestamp.

In any case I doubt support for timestamp difference below 1ms is really what you are looking for.

--
 
---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.