Only the first frame is captured by the DesktopCapturer on mac

[Olivier BERTHONNEAU]

Hi,

I have a C++ application on mac capturing the desktop using WebRTC's desktop capturer and sending it as video frames to a connected browser.

All the connection seems to work as no errors are reported in the logs.

The DesktopCapturer::CaptureFrame() method is called periodically from another thread and OnCaptureResult is called back accordingly with a std::unique_ptr<webrtc::DesktopFrame> as parameter.

Here is my implementation of OnCaptureResult:

void Capturer::OnCaptureResult(webrtc::DesktopCapturer::Result result,

                              std::unique_ptr<webrtc::DesktopFrame> desktopFrame)

{

     if (result != webrtc::DesktopCapturer::Result::SUCCESS)

   {

       return; // Obviously never called

   }

   int width = desktopFrame->size().width();

   int height = desktopFrame->size().height();

   rtc::scoped_refptr<webrtc::I420Buffer> res_i420_frame = webrtc::I420Buffer::Create(width, height);

   webrtc::ConvertToI420(webrtc::VideoType::kARGB,

                      desktopFrame->data(),

                      0, 0,

                      width, height,

                      0,

                      webrtc::kVideoRotation_0,

                      res_i420_frame);

       

   webrtc::VideoFrame frame = webrtc::VideoFrame(res_i420_frame, 0, 0, webrtc::kVideoRotation_0);

   this->OnFrame(frame, width, height);

}

This code basically re-encode the ARBG DesktopFrame into a I420 frame then feed it to OnFrame.

When my browser connects to my C++ peer, I can immediately see a mirror of my screen in the video tag.

However, only the first frame of the video is displayed, even though OnCaptureResult (and thus OnFrame) is called periodically.

I first suspected my video was simply not playing on the browser side. So I modified the above code to send a black frame on one every two frame.

As expected my "video" blinks showing that why video tag is indeed receiving data from my C++ app.

Is there something I'm missing ? Why is DesktopCapturer always returning the same frame even though I call DesktopCapturer::CaptureFrame() periodically ?

[Olivier BERTHONNEAU]

I investigated further more. It looks like the CGDisplayStreamFrameAvailableHandler is never called, which means no dirty regions are ever reported back to the capturer.

I can see this because the handler defined in ScreenCapturerMac::RegisterRefreshAndMoveHandlers is never called. Therefor, after the first frame is sent, ScreenCapturerHelper::TakeInvalidRegion will always return an empty invalid region, leading to no DesktopRect ever being pulled back from the screen.

Looking at how the display stream is started we can see it uses CGDisplayStreamCreate (https://developer.apple.com/documentation/coregraphics/1455170-cgdisplaystreamcreate).

There is another function for creating a display stream called CGDisplayStreamCreateWithDispatchQueue (https://developer.apple.com/documentation/coregraphics/1454968-cgdisplaystreamcreatewithdispatc).

Both seem to achieve the same thing but using a different implementation.

However, for some reason, implementing the capturer with this last function works on my macOS Sierra 10.12.5.

I am not sure of the implications of switching those functions. Nor do I understand why the current code doesn't work on my setup.

Is there anyone aware of the DesktopCapturer implementation on mac who could help me before I file a bug report ? 

[Alexandre GOUAILLARD]

hi olivier,

I can help.

Is your problem in webrtc standalone, or in chrome? (not the same bug tracker).

Alex.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e7562ace-a08c-4fc5-bff0-f50b94cfdb53%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex. Gouaillard, PhD, PhD, MBA

------------------------------------------------------------------------------------

President - CoSMo Software Consulting, Singapore

------------------------------------------------------------------------------------

sg.linkedin.com/agouaillard

• 
  

[Olivier BERTHONNEAU]

Hi Alex,

Yes my problem is in WebRTC standalone.

It is a C++ app capturing frames on a mac and transmitting to a chrome client. The chrome peer is just a viewer and do no capture frames.

Olivier

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e7562ace-a08c-4fc5-bff0-f50b94cfdb53%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alexandre GOUAILLARD]

then the bug tracker is here:

https://bugs.chromium.org/p/webrtc/issues/list

and the explanation about how to proceed for the content is here:

https://webrtc.org/bugs/

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e5f7a172-b9b1-46ea-a950-194d9d3979cf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Xander Dumaine]

Just for a sanity check: does your viewer <video/> tag have autoplay=true? If not, it would only show the first frame and no updates until calling .play() on the element. I've seen this be the cause of the same symptoms for others a few times in the past.

[Olivier BERTHONNEAU]

Hi Xander,

Yes, autoplay is enabled.

I can verify the encoding, transport and playback in the browser work if I modify the above code to insert a black frame one every two frame.

[Olivier BERTHONNEAU]

The issue has been reported here: https://bugs.chromium.org/p/webrtc/issues/detail?id=7960

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/e5f7a172-b9b1-46ea-a950-194d9d3979cf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Olivier BERTHONNEAU]

Ok so I found what happens thanks to people on the bug tracker.

Turns out the DesktopCapturer on Mac needs a Core Foundation run loop to process events on the capture thread.

So I just ran a little 

CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0, true);

In my main loop and got it sorted out.

[sajid....@cosmosoftware.io]

Hi Olivier,

I am trying to do the same but in my case, if I call CaptureFrame of the DesktopCapturer repeatedly with delay of less than 0.5 seconds, I get the following error:

#

# Fatal error in ../../modules/desktop_capture/screen_capturer_mac.mm, line 381

# last system error: 316

# Check failed: !queue_.current_frame() || !queue_.current_frame()->IsShared()

# 

#

So potentially, I am able to get 2 fps. Did you encounter any such problem?

Moreover I'd be interested to see the code using CFRunLoop.

Thanks!

[Olivier BERTHONNEAU]

HI,

I haven't work on this matter for almost a year. Here are some informations I can think of that may be able to guide you.

I never encountered the fatal error you mentioned.

Last time my test program has been tested was on WebRTC 63.

I added this line in my main event loop to have it working. In my case it's in a different thread but I would try to call that before capturing a frame periodically. 

CFRunLoopRunInMode(kCFRunLoopDefaultMode, 1, false);

Hope this helps,

Olivier

[Spencer Dixon]

Did you ever find a solution for this?  I'm running into the same issue :(

[Jacques-Olivier Haché]

I successfully applied Olivier's solution.

This was very useful, thanks !

A couple of additional points: 

• used CFRunLoopRunInMode(kCFRunLoopDefaultMode, 0, true); 
• I placed the all right above my call to captureFrame();
  
• must run on the same thread as the thread that set up the CFRunLoopAddSource. In my case, the worker thread.
  
• I had to do one call per frame captured (hence above captureFrame();)
• I don't advice setting the loop for 1 sec, as it would block the worker thread

Cheers,

JOH

[The V]

Those instructions didn't change anything for me, still got the first frame only. Looks like I never get CGDisplayStream notifications from macOS (I'm talking about handlers set up in RegisterRefreshAndMoveHandlers), so for me the only solution was to invalidate the whole screen in every ScreenCapturerMac::CaptureFrame:

  helper_.InvalidateScreen(screen_pixel_bounds_.size());

No idea what I'm doing wrong. What I'm working on is not WebRTC solution from the ground up though, I just extract some VOIP/SS features from WebRTC code base.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/2b445aa4-0996-47ad-ab3b-c427e56877a7%40googlegroups.com.

[Tien Nguyen]

If you call CapturerFrame() in a worker thread, try to use a dispatch_queue_t within CGDisplayStreamCreateWithDispatchQueue instead of using CGDisplayStreamCreate with 

CFRunLoop. Maybe it's works.

Vào 17:26:21 UTC+7 Thứ Sáu, ngày 30 tháng 6 năm 2017, Olivier BERTHONNEAU đã viết: