VideoCaptureMacQTKitObjC.mm and VideoCaptureMacQTKitInfoObjC.mm

[TJ Grant]

Hi all…

So I've made some more progress pouring through the Mac part of the WebRTC codebase, and I see that for some of the places I'm hanging at (if I choose to do things in a "properly threaded" way) are in two files:

VideoCaptureMacQTKitObjC.mm

VideoCaptureMacQTKitInfoObjC.mm

I notice that there appears to be notes here and there that use of QTKit for video capture is not thread aware or thread safe.

I get the general idea of what these classes are trying to do, however I'm really scratching my head…

What's the general idea on the design of these classes? What would be the suggested route to "fix" these to become thread aware?

I'm also curious though, why are there two classes that are essentially doing the same thing-- why is there an "info" class when you can get the same stuff from the "non-info" class?

Any hints would be much appreciated.

Best regards,
--TJ Grant
tjg...@tatewake.com

[Magnus Flodman]

Hi TJ,

> So I've made some more progress pouring through the Mac part of the WebRTC
> codebase, and I see that for some of the places I'm hanging at (if I choose
> to do things in a "properly threaded" way) are in two files:
>
> VideoCaptureMacQTKitObjC.mm
> VideoCaptureMacQTKitInfoObjC.mm

Hanging as in a dead-lock? If that is the case, please file a bug
code.webrtc.org and include call stacks if possible.

> I'm also curious though, why are there two classes that are essentially
> doing the same thing-- why is there an "info" class when you can get the
> same stuff from the "non-info" class?

The idea behind the two classes was to have one class dealing only
with device info and one class working with the actual device and
media capture. And to complicate things, the API should be as for Win
and Linux as well as the SequenceGrabber version of Mac Capture
Module. But I agree with your point and I have several suggestions for
improving the WebRTC QTKit usage as well as the capture module in
general, so this is on my todo list.


-Magnus

[TJ Grant]

Magnus…

It's not a deadlock within WebRTC's code; the program (the Mac version of Peer Connection Client that I'm writing) hangs on Apple's QTKit calls.

My research on this indicates that whatever thread you start calling QTKit with must be the same thread you use throughout, and any calls from a different thread may corrupt the "QTKit state" in use. (Although I could be mistaken.)

What's I observe is that from within (an unmodified) P.C.C. the main thread starts initialization of some of the QTKit components (through WebRTC), however worker_thread (from P.C.C.) calls into these methods as well.

Particularly (within video_capture_qtkit_objc.mm), this gets called from worker_thread and hangs:

    success = [_captureSession addOutput:_captureDecompressedVideoOutput

               error:&error];

However, commenting this out (as a test) you'll wind up hanging at a different QTKit call.

Forcing everything to go through the main thread (by making worker_thread an instance of the main thread) doesn't have this issue (in fact, it completes everything successfully), but obviously this isn't ideal moving forward.

My thought at this point is to wrap all the calls to QTKit into one background thread, which might solve the threading issue.

(Any thoughts on that?)

Best regards,

--TJ Grant
tjg...@tatewake.com

[Bob Wolff]

Magnus and TJ,

We have found the exact same result. I have written (recently) a full blown QTKit capture -> encode -> rtsp streaming solution and so I'm familiar with the calls and such. So I was surprised to see this call die without ever returning. But it seems TJ has the threading answer to 'why' and in fact he may have answered a question for me with regards to why sometimes my version of the same pipeline will work from the UI (clicking buttons) but sometimes will not fully function when it is started from another thread which is web-controlled through sockets.  @synchronize possibly as a simpler answer rather than winding back to a single background thread TJ? I have not done that protection myself but it is ringing a faint bell from reading the QTKit docs.

We were planning to file a bug on it today, but I won't go to the trouble if this discussion is going to roll along to a potential solution-path quickly. :-)  Will wait and see later today and tomorrow I suppose.

Bob Wolff

[TJ Grant]

Bob…

Well interestingly enough, I've tried a couple of different tricks as possible quick fixes-- NSLock'ing around all the QTKit methods, performSelectorOnMainThread with helper methods (which is kind of a hack I think, but solved a similar problem with OpenGL), and I believe I tried @synchronize initially (I might give it another go just for kicks though.)

However, I'm not really familiar with Mac threading though, so it may be a combination of things (perhaps locking and synchronization) that's missing.

>We were planning to file a bug on it today, but I won't go to the trouble if this discussion is going to roll along to a potential solution-path quickly. :-)  Will wait and see later today and tomorrow I suppose.

Hmm, define "quick" :-)

Actually, if you're game, please still file the bug, I imagine someone would have to do so to get a fix in anyway…

Best,
--TJ Grant
tjg...@tatewake.com