racket/gui and the Cocoa NSRunLoop

[Mark Wunsch]

Hi there. I've run into a bit of a conundrum and I'm not quite sure how to proceed. This code intersects between Racket, C (through FFI), and Cocoa and I'm by no means an expert in any of those, so I'm hoping someone can help explain an odd set of bugs I'm seeing.

The main question I have is: How does racket/gui set up and interact with the NSApplication's main thread?

I have some code that bridges Racket to GStreamer (http://gstreamer.freedesktop.org) via FFI: https://github.com/mwunsch/overscan

For the purposes of this discussion, it will help to have that code nearby.

Here's a small Racket program that uses the above-mentioned module to create a GStreamer pipeline using an osxvideosink (Objective-C source code for this element can be found here: https://github.com/GStreamer/gst-plugins-good/blob/master/sys/osxvideo/osxvideosink.m)

I'm running this code in High Sierra, with the GST_DEBUG ENV variable set to "osxvideosink:5"

#lang overscan

(broadcast (videotestsrc) (element-factory%-make "osxvideosink"))

This code seems to run fine. When I add a line above the broadcast call to (require racket/gui/base), the code will crash. Looking at the backtrace, I see that the crashed thread attempts to receive an event from a Core Foundation CFRunLoop and calls out to an observer callback function.

My sense of this crash is that there's something inside the racket/gui internals (my hunch is that this code snippet is in the right neighborhood: https://github.com/racket/gui/blob/master/gui-lib/mred/private/wx/cocoa/queue.rkt#L288-L293) that works around a "typical" NSApplication start-up sequence that makes the osxvideosink code believe that it's running outside of the main NSApp thread. "Typical" is in quotes because I honestly have no idea what I'm talking about.

Another example of this odd behavior is using the glimagesink (associated source code in this neck of the woods: https://github.com/GStreamer/gst-plugins-bad/blob/8d99867c13944be9ba75a892682204955f16f586/gst-libs/gst/gl/cocoa/gstglwindow_cocoa.m#L332-L354)

Running the code, but this time with GST_DEBUG set to "gl*:5"

#lang overscan

(require racket/gui/base)
(broadcast (videotestsrc) (element-factory%-make "glimagesink"))

For me, this code will display a window with a still video frame, and not animate beyond this. My hunch is, similar to the osxvideosink code, the glimagesink code expects frame updates to be enqueued on the main thread, but the creation of the racket/gui internals prevents these events from being dequeued.

Can someone who has a better understanding of the Cocoa foundations (pun somewhat intentional) of racket/gui help me better understand why this code is behaving oddly, and what, if anything can be done to ensure that the application's main thread processes events (if that's the kind of thing that should be ensured)?

Thanks for going down this rabbit hole.

- Mark

[Matthew Flatt]

At Wed, 10 Jan 2018 11:17:14 -0800 (PST), Mark Wunsch wrote:
> The main question I have is: How does racket/gui set up and interact with
> the NSApplication's main thread?

> [...]

> My sense of this crash is that there's something inside the racket/gui

> internals [....]

> that works around a "typical" NSApplication start-up sequence that makes
> the osxvideosink code believe that it's running outside of the main NSApp
> thread. "Typical" is in quotes because I honestly have no idea what I'm
> talking about.

It's true that `racket/gui` avoids the usual run loop, so I expect that
"osxvideosink.m" will think that it needs its own. Specifically,
`[[NSRunLoop mainRunLoop] currentMode]` will return nil. And if
`gst_osx_videosink_check_main_run_loop` runs in the main OS thread via
a foreign call, then no events will be dispatched while it waits to
check in teh backup way, because the Racket-level thread that pulses
the main run loop will be blocked until the foreign call returns.
Finally, if "osxvideosink.m" sets up its own run loop and hacks
`isThread`, that would interact badly with `racket/gui` and could
explain the crash.

Does it help to wrap the `broadcast` call with

(call-atomically-in-run-loop
(lambda ()
....))

using the implementation of `call-atomically-in-run-loop` below?

----------------------------------------

#lang racket/base
(require racket/gui/base
ffi/unsafe
ffi/unsafe/objc
ffi/unsafe/nsalloc)

(provide call-atomically-in-run-loop)

(import-class NSObject NSArray NSRunLoop)
(define NSDefaultRunLoopMode (get-ffi-obj 'NSDefaultRunLoopMode #f _id))

(define-objc-class CallerContainer NSObject
[proc]
(-a _void (call) (proc)))

(define (call-atomically-in-run-loop proc)
(define result #f)
(define s (make-semaphore))
(call-with-autorelease
(lambda ()
(define obj (tell CallerContainer alloc))
(set-ivar! obj proc (lambda () (set! result (proc)) (semaphore-post s)))
(tellv (tell NSRunLoop mainRunLoop)
performSelector: #:type _SEL (selector call)
target: obj
argument: #f
order: #:type _uint 0
modes: (tell (tell NSArray alloc)
initWithObject: NSDefaultRunLoopMode))))
(yield s)
result)

;; Example, gets a non-#f result:
#;
(call-atomically-in-run-loop
(lambda ()
(tell (tell NSRunLoop mainRunLoop) currentMode)))

[Mark Wunsch]

Yes — that's perfect, thank you! And it also gives me a better sense of how racket/gui is implemented for macOS. This snippet of code is now included in my library, and I wonder if there's benefit to having this be available in racket/gui for cross-platform usage?

Now that I understand this a bit, I tried solving the second use-case I wrote about, using the glimagesink. My understanding of this code is that it creates a subview with a CAOpenGLLayer, and then enqueues a `setNeedsDisplay`: https://github.com/GStreamer/gst-plugins-bad/blob/8d99867c13944be9ba75a892682204955f16f586/gst-libs/gst/gl/cocoa/gstglwindow_cocoa.m#L332-L354

When I run equivalent code outside of racket (e.g. through the gst-launch tool), I see that I continuously see calls to draw (https://developer.apple.com/documentation/quartzcore/caopengllayer/1522316-draw). Those calls are not being made when run from Racket.

I realize we're venturing outside of the Realm of Racket (intentional pun) and closer to Cocoa, but I'd like to have a better sense of how to dequeue these drawing actions. Based on your previous code, I thought that I could yield to the main thread by calling `run` or `runUntilDate` on the main NSRunLoop, but that didn't seem to accomplish this goal.

Any ideas of how to "yield" to whatever queue or thread GStreamer has running?

Thank you so much for your help!

[Matthew Flatt]

Sorry for the slow response!

At Thu, 11 Jan 2018 12:56:02 -0500, Mark Wunsch wrote:
> Yes — that's perfect, thank you! And it also gives me a better sense of how
> racket/gui is implemented for macOS. This snippet of code is now included in
> my library, and I wonder if there's benefit to having this be available in
> racket/gui for cross-platform usage?

The snippet seems fairly specialized to me, and I'm not sure there's a
useful cross-platform interpretation.

> Now that I understand this a bit, I tried solving the second use-case I wrote
> about, using the glimagesink. My understanding of this code is that it creates
> a subview with a CAOpenGLLayer, and then enqueues a `setNeedsDisplay`:
> https://github.com/GStreamer/gst-plugins-bad/blob/8d99867c13944be9ba75a89268220
> 4955f16f586/gst-libs/gst/gl/cocoa/gstglwindow_cocoa.m#L332-L354
>
> When I run equivalent code outside of racket (e.g. through the gst-launch
> tool), I see that I continuously see calls to draw
> (https://developer.apple.com/documentation/quartzcore/caopengllayer/1522316-dra
> w). Those calls are not being made when run from Racket.
>
> I realize we're venturing outside of the Realm of Racket (intentional pun)
> and closer to Cocoa, but I'd like to have a better sense of how to dequeue
> these drawing actions. Based on your previous code, I thought that I could
> yield to the main thread by calling `run` or `runUntilDate` on the main
> NSRunLoop, but that didn't seem to accomplish this goal.
>
> Any ideas of how to "yield" to whatever queue or thread GStreamer has running?

I'm not sure I'm following, but I think a `setNeedsDisplay` on a view
within a `racket/gui` frame will triggers a refresh callback in the
normal `racket/gui` event queue.

Does the `yield` function from `racket/gui` allow the callback to run
in the way you expect? I'd try `yield` applied to a semaphore that's
posted by a low-priority callback installed with `queue-callback`,
since refresh events will happen with higher priority.

There's also `flush-display`. For various reasons, `flush-display`
rarely helps, but it may be worth a try.

[Mark Wunsch]

Thanks for the response. Since writing this, I've tried to better understand this C code, and my understanding of the problems here might have not been correct. They still probably aren't correct, but I'll try to better articulate the challenge.
I tried all of the techniques you suggested, none of them seemed to work. With my limited understanding of this C GStreamer code, I think the problem area is in this function of the codebase: https://github.com/GStreamer/gst-plugins-bad/blob/8d99867c13944be9ba75a892682204955f16f586/gst-libs/gst/gl/cocoa/gstglwindow_cocoa.m#L421-L448

When initialized this code seems to create a new GCD dispatch queue:

(__bridge_retained gpointer)
(dispatch_queue_create ("org.freedesktop.gstreamer.glwindow", NULL))

It then periodically dispatches work to this queue.

My understanding of how GCD dispatch queues work is practically nonexistent, but I'm under the impression that work is executed off these queues through some Cocoa mechanism off the main run loop, as before. Do you know how I might ensure that this work is executing or am I way off the mark?

Thanks.

On Mon, Jan 22, 2018, at 10:31 PM, Matthew Flatt wrote:
> Sorry for the slow response!
>
> At Thu, 11 Jan 2018 12:56:02 -0500, Mark Wunsch wrote:

> > Yes — that's perfect, thank you! And it also gives me a better sense of how
> > racket/gui is implemented for macOS. This snippet of code is now included in
> > my library, and I wonder if there's benefit to having this be available in
> > racket/gui for cross-platform usage?
>

> The snippet seems fairly specialized to me, and I'm not sure there's a
> useful cross-platform interpretation.
>

> > Now that I understand this a bit, I tried solving the second use-case I wrote
> > about, using the glimagesink. My understanding of this code is that it creates
> > a subview with a CAOpenGLLayer, and then enqueues a `setNeedsDisplay`:
> > https://github.com/GStreamer/gst-plugins-bad/blob/8d99867c13944be9ba75a89268220
> > 4955f16f586/gst-libs/gst/gl/cocoa/gstglwindow_cocoa.m#L332-L354
> >
> > When I run equivalent code outside of racket (e.g. through the gst-launch
> > tool), I see that I continuously see calls to draw
> > (https://developer.apple.com/documentation/quartzcore/caopengllayer/1522316-dra
> > w). Those calls are not being made when run from Racket.
> >
> > I realize we're venturing outside of the Realm of Racket (intentional pun)
> > and closer to Cocoa, but I'd like to have a better sense of how to dequeue
> > these drawing actions. Based on your previous code, I thought that I could
> > yield to the main thread by calling `run` or `runUntilDate` on the main
> > NSRunLoop, but that didn't seem to accomplish this goal.
> >
> > Any ideas of how to "yield" to whatever queue or thread GStreamer has running?
>