the challenge of factoring/threading crankd

[Preston Holmes]

Over the past few weeks as I've had bits of time I've been trying a
few scratch paper experiments to see how one might factor out the
event listening portion of crankd into a more general use module.

The challenge/goal that I was working with was this:

allow for asynchronous python callback on system event from inside
pure python code (and the corollary became: without adverse memory
impact of multiple Cocoa loading python processes)

The core of crankd's functionality revolves around the use of an
NSRunloop

Ideally the runloop blocks till an event occurs, but that makes crankd
unresponsive to signals etc - hence the timer

The same runloop behavior makes it impossible to keep things truly
asynchronous if the runloop is being run anywhere in the main thread
of your code (your code meaning - the calling code / user code), and
so no form of standard import and function calling would work.

also to have this all happening in the main thread means that one has
to mix Cocoa and python in the same program

To allow segregation of the Cocoa and Python, the Cocoa runloop has to
occur is a separate thread of execution. Problems with that are:

python threading, and multiprocessing modules use os.fork and fork()
is not compatible with Cocoa:
http://lists.apple.com/archives/cocoa-dev/2005/Jan/msg00676.html

The next approach attempted was to use NSThread so that Cocoa was
always running in a non-forked process and that the threading was
handled by Cocoa directly. However, only a subset of Cocoa frameworks
are thread safe and those do not include critical ones like
NSWorkplace etc.
http://www.mail-archive.com/coco...@lists.apple.com/msg26099.html

As a side note, I was able to get a NSThread spawned with its own
runloop and callbacks working in python - but only for a simple timer.

So what I've come up with is a method whereby the runloop code runs
inside its own process - and communicates with standard python code
through named pipes.

This results then in three layers:

1) end user code - read as simple python code (no pyobjc), uses
EventListener class

2) EventListener class - provides interface to associate callable
functions with system events. This class handles communication with
the runloop watcher through named pipes, adding events to be watched
and receiving event occurrences and running the callbacks

3) eventwatcher tool - a program that is run and watches for events -
communicates with one or more EventListener instances via named
pipes. A single process would handle all EventListener instances in
any user code - events can be added or removed while running (not
avail in crankd currently).

I've have outlined the design of 2 and 3 pretty thoroughly with some
aspects spot tested with experiments.

So now the question is whether to proceed.

I know this group is rightfully averse to complexity - but the outline
I have for dispatching events through named pipes is pretty efficient
and I haven't been able to think of any other way to factor out the
runloop dependent parts of crankd.

Do others feel that the functionality of listening for various system
events and executing python code in response is a feature that would
be useful to have as a general API abstracted from the context of the
crankd utility? Note that such a refactor I'm proposing is largely
compatible and independent from the work Nigel is doing - as his
improvements would just be split into those that are runloop-code
related and those that are crankd related.

Please weigh in with your opinion

-Preston

[Nigel Kersten]

So I haven't had time to do more than read this once and have a
cursory think about it.

I'll digest it more later, but my gut feeling is that this will
eventually be a useful API to abstract out, but the best idea is to
keep it in crankd until we are clearer about the functionality it
provides.

I'm not particularly wedded to this point of view though.

I do like the approach you're taking though.

[Chris Adams]

I generally like the idea - one problem with the current design is
that it doesn't handle asynchronous events well (you'd have to handle
everything manually) and it would allow you to do things like kill
unresponsive listeners or simultaneous processing. It also opens the
possibility of allowing things like running event handlers as a user,
which could be interesting in some scenarios.

As far as a general purpose API goes, I can see where it could be
valuable in the future but I'd recommend starting with a major crankd
upgrade and breaking out the API the first time we come up with a
solid use for it - otherwise I'm afraid it might end up being a large
amount of work with limited utility.

Chris