Adding a native function to julia's scheduler

[yigiter...@gmail.com]

I need to call "waitformultipleobjects" (windows equivalent of "select" call in linux) from julia using ccall. As this is a blocking function, I would like to call it within another coroutine (task).

 

The problem is that the “Taks” in Julia only function effectively if all the blocking calls within it emanates from julia's own I/O interface. It cannot deal with any blocking call to native C-functions.

 

As far as I can see julia is based on Libuv. I guess every time a blocking call (from defined I/O interface) is issued, julia internally calls a corresponding function from the asynchronous libuv and then waits() for a notify() from the libuv. I guess the entire scheduler of julia is based on this paradigm so that It can deal with asynch operation within a single thread.

 

My question is, is it possible to extent this wait() - notify() paradigm for any arbitrary blocking ccall call?

 

I have tried the following solution, but it fails miserably:

 

0) Start a task which calls a non-blocking function from the dll and then wait() for a notify().

1)  (In C) Implement a dll which creates another thread to call the real blocking function whenever julia calls the non-blocking function in the previous step.

2) Provide a Julia callback function to the dll which is called at the finalizing step of the thread by the dll.

3) (In Julia) the callback function calls the notify() function.

However, it turned out that notify() function itself is not thread safe and Julia’s respond to notify() from another thread (created in C) is totally random.

 

Is it possible to make the julia’s scheduler handle the arbitrary blocking calls?

 

(PS: I was previously advised a solution based on parallel processes. However, for several reasons, multi-process paradigm is not a suitable option for me right now.)

[Tim Holy]

Not sure it will help your specific use case, but see
http://docs.julialang.org/en/release-0.3/manual/calling-c-and-fortran-code/#thread-safety
and an example in
https://github.com/JuliaGPU/CUDArt.jl/blob/master/src/stream.jl

--Tim

[Isaiah Norton]

You could potentially yield to Julia from C instead. See https://github.com/JuliaLang/julia/issues/6006 (use with caution...)

[Keno Fischer]

The generic solution here is to use a thread and uv's async primitive which allows you to queue an event on julia's event loop (and is represented by a SingleAsyncWork at the julia level - ZMQ does this for example). Depending on what you're waiting on, there may also be deeper integration available at the libuv level, but that would depend on the application.

[Stefan Karpinski]

At some point I wrote this code which allows you to create a separate thread and make arbitrary ccalls on that thread. It uses ZMQ for communication though, and the Julia process segfaults when it quits, so it's not ideal. It might be good to have an API for spawning threads and running ccalls in them. You probably don't want to spawn the thread for each call, so it would look something like this:

1. create an object to represent the worker thread
   
2. make a thread_ccall with the thread object as an additional argument c.f. ccall
   
3. the co-routine where the call occurs blocks waiting for the call to finish
   

So the usage largely looks just like making a blocking ccall, but it allows the scheduler to do other work while the ccall happens in the other thread.