ccall from gc finalizer (task switch)

[remi....@gmail.com]

Hi guys,

While creating a tiny wrapper over the C/C++ primesieve library, I encountered an error while trying to finalize the Ptr returned by a function. This pointer must be freed by another function from the library, so I tried to wrap it into a finalizer function but it doesn't work because of the following error: "task switch not allowed from inside gc finalizer". Could someone explain how this could be done properly? I would try to avoid the returned array if possible. Here is the code so far:

const LIB_PRIMESIEVE = "libprimesieve.so"

type C_primesieve_array
    handle::Ptr{Int32}
end

function primes(n::Int)
    size = Cint[0]
    primes_ptr = ccall(
        (:primesieve_generate_primes, LIB_PRIMESIEVE),
        C_primesieve_array,
        (UInt64, UInt64, Ptr{Cint}, Int),
        1, n, size, 3)

    function primesieve_array_finalizer(primesieve_arr::C_primesieve_array)
        ccall(
            (:primesieve_free, LIB_PRIMESIEVE),
            Void, (Ptr{Int32},), primesieve_arr.handle)
    end

    finalizer(primes_ptr, primesieve_array_finalizer)
    return pointer_to_array(primes_ptr.handle, size[1], false)
end

Thank you very much,
Rémi

[Jameson Nash]

note, if a c functions returns a bits type (such as Ptr), it is invalid to declare it to return a struct (such as C_primesieve_array). this will soon cause failures on the x86 linux ABI.

is the `primesieve_free` c-function itself attempting to execute julia code?

note, that since you didn't return `primes_ptr`, it will be garbage collected (and finalized) shortly after your primes function returns

[Rémi Berson]

Hi Jameson, Thank you for your answer,

`primesieve_sieve` is not calling Julia code (juste somme pointer arithmetics and a delete). But it seems that call to `println` in the finalizer is causing the error, strangely.

What would be the prefered way to handle memory deallocation in this case? Copying the result from the C function and immediately deleting the memory allocated by it would be the easiest way, but it will impact the performance. Is there any way to avoid that?

Also, I don't know if it's the right explanation, but I have the feeling that because the GC doesn't know how much memory has been allocated by the C function, he doesn't know when deallocating memory becomes "urgent". Is there any way to inform the GC how much memory is hold by a given pointer?

[Jameson Nash]

I believe there is some sort of gc_counted_malloc function that you can call to inform the gc of significant memory allocations that occur outside of the Julia gc.

I suspect that making the copy may actually be your best option at negligible performance cost. Sometimes, you can provide an alternative malloc to c libraries, allowing you to take ownership of the array data afterwards. I don't know if that is the case here.

[Kirill Ignatiev]

So is there some resolution for this question? I also ran into the problem of not being able to call a C function inside a finalizer (cudaEventDestroy), so I have exactly the same question.

[Rémi Berson]

From what I know until know there are two options:

1. Make a copy immediately after the call to the C function and free the allocated memory by it
2. Wrap the returned pointer into another type and register a finalizer for it (and you must return it, otherwise it will be garbage collected immediately). I have no further information about a way to inform the garbage collector about the amount of memory allocated by the foreign function.

More ideas are always welcome!
And thank you for the help so far.

Rémi

[Kirill Ignatiev]

On Saturday, 14 February 2015 08:22:57 UTC-5, Rémi Berson wrote:

Wrap the returned pointer into another type and register a finalizer for it (and you must return it, otherwise it will be garbage collected immediately).

The problem still exists, this doesn't address it: how do I run a C function in a finalizer? I cannot run the finalizer myself immediately, and the object returned will live for an unknown amount of time. If the finalizer needs to call a C function to free the object's internal state, how do I assign such a finalizer to a Julia object?

[Jameson Nash]

Kirill, I am sure what you are asking. Remi's initial question was resolved to be due to calling print in a finalizer, and unrelated to the ccall usage (which works fine).

[Kirill Ignatiev]

On Saturday, 14 February 2015 17:32:55 UTC-5, Jameson wrote:

Kirill, I am sure what you are asking. Remi's initial question was resolved to be due to calling print in a finalizer, and unrelated to the ccall usage (which works fine).

But there is no print in the original code, or at least I'm not seeing it. Was it there at first but then scrubbed before posting? Also, do you know what behaviour is actually forbidden for finalizers? I am working with an existing library, and tracking down finalizer bugs is somewhat tricky. It may be trying to print something, I cannot tell yet.

If ccall should work fine, that's good to know.

[Jameson Nash]

Just task switches are disallowed (and thus any IO). Perhaps after the threads branch merged, it'll be worth reconsidering that restriction too. The reason to ban task switch is that gc could happen at any time in user's code due to an allocation. It's not generally advisable therefore to allow task switches at that time, since that it necessary for the user to be much more concerned with syncronization.

[Kirill Ignatiev]

On Saturday, 14 February 2015 19:41:25 UTC-5, Jameson wrote:

Just task switches are disallowed (and thus any IO). Perhaps after the threads branch merged, it'll be worth reconsidering that restriction too. The reason to ban task switch is that gc could happen at any time in user's code due to an allocation. It's not generally advisable therefore to allow task switches at that time, since that it necessary for the user to be much more concerned with syncronization.

I see, thanks for explaining. 

[andrew cooke]

Since this is a top hit on google for this issue, I'm posting here just to add that you see this error if you add a println() to your finalize function, to check whether it is called :o)

One solution is to increment a global counter in the finalizer and print the counter before and after gc().

Andrew