cfunction result garbage collected

115 views
Skip to first unread message

Christian Rorvik

unread,
Sep 28, 2016, 12:22:49 PM9/28/16
to julia-users
I'm not yet a 100% sure what is happening in my application, but I'm embedding Julia and using cfunction() to generate C-callable functions that at a later time invoke the registered julia code. I use a fairly standard pattern I think, of passing a pointer to a julia object that is retain in C and later passed down to the callback created by cfunction, and from there resolved to the original real Julia reference type, invoking a closure. The closure itself is retained in some global state to prevent garbage collection. What appears to be happening however, is the code at the landing site for the cfunction returned pointer is at some point garbage collection (or at least corrupted), as my program, after a while of running, will segfault upon invoking the callback (after many previously successful callbacks). It segfaults because it hits invalid code, and it's not that some state is missing when running the code.

Is this to be be expected, and what's the right way to ensure the code isn't GC'd?

Christian Rorvik

unread,
Sep 28, 2016, 4:36:35 PM9/28/16
to julia-users
GC of code turned out to be a red herring. After isolating the instance of a call that was crashing, and getting right up to it, GDB was kind enough to get its knickers in a twist and segfault itself every time I stepped instruction by instruction to find the precise point of failure (backtrace after segfault looked pretty meaningless).

What it tunred out to be a was a Ref vs Ptr problem, where in Julia I have a callback wrapper that receives a pointer to the closure. I was taking this as Ref{T} and invoking it with a pointer from the C code. This worked all fine and dandy, but somehow caused the object to be collected (I'm guessing). I can see how what I did was wrong, but I can't see how it would lead to something being collected when there still exists roots pointing to it. If anything I would possibly expect a leak. Nonetheless, bit of a wild goose chase.

Yichao Yu

unread,
Sep 28, 2016, 4:50:47 PM9/28/16
to Julia Users
On Wed, Sep 28, 2016 at 4:36 PM, Christian Rorvik
<christia...@gmail.com> wrote:
> GC of code turned out to be a red herring. After isolating the instance of a
> call that was crashing, and getting right up to it, GDB was kind enough to
> get its knickers in a twist and segfault itself every time I stepped
> instruction by instruction to find the precise point of failure (backtrace
> after segfault looked pretty meaningless).
>
> What it tunred out to be a was a Ref vs Ptr problem, where in Julia I have a
> callback wrapper that receives a pointer to the closure. I was taking this
> as Ref{T} and invoking it with a pointer from the C code. This worked all
> fine and dandy, but somehow caused the object to be collected (I'm
> guessing). I can see how what I did was wrong, but I can't see how it would
> lead to something being collected when there still exists roots pointing to
> it. If anything I would possibly expect a leak. Nonetheless, bit of a wild
> goose chase.

It's unclear what exactly you mean but just to be clear `cfunction(f,
Ret, Tuple{Ref{Int}})` expect a pointer to a julia boxed `Int` that is
rooted somewhere during the execution of the julia callback function.
It is illegal to pass it a bare C `intptr_t*` (that doesn't point to
julia memory of a boxed Int) and it is illegal to generates it with
`ccall(..., (Ptr{Int},), &i)`.

Cristóvão Duarte Sousa

unread,
Sep 28, 2016, 4:58:13 PM9/28/16
to julia-users
I can't help you at all, but I would like to ask if you pass the cfunction() result to the C side.
I've been playing with embedding Julia and discovered I could call a cfunction() directly  from C, although I'm not sure that's a good practice.

https://github.com/cdsousa/embedding_julia/blob/master/embedding.cpp#L40


On Wednesday, September 28, 2016 at 5:22:49 PM UTC+1, Christian Rorvik wrote:

Christian Rorvik

unread,
Sep 28, 2016, 5:00:01 PM9/28/16
to julia-users
I don't have the code at hand right now (it's at work), but what I was doing was something like

type Wrapper{Arg}
    cfun::Ptr{Void}
    cobj::Ptr{Void}
    obj

    function Wrapper{T}(f::T)
        return new(cfunction(call_wrapper, Void, (Ref{T}, Arg)), pointer_from_objref(f), f)
    end
end

Where call_wrapper would invoke the closure. I had to change call_wrapper to receive the closure as Ptr{T} and call unsafe_pointer_to_objref and then call it.

Christian Rorvik

unread,
Sep 28, 2016, 5:02:33 PM9/28/16
to julia-users
Yeah that's what I do. I think that's kind of the point of it, although I think most people use it perhaps as in-scope callbacks, whereas I persist the callbacks and invoke them later, so additionally I have to retain the wrapped closures to prevent them being collected.

Yichao Yu

unread,
Sep 28, 2016, 5:06:51 PM9/28/16
to Julia Users
On Wed, Sep 28, 2016 at 5:00 PM, Christian Rorvik
<christia...@gmail.com> wrote:
> I don't have the code at hand right now (it's at work), but what I was doing
> was something like
>
> type Wrapper{Arg}
> cfun::Ptr{Void}
> cobj::Ptr{Void}
> obj
>
> function Wrapper{T}(f::T)
> return new(cfunction(call_wrapper, Void, (Ref{T}, Arg)),
> pointer_from_objref(f), f)
> end
> end

This is wrong, You shouldn't ever call `pointer_from_objref` for
unknown object. The right way to handl this is

type Wrapper{Arg}
cfun::Ptr{Void}
cobj::Ptr{Void}
obj
function Wrapper{T}(f::T)
to_root = Base.cconvert(Ref{T}, f)
ptr = Base.unsafe_convert(Ref{T}, to_root)
return new(cfunction(call_wrapper, Void, (Ref{T}, Arg)),
Ptr{Void}(ptr), to_root)

Christian Rorvik

unread,
Sep 28, 2016, 5:16:03 PM9/28/16
to julia-users
Thanks! I'll try this tomorrow. So similar to what you've done here? https://github.com/yuyichao/FunctionWrappers.jl/blob/master/src/FunctionWrappers.jl#L40

I can't remember if I actually called that explicitly, but I believe what I ended up with was similar to here: http://julialang.org/blog/2013/05/callback, where they use unsafe_pointer_to_objref to recover the function object.

Yichao Yu

unread,
Sep 28, 2016, 5:33:04 PM9/28/16
to Julia Users
On Wed, Sep 28, 2016 at 5:16 PM, Christian Rorvik
<christia...@gmail.com> wrote:
> Thanks! I'll try this tomorrow. So similar to what you've done here?
> https://github.com/yuyichao/FunctionWrappers.jl/blob/master/src/FunctionWrappers.jl#L40

Correct. Also https://github.com/JuliaPy/PyCall.jl/pull/267

>
> I can't remember if I actually called that explicitly, but I believe what I
> ended up with was similar to here:
> http://julialang.org/blog/2013/05/callback, where they use
> unsafe_pointer_to_objref to recover the function object.

unsafe_pointer_to_objref is ok. It's unsafe in the sense that you
almost guarantee a segfault if it is not used correctly.
pointer_from_objref is less unsafe in that you might not get a
segfault if the GC is nice to you today.

Ref https://github.com/JuliaLang/julia/issues/15857 for why
pointer_from_objref is bad if you call it on arbitrary object (mostly
immutables). Essentially the compiler is free to give you garbage if
you call it on immutables or the pointer you get back might not be the
pointer that you stored somewhere else since the pointer value is
supposed to be insignificant, only the content is, for immutables.

Christian Rorvik

unread,
Sep 29, 2016, 8:54:37 AM9/29/16
to julia-users
That's very helpful, thanks very much!
Reply all
Reply to author
Forward
0 new messages