Debugging GC: where is a reference being held?

[Tim Holy]

Over in Reactive.jl we're trying to figure out why certain objects are not
being garbage-collected, and the natural presumption is that something is
still holding a reference to them. Looking at the source of gc.c, I see
there's an old stub `lookforme` which seems to be unused. There's a
`gc_findval` which may be more promising, but I don't quite understand how one
might go about using it in practice. Anyone have any tips?

--Tim

[Yichao Yu]

I usally debug the other kind of GC problems so I'm afraid I can't
offer too many tips so here's my understanding of the intended use of
`gc_findval` without actually running the code.

When you have a value that is not being GC'd, you can interupt the
program and set the gc_findval value in the debugger (you may need to
set a breakpoint in gc.c since it is not exported, alternatively you
can DLLEXPORT that symbol or if you can turn ASLR off, you can compile
that address in directly).

The debugger should be triggered (by SIGINT on unix) on the next GC
(or you can `p jl_gc_collect(1)` to trigger one manually from gdb) and
when you look at the backtrace, you should see a series of `push_root`
and/or `gc_push_root` (depending on inlining...) the first argument
(`v`) for each of the function is the object currently being marked by
the GC and you can then trace back the object reference chain.

If the reference chain is too long, it might get pushed to the remset
(to avoid stack overflow) and you might want to break on the remset
value this time to see the complete chain.

>
> --Tim
>

[Tim Holy]

Thanks Yichao, this was very helpful. I temporarily added

diff --git a/src/gc.c b/src/gc.c
index c02bbbd..984bc2e 100644
--- a/src/gc.c
+++ b/src/gc.c
@@ -1468,6 +1468,10 @@ void gc_queue_binding(jl_binding_t *bnd)
static int push_root(jl_value_t *v, int d, int);
#ifdef JL_DEBUG_BUILD
static void *volatile gc_findval; // for usage from gdb, for finding the gc-
root for a value
+DLLEXPORT void jl_set_gc_findval(jl_value_t *val)
+{
+ gc_findval = val;
+}
#endif
static inline int gc_push_root(void *v, int d) // v isa jl_value_t*
{

to make it easy to choose the variable to be traced from julia.

Sadly the backtrace was truncated so I didn't really get very far with the
overall question, but now I know how to go about this in the future.

Best,
--Tim

[Yichao Yu]

On Sat, Nov 14, 2015 at 8:18 AM, Tim Holy <tim....@gmail.com> wrote:
> Thanks Yichao, this was very helpful. I temporarily added
>
> diff --git a/src/gc.c b/src/gc.c
> index c02bbbd..984bc2e 100644
> --- a/src/gc.c
> +++ b/src/gc.c
> @@ -1468,6 +1468,10 @@ void gc_queue_binding(jl_binding_t *bnd)
> static int push_root(jl_value_t *v, int d, int);
> #ifdef JL_DEBUG_BUILD
> static void *volatile gc_findval; // for usage from gdb, for finding the gc-
> root for a value
> +DLLEXPORT void jl_set_gc_findval(jl_value_t *val)
> +{
> + gc_findval = val;
> +}
> #endif
> static inline int gc_push_root(void *v, int d) // v isa jl_value_t*
> {
>
> to make it easy to choose the variable to be traced from julia.

I think this is useful and we can have that mainlined. (probably move
this whole #ifdef JL_DEBUG_BUILD block to gc-debug.c)

>
> Sadly the backtrace was truncated so I didn't really get very far with the
> overall question, but now I know how to go about this in the future.

What do you mean by truncated? The GDB backtrace? Or the gc trace. If
the GC trace starts from the remset, you can trigger on that value
instead (although the most reliable way I can think of to do this is
to turn off ASLR[1]) and use the bare address in the debugger. (unless
you know how to find that object in julia of course)

[1] https://github.com/JuliaLang/julia/blob/eddd5b8c7e419c6eff1b2b93b4b6f98b9f314faf/src/gc-debug.c#L119