Help testing the debugger

[Keno Fischer]

Hi everyone,

I'm about to start making a big push on the debugger again and I'd like to have everybody come along and see where I'm going. The first thing to do is to have a version of LLDB that understands julia's JITed stack frames.

- Build julia with instructions from https://github.com/Keno/Cxx.jl (just the julia parts, the Cxx part won't be needed until I release the UI).

Additionally, on OS X

- Enable FORCE_ELF in src/options (just realized I haven't pushed this yet - will do)
- Comment out `if (arch.GetTriple().getVendor() != llvm::Triple::Apple)` in deps/llvm-svn/tools/lldb/source/Plugins/JITLoader/GDB/JITLoaderGDB.cpp (I am in the processing of adding an LLDB setting that does this).

- Apply http://reviews.llvm.org/D6185 if it hasn't been merged then.

Then, you can use LLDB just as you would usually, but note that on OS X, you also have to 

- export LLDB_DEBUGSERVER_PATH=/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Resources/debugserver

This should at least get you stack traces and the ability to step through julia code. Please file any issues as bug reports on JuliaLang/julia. For local variables, you may get some, and you'll get more if you set DISABLE_OPT in src/options.h. If you set the latter, I'd love to collect a set of test cases that don't get local variables even though they should, so I can gradually start fixing them in LLVM (I haven't really started collecting test cases, so you'll probably find a lot of functions don't work, yet).

Let me know if you have any questions.

Keno

[Yichao Yu]

On Tue, Apr 28, 2015 at 12:44 PM, Keno Fischer
<kfis...@college.harvard.edu> wrote:
> Hi everyone,
> I'm about to start making a big push on the debugger again and I'd like to
> have everybody come along and see where I'm going. The first thing to do is
> to have a version of LLDB that understands julia's JITed stack frames.
>
> - Build julia with instructions from https://github.com/Keno/Cxx.jl (just
> the julia parts, the Cxx part won't be needed until I release the UI).

The README says that LLVM 3.7 is required. Just curious how much will
be working if I'm using LLVM 3.6 on Linux?

[Keno Fischer]

LLDB might partially. I'm not planning to support the Cxx based UI on anything less than 3.7. I also expect that future improvements to the debug info will only be available with LLVM 3.7.

[Tracy Wadleigh]

There's no mention of windows. Is it possible it might be made to work there?

[Keno Fischer]

Windows support will have to wait for LLDB to catch up. The good news is that people from Google and Microsoft are actively working on it, but it's not there yet.

[Chris Foster]

I'm trying to get the build to work on ubuntu 14.04 with a very recent
git revision of julia (74a0253) but I'm hitting issues with undefined
references to pthreads in the lldb dependency. For anyone else
encountering this problem, it's tracked here:

https://github.com/Keno/Cxx.jl/issues/94

(For some stupid reason, this was not the first place I looked so I
thought I'd point it out.)

~Chris

[Scott Jones]

Yay!!! +100!  Debugging Julia so far has not been exactly pleasant... I'll definitely try this out.

[David Anthoff]

Microsoft had its big announcement day today. The following two items sound good for the prospects of debugging support on Windows:

 

http://blogs.msdn.com/b/vcblog/archive/2015/04/29/open-sourcing-visual-studio-s-gdb-lldb-debug-engine.aspx

http://blogs.msdn.com/b/vcblog/archive/2015/04/29/debug-c-code-on-linux-from-visual-studio.aspx

 

I don’t understand enough about debuggers to really tell whether that will help Julia, but these posts seemed to have all the right buzz words ;)

 

Cheers,

David

[Avik Sengupta]

I've been trying to follow these instructions for OSX on the latest julia master, and am getting the following compile error in codegen.cpp. This is with an existing git checkout, but with cleanall and distclean in root and deps. Any suggestions before I "rm -rf" ?

CC src/codegen.o
codegen.cpp:218:8: error: unknown type name 'MDCompositeType'
static MDCompositeType *jl_value_dillvmt;
       ^
codegen.cpp:219:8: error: unknown type name 'MDDerivedType'; did you mean 'DIDerivedType'?
static MDDerivedType *jl_pvalue_dillvmt;
       ^~~~~~~~~~~~~
       DIDerivedType
/Users/aviks/dev/julia/julia4/usr/include/llvm/IR/DebugInfoMetadata.h:677:7: note:
      'DIDerivedType' declared here
class DIDerivedType : public DIDerivedTypeBase {
      ^
codegen.cpp:220:8: error: unknown type name 'MDDerivedType'; did you mean 'DIDerivedType'?
static MDDerivedType *jl_ppvalue_dillvmt;
       ^~~~~~~~~~~~~
       DIDerivedType
/Users/aviks/dev/julia/julia4/usr/include/llvm/IR/DebugInfoMetadata.h:677:7: note:
      'DIDerivedType' declared here
class DIDerivedType : public DIDerivedTypeBase {
      ^
codegen.cpp:221:8: error: unknown type name 'MDSubroutineType'; did you mean
      'DISubroutineType'?
static MDSubroutineType *jl_di_func_sig;
       ^~~~~~~~~~~~~~~~
       DISubroutineType
/Users/aviks/dev/julia/julia4/usr/include/llvm/IR/DebugInfoMetadata.h:921:7: note:
      'DISubroutineType' declared here
class DISubroutineType : public DICompositeTypeBase {
      ^
codegen.cpp:347:5: error: unknown type name 'MDLocalVariable'; did you mean
      'DILocalVariable'?
    MDLocalVariable *dinfo;
    ^~~~~~~~~~~~~~~
    DILocalVariable
/Users/aviks/dev/julia/julia4/usr/include/llvm/IR/DebugInfoMetadata.h:1855:7: note:
      'DILocalVariable' declared here
class DILocalVariable : public DIVariable {
      ^
In file included from codegen.cpp:586:
./cgutils.cpp:259:8: error: unknown type name 'MDType'
static MDType *julia_type_to_di(jl_

[Andreas Lobinger]

Just a quick question:

I ran into the -pthreads problem on LLVM building (yes, i'm ubuntu 14.04) and run into similar problem in codegen.cpp
lobi@orange4:~/julia04$ make
    CC src/codegen.o
codegen.cpp:168:8: error: ‘DataLayoutPass’ does not name a type
 static DataLayoutPass *jl_data_layout;

is this to be reported as issue on julia (0.4 dev). Or Cxx. Or LLVM-3.7?

[Elliot Saba]

I'd just like to second that I'm having the same trouble as Avik in testing this out.

-E

[Jameson Nash]

If you've previously build llvm-svn in your working directly, you might need to run `make -C deps update-llvm LLVM_VER=svn && make -C deps reinstall-llvm` to actually upgrade it. Stuff changes (and breaks the ABI used by Julia) pretty often on llvm-svn, so it's often a good first step to check that your llvm-svn folder is actually up-to-date.

[Keno Fischer]

Yes, they renamed a bunch of things a couple of days ago. I'm on it.

[Andreas Lobinger]

Thank you for that.

On Friday, May 1, 2015 at 8:28:28 PM UTC+2, Keno Fischer wrote:

Yes, they renamed a bunch of things a couple of days ago. I'm on it.

in the meanwhile: Could you/anyone please post a version number (julia+LLVM) when it was still working, or just the date.

Wishing a happy day,
       Andreas

[Elliot Saba]

I've made it past the compiler errors now with the latest commit.  Thanks Keno.

I'm now getting stuck on this:

    JULIA usr/lib/julia/sys0.o

Assertion failed: (Section && "Cannot switch to a null section!"), function SwitchSection, file /Users/sabae/src/julia/deps/llvm-svn/lib/MC/MCStreamer.cpp, line 668.

/bin/sh: line 1: 72564 Abort trap: 6           /Users/sabae/src/julia/usr/bin/julia -C native --build /Users/sabae/src/julia/usr/lib/julia/sys0 sysimg.jl

make[1]: *** [/Users/sabae/src/julia/usr/lib/julia/sys0.o] Error 134

[Keno Fischer]

This is http://reviews.llvm.org/D6185

[Avik Sengupta]

So I'm trying to get this working on linux now, and hit the pthreads issue. I used the workaround in this issue to successfully compile lldb (add a -lpthread). However, julia now crashes on sysimg generation. gdb shows the following. Any ideas on what could be wrong? Thanks!

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x00007ffff7ff603e in julia.new_0 ()
#0  0x00007ffff7ff603e in julia.new_0 ()
#1  0x00007ffff78e2d38 in jl_apply (f=0x7ffdf2f40a70, args=0x7fffffffd9c8, nargs=2) at julia.h:1280
#2  0x00007ffff78e6dcf in jl_trampoline (F=0x7ffdf2f40a70, args=0x7fffffffd9c8, nargs=2) at builtins.c:961
#3  0x00007ffff78d6962 in jl_apply (f=0x7ffdf2f40a70, args=0x7fffffffd9c8, nargs=2) at julia.h:1280
#4  0x00007ffff78dcccc in jl_apply_generic (F=0x7ffdf2f409f0, args=0x7fffffffd9c8, nargs=2) at gf.c:1722
#5  0x00007ffff7986a17 in jl_apply (f=0x7ffdf2f409f0, args=0x7fffffffd9c8, nargs=2) at julia.h:1280
#6  0x00007ffff7986f08 in do_call (f=0x7ffdf2f409f0, args=0x7ffdf2f44468, nargs=2, eval0=0x0, locals=0x0, nl=0,
    ngensym=0) at interpreter.c:65
#7  0x00007ffff7987c3b in eval (e=0x7ffdf2f40a30, locals=0x0, nl=0, ngensym=0) at interpreter.c:212
#8  0x00007ffff7986b8b in jl_interpret_toplevel_expr (e=0x7ffdf2f40a30) at interpreter.c:27
#9  0x00007ffff79a27a3 in jl_toplevel_eval_flex (e=0x7ffdf2f40a10, fast=1) at toplevel.c:507
---Type <return> to continue, or q <return> to quit---
#10 0x00007ffff79a2ace in jl_parse_eval_all (fname=0x7ffff7a0684b "boot.jl", len=7) at toplevel.c:555
#11 0x00007ffff79a2d09 in jl_load (fname=0x7ffff7a0684b "boot.jl") at toplevel.c:594
#12 0x00007ffff7991390 in _julia_init (rel=JL_IMAGE_JULIA_HOME) at init.c:1041
#13 0x00007ffff7992ac4 in julia_init (rel=JL_IMAGE_JULIA_HOME) at task.c:253
#14 0x000000000040269d in main (argc=1, argv=0x7fffffffe4f0) at repl.c:493

[Avik Sengupta]

Given that the error is a SIGILL , it may be relevant that I am trying to run this with a vagrant/virtualbox session. I will note that I was using a default, self compiled, v0.4 julia on this box successfully.

[Avik Sengupta]

I've managed to build everything on OSX now (thanks for the fixes Keno).

I am now stuck on https://llvm.org/bugs/show_bug.cgi?id=22543 . Keno, I see that you've seen this as well, how did you move past it? (I presume I should be running <julia-home>/usr/bin/lldb , and not the version that is already installed in /usr/bin on my mac).

Regards
-
Avik

[Keno Fischer]

Hmm, I don't know, it just suddenly went away (sorry that's not particularly helpful). You could leave a comment on the LLVM bug. Maybe somebody will look at it again.

[Avik Sengupta]

Has anyone else had success building and running the debugger on OSX? I am still stuck on the "enable-objc-arc-opts registered more than once!" error, even after a refresh today. 

Regards

-

Avik

[Chris Foster]

I had this problem on linux. I worked around it by commenting out the
EnableARCOptimizations object in lib/Transforms/ObjCARC/ObjCARC.cpp.

[Avik Sengupta]

Phew, thanks Chris, that works... 

Now the next question is probably dumb, but.... 

So I'm at the lldb prompt, having recieved a SIGSEGV within my Julia program. When I view backtraces, am I expecting to see julia functions in the backtrace? Currently, all I can see are a series of jl_apply/jl_trampoline framesbt in the bt, which is similar to what I think I see in stock GDB. What am I missing here?

Thanks

-

Avik

[Keno Fischer]

Yes, you should see julia function (they are indicated by "JIT" for the file name). Did you do the edits to lldb source that I described in the original email?

[Avik Sengupta]

So I spent a couple of days rechecking and recompiling everything 

So I now have:

julia> versioninfo()

Julia Version 0.4.0-dev+5236

Commit 01abdc3* (2015-06-06 15:02 UTC)

Platform Info:

  System: Darwin (x86_64-apple-darwin13.4.0)

  CPU: Intel(R) Core(TM) i5-4258U CPU @ 2.40GHz

  WORD_SIZE: 64

  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)

  LAPACK: libopenblas

  LIBM: libopenlibm

  LLVM: libLLVM-svn

My LLVM is from 3rd June. I have checked that the llvm build artefacts in usr/lib have the correct time.  Note that my julia build includes #11515

My Make.user is 

override LLDB_VER=master

override LLVM_VER=svn

override LLVM_ASSERTIONS=1

override BUILD_LLVM_CLANG=1

override BUILD_LLDB=1

override USE_LLVM_SHLIB=1

override LLDB_DISABLE_PYTHON=1

I have the following changes: 

In Julia/src/options.h

         Enable FORCE_ELF

In LLVM

          comment out EnableARCOptimizations in  lib/Transforms/ObjCARC/ObjCARC.cpp

In LLDB

     Comment out `if (arch.GetTriple().getVendor() != llvm::Triple::Apple)` in deps/llvm-svn/tools/lldb/source/Plugins/JITLoader/GDB/JITLoaderGDB.cpp 

(D6185 has already been applied)

And with that, I can't see Julia in the lldb backtrace. Even worse, I dont see backtraces in Julia either: 

julia> sqrt(-1)

ERROR: DomainError:

So clearly I am doing something wrong, but I am at a loss as to how to troubleshoot this. 

Regards

-

Avik

[Elliot Saba]

Revisited this today, I'm getting good backtraces on OSX 10.10.3 I think, but I'm not getting any backtraces that have JIT code in them, as far as I can tell.

I have:

* enabled FORCE_ELF in src/options.h

* Commented out `if (arch.GetTriple().getVendor() != llvm::Triple::Apple) in deps/llvm-svn/tools/lldb/source/Plugins/JITLoader/GDB/JITLoaderGDB.cpp

My Make.user is:

override LLDB_VER=master

override LLVM_VER=svn

override LLVM_ASSERTIONS=1

override BUILD_LLVM_CLANG=1

override BUILD_LLDB=1

override USE_LLVM_SHLIB=1

override LLDB_DISABLE_PYTHON=1

My llvm-svn gitsha is: bc482286fcf06ba573338c5712daac85eaf48722, which is from June 11th, 2015.

If I run the export command to set LLDB_DEBUGSERVER_PATH and then run `lldb ./julia`, run `peakflops()` and CTRL-C during the execution, I get this kind of a backtrace.

-E

 

[Keno Fischer]

Try something that isn't precompiled :P (or remove sys.dylib).

[Elliot Saba]

I thought that `peakflops()` wasn't precompiled (since the first time I paused it, I was in the middle of a bunch of compilation calls), but just in case, I tried getting backtraces from this function:

function foo(N)

  a = randn(N,N)

  b = randn(N,N)

  c = zeros(N,N)

  for x in 1:N

    c += a * b

  end

  return sum(c)

end

However, it seems that lldb doesn't like to give me full backtraces from this.

-E

[Matt Bauman]

Just for anyone else that might have been making the same mistake I was:

I was having the same issues in getting the JIT frames.  It turns out that there's a purpose to building LLDB.  You need to actually run it.  That is, don't use the system LLDB or any other LLDB that you might have hanging around.  You want to use the one that ends up alongside julia in ./usr/bin.  To be specific:

$ ./usr/bin/lldb ./usr/bin/julia

(It seems like some recent LLDB/julia combination may have difficulty competing for a prompt, but you can always attach from a second terminal.  And knowing the folks involved, this will probably be transient.)

[Avik Sengupta]

>It seems like some recent LLDB/julia combination may have difficulty competing for a prompt

I found that doing a "r" from lldb has issues getting a julia prompt, but "process launch" works, and gives me a julia prompt from within lldb. 

Regards

-

Avik