Fun with Julia, Cxx.jl and LLDB

[Keno Fischer]

As you know I'm working on brining debugging support to julia using LLDB and while this email is not directly about that (sorry if the title got you excited), it does show of some of the infrastructure I've been building, and I think it's incredibly fun.

For background, I'm helping teach Harvard's operating systems course this semester (CS161). The idea of the class is to have the students build a significant chunk of an operating system for a MIPS processor (with some simplifications) over the course of the semester. As you might imagine a significant chunk of the time for the class is usually spent in GDB, tracking down subtle memory/synchronization/other bugs.

I took this class two years ago myself (this was before we had IJulia, Compose, Gadfly - wow, we have come quite a way since) and I ended up spending quite a significant chunk working on julia packages (in particular Terminals.jl, REPL.jl, NCurses.jl, ELF.jl and DWARF.jl) to help me visualize parts of the system to allow me to be more efficient at debugging (the final result from two years ago is here: https://www.youtube.com/watch?v=n2ErEoUtaus). This was a really hacked together project where I'd load the mips emulator as a shared library and use DWARF.jl to find appropriate offsets.

Anyway, I figured this would be the perfect target to try to out some of the infrastructure I've been building for debugging. sys161 (the MIPS emulator used in the class) exposes a GDB compatible debugging interface, so I figured I should just be able to connect LLDB to it and get going (this would have worked if LLDB came with MIPS support which it currently doesn't, so I had to quickly port it to MIPS).

The first thing I wanted to do was to do some visualization of the synchronization structure (since that's what's currently relevant), so here's what I came up with:

To explain the graph a little, what I'm basically doing is (from julia) setting a breakpoint on a thread switch and recording which thread switches to what other thread (and when threads get created or destroyed). Red lines indicate times at which a thread is blocked on another thread. Since the graph above is a little, dense, here's zooming in on one of the steps early in the graph:

From this pattern you can probably guess the scheduling algorithm used while this was running (it's a round robin scheduler running every thread in order and then going back to the first), also note that new thread getting added in the middle of the picture. Also note that blocked thread get skipped in execution:

​​​

I put up the code at https://gist.github.com/Keno/ef9b38f5b50b666aff87 for those interested, but I'll have some work to do, to get everything organized properly in the right places, but I was pretty excited making it work, so I wanted to share it now ;). I'm hoping the debugging work, once it's completed, will enable folks to think about new ways to explore what's really going inside the system and make this kind of thing very easy to do (the code is still a little more clunky than I want it to be, but I think the 150 lines it is currently isn't terrible either). Of course one could probably also do this in many other language, but I can't think of any language in which it would be this easy. I didn't have to write a single line of glue C or C++ to interface with LLDB, since I used Cxx.jl and everything basically just works. Also, as always Compose/Gadfly are fantastic of course ;).

Lastly, I need to of course give credit to the LLVM/Clang/LLDB folks, whose work makes all of this possible. The example above, doesn't yet add much on top of the foundation (so all credit to them ;)! Let's some cool stuff on top if it.

Keno

[Elliot Saba]

> (this would have worked if LLDB came with MIPS support which it currently doesn't, so I had to quickly port it to MIPS).

Sounds like a nice little project to do before breakfast in the morning.

In all seriousness, that's pretty cool, and a great testament to the power of Cxx.jl.  :P

-E