[llvm-dev] ORC JIT Weekly #7 -- JITEventListener support and Swift Immediate Mode Migration

[Lang Hames via llvm-dev]

Hi All,

Just a couple of updates this week:

First: there is a preview patch up at https://reviews.llvm.org/D75838 to enable use of JITEventListeners in RTDyldObjectLinkingLayer. I've only done very preliminary testing on it, but was able to debug simple JIT'd programs on Linux using the GDB registration listener. If you've been wanting to move to ORC but held up by lack of debugger / profiler registration, chime in on the review -- I'd appreciate feedback and help with testing. I will also post a bug tomorrow for investigating similar (or hopefully better) debugger registration functionality for JITLink.

The other news is that Swift's immediate mode has moved from MCJIT  to LLJIT. The first patch for this was posted on GitHub a few weeks back, but blocked while I tracked down some failures in Swift script test cases on Linux. Those failures have now been fixed and Swift's immediate mode moved to LLJIT in https://github.com/apple/swift/commit/7a0754b127b5fdfeaa2461be44265059c1fb162d.

-- Lang.

[Frank Tetzel via llvm-dev]

Hi Lang,

> First: there is a preview patch up at https://reviews.llvm.org/D75838
> to enable use of JITEventListeners in RTDyldObjectLinkingLayer. I've
> only done very preliminary testing on it, but was able to debug
> simple JIT'd programs on Linux using the GDB registration listener.
> If you've been wanting to move to ORC but held up by lack of debugger
> / profiler registration, chime in on the review -- I'd appreciate
> feedback and help with testing. I will also post a bug tomorrow for
> investigating similar (or hopefully better) debugger registration
> functionality for JITLink.

Does this also include support for PerfEventListener for profiling with
perf?

I think, debugging and profiling support is very important for a JIT
engine. I could never get it to work with older LLVM versions. Is there
example code somewhere available?

Best regards,
Frank
_______________________________________________
LLVM Developers mailing list
llvm...@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev

[Geoff Levner via llvm-dev]

+1. Once ORC JIT supports debugging and profiling, it will have everything we need.

Thanks, Lang!

[Stefan Gränitz via llvm-dev]

Great, this should work with GDB on Linux now! Hope to find the time soon to double-check.

Note that for LLDB we have to fix the second part of this regression (source-level debugging):
https://bugs.llvm.org/show_bug.cgi?id=36209

For MachO support on macOS we need to implement `getObjectForDebug()` here:
https://code.woboq.org/llvm/llvm/lib/ExecutionEngine/RuntimeDyld/RuntimeDyldMachO.cpp.html#_ZNK12_GLOBAL__N_121LoadedMachOObjectInfo17getObjectForDebugERKN4llvm6object10ObjectFileE

Best,
Stefan

-- 
https://flowcrypt.com/pub/stefan....@gmail.com

[Lang Hames via llvm-dev]

Hi Frank,

Does this also include support for PerfEventListener for profiling with perf?

This patch just enables use of the JITEventListeners with RTDyldObjectLinkingLayer. In theory it should bring RTDyldObjectLinkingLayer users up to parity with MCJIT's support. (Note: Windows and Linux still use RTDyldObjectLinkingLayer by default, MacOS uses JITLink but can be manually switched back to RTDyldObjectLinkingLayer if event listeners are needed).

So if PerfEventListener wasn't working for you before I wouldn't expect this to fix the issue. On the other hand it makes sense to try it and file a bug, since it *should* work.

What OS have you been seeing problems with?

I think, debugging and profiling support is very important for a JIT engine. I could never get it to work with older LLVM versions. Is there example code somewhere available?

Agreed -- We definitely want this. My plan at the moment is to make sure the JITEventListener API is at parity with MCJIT to make it easy for clients to transition. The long term plan is to re-implement debugging and profiling support from the ground up via ObjectLinkingLayer::Plugins, which are much more flexible than JITEventListeners.

I was hoping to land some example code today, but hit a bug with object buffer ownership. I hope to sort this out and (re)commit the event listener support and example code tomorrow.

Cheers,

Lang.

[Andres Freund via llvm-dev]

Hi,

On 2020-03-09 21:20:44 +0100, Frank Tetzel via llvm-dev wrote:
> I think, debugging and profiling support is very important for a JIT
> engine. I could never get it to work with older LLVM versions. Is there
> example code somewhere available?

Since I added the perf listener I probably can help you with that. What
exactly was the problem you were hitting?

I don't have isolated example code, but it really shouldn't take much
code to set up the event listener. All that PG (which is what I added
the perf listener for) is doing is:
https://github.com/postgres/postgres/blob/master/src/backend/jit/llvm/llvmjit.c#L676

When taking a profile of a program you need to specify -k 1 to perf, to
have compatible clocks. E.g. you need to take it like
$ perf record -k 1 --call-graph dwarf -o /tmp/perf.data -p 22950
and then
you need to inject the JIT information into the profile:
$ perf inject -v --jit -i /tmp/perf.data -o /tmp/perf.jit.data
after that perf.jit.data is a perf profile enriched with the necessary
information, and can normally be inspected by perf report (using -i
/tmp/perf.jit.data).

If you want debug information within the JITed code (e.g. to see
associated source lines or such) that's a separate task - you need to
emit debug information, which is unrelated to the listener itself.

Greetings,

Andres Freund

[Frank Tetzel via llvm-dev]

> On 2020-03-09 21:20:44 +0100, Frank Tetzel via llvm-dev wrote:
> > I think, debugging and profiling support is very important for a JIT
> > engine. I could never get it to work with older LLVM versions. Is
> > there example code somewhere available?
>
> Since I added the perf listener I probably can help you with that.
> What exactly was the problem you were hitting?
>
> I don't have isolated example code, but it really shouldn't take much
> code to set up the event listener. All that PG (which is what I added
> the perf listener for) is doing is:
> https://github.com/postgres/postgres/blob/master/src/backend/jit/llvm/llvmjit.c#L676
>
> When taking a profile of a program you need to specify -k 1 to perf,
> to have compatible clocks. E.g. you need to take it like
> $ perf record -k 1 --call-graph dwarf -o /tmp/perf.data -p 22950
> and then
> you need to inject the JIT information into the profile:
> $ perf inject -v --jit -i /tmp/perf.data -o /tmp/perf.jit.data
> after that perf.jit.data is a perf profile enriched with the necessary
> information, and can normally be inspected by perf report (using -i
> /tmp/perf.jit.data).

I followed all these steps. During recording, dump files are created in
~/.debug/. Unfortunately, perf report never allowed me to "zoom" into
the jitted function and get an assembly view with profile information.
Either inject or report do not work for some reason. The tools work
fine when I'm using a different JIT library (AsmJit). Maybe it's just
some path issue. I do not really like that the dump files are created
in ~/.debug/.

Last time I tried it was some months ago with LLVM 9. I'll try it
shortly with LLVM trunk/master and report back.

> If you want debug information within the JITed code (e.g. to see
> associated source lines or such) that's a separate task - you need to
> emit debug information, which is unrelated to the listener itself.

Profiling on the assembly level would already be very helpful.

Best regards,
Frank

[Frank Tetzel via llvm-dev]

> Last time I tried it was some months ago with LLVM 9. I'll try it
> shortly with LLVM trunk/master and report back.

Finally, I found the time to try it out again. And it works fine in LLVM
master. I haven't tried any other LLVM version.

Profiling with perf on assembly level works fine. And so does gdb. One
can set pending breakpoints on the to be generated function and
execution stops when calling the function which shows that the jit
integration in gdb works. I have not played around with debug
information.

For the perf support, it's a bit of a pain to remember to enable
LLVM_USE_PERF when building LLVM. Why not enable it by default?

[Owen Anderson via llvm-dev]

> On Mar 31, 2020, at 4:51 AM, Frank Tetzel via llvm-dev <llvm...@lists.llvm.org> wrote:
>
>> Last time I tried it was some months ago with LLVM 9. I'll try it
>> shortly with LLVM trunk/master and report back.
>
> Finally, I found the time to try it out again. And it works fine in LLVM
> master. I haven't tried any other LLVM version.

Have you had any success in identifying the root cause of this problem? I'm being bitten by it to, and moving to LLVM master is not currently an option for my project.

--Owen

[Frank Tetzel via llvm-dev]

> >> Last time I tried it was some months ago with LLVM 9. I'll try it
> >> shortly with LLVM trunk/master and report back.
> >
> > Finally, I found the time to try it out again. And it works fine in
> > LLVM master. I haven't tried any other LLVM version.
>
> Have you had any success in identifying the root cause of this
> problem? I'm being bitten by it to, and moving to LLVM master is not
> currently an option for my project.

No, I have not investigated the issue anymore.

Regards,
Frank