kernel tracing question

[prasun3]

Hi

I was trying out the kernel tracing feature for the first time. I noticed (while looking at issue 6486) that there are some kernel tests added, but I don't see them when I run ctest. The following cmd returns an empty output. Is there anything I need to do to enable these kernel tests? I can see a total of 482 tests.

$ sudo ctest -N | grep kernel

I am able to run drcachesim with the enable_kernel_tracing option. From the view output, it looks like we expect to see only ifetch records from the kernel presently. Is this correct?

I also noticed that there are multiple sysrets per syscall. Possibly there is some other user code that's making these syscalls - some other process or DR itself? This is what I had run:

$ sudo bin64/drrun -t drcachesim -enable_kernel_tracing -offline -- /bin/ls

$ sudo clients/bin64/drraw2trace -indir drmemtrace.ls.261760.3345.dir/

$ bin64/drrun -t drcachesim -simulator_type view -infile drmemtrace.ls.261760.3345.dir/trace/drmemtrace.ls.261760.1516.trace.zip 2>&1 | grep -m 20 "syscall\|sysret\|iret"
       11299        9244:      261760 ifetch       2 byte(s) @ 0x00007f96131b5f49 0f 05                syscall  -> %rcx %r11
       11752        9693:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       12021        9962:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       13889       11570:      261760 ifetch       2 byte(s) @ 0x00007f96131b4bf3 0f 05                syscall  -> %rcx %r11
       14342       12019:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       14548       12225:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       14817       12494:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       24488       19885:      261760 ifetch       2 byte(s) @ 0x00007f96131b6df9 0f 05                syscall  -> %rcx %r11
       24941       20334:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       27742       23135:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       28011       23404:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       29869       24959:      261760 ifetch       2 byte(s) @ 0x00007f96131b6f66 0f 05                syscall  -> %rcx %r11
       30323       25408:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       34484       29569:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       34753       29838:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       35513       30593:      261760 ifetch       2 byte(s) @ 0x00007f96131b6d37 0f 05                syscall  -> %rcx %r11
       35966       31042:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       36694       31770:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       36963       32039:      261760 ifetch       3 byte(s) @ 0xffffffff8d800197 48 0f 07             sysret %rcx %r11
       37739       32806:      261760 ifetch       2 byte(s) @ 0x00007f96131b7184 0f 05                syscall  -> %rcx %r11

[Abhinav Sharma]

Hi,

> Is there anything I need to do to enable these kernel tests?

To enable all kernel tracing tests, you need to pass -DRUN_SUDO_TESTS=ON to cmake during build. Also, note that because kernel tracing tests require PT hardware support, they are not run on our Github Action workflows yet, and can be run only locally on a system that provides the required support.

> From the view output, it looks like we expect to see only ifetch records from the kernel presently. Is this correct?

That's correct. We have an in-progress design to "synthesize" memory addresses, in a way that preserves usefulness of the memory access pattern, but that is not available yet.

> I also noticed that there are multiple sysrets per syscall. Possibly there is some other user code that's making these syscalls - some other process or DR itself?

We are aware of some "noise" instructions in each system call's trace. Some of these are from the ioctl call that DR makes to resume/pause PT tracing, write calls that DR may make to output logs, or from other code that the kernel happened to execute during the system call (perhaps unrelated execution like handling of interrupts). We also have a noise filter in-progress that identifies and removes such unrelated and non-app execution from the trace.

Since kernel tracing in general is still experimental, we are using the parent issue i#5505 for tracking work on these additional features/bugs and haven't filed separate issues in the Github tracker.

Abhinav