eBPF tools and troubleshooting?

[Igor Cherkaev]

Hello,

With kernel 5.4 delivered to stable channel, what's the right way to test/troubleshoot BPF? I tried going to toolbox, install kernel-devel and link modules, and install bcc-tools, but they wouldn't work. What's the right way to debug/troubleshoot/develop BPF stuff on Flatcar Linux?

Thank you.

[Alban Crequy]

Hi Igor,

> With kernel 5.4 delivered to stable channel, what's the right way to test/troubleshoot BPF? I tried going to toolbox, install kernel-devel and link modules, and install bcc-tools, but they wouldn't work. What's the right way to debug/troubleshoot/develop BPF stuff on Flatcar Linux?

You can run the following docker command:

docker run -ti --rm --privileged -v /lib/modules:/lib/modules -v
/sys/fs/bpf:/sys/fs/bpf --pid=host kinvolk/bcc:latest
/usr/share/bcc/tools/tcptracer

The volume /lib/modules is necessary to let bcc access the kernel
headers. On Flatcar, the kernel headers are directly available in
/lib/modules (it is not a symlink like on other distros) so no need
to mount /usr/src.

You can also use the toolbox, but you also have to share /lib/modules:

Add the following in /etc/default/toolbox or in ${HOME}/.toolboxrc:

TOOLBOX_BIND="--bind=/:/media/root --bind=/usr:/media/root/usr
--bind=/run:/media/root/run --bind=/sys/fs/bpf:/sys/fs/bpf
--bind=/lib/modules:/lib/modules
--bind=/sys/kernel/debug:/sys/kernel/debug"
TOOLBOX_FLAGS="--system-call-filter=bpf --system-call-filter=perf_event_open"

We could add those flags by default in toolbox upstream. We started
with https://github.com/flatcar-linux/toolbox/pull/3/files but as you
see, we still have to add /lib/modules, debugfs and perf_event_open.

Then, run toolbox:

$ toolbox
# dnf install bcc-tools
# /usr/share/bcc/tools/tcptracer

See https://github.com/flatcar-linux/toolbox/blob/master/toolbox if
you're interested to see how the flags are used.

Note that the Fedora 32 toolbox and the kinvolk/bcc Docker image use
different versions of bcc-tools, so there could be different
behaviour. For example, at the moment, I noticed that execsnoop works
with kinvolk/bcc but not with the Fedora 32 toolbox.

In addition to bcc, you can also try bpftrace:

docker run -ti --rm --privileged -v /lib/modules:/lib/modules -v
/sys/fs/bpf:/sys/fs/bpf -v /sys/kernel/debug:/sys/kernel/debug
--pid=host quay.io/iovisor/bpftrace bpftrace -e
'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

And for BPF tracing in Kubernetes, you can look at Inspektor Gadget
and kubectl-trace:
https://github.com/kinvolk/inspektor-gadget
https://github.com/iovisor/kubectl-trace

Cheers,
Alban

[Igor Cherkaev]

Thank you, Alban! You're a life saver!