Bug#911815: /usr/bin/perf_4.18: Please build perf against libbfd

[Mike Hommey]

Package: linux-perf-4.18
Version: 4.18.10-2
Severity: wishlist
File: /usr/bin/perf_4.18

Dear Maintainer,

Running e.g. perf report with dwarf call graph info can take a long time
depending on the size of the profile and the size of dwarf info in the
binaries being profiled. That's because each address in each library is
handled by forking and executing a new addr2line process. Each addr2line
process has to parse the dwarf info of the library it's given just to
find the location of one address. Multiply by the number of addresses,
and this can quickly become ridiculous.

Perf, however, has an alternative implementation that just uses libbfd,
so it would be much faster than spawning a large amount of new
processes, each with a large overhead.

Mike


-- System Information:
Debian Release: buster/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'testing'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.18.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=ja_JP.UTF-8, LC_CTYPE=ja_JP.UTF-8 (charmap=UTF-8), LANGUAGE=ja_JP.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-perf-4.18 depends on:
ii libbabeltrace1 1.5.6-1
ii libc6 2.27-6
ii libdw1 0.170-0.5
ii libelf1 0.170-0.5
ii liblzma5 5.2.2-1.3
ii libnuma1 2.0.12-1
ii libperl5.26 5.26.2-7+b1
ii libpython3.6 3.6.7-1
ii libslang2 2.3.2-1+b1
ii libunwind8 1.2.1-8
ii zlib1g 1:1.2.11.dfsg-1

Versions of packages linux-perf-4.18 recommends:
ii linux-base 4.5

Versions of packages linux-perf-4.18 suggests:
pn linux-doc-4.18 <none>

-- no debconf information

[Mike Hommey]

On Thu, Oct 25, 2018 at 03:20:11PM +0900, Mike Hommey wrote:
> Package: linux-perf-4.18
> Version: 4.18.10-2
> Severity: wishlist
> File: /usr/bin/perf_4.18
>
> Dear Maintainer,
>
> Running e.g. perf report with dwarf call graph info can take a long time
> depending on the size of the profile and the size of dwarf info in the
> binaries being profiled. That's because each address in each library is
> handled by forking and executing a new addr2line process. Each addr2line
> process has to parse the dwarf info of the library it's given just to
> find the location of one address. Multiply by the number of addresses,
> and this can quickly become ridiculous.
>
> Perf, however, has an alternative implementation that just uses libbfd,
> so it would be much faster than spawning a large amount of new
> processes, each with a large overhead.

I found https://salsa.debian.org/kernel-team/linux/blob/master/debian/rules.d/tools/perf/Makefile#L27-31

This sucks badly :(
I have some massive perf data that take *hours* to deal with without
libbfd. With a reduced perf data, this is the kind of difference this
makes:

$ time perf script > /dev/null # addr2line
real 3m8.718s
user 1m12.606s
sys 1m56.649s

$ time perf script > /dev/null # libbfd
real 0m4.141s
user 0m3.425s
sys 0m0.894s

Mike

[Ben Hutchings]

Control: tag -1 wontfix

On Thu, 2018-10-25 at 18:55 +0900, Mike Hommey wrote:
[...]
> I found https://salsa.debian.org/kernel-team/linux/blob/master/debian/rules.d/tools/perf/Makefile#L27-31
[...]

For future reference, that's the comment:

# perf can link against libbfd if available, but the result is
# undistributable as they are licenced under GPL v2 and v3+
# respectively. Override detection of libbfd and insist that
# cplus_demangle() can be found in libiberty (LGPL v2.1+).

Tagging this wontfix since we can't fix that problem.

Ben.

--
Ben Hutchings
The obvious mathematical breakthrough [to break modern encryption]
would be development of an easy way to factor large prime numbers.
- Bill Gates

[Debian Bug Tracking System]

Processing control commands:

> tag -1 wontfix
Bug #911815 [linux-perf-4.18] /usr/bin/perf_4.18: Please build perf against libbfd
Added tag(s) wontfix.

--
911815: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=911815
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

[Steinar H. Gunderson]

On Fri, Oct 26, 2018 at 01:12:51AM +0100, Ben Hutchings wrote:
> For future reference, that's the comment:
>
> # perf can link against libbfd if available, but the result is
> # undistributable as they are licenced under GPL v2 and v3+
> # respectively. Override detection of libbfd and insist that
> # cplus_demangle() can be found in libiberty (LGPL v2.1+).
>
> Tagging this wontfix since we can't fix that problem.

But we can probably make the addr2line solution much faster?
perf runs:

scnprintf(cmd, sizeof(cmd), "addr2line -e %s %016"PRIx64,
dso_name, addr);

fp = popen(cmd, "r");

but the normal way of running addr2line is to run it and then start feeding
it addresses on stdin (ie. don't start the program anew for each and every
address we want to look up). I haven't tried, but it sounds like that would
reduce overhead significantly?

I also don't know if there's a cache somewhere in front of this? It seems to
look up the same addresses over and over and over again, at least in my case
(decoding a processor trace).

/* Steinar */
--
Homepage: https://www.sesse.net/

[Tony Garnock-Jones]

Dear Steinar, Ben, Mike, others:

On Tue, 18 Aug 2020 00:35:53 +0200 "Steinar H. Gunderson"
<se...@debian.org> wrote:
> But we can probably make the addr2line solution much faster?
> perf runs:

> [...]

> but the normal way of running addr2line is to run it and then start feeding
> it addresses on stdin (ie. don't start the program anew for each and every
> address we want to look up). I haven't tried, but it sounds like that would
> reduce overhead significantly?

Thanks, Steinar, for your suggestion! I've written a patch against the
non-libbfd code in perf to try it out.

It works very well. What used to take endless minutes now takes a few
seconds.

Please find my small patch (against "apt source linux-perf-5.10") attached.

I have also started the process of submitting it upstream.

Best Regards,
Tony

[Steinar H. Gunderson]

On Thu, Sep 09, 2021 at 12:54:50PM +0200, Tony Garnock-Jones wrote:
> Thanks, Steinar, for your suggestion! I've written a patch against the
> non-libbfd code in perf to try it out.
>
> It works very well. What used to take endless minutes now takes a few
> seconds.

Thanks for doing this! I can confirm; I tested this on “perf report” against
a perf.data with DWARF tracebacks, with perf 5.13.4 (the same file every
time), and here are the results:

non-bfd, without patch: 7m59s
non-bfd, with patch: 15s
bfd: 15s

[Tony Garnock-Jones]

Just to follow up again, here's an improved version of the patch that
doesn't hand-roll quite so much socketpair/fork/exec code, reusing
existing tools/ code instead!

Also, here's the upstream discussion of the patch on the
linux-perf-users mailing list:
https://lore.kernel.org/linux-perf-users/20210910102307....@leastfixedpoint.com/T/#u

Cheers,
Tony

[Salvatore Bonaccorso]

Hi,

Unless mistaken, then this has landed as
https://git.kernel.org/linus/be8ecc57f180415e8a7c1cc5620c5236be2a7e56
in mainline.

Regards,
Salvatore

[Debian Bug Tracking System]

Your message dated Mon, 15 Nov 2021 10:00:20 +0000
with message-id <E1mmYmi-...@fasolo.debian.org>
and subject line Bug#911815: fixed in linux 5.15.2-1~exp1
has caused the Debian Bug report #911815,
regarding /usr/bin/perf_4.18: Please build perf against libbfd
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ow...@bugs.debian.org
immediately.)