fix callgraphs of 32-bit processes on 64-bit kernels.

Török Edwin

unread,

Mar 15, 2010, 11:40:02 AM3/15/10

to

Hi,

Callgraph profiling 32-bit apps on a 64-bit kernel doesn't work.
The reason is that perf_callchain_user tries to read a stackframe with 64-bit
pointers, which is wrong for a 32-bit process.

This patch fixes that, and I am almost able to get nice callgraph profiles
from 32-bit apps now! (except for some problems with perf itself when tracing
kernel modules, see [1])

Page-faults can be traced nicely (sid-ia32 is a 32-bit chroot):

$ sudo perf record -e page-faults -f -g /home/edwin/sid-ia32/usr/bin/glxgears
$ sudo perf report
...
45.33% libc-2.10.2.so [.] __GI_memcpy
|
--- __GI_memcpy
_mesa_BufferDataARB
_mesa_meta_Clear
radeonUserClear
r700Clear
_mesa_Clear
0x8049367
0x804a6ba
__libc_start_main
0x8049111

16.96% libc-2.10.2.so [.] __GI_memset
|
--- __GI_memset
_tnl_init_vertices
_swsetup_CreateContext
r600CreateContext
driCreateNewContext
dri2CreateNewContext
0xf77ab7dd
0xf7783c67
0xf778514c
0x804974f
0x804a33d
__libc_start_main
0x8049111

[1] But there is a problem with the perf tool: it can't trace addresses in
kernel modules. This is a problem regardless if the traced app is 32-bit or
64-bit; and regardless if I do callgraph profiling or not.
See the above trace, where the kernel addresses have all ffffffffa0* without a
symbol name.

If I look at /proc/kallsyms I can guess the symbols, for example
0xffffffffa01b4493 is probably this one:
ffffffffa01b4411 t r600_cs_packet_parse [radeon]

If I record/report without callgraph its the same problem:
[...]
24.01% glxgears [kernel] [k] 0xffffffffa01b4ee9
3.96% glxgears libdrm_radeon.so.1.0.0 [.] cs_gem_write_reloc
3.53% glxgears r600_dri.so [.] r700SendSPIState
2.77% glxgears r600_dri.so [.] r700DrawPrims
1.99% glxgears r600_dri.so [.] r700SendVSConsts

Kernel symbol for 0xffffffffa01b4ee9 is not shown, I can guess it is this one
(hey it was an exact match!):
ffffffffa01b4ee9 t r600_packet3_check [radeon]

It would be good if perf knew how to lookup symbols in kernel modules!

Best regards,
--Edwin
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Török Edwin

unread,

Mar 15, 2010, 12:30:02 PM3/15/10

to

BTW perf report -m -k /home/edwin/builds/linux-2.6/vmlinux doesn't show
the symbols either.

Török Edwin

unread,

Mar 16, 2010, 4:20:02 AM3/16/10

to

I always forget that, unlike every other program, perf doesn't install
by default to /usr/local!
So I was running the wrong version of perf (from an older kernel), since
perf was installed to $HOME/bin (which of course isn't in sudo's path).

| 0xf76ee270
| 0x804a6d5
| __libc_start_main
| 0x8049111
[...]

Ingo Molnar

unread,

Mar 16, 2010, 4:50:02 AM3/16/10

to

* T??r??k Edwin <edwin...@gmail.com> wrote:

> On 03/15/2010 06:23 PM, T??r??k Edwin wrote:

> > On 03/15/2010 05:34 PM, T??r??k Edwin wrote:
> >>
> >> It would be good if perf knew how to lookup symbols in kernel modules!
> >
> > BTW perf report -m -k /home/edwin/builds/linux-2.6/vmlinux doesn't show
> > the symbols either.
>
> I always forget that, unlike every other program, perf doesn't install
> by default to /usr/local!
> So I was running the wrong version of perf (from an older kernel), since
> perf was installed to $HOME/bin (which of course isn't in sudo's path).
>
> Sorry for the confusion, the 2.6.33 perf DOES know how to lookup the
> symbols:
> 9.92% glxgears [radeon] [k]
> r600_packet3_check
> |
> --- r600_packet3_check
> |
> |--96.80%-- r600_cs_parse

Ok, great!

I suspect we could install into /usr/local too. Do you want to send a patch
for that?

Ingo

Török Edwin

unread,

Mar 16, 2010, 6:00:02 AM3/16/10

to

It was confusing to install into $(HOME)/bin, especially since there was
no documentation mentioning where perf gets installed by default.
So install to /usr/local by default, as other programs do, and allow users to
override the install location by specifying the prefix explicitly.

Signed-off-by: Török Edwin <edwin...@gmail.com>
---
tools/perf/Makefile | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 2e7fa3a..8e8c199 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -216,7 +216,7 @@ STRIP ?= strip
# runtime figures out where they are based on the path to the executable.
# This can help installing the suite in a relocatable way.

-prefix = $(HOME)
+prefix = /usr/local
bindir_relative = bin
bindir = $(prefix)/$(bindir_relative)
mandir = share/man
--
1.7.0

Ingo Molnar

unread,

Mar 16, 2010, 6:20:02 AM3/16/10

to

* T??r??k Edwin <edwin...@gmail.com> wrote:

> It was confusing to install into $(HOME)/bin, especially since there was
> no documentation mentioning where perf gets installed by default.
> So install to /usr/local by default, as other programs do, and allow users to
> override the install location by specifying the prefix explicitly.
>

> Signed-off-by: T??r??k Edwin <edwin...@gmail.com>

> ---
> tools/perf/Makefile | 2 +-
> 1 files changed, 1 insertions(+), 1 deletions(-)
>
> diff --git a/tools/perf/Makefile b/tools/perf/Makefile
> index 2e7fa3a..8e8c199 100644
> --- a/tools/perf/Makefile
> +++ b/tools/perf/Makefile
> @@ -216,7 +216,7 @@ STRIP ?= strip
> # runtime figures out where they are based on the path to the executable.
> # This can help installing the suite in a relocatable way.
>
> -prefix = $(HOME)
> +prefix = /usr/local
> bindir_relative = bin
> bindir = $(prefix)/$(bindir_relative)
> mandir = share/man

Btw., we inherited that default prefix from the Git project.

Is there a way to get it into ~/bin/ if the user does not have permission to
/usr/local ? (i.e. doesnt run it as root)

That's a really convenient aspect of doing a 'make install' as user. (Which i
tend to do in most cases)

Ingo

Török Edwin

unread,

Mar 16, 2010, 6:20:02 AM3/16/10

to

On 03/16/2010 10:47 AM, Ingo Molnar wrote:
> * T??r??k Edwin <edwin...@gmail.com> wrote:
>
>> On 03/15/2010 06:23 PM, T??r??k Edwin wrote:
>>> On 03/15/2010 05:34 PM, T??r??k Edwin wrote:
>>>> It would be good if perf knew how to lookup symbols in kernel modules!
>>> BTW perf report -m -k /home/edwin/builds/linux-2.6/vmlinux doesn't show
>>> the symbols either.
>> I always forget that, unlike every other program, perf doesn't install
>> by default to /usr/local!
>> So I was running the wrong version of perf (from an older kernel), since
>> perf was installed to $HOME/bin (which of course isn't in sudo's path).
>>
>> Sorry for the confusion, the 2.6.33 perf DOES know how to lookup the
>> symbols:
>> 9.92% glxgears [radeon] [k]
>> r600_packet3_check
>> |
>> --- r600_packet3_check
>> |
>> |--96.80%-- r600_cs_parse
>
> Ok, great!

BTW the patch I sent yesterday for tracing 32-bit apps is still needed,
since that is a kernel patch, and it wasn't due to using the wrong perf.

>
> I suspect we could install into /usr/local too. Do you want to send a patch
> for that?

Sent.

BTW I think perf would need some documentation on how to install, and
what packages you need to build everything, what permissions it needs to
run, etc.

1. manpages
For example by default the manpages don't get built and installed, so
perf report --help doesn't work. It needs a 'make man', and 'make
install-man'.
This is fine, because they need asciidoc and xmlto which aren't usually
installed on every system. But there should be some documentation
mentioning this.

2. privileges
I just found out that perf works without root privileges (I just
assumed it needed root, since oprofile needs it).

3. non-working targets?
Also there are some targets in Documentation that can't be built due to
missing files, like pdf which needs a non-existent user-manual.xml.

4. unresolved symbols
Sometimes I get symbol addresses that are not resolved, like this:
57.03% :32216 7fc7dc0acfa6 [.] 0x007fc7dc0acfa6
12.39% :32216 [radeon] [k] r600_packet3_check
4.92% :32216 [radeon] [k] r600_cs_packet_parse
2.70% :32216 [radeon] [k] r600_cs_parse

Is this due to ASLR? Does perf need ASLR disabled?
That address corresponds to this:
7fc7dc07e000-7fc7dc2ef000 r-xp 00000000 fd:03 27713
/opt/xorg/lib/dri/r600_dri.so

It'd of course be nice if there was a distro package for perf, I think
I'll file a RFP in Debian for that.

Best regards,
--Edwin

Ingo Molnar

unread,

Mar 16, 2010, 6:30:02 AM3/16/10

to

* Avi Kivity <a...@redhat.com> wrote:

> What about people (like me) who do 'make && sudo make install'?

I'd like everyone to be happy :-)

In case of irreconcilable differences i prefer the creation of two parallel
universes, one for each variant. This does not seem to be such a case though:

> Can we make it position independent and derive the path from /proc/$$/exe?

Sounds like a useful approach ...

Avi Kivity

unread,

Mar 16, 2010, 6:30:02 AM3/16/10

to

On 03/16/2010 12:10 PM, Ingo Molnar wrote:

What about people (like me) who do 'make && sudo make install'?

Can we make it position independent and derive the path from /proc/$$/exe?

--
error compiling committee.c: too many arguments to function

Török Edwin

unread,

Mar 16, 2010, 6:30:02 AM3/16/10

to

On 03/16/2010 12:10 PM, Ingo Molnar wrote:

That is complicated, I usually run make as a normal user, and only do
make install as root (or sudo make install).
I do that for the kernel itself, and usually every program I build (I
don't like compiling as root).

>
> That's a really convenient aspect of doing a 'make install' as user. (Which i
> tend to do in most cases)

On 03/16/2010 12:20 PM, Avi Kivity wrote:
> What about people (like me) who do 'make && sudo make install'?
>
> Can we make it position independent and derive the path from /proc/$$/exe?
>

There is a RUNTIME_PREFIX define (undocumented...) that seems to do
something like that.
Are there any security implications of using that by default?

Best regards,
--Edwin

Ingo Molnar

unread,

Mar 16, 2010, 6:30:03 AM3/16/10

to

* T?r?k Edwin <edwin...@gmail.com> wrote:

> On 03/16/2010 10:47 AM, Ingo Molnar wrote:
> > * T??r??k Edwin <edwin...@gmail.com> wrote:
> >
> >> On 03/15/2010 06:23 PM, T??r??k Edwin wrote:
> >>> On 03/15/2010 05:34 PM, T??r??k Edwin wrote:
> >>>> It would be good if perf knew how to lookup symbols in kernel modules!
> >>> BTW perf report -m -k /home/edwin/builds/linux-2.6/vmlinux doesn't show
> >>> the symbols either.
> >> I always forget that, unlike every other program, perf doesn't install
> >> by default to /usr/local!
> >> So I was running the wrong version of perf (from an older kernel), since
> >> perf was installed to $HOME/bin (which of course isn't in sudo's path).
> >>
> >> Sorry for the confusion, the 2.6.33 perf DOES know how to lookup the
> >> symbols:
> >> 9.92% glxgears [radeon] [k]
> >> r600_packet3_check
> >> |
> >> --- r600_packet3_check
> >> |
> >> |--96.80%-- r600_cs_parse
> >
> > Ok, great!
>
> BTW the patch I sent yesterday for tracing 32-bit apps is still needed,
> since that is a kernel patch, and it wasn't due to using the wrong perf.

I've Cc:-ed Frederic for that bug. (Frederic has written a good deal of that
code)

> > I suspect we could install into /usr/local too. Do you want to send a patch
> > for that?
>
> Sent.
>
> BTW I think perf would need some documentation on how to install, and what
> packages you need to build everything, what permissions it needs to run,
> etc.

Agreed. (I've Cc:-ed Arnaldo who has a pending fix in this area.)

Ingo