perf probe: request: Better message when no debug info is found on vmlinux

[Arnaldo Carvalho de Melo]

Hi Masami,

I was trying to figure out why sys_perf_event_open was returning
EACCES when trying to record, as a non priviledged user, a 'ping'
process started by this user, while it has no problem doing the same
thing if the monitored process is 'sleep' (reusing of ptrace perms seems
to be what is causing this, but I digress), when I stumbled in something
annoying:

[root@ssdandy ~]# perf probe -v -L icmp_rcv
Looking at the vmlinux_path (6 entries long)
Using /lib/modules/3.16.0+/build/vmlinux for symbols
Failed to open debuginfo file.
Error: Failed to show lines. (-2)
[root@ssdandy ~]#

Huh? Why? So I went to look to see why it was "failing to open debuginfo
file":

[root@ssdandy ~]# ls -la /lib/modules/3.16.0+/build/vmlinux
-rwxrwxr-x. 3 acme acme 22505817 Aug 13 11:54
/lib/modules/3.16.0+/build/vmlinux
[root@ssdandy ~]# ls -la /lib/modules/3.16.0+/build
lrwxrwxrwx. 1 root root 29 Aug 13 12:10 /lib/modules/3.16.0+/build ->
/home/acme/git/build/v3.16.0+

The file is there, but... I forgot I had disabled CONFIG_DEBUG_INFO on
this machine:

[root@ssdandy ~]# ls -la /lib/modules/3.16.0+/build/.config
-rw-rw-r--. 1 acme acme 136217 Aug 13 11:40
/lib/modules/3.16.0+/build/.config
[root@ssdandy ~]# grep CONFIG_DEBUG_INFO
/lib/modules/3.16.0+/build/.config
# CONFIG_DEBUG_INFO is not set
[root@ssdandy ~]#

So, only the CFI ELF section was there, not the other with the -g stuff:

[root@ssdandy ~]# readelf -SW /lib/modules/3.16.0+/build/vmlinux | grep \.debug_
[29] .debug_frame PROGBITS 0000000000000000 1266030 0021d0 00 0 0 8
[root@ssdandy ~]#

So, I suggest that we improve that error message to be more clear:

[root@ssdandy ~]# perf probe -L icmp_rcv
Failed to open debuginfo file.
Error: The /lib/modules/3.16.0+/build/vmlinux file has no
debugging information, rebuild with CONFIG_DEBUG_INFO=y

[root@ssdandy ~]#

Since it knows it is a kernel image file, and that the config option to
enable it is CONFIG_DEBUG_INFO.

Thanks,

- Arnaldo

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Masami Hiramatsu]

OK, it seems better for me too.
It will be a bit different message, but I'm sure to mention
that config option for users.

Thank you!

--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hi...@hitachi.com

[Masami Hiramatsu]

Warn user to rebuild target with debuginfo when the perf probe
fails to find debug information in the target binary.
Without this, perf probe just reports the failure, but it's
no hint for users. This gives more hint for users.

Without this,

$ strip perf
$ ./perf probe -x perf -L argv_split

Failed to open debuginfo file.
Error: Failed to show lines.

With this,

$ strip perf
$ ./perf probe -x perf -L argv_split
The /home/fedora/ksrc/linux-3/tools/perf/perf file has no debug information, rebuild with -g.

Error: Failed to show lines.

The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
if the target is the kernel or a kernel module.

Signed-off-by: Masami Hiramatsu <masami.hi...@hitachi.com>
Reported-by: Arnaldo Carvalho de Melo <ac...@kernel.org>
Cc: Jiri Olsa <jo...@redhat.com>
Cc: Namhyung Kim <namh...@gmail.com>
Cc: David Ahern <dsa...@gmail.com>
---
tools/perf/util/probe-event.c | 40 ++++++++++++++++++++++------------------
1 file changed, 22 insertions(+), 18 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 784ea42..57eb343 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -258,21 +258,32 @@ static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)
#ifdef HAVE_DWARF_SUPPORT

/* Open new debuginfo of given module */
-static struct debuginfo *open_debuginfo(const char *module)
+static struct debuginfo *open_debuginfo(const char *module, bool silent)
{
const char *path = module;
+ struct debuginfo *ret;

if (!module || !strchr(module, '/')) {
path = kernel_get_module_path(module);
if (!path) {
- pr_err("Failed to find path of %s module.\n",
- module ?: "kernel");
+ if (!silent)
+ pr_err("Failed to find path of %s module.\n",
+ module ?: "kernel");
return NULL;
}
}
- return debuginfo__new(path);
+ ret = debuginfo__new(path);
+ if (!ret && !silent) {
+ pr_warning("The %s file has no debug information, ", path);
+ if (!module || !strtailcmp(path, ".ko"))
+ pr_warning("rebuild with CONFIG_DEBUG_INFO=y.\n");
+ else
+ pr_warning("rebuild with -g.\n");
+ }
+ return ret;
}

+
static int get_text_start_address(const char *exec, unsigned long *address)
{
Elf *elf;
@@ -333,15 +344,13 @@ static int find_perf_probe_point_from_dwarf(struct probe_trace_point *tp,
pr_debug("try to find information at %" PRIx64 " in %s\n", addr,
tp->module ? : "kernel");

- dinfo = open_debuginfo(tp->module);
+ dinfo = open_debuginfo(tp->module, verbose == 0);
if (dinfo) {
ret = debuginfo__find_probe_point(dinfo,
(unsigned long)addr, pp);
debuginfo__delete(dinfo);
- } else {
- pr_debug("Failed to open debuginfo at 0x%" PRIx64 "\n", addr);
+ } else
ret = -ENOENT;
- }

if (ret > 0) {
pp->retprobe = tp->retprobe;
@@ -457,13 +466,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,
struct debuginfo *dinfo;
int ntevs, ret = 0;

- dinfo = open_debuginfo(target);
+ dinfo = open_debuginfo(target, !need_dwarf);

if (!dinfo) {
- if (need_dwarf) {
- pr_warning("Failed to open debuginfo file.\n");
+ if (need_dwarf)
return -ENOENT;
- }
pr_debug("Could not open debuginfo. Try to use symbols.\n");
return 0;
}
@@ -620,11 +627,9 @@ static int __show_line_range(struct line_range *lr, const char *module)
char *tmp;

/* Search a line range */
- dinfo = open_debuginfo(module);
- if (!dinfo) {
- pr_warning("Failed to open debuginfo file.\n");
+ dinfo = open_debuginfo(module, false);
+ if (!dinfo)
return -ENOENT;
- }

ret = debuginfo__find_line_range(dinfo, lr);
debuginfo__delete(dinfo);
@@ -772,9 +777,8 @@ int show_available_vars(struct perf_probe_event *pevs, int npevs,
if (ret < 0)
return ret;

- dinfo = open_debuginfo(module);
+ dinfo = open_debuginfo(module, false);
if (!dinfo) {
- pr_warning("Failed to open debuginfo file.\n");
ret = -ENOENT;
goto out;

[Arnaldo Carvalho de Melo]

Em Thu, Aug 14, 2014 at 06:29:38PM +0000, Masami Hiramatsu escreveu:
> Warn user to rebuild target with debuginfo when the perf probe
> fails to find debug information in the target binary.
> Without this, perf probe just reports the failure, but it's
> no hint for users. This gives more hint for users.
>
> Without this,

That was quick!

The changelog also was nicely written, stating the problem, then
providing an example to exhibit the problem before the patch, then the
end result, with the intended behaviour, after the patch.

Thanks a lot!

- Arnaldo

[Brendan Gregg]

On Thu, Aug 14, 2014 at 11:29 AM, Masami Hiramatsu
<masami.hi...@hitachi.com> wrote:
[...]

> The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
> if the target is the kernel or a kernel module.

Thanks, definitely an improvement! Should the kernel message also
mention kernel debuginfo packages? Depends on the distribution and
environment, but I think for some users the solution is to add the
package.

Brendan

[Arnaldo Carvalho de Melo]

Em Thu, Aug 14, 2014 at 01:07:28PM -0700, Brendan Gregg escreveu:
> On Thu, Aug 14, 2014 at 11:29 AM, Masami Hiramatsu
> <masami.hi...@hitachi.com> wrote:
> [...]
> > The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
> > if the target is the kernel or a kernel module.

> Thanks, definitely an improvement! Should the kernel message also
> mention kernel debuginfo packages? Depends on the distribution and
> environment, but I think for some users the solution is to add the
> package.

Yeah, something like what is suggested by gdb and documented here:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Developer_Guide/intro.debuginfo.html

--------------------------------------------------------------------
In some cases (such as loading a core file), GDB does not know the
name, version, or release of a name-debuginfo-version-release.rpm
package; it only knows the build-id. In such cases, GDB suggests a
different command:

gdb -c ./core
[...]
Missing separate debuginfo for the main executable filename
Try: yum --disablerepo='*' --enablerepo='*debug*' install /usr/lib/debug/.build-id/ef/dd0b5e69b0742fa5e5bad0771df4d1df2459d1
---------------------------------------------------------------------

This is something I want to have eventually, i.e. to have per distro
plugins to automatically download packages required for some features,
like probing and annotation, for instance.

E.g:

[acme@zoo ~]$ perf record usleep 1
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 0.014 MB perf.data (~615 samples) ]
[acme@zoo ~]$ perf buildid-list
34412145145a7561db0d0063a925c17d9af67a1f [kernel.kallsyms]
efc76c94d401b3b7f0f8ecb893c829d82f10e4b2 /usr/lib64/libc-2.18.so
[acme@zoo ~]$ sudo yum --disablerepo='*' --enablerepo='*debug*' install /usr/lib/debug/.build-id/34/412145145a7561db0d0063a925c17d9af67a1f
Loaded plugins: auto-update-debuginfo, langpacks, refresh-packagekit
Resolving Dependencies
--> Running transaction check
---> Package kernel-debuginfo.x86_64 0:3.15.8-200.fc20 will be installed
--> Processing Dependency: kernel-debuginfo-common-x86_64 =
3.15.8-200.fc20 for package: kernel-debuginfo-3.15.8-200.fc20.x86_64
--> Running transaction check
---> Package kernel-debuginfo-common-x86_64.x86_64 0:3.15.8-200.fc20
will be installed
--> Finished Dependency Resolution

Dependencies Resolved
<SNIP>

- Arnaldo

[Masami Hiramatsu]

(2014/08/15 10:07), Arnaldo Carvalho de Melo wrote:
> Em Thu, Aug 14, 2014 at 01:07:28PM -0700, Brendan Gregg escreveu:
>> On Thu, Aug 14, 2014 at 11:29 AM, Masami Hiramatsu
>> <masami.hi...@hitachi.com> wrote:
>> [...]
>>> The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
>>> if the target is the kernel or a kernel module.
>
>> Thanks, definitely an improvement! Should the kernel message also
>> mention kernel debuginfo packages? Depends on the distribution and
>> environment, but I think for some users the solution is to add the
>> package.

I see, and at least fedora/rhel has debuginfo for all packages.
So, not only for the kernel, but also for user applications,
we'll need to do that.

> Yeah, something like what is suggested by gdb and documented here:
>
> https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Developer_Guide/intro.debuginfo.html
>
> --------------------------------------------------------------------
> In some cases (such as loading a core file), GDB does not know the
> name, version, or release of a name-debuginfo-version-release.rpm
> package; it only knows the build-id. In such cases, GDB suggests a
> different command:
>
> gdb -c ./core
> [...]
> Missing separate debuginfo for the main executable filename
> Try: yum --disablerepo='*' --enablerepo='*debug*' install /usr/lib/debug/.build-id/ef/dd0b5e69b0742fa5e5bad0771df4d1df2459d1
> ---------------------------------------------------------------------

ah, that's nice :)

>
> This is something I want to have eventually, i.e. to have per distro
> plugins to automatically download packages required for some features,
> like probing and annotation, for instance.

Yeah, however, it depends on the distro. AFAIK, ubuntu provides
debuginfo package only for the kernel. So, at this point, I think
what we can do is just say "please install debuginfo package"
as below.

$ ./perf probe -x perf -L argv_split
The /home/fedora/ksrc/linux-3/tools/perf/perf file has no debug information, rebuild with -g.

Or install appropriate debuginfo package.

Error: Failed to show lines.

Thank you,

--
Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hi...@hitachi.com

[Masami Hiramatsu]

Warn user to rebuild target with debuginfo when the perf probe
fails to find debug information in the target binary.
Without this, perf probe just reports the failure, but it's
no hint for users. This gives more hint for users.

Without this,

$ strip perf

$ ./perf probe -x perf -L argv_split

Failed to open debuginfo file.

Error: Failed to show lines.

With this,

$ strip perf

$ ./perf probe -x perf -L argv_split

The /home/fedora/ksrc/linux-3/tools/perf/perf file has no debug information.
Rebuild with -g, or install an appropriate debuginfo pacakge.

Error: Failed to show lines.

The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
if the target is the kernel or a kernel module.

Signed-off-by: Masami Hiramatsu <masami.hi...@hitachi.com>
Reported-by: Arnaldo Carvalho de Melo <ac...@kernel.org>
Cc: Jiri Olsa <jo...@redhat.com>
Cc: Namhyung Kim <namh...@gmail.com>
Cc: David Ahern <dsa...@gmail.com>

Cc: Brendan Gregg <brendan...@gmail.com>
---
tools/perf/util/probe-event.c | 41 +++++++++++++++++++++++------------------
1 file changed, 23 insertions(+), 18 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index 784ea42..9a29c72 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -258,21 +258,33 @@ static void clear_probe_trace_events(struct probe_trace_event *tevs, int ntevs)

#ifdef HAVE_DWARF_SUPPORT

/* Open new debuginfo of given module */
-static struct debuginfo *open_debuginfo(const char *module)
+static struct debuginfo *open_debuginfo(const char *module, bool silent)
{
const char *path = module;
+ struct debuginfo *ret;

if (!module || !strchr(module, '/')) {
path = kernel_get_module_path(module);
if (!path) {
- pr_err("Failed to find path of %s module.\n",
- module ?: "kernel");
+ if (!silent)
+ pr_err("Failed to find path of %s module.\n",
+ module ?: "kernel");
return NULL;
}
}
- return debuginfo__new(path);
+ ret = debuginfo__new(path);
+ if (!ret && !silent) {

+ pr_warning("The %s file has no debug information.\n", path);

+ if (!module || !strtailcmp(path, ".ko"))

+ pr_warning("Rebuild with CONFIG_DEBUG_INFO=y, ");
+ else
+ pr_warning("Rebuild with -g, ");
+ pr_warning("or install an appropriate debuginfo pacakge.\n");

+ }
+ return ret;
}

+
static int get_text_start_address(const char *exec, unsigned long *address)
{
Elf *elf;

@@ -333,15 +345,13 @@ static int find_perf_probe_point_from_dwarf(struct probe_trace_point *tp,

pr_debug("try to find information at %" PRIx64 " in %s\n", addr,
tp->module ? : "kernel");

- dinfo = open_debuginfo(tp->module);
+ dinfo = open_debuginfo(tp->module, verbose == 0);
if (dinfo) {
ret = debuginfo__find_probe_point(dinfo,
(unsigned long)addr, pp);
debuginfo__delete(dinfo);
- } else {
- pr_debug("Failed to open debuginfo at 0x%" PRIx64 "\n", addr);
+ } else
ret = -ENOENT;
- }

if (ret > 0) {
pp->retprobe = tp->retprobe;

@@ -457,13 +467,11 @@ static int try_to_find_probe_trace_events(struct perf_probe_event *pev,

struct debuginfo *dinfo;
int ntevs, ret = 0;

- dinfo = open_debuginfo(target);
+ dinfo = open_debuginfo(target, !need_dwarf);

if (!dinfo) {
- if (need_dwarf) {
- pr_warning("Failed to open debuginfo file.\n");
+ if (need_dwarf)
return -ENOENT;
- }
pr_debug("Could not open debuginfo. Try to use symbols.\n");
return 0;
}

@@ -620,11 +628,9 @@ static int __show_line_range(struct line_range *lr, const char *module)

char *tmp;

/* Search a line range */
- dinfo = open_debuginfo(module);
- if (!dinfo) {
- pr_warning("Failed to open debuginfo file.\n");
+ dinfo = open_debuginfo(module, false);
+ if (!dinfo)
return -ENOENT;
- }

ret = debuginfo__find_line_range(dinfo, lr);
debuginfo__delete(dinfo);

@@ -772,9 +778,8 @@ int show_available_vars(struct perf_probe_event *pevs, int npevs,

if (ret < 0)
return ret;

- dinfo = open_debuginfo(module);
+ dinfo = open_debuginfo(module, false);
if (!dinfo) {
- pr_warning("Failed to open debuginfo file.\n");
ret = -ENOENT;
goto out;
}

[Masami Hiramatsu]

Here is v2 patch, which I've added "or install an appropriate debuginfo pacakge." :)

Thank you,

Masami HIRAMATSU
Software Platform Research Dept. Linux Technology Research Center
Hitachi, Ltd., Yokohama Research Laboratory
E-mail: masami.hi...@hitachi.com

[Brendan Gregg]

On Thu, Aug 14, 2014 at 6:51 PM, Masami Hiramatsu
<masami.hi...@hitachi.com> wrote:
> Here is v2 patch, which I've added "or install an appropriate debuginfo pacakge." :)

[...]

Looks good, thanks.

Brendan

[Arnaldo Carvalho de Melo]

Em Thu, Aug 14, 2014 at 08:39:51PM -0700, Brendan Gregg escreveu:
> On Thu, Aug 14, 2014 at 6:51 PM, Masami Hiramatsu
> <masami.hi...@hitachi.com> wrote:
> > Here is v2 patch, which I've added "or install an appropriate debuginfo pacakge." :)
> [...]
>
> Looks good, thanks.

Thanks, applied.

- Arnaldo

[tip-bot for Masami Hiramatsu]

Commit-ID: 92561cb7883194714475c7a7775a11a9c40f75cb
Gitweb: http://git.kernel.org/tip/92561cb7883194714475c7a7775a11a9c40f75cb
Author: Masami Hiramatsu <masami.hi...@hitachi.com>
AuthorDate: Fri, 15 Aug 2014 01:44:32 +0000
Committer: Arnaldo Carvalho de Melo <ac...@redhat.com>
CommitDate: Fri, 15 Aug 2014 10:32:43 -0300

perf probe: Warn user to rebuild target with debuginfo

Warn user to rebuild target with debuginfo when the perf probe fails to
find debug information in the target binary.

Without this, perf probe just reports the failure, but it's no hint for
users. This gives more hint for users.

Without this:

$ strip perf
$ ./perf probe -x perf -L argv_split
Failed to open debuginfo file.
Error: Failed to show lines.

With this:

$ strip perf
$ ./perf probe -x perf -L argv_split
The /home/fedora/ksrc/linux-3/tools/perf/perf file has no debug information.

Rebuild with -g, or install an appropriate debuginfo package.

Error: Failed to show lines.

The "rebuild with ..." part changes to "rebuild with CONFIG_DEBUG_INFO"
if the target is the kernel or a kernel module.

Suggested-by: Arnaldo Carvalho de Melo <ac...@kernel.org>
Signed-off-by: Masami Hiramatsu <masami.hi...@hitachi.com>
Cc: Brendan Gregg <brendan...@gmail.com>

Cc: David Ahern <dsa...@gmail.com>
Cc: Jiri Olsa <jo...@redhat.com>
Cc: Namhyung Kim <namh...@gmail.com>

Cc: yrl.pp-m...@hitachi.com
Link: http://lkml.kernel.org/r/20140815014432.2...@kbuild-fedora.novalocal
Signed-off-by: Arnaldo Carvalho de Melo <ac...@redhat.com>

---
tools/perf/util/probe-event.c | 41 +++++++++++++++++++++++------------------
1 file changed, 23 insertions(+), 18 deletions(-)

diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c

index 784ea42..ac15ff7 100644

+ pr_warning("or install an appropriate debuginfo package.\n");