sample-bt fails
487 views
Skip to first unread message
S
Jun 25, 2015, 6:53:32 PM6/25/15
to openre...@googlegroups.com
Hi agentzh,
Thanks for the cool sample-bt tool. When I use it with some applications, sample-bt fails with "ERROR: too many pending (error) messages". Could you please comment how we can fix this? Detailed output is as below. Thanks!
....
undefined symbol: _ZN5boost15program_options6detail7cmdline5styleEi (/home/..) --> I have complied both boost and google protobuf, but I still see the underfined symbol issues (but stap seems to be able to grab the symbols anyway)
undefined symbol: _ZNK6google8protobuf7Message9SpaceUsedEv (/home/..)
....
WARNING: Tracing 78768 (/home/...) in both user-space and kernel-space...
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search for stap_c96f67434fd8f68fcbe858907933cae_84312
WARNING: DWARF expression stack underflow in CFI
WARNING: too many pending (warning) messages
ERROR: too many pending (error) messages
WARNING: Number of errors: 6, skipped probes: 10007
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run failed. [man error::pass5]
Yichun Zhang (agentzh)
Jun 25, 2015, 11:38:23 PM6/25/15
to openresty-en
Hello!
On Fri, Jun 26, 2015 at 6:53 AM, S wrote:
> Thanks for the cool sample-bt tool. When I use it with some applications,
> sample-bt fails with "ERROR: too many pending (error) messages". Could you
> please comment how we can fix this? Detailed output is as below. Thanks!
Well, some suggestions:
1. Try the latest version of systemtap and elfutils from their
official releases (please see http://openresty.org/#BuildSystemtap for
instructions)
2. Ensure your boost and protobuf builds have debug symbols enabled by
passing -g to gcc/g++ while compiling *every* single source file in
those projects.
3. Ensure your gcc/g++ version is recent enough. The quality of the
debug symbols generated by gcc 4.5+ is significantly better than older
versions. Generally, the newer, the better.
4. Try to use shorter sampling intervals if it makes sense.
> undefined symbol: _ZN5boost15program_options6detail7cmdline5styleEi (/home/..) --> ...
> undefined symbol: _ZNK6google8protobuf7Message9SpaceUsedEv (/home/..)
Using c++filt can get the original C++ names:
$ echo _ZN5boost15program_options6detail7cmdline5styleEi | c++filt
boost::program_options::detail::cmdline::style(int)
$ echo _ZNK6google8protobuf7Message9SpaceUsedEv | c++filt
google::protobuf::Message::SpaceUsed() const
These names may give you directions.
Generally, these "undefined symbol" warnings can be safely ignored.
Are you sure you're not getting other things?
Regards,
-agentzh
On Fri, Jun 26, 2015 at 6:53 AM, S wrote:
> Thanks for the cool sample-bt tool. When I use it with some applications,
> sample-bt fails with "ERROR: too many pending (error) messages". Could you
> please comment how we can fix this? Detailed output is as below. Thanks!
1. Try the latest version of systemtap and elfutils from their
official releases (please see http://openresty.org/#BuildSystemtap for
instructions)
2. Ensure your boost and protobuf builds have debug symbols enabled by
passing -g to gcc/g++ while compiling *every* single source file in
those projects.
3. Ensure your gcc/g++ version is recent enough. The quality of the
debug symbols generated by gcc 4.5+ is significantly better than older
versions. Generally, the newer, the better.
4. Try to use shorter sampling intervals if it makes sense.
> undefined symbol: _ZN5boost15program_options6detail7cmdline5styleEi (/home/..) --> ...
> undefined symbol: _ZNK6google8protobuf7Message9SpaceUsedEv (/home/..)
Using c++filt can get the original C++ names:
$ echo _ZN5boost15program_options6detail7cmdline5styleEi | c++filt
boost::program_options::detail::cmdline::style(int)
$ echo _ZNK6google8protobuf7Message9SpaceUsedEv | c++filt
google::protobuf::Message::SpaceUsed() const
These names may give you directions.
Generally, these "undefined symbol" warnings can be safely ignored.
Are you sure you're not getting other things?
Regards,
-agentzh
Sheng Li
Jun 26, 2015, 2:47:17 PM6/26/15
to openre...@googlegroups.com
Thanks, agentzh!
The results I shared were with gcc/g++ is 4.8.3 and systemtap 2.6/0.160 (both are very very recent). Both boost and protobuffer were compiled with -g option. And I still had those problems.
What is the best way to check whether the debug symbols are generated correctly? Using nm or gdb or something better?
The only messages stap thrown out were the undefined symbol: (there were lots of them). Then it just quit with the following messages. Could you provide any insights from the warning and error messages below (as you can see there is an "ERROR: too many pending (error) messages" in the middle)? How can I get more information about this ERROR message? I googled but did find anything useful.
....
undefined symbol: ..... (lot of them with different functions)
....
WARNING: Tracing 78768 (/home/....) in both user-space and kernel-space...
WARNING: no or bad debug frame hdr
WARNING: No binary search table for eh frame, doing slow linear search for stap_c96f67434fd8f68fcbe858907933cae_8
WARNING: DWARF expression stack underflow in CFI
WARNING: too many pending (warning) messages
ERROR: too many pending (error) messages
WARNING: Number of errors: 6, skipped probes: 10007
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run failed. [man error::pass5]
--
You received this message because you are subscribed to a topic in the Google Groups "openresty-en" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/openresty-en/ELvy9lOZBh8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to openresty-en...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Yichun Zhang (agentzh)
Jun 26, 2015, 10:40:13 PM6/26/15
to openresty-en
Hello!
On Sat, Jun 27, 2015 at 2:46 AM, Sheng Li wrote:
> The results I shared were with gcc/g++ is 4.8.3 and systemtap 2.6/0.160
> (both are very very recent). Both boost and protobuffer were compiled with
> -g option. And I still had those problems.
>
I believe you :)
> What is the best way to check whether the debug symbols are generated
> correctly? Using nm or gdb or something better?
>
Usually I just use objdump or dwarfdump. gdb is fine too.
> The only messages stap thrown out were the undefined symbol: (there were
> lots of them).
I used to see such warnings before and they are always harmless on my
side. So I didn't bother looking into this.
I suggest reporting it to the official systemtap mailing list:
https://sourceware.org/systemtap/getinvolved.html
The systemtap team is usually very helpful according to my experience.
I can take a look at it too, but not any time soon.
Best regards,
-agentzh
On Sat, Jun 27, 2015 at 2:46 AM, Sheng Li wrote:
> The results I shared were with gcc/g++ is 4.8.3 and systemtap 2.6/0.160
> (both are very very recent). Both boost and protobuffer were compiled with
> -g option. And I still had those problems.
>
> What is the best way to check whether the debug symbols are generated
> correctly? Using nm or gdb or something better?
>
> The only messages stap thrown out were the undefined symbol: (there were
> lots of them).
side. So I didn't bother looking into this.
I suggest reporting it to the official systemtap mailing list:
https://sourceware.org/systemtap/getinvolved.html
The systemtap team is usually very helpful according to my experience.
I can take a look at it too, but not any time soon.
Best regards,
-agentzh
Sheng Li
Jun 29, 2015, 5:31:30 PM6/29/15
to openre...@googlegroups.com
Thanks, agentzh!
Now i have a weirder problem, systamtap suddenly stopped working. The error message is as below. I tried both the stap (v2.6) from the packet manager, and the latest stap (v 2.8), and both of the versions of stap failed with the same error. Could you please share your insights?
$ sudo ./sample-bt -p 9149 -t 1 -u -k
WARNING: missing unwind/symbol data for module 'stap_045251fe2e371aedb286f29ba526b824_4580'
WARNING: missing unwind/symbol data for module 'stap_1733e84de62237055a84a9ca367b1e09_9390'
WARNING: /opt/stap/bin/staprun exited with signal: 9 (Killed)
Pass 5: run failed. [man error::pass5]
Thanks!
-agentzh
Yichun Zhang (agentzh)
Jun 30, 2015, 12:27:35 AM6/30/15
to openresty-en
Hello!
On Tue, Jun 30, 2015 at 5:31 AM, Sheng Li wrote:
> Now i have a weirder problem, systamtap suddenly stopped working. The error
> message is as below. I tried both the stap (v2.6) from the packet manager,
> and the latest stap (v 2.8), and both of the versions of stap failed with
> the same error. Could you please share your insights?
>
> $ sudo ./sample-bt -p 9149 -t 1 -u -k
> WARNING: missing unwind/symbol data for module
> 'stap_045251fe2e371aedb286f29ba526b824_4580'
> WARNING: missing unwind/symbol data for module
> 'stap_1733e84de62237055a84a9ca367b1e09_9390'
> WARNING: /opt/stap/bin/staprun exited with signal: 9 (Killed)
Seems like something in your system is killing stap. Maybe your system
does not have enough memory and your OOM killer kills the stap process
with force.
Regards,
-agentzh
On Tue, Jun 30, 2015 at 5:31 AM, Sheng Li wrote:
> Now i have a weirder problem, systamtap suddenly stopped working. The error
> message is as below. I tried both the stap (v2.6) from the packet manager,
> and the latest stap (v 2.8), and both of the versions of stap failed with
> the same error. Could you please share your insights?
>
> $ sudo ./sample-bt -p 9149 -t 1 -u -k
> WARNING: missing unwind/symbol data for module
> 'stap_045251fe2e371aedb286f29ba526b824_4580'
> WARNING: missing unwind/symbol data for module
> 'stap_1733e84de62237055a84a9ca367b1e09_9390'
> WARNING: /opt/stap/bin/staprun exited with signal: 9 (Killed)
does not have enough memory and your OOM killer kills the stap process
with force.
Regards,
-agentzh
Reply all
Reply to author
Forward
0 new messages