Perf having trouble finding symbols with .debug extension
3,706 views
Skip to first unread message
Matthew Redmond
Aug 3, 2012, 6:26:33 PM8/3/12
to Chromium OS dev, Stephane Eranian, Mike Frysinger
Hello all,
I've run into a minor problem with perf report - specifically, with the --symfs option.
I ran perf record and generated a perf.data file on a remote machine, then sent that perf.data over to my workstation. I've downloaded a copy of the appropriate debug symbols for a ChromeOS build:
mrdmnd@mrdmnd:~$ gsutil cp gs://chromeos-image-archive/x86-zgb-release/R22-2630.0.0/debug.tgz . && tar -zxvf debug.tgz
Now, I want to symbolize the perf.data file:
mrdmnd@mrdmnd:~$ perf report -i perf.data --symfs debug/
[kernel.kallsyms] with build id c25185ad8bec1b1a97b9e46e40ca4c2c706c3955 not found, continuing without symbols
/lib/libc-2.15.so with build id 35e4883d0347a583d9a2bab3a5dec0c5a222081d not found, continuing without symbols
/opt/google/chrome/chrome with build id a392bdfb0c1cbc32501e9ec8b687b3c368853c07 not found, continuing without symbols
/lib/libpthread-2.15.so with build id d0552f28e546990859d6e21862d66dc3379ee442 not found, continuing without symbols
/lib/librt-2.15.so with build id 63b49063c3c6d733a2917d5070b21b401f421998 not found, continuing without symbols
# ========
# captured on: Fri Aug 3 11:28:01 2012
# ========
#
# hostname : localhost
# os release : 3.4.0
# perf version : 3.4.1111.gc604447
# arch : i686
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Atom(TM) CPU N570 @ 1.66GHz
# cpuid : GenuineIntel,6,28,10
# total memory : 1935020 kB
# cmdline : /usr/local/sbin/perf record -a -F 100 -- sleep 2
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 82, 83, 84, 85 }
# CPU_TOPOLOGY info available, use -I to display
# Events: 109 cycles
#
# Overhead Command Shared Object Symbol
# ........ ........... .................. ......
#
53.66% swapper [kernel.kallsyms] [k] 0x00000000811e31f7
26.65% chrome chrome [.] 0x0000000001dd40e8
8.00% perf [kernel.kallsyms] [k] 0x000000008104b1ee
6.24% chrome [kernel.kallsyms] [k] 0x000000008104a08a
1.65% chrome librt-2.15.so [.] 0x0000000000004376
1.34% kworker/0:2 [kernel.kallsyms] [k] 0x0000000081037ab2
0.74% kworker/3:1 [kernel.kallsyms] [k] 0x00000000810480ab
0.67% perf libc-2.15.so [.] 0x0000000000140dd4
0.53% chrome libc-2.15.so [.] 0x000000000012f990
0.52% chrome libpthread-2.15.so [.] 0x0000000000008f30
Which is not great.
Now, I rename one of the symbols from debug/opt/google/chrome/chrome.debug to debug/opt/google/chrome/chrome
mrdmnd@mrdmnd:~/debug/opt/google/chrome$ mv chrome.debug chrome
And then, perf can "find" the appropriate symbols:
mrdmnd@mrdmnd:~$ perf report -i perf.data --symfs debug/
[kernel.kallsyms] with build id c25185ad8bec1b1a97b9e46e40ca4c2c706c3955 not found, continuing without symbols
/lib/libc-2.15.so with build id 35e4883d0347a583d9a2bab3a5dec0c5a222081d not found, continuing without symbols
/lib/libpthread-2.15.so with build id d0552f28e546990859d6e21862d66dc3379ee442 not found, continuing without symbols
/lib/librt-2.15.so with build id 63b49063c3c6d733a2917d5070b21b401f421998 not found, continuing without symbols
# ========
# captured on: Fri Aug 3 11:28:01 2012
# ========
#
# hostname : localhost
# os release : 3.4.0
# perf version : 3.4.1111.gc604447
# arch : i686
# nrcpus online : 4
# nrcpus avail : 4
# cpudesc : Intel(R) Atom(TM) CPU N570 @ 1.66GHz
# cpuid : GenuineIntel,6,28,10
# total memory : 1935020 kB
# cmdline : /usr/local/sbin/perf record -a -F 100 -- sleep 2
# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 82, 83, 84, 85 }
# CPU_TOPOLOGY info available, use -I to display
# Events: 109 cycles
#
# Overhead Command Shared Object
# ........ ........... .................. ................................................................................................................................................................................................
#
53.66% swapper [kernel.kallsyms] [k] 0x00000000811e31f7
8.00% perf [kernel.kallsyms] [k] 0x000000008104b1ee
6.24% chrome [kernel.kallsyms] [k] 0x000000008104a08a
2.04% chrome chrome [.] __x86.get_pc_thunk.bx
2.01% chrome chrome [.] WebCore::StyleResolver::collectMatchingRulesForList(WTF::Vector<WebCore::RuleData, 0u> const*, int&, int&, WebCore::StyleResolver::MatchOptions const&)
1.96% chrome chrome [.] WebCore::enclosingIntRect(WebCore::FloatRect const&)
1.65% chrome librt-2.15.so [.] 0x0000000000004376
1.37% chrome chrome [.] MessageLoop::DoDelayedWork(base::TimeTicks*)
1.34% kworker/0:2 [kernel.kallsyms] [k] 0x0000000081037ab2
1.34% chrome chrome [.] WebKit::WebTransformationMatrix::WebTransformationMatrix(WebKit::WebTransformationMatrix const&)
1.32% chrome chrome [.] WebKit::WebTransformationMatrix::operator==(WebKit::WebTransformationMatrix const&) const
1.20% chrome chrome [.] MessageLoop::PostDelayedTask(tracked_objects::Location const&, base::Callback<void ()()> const&, base::TimeDelta)
1.04% chrome chrome [.] WTF::HashMap<WebCore::RenderObject*, WTF::RefPtr<WebCore::CompositeAnimation>, WTF::PtrHash<WebCore::RenderObject*>, WTF::HashTraits<WebCore::RenderObject*>, WTF::HashTraits<WTF::RefPtr<We
1.01% chrome chrome [.] WebCore::CCLayerTilingData::tileAt(int, int) const
1.00% chrome chrome [.] tcmalloc::FL_Pop(void**)
0.74% kworker/3:1 [kernel.kallsyms] [k] 0x00000000810480ab
0.67% perf libc-2.15.so [.] 0x0000000000140dd4
0.66% chrome chrome [.] WebCore::CCThreadProxy::scheduledActionUpdateMoreResources()
0.63% chrome chrome [.] WebCore::RenderLayer::convertToLayerCoords(WebCore::RenderLayer const*, WebCore::FractionalLayoutPoint&) const
0.63% chrome chrome [.] WebCore::Element::recalcStyle(WebCore::Node::StyleChange)
0.62% chrome chrome [.] WebCore::RenderObject::isSVGRoot() const
0.60% chrome chrome [.] WebCore::KeyframeAnimation::affectsProperty(WebCore::CSSPropertyID) const
0.60% chrome chrome [.] void WTF::Vector<int, 32u>::appendRange<int const*>(int const*, int const*)
0.59% chrome chrome [.] WebCore::RenderBoxModelObject::relativePositionOffset() const
0.58% chrome chrome [.] WTF::Vector<std::pair<void (*)(WebCore::Node*, unsigned int), std::pair<WTF::RefPtr<WebCore::Node>, unsigned int> >, 0u>::shrinkCapacity(unsigned int)
0.57% chrome chrome [.] bool WebCore::Region::Shape::compareShapes<WebCore::Region::Shape::CompareContainsOperation>(WebCore::Region::Shape const&, WebCore::Region::Shape const&)
0.57% chrome chrome [.] char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag)
0.57% chrome chrome [.] WebCore::LayerChromium::opacity() const
0.56% chrome chrome [.] .L90
0.55% chrome chrome [.] WebCore::StyleResolver::sortAndTransferMatchedRules(WebCore::StyleResolver::MatchResult&)
0.55% chrome chrome [.] base::StatisticsRecorder::FindHistogram(std::string const&, base::Histogram**)
0.55% chrome chrome [.] WebCore::RenderObject::animation() const
0.55% chrome chrome [.] WebCore::Region::Region(WebCore::IntRect const&)
0.54% chrome chrome [.] WebCore::RenderLayer::styleChanged(WebCore::StyleDifference, WebCore::RenderStyle const*)
0.53% chrome libc-2.15.so [.] 0x000000000012f990
0.52% chrome libpthread-2.15.so [.] 0x0000000000008f30
Which is clearly working.
So I think the problem is that perf isn't recognizing symbols that end with the .debug suffix.
Am I doing something wrong?
Matthew Redmond
Aug 3, 2012, 7:58:45 PM8/3/12
to Stephane Eranian, Chromium OS dev, Mike Frysinger
We didn't realize that the path from stripped binaries had to start with /usr/lib/debug - strace pointed that out. Good insight.
The solution was to create a /usr/lib/debug subdirectory inside of a random subdirectory, then paste the actual symbol directory in there.
The buildids were pointing to /TLD/usr/lib/debug/whatever, so we called perf report ... --symfs TLD/ and it worked.
Now we'd like to get the kernel symbols working - the tarball that I downloaded (gs://chromeos-image-archive/x86-zgb-release/R22-2630.0.0/debug.tgz) doesn't have a System.map file
(Ahmad suggested the flag --kallsyms /boot/System.map but the only thing in the tarball's /boot/ is vmlinux)
On Fri, Aug 3, 2012 at 4:12 PM, Stephane Eranian <era...@google.com> wrote:
What would be interesting to see would be a strace of your perf report
to see where it is looking in the symfs.
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages