No symbol traces from stripped binaries

[Iman Ahmadvand]

Hi everybody.

From the documentation, google breakpad is:

> a library and tool suite that allows you to distribute an application

> to users with compiler-provided debugging information removed

For the proof of the above quote we will try it with this minimal c++17 sample:

    #include <thread>

    #include <filesystem>

    

    #include <client/linux/handler/exception_handler.h>

    

    namespace breakpad = google_breakpad;

    

    static bool DumpCallback(const breakpad::MinidumpDescriptor& md,

                             void* context,

                             bool success) {

        (void)md;

        (void)context;

        return success;

    }

    

    static void fault(unsigned after) {

        std::this_thread::sleep_for(std::chrono::seconds{after});

        delete reinterpret_cast<std::string*>(0xFEE1DEAD);

    }

    

    int32_t main(int argc, char** argv) {

        (void)argc;

        (void)argv;

    

        auto pwd = std::filesystem::current_path();

        const auto dumpDir = pwd.string() + "/dumps";

        std::filesystem::create_directory(dumpDir);

        breakpad::MinidumpDescriptor md(dumpDir);

        new google_breakpad::ExceptionHandler(

            md,

            /* FilterCallback */ nullptr,

            DumpCallback,

            /* callback_context */ nullptr,

            true,

            -1

        );

    

        fault(1U);

    

        return EXIT_SUCCESS;

    }

With debugging symbols integrated, it's what is expected to be, so if we try to run it and process the output dump file (with help of main utilities such as `dump_syms` and `minidump_stackwalk`) the result is a nice symbol trace:

    Operating system: Linux

                      0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64

    CPU: amd64

         family 6 model 58 stepping 9

         1 CPU

    

    GPU: UNKNOWN

    

    Crash reason:  SIGSEGV /SEGV_MAPERR

    Crash address: 0xfee1dead

    Process uptime: not available

    

    Thread 0 (crashed)

     0  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_data() const [basic_string.h : 176 + 0x4]

        rax = 0x00000000fee1dead   rdx = 0x00007ffc289e7e40

        rcx = 0x00007f559c571bc1   rbx = 0x000055d0d25df6f0

        rsi = 0x00007ffc289e7e40   rdi = 0x00000000fee1dead

        rbp = 0x00007ffc289e7e00   rsp = 0x00007ffc289e7e00

         r8 = 0x0000000000000000    r9 = 0x000055d0d25e38d8

        r10 = 0x0000000000000000   r11 = 0x0000000000000246

        r12 = 0x000055d0d176cfd0   r13 = 0x00007ffc289e8100

        r14 = 0x0000000000000000   r15 = 0x0000000000000000

        rip = 0x000055d0d176ddf0

        Found by: given as instruction pointer in context

     1  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_is_local() const [basic_string.h : 211 + 0xc]

        rbx = 0x000055d0d25df6f0   rbp = 0x00007ffc289e7e30

        rsp = 0x00007ffc289e7e10   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x000055d0d176e021

        Found by: call frame info

     2  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose() [basic_string.h : 220 + 0xc]

        rbx = 0x000055d0d25df6f0   rbp = 0x00007ffc289e7e50

        rsp = 0x00007ffc289e7e40   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x000055d0d176de0e

        Found by: call frame info

     3  core!std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() [basic_string.h : 657 + 0xc]

        rbx = 0x000055d0d25df6f0   rbp = 0x00007ffc289e7e70

        rsp = 0x00007ffc289e7e60   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x000055d0d176dae0

        Found by: call frame info

     4  core!fault [main.cpp : 18 + 0xa]

        rbx = 0x000055d0d25df6f0   rbp = 0x00007ffc289e7ea0

        rsp = 0x00007ffc289e7e80   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x000055d0d176e770

        Found by: call frame info

     5  core!main [main.cpp : 38 + 0xa]

        rbx = 0x000055d0d25df6f0   rbp = 0x00007ffc289e8020

        rsp = 0x00007ffc289e7eb0   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x000055d0d176e882

        Found by: call frame info

     6  libc.so.6 + 0x2409b

        rbx = 0x0000000000000000   rbp = 0x000055d0d1796980

        rsp = 0x00007ffc289e8030   r12 = 0x000055d0d176cfd0

        r13 = 0x00007ffc289e8100   r14 = 0x0000000000000000

        r15 = 0x0000000000000000   rip = 0x00007f559c0a009b

        Found by: call frame info

     7  core!fault [main.cpp : 19 + 0x3]

        rsp = 0x00007ffc289e8050   rip = 0x000055d0d176e782

        Found by: stack scanning

     8  core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]

        rsp = 0x00007ffc289e8068   rip = 0x000055d0d176cfd0

        Found by: stack scanning

     9  ld-linux-x86-64.so.2 + 0xf476

        rsp = 0x00007ffc289e80c0   rip = 0x00007f559d0d2476

        Found by: stack scanning

    10  core!google_breakpad::FileID::ElfFileIdentifier(google_breakpad::wasteful_vector<unsigned char>&) [file_id.cc : 158 + 0x10]

        rsp = 0x00007ffc289e80d8   rip = 0x000055d0d176cfd0

        Found by: stack scanning

    

    Loaded modules:

    0x55d0d1765000 - 0x55d0d1796fff  core  ???  (main)

    0x7f55994d6000 - 0x7f55994dbfff  librt.so.1  ???

    0x7f55994e0000 - 0x7f55994f2fff  libbsd.so.0  ???

    0x7f55994fc000 - 0x7f5599500fff  libXdmcp.so.6  ???

    0x7f5599702000 - 0x7f5599703fff  libXau.so.6  ???

    0x7f5599906000 - 0x7f5599924fff  libxcb.so.1  ???

    0x7f5599930000 - 0x7f5599940fff  libXext.so.6  ???

    0x7f5599b42000 - 0x7f5599be7fff  libX11.so.6  ???

    0x7f5599c85000 - 0x7f5599cd8fff  libpcre.so.3  ???

    0x7f5599cf9000 - 0x7f5599cfafff  libicudata.so.63  ???

    0x7f559b6e9000 - 0x7f559b70dfff  libgraphite2.so.3  ???

    0x7f559b716000 - 0x7f559b79ffff  libfreetype.so.6  ???

    0x7f559b7d2000 - 0x7f559b830fff  libGLdispatch.so.0  ???

    0x7f559b891000 - 0x7f559b89efff  libGLX.so.0  ???

    0x7f559b8c5000 - 0x7f559b95dfff  libglib-2.0.so.0  ???

    0x7f559b9e4000 - 0x7f559b9e5fff  libdl.so.2  ???

    0x7f559b9e9000 - 0x7f559b9f7fff  libdouble-conversion.so.1  ???

    0x7f559ba00000 - 0x7f559ba53fff  libpcre2-16.so.0  ???

    0x7f559ba7b000 - 0x7f559bbb0fff  libicuuc.so.63  ???

    0x7f559bc4c000 - 0x7f559be92fff  libicui18n.so.63  ???

    0x7f559bf27000 - 0x7f559bf3efff  libz.so.1  ???

    0x7f559bf48000 - 0x7f559bff9fff  libharfbuzz.so.0  ???

    0x7f559c043000 - 0x7f559c06dfff  libpng16.so.16  ???

    0x7f559c07c000 - 0x7f559c1e5fff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)

    0x7f559c23d000 - 0x7f559c250fff  libgcc_s.so.1  ???

    0x7f559c259000 - 0x7f559c304fff  libm.so.6  ???

    0x7f559c3dc000 - 0x7f559c510fff  libstdc++.so.6  ???

    0x7f559c560000 - 0x7f559c574fff  libpthread.so.0  ???

    0x7f559c581000 - 0x7f559c5e2fff  libGL.so.1  ???

    0x7f559c615000 - 0x7f559c937fff  libQt5Core.so.5  ???

    0x7f559cb10000 - 0x7f559cfadfff  libQt5Gui.so.5  ???

    0x7f559d0c3000 - 0x7f559d0e1fff  ld-linux-x86-64.so.2  ???  (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)

    0x7ffc289fe000 - 0x7ffc289fffff  linux-gate.so  ???

However, when we try the exact same routine as above but instead with the dump file generated from the release binary (without debugging symbols) which is typically deployed to the client, the output is not desired:

    Operating system: Linux

                      0.0.0 Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64

    CPU: amd64

         family 6 model 58 stepping 9

         1 CPU

    

    GPU: UNKNOWN

    

    Crash reason:  SIGSEGV /SEGV_MAPERR

    Crash address: 0xfee1dead

    Process uptime: not available

    

    Thread 0 (crashed)

     0  core + 0x73da

        rax = 0x0000000000000000   rdx = 0x000055bd94854375

        rcx = 0x00007f9739ab4bc1   rbx = 0x00007ffe3d22cf70

        rsi = 0x00007ffe3d22cf00   rdi = 0x00007ffe3d22cf00

        rbp = 0x00007ffe3d22d030   rsp = 0x00007ffe3d22ced0

         r8 = 0x0000000000000000    r9 = 0x000055bd94c7d9b8

        r10 = 0x0000000000000000   r11 = 0x0000000000000246

        r12 = 0x00007ffe3d22cf10   r13 = 0x00007ffe3d22cf30

        r14 = 0x00007ffe3d22cf00   r15 = 0x000055bd94c79720

        rip = 0x000055bd948543da

        Found by: given as instruction pointer in context

     1  core + 0x2f1b5

        rbp = 0x00007ffe3d22d030   rsp = 0x00007ffe3d22d000

        rip = 0x000055bd9487c1b5

        Found by: stack scanning

     2  ld-linux-x86-64.so.2 + 0xf530

        rbp = 0x00007ffe3d22d030   rsp = 0x00007ffe3d22d008

        rip = 0x00007f973a615530

        Found by: stack scanning

     3  core + 0x7660

        rbp = 0x00007ffe3d22d030   rsp = 0x00007ffe3d22d018

        rip = 0x000055bd94854660

        Found by: stack scanning

     4  core + 0x2f170

        rsp = 0x00007ffe3d22d038   rip = 0x000055bd9487c170

        Found by: stack scanning

     5  libc.so.6 + 0x2409b

        rsp = 0x00007ffe3d22d040   rip = 0x00007f97395e309b

        Found by: stack scanning

     6  core + 0x70a0

        rsp = 0x00007ffe3d22d060   rip = 0x000055bd948540a0

        Found by: stack scanning

     7  core + 0x7660

        rsp = 0x00007ffe3d22d078   rip = 0x000055bd94854660

        Found by: stack scanning

     8  ld-linux-x86-64.so.2 + 0xf476

        rsp = 0x00007ffe3d22d0d0   rip = 0x00007f973a615476

        Found by: stack scanning

     9  core + 0x7660

        rsp = 0x00007ffe3d22d0e8   rip = 0x000055bd94854660

        Found by: stack scanning

    10  core + 0x768a

        rsp = 0x00007ffe3d22d100   rip = 0x000055bd9485468a

        Found by: stack scanning

    

    Loaded modules:

    0x55bd9484d000 - 0x55bd9487cfff  core  ???  (main)  (WARNING: No symbols, core, 1003D7A4BBF22C584BDC06B4C9D10B7D0)

    0x7f9736a19000 - 0x7f9736a1efff  librt.so.1  ???

    0x7f9736a23000 - 0x7f9736a35fff  libbsd.so.0  ???

    0x7f9736a3f000 - 0x7f9736a43fff  libXdmcp.so.6  ???

    0x7f9736c45000 - 0x7f9736c46fff  libXau.so.6  ???

    0x7f9736e49000 - 0x7f9736e67fff  libxcb.so.1  ???

    0x7f9736e73000 - 0x7f9736e83fff  libXext.so.6  ???

    0x7f9737085000 - 0x7f973712afff  libX11.so.6  ???

    0x7f97371c8000 - 0x7f973721bfff  libpcre.so.3  ???

    0x7f973723c000 - 0x7f973723dfff  libicudata.so.63  ???

    0x7f9738c2c000 - 0x7f9738c50fff  libgraphite2.so.3  ???

    0x7f9738c59000 - 0x7f9738ce2fff  libfreetype.so.6  ???

    0x7f9738d15000 - 0x7f9738d73fff  libGLdispatch.so.0  ???

    0x7f9738dd4000 - 0x7f9738de1fff  libGLX.so.0  ???

    0x7f9738e08000 - 0x7f9738ea0fff  libglib-2.0.so.0  ???

    0x7f9738f27000 - 0x7f9738f28fff  libdl.so.2  ???

    0x7f9738f2c000 - 0x7f9738f3afff  libdouble-conversion.so.1  ???

    0x7f9738f43000 - 0x7f9738f96fff  libpcre2-16.so.0  ???

    0x7f9738fbe000 - 0x7f97390f3fff  libicuuc.so.63  ???

    0x7f973918f000 - 0x7f97393d5fff  libicui18n.so.63  ???

    0x7f973946a000 - 0x7f9739481fff  libz.so.1  ???

    0x7f973948b000 - 0x7f973953cfff  libharfbuzz.so.0  ???

    0x7f9739586000 - 0x7f97395b0fff  libpng16.so.16  ???

    0x7f97395bf000 - 0x7f9739728fff  libc.so.6  ???  (WARNING: No symbols, libc.so.6, A8A9B91823C5CFE5E5B5D946D605D0920)

    0x7f9739780000 - 0x7f9739793fff  libgcc_s.so.1  ???

    0x7f973979c000 - 0x7f9739847fff  libm.so.6  ???

    0x7f973991f000 - 0x7f9739a53fff  libstdc++.so.6  ???

    0x7f9739aa3000 - 0x7f9739ab7fff  libpthread.so.0  ???

    0x7f9739ac4000 - 0x7f9739b25fff  libGL.so.1  ???

    0x7f9739b58000 - 0x7f9739e7afff  libQt5Core.so.5  ???

    0x7f973a053000 - 0x7f973a4f0fff  libQt5Gui.so.5  ???

    0x7f973a606000 - 0x7f973a624fff  ld-linux-x86-64.so.2  ???  (WARNING: No symbols, ld-linux-x86-64.so.2, 7BFD5DF2BE95A34B86FD71080ACCAE8C0)

    0x7ffe3d31a000 - 0x7ffe3d31bfff  linux-gate.so  ???

Is there anything else we need to consider?

[Mike Frysinger]

you didn't show the commands you ran.  but you need to provide the symbols to the dump tools.  basically, you need to fix this error from your output:

    0x55bd9484d000 - 0x55bd9487cfff  core  ???  (main)  (WARNING: No symbols, core, 1003D7A4BBF22C584BDC06B4C9D10B7D0)

--
You received this message because you are subscribed to the Google Groups "google-breakpad-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-breakpad-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-breakpad-discuss/556f6279-eef7-4e9b-96f6-aa9e9731fd04n%40googlegroups.com.

[Iman Ahmadvand]

Hi Mike.

This is the dump.sh we use:

    #!/bin/bash

    

    #

    # e.g ./dump.sh ./exec $PWD/dumps

    #

    

    set -e

    set -u

    

    DBG_INFO=$(realpath ${1})

    DUMPS_DIR=$(realpath ${2:-$PWD/dumps})

    DUMP_SYMS=${3:-~/WorkSpace/libraries/breakpad/src/tools/linux/dump_syms/dump_syms}

    STAK_WALK=${4:-~/WorkSpace/libraries/breakpad/src/processor/minidump_stackwalk}

    

    #

    # Generate debug symbols

    #

    base=$(basename $DBG_INFO)

    $DUMP_SYMS $DBG_INFO > $DUMPS_DIR/$base.sym

    

    #

    # Create dump dir structure

    #

    list=($(head -n1 $DUMPS_DIR/$base.sym))

    hash=${list[3]}

    mkdir -p $DUMPS_DIR/symbols/$base/$hash

    mv $DUMPS_DIR/$base.sym $DUMPS_DIR/symbols/$base/$hash

    

    #

    # Produce stack trace

    #

    RED='\033[0;36m'

    NC='\033[0m' # No Color

    tree $DUMPS_DIR

    for dmp in $DUMPS_DIR/*.dmp ; do

    filename=$(basename -- "${dmp}")

    filename="${filename%.*}"

    echo -e "generating stack trace for -> ${RED}${dmp}${NC}"

    $STAK_WALK ${dmp} $DUMPS_DIR/symbols > $DUMPS_DIR/${filename}.txt 2>/dev/null

    done

And actually, we did, we provide the debug version of our binary + client-generated dump file to the above script.

Don't know where that warning comes from.

[iman ahmadvand]

Hi Mike.

Any hint you can give us?

There's a weird issue, I have a symbol trace for a  debug build version of my binary(-g)

but when I add an optimization level (-g -O1) there no symbols anymore with exact same routine!

Regards.

You received this message because you are subscribed to a topic in the Google Groups "google-breakpad-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-breakpad-discuss/JmhhdZkWJtU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-breakpad-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-breakpad-discuss/CAAbOSc%3Dgb4GQUj8GpvCEDMTjA1gvPWJ_FZrPAzp3m2USZfzfGw%40mail.gmail.com.