Getting GDB to use debug symbols from debug.tgz

[Rick Byers]

Hi,

I'm trying to remotely debug a ChromeOS image produced by an official build.  I've had no problem remotely debugging a build I produced (where I keep the original un-stripped binaries on the host and point GDB to them with set sysroot), but I haven't been able to get this to work when running an image produced by the build bots.

In order to get symbols I need GDB to load the debug versions of the libraries, which I found in the debug.tgz next to the image.zip I'm using.  But how can I get GDB to load these?  I've tried 'set debug-file-directory' and 'set sysroot' pointing at where I extracted debug.tgz, but it doesn't seem to find them (info share shows most libraries don't have symbols loaded).  

If I rename the files to match what GDB expects (eg. libX11.so.6.3.0.debug -> libX11.so.6) then it appears to work - so this looks like just a matter of the files in debug.tgz not having the right names.  Presumably I could write a script to add all the possible symlinks to satisfy GDB, but surely this is a common scenario - no?  If this is what's necessary, why doesn't the tar file contain the necessary sym links already?  Is there some other process people follow for debugging build-bot builds?

Thanks!

   Rick

[Satoru Takabayashi]

+cc people I found from "git log" in google-breakpad.

I was also doing something similar today but had no luck. FWIW, Here's what I did:

- downloaded a minidump from a crash page
- converted the minidump file to a core file using minidump-2-core tool. you can build the tool in the chromium tree by ninja -C out/Release minidump-2-core).

- obtained the corresponding official build image and debug.tgz.

- extracted the root partition tree from the image. this is a bit tricky but you can do this using src/scripts/mount_gpt_image.sh in the chromium os tree. /tmp/m is the root partition.

- extracted debug.tgz.

- ran gdb like:

% gdb m/opt/google/chrome/chrome  core   # m is the root partition's root directory

(gdb) symbol-file debug/opt/google/chrome/chrome.debug

(gdb) set sysroot m 

(gdb) bt

- no symbol names were resolved...

So I suspected the core file created by minidump-2-core was broken. "info sharedlibrary" says:

(gdb) info sharedlibrary 
warning: while parsing target library list: no element found
From        To          Syms Read   Shared Object Library
0x78ba4760  0x78bbbf67  Yes (*)     m/lib/ld-linux.so.2
(*): Shared library is missing debugging information.

This is not correct. There should be many more shared libraries, but only ld-linux.so.2 was listed, and the address also looks wrong. The minidump file says:

module[91]

MDRawModule

  base_of_image                   = 0x73b6f000

  size_of_image                   = 0x1f000

  ...

  (code_file)                     = "/lib/ld-2.10.1.so"

These addresses don't match. Note that ld-linux.so.2 is a symlink to ld-2.10.1.so.

I also built minidump-2-core from platform/google-breakpad in the chromium os tree and used it to get a core file, but the result was the same.

% cd src/platform/google-breakpad/src/tools/linux/md2core

% g++ -m32 minidump-2-core.cc -I../../.. # -m32 to build a 32-bit binary

% ./a.out minidump.dmp > core

Maybe minidump-2-core is broken?

What I wanted to do today was to check local variables in a function in the Chrome binary using GDB, but I got stuck here.

Satoru

> --
> Chromium OS Developers mailing list: chromiu...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-os-dev?hl=en

[Raymes Khoury]

+cc cmtice

[Raymes Khoury]

[Rick Byers]

Satoru, Trying to do this with a minidump adds another layer of challenges - lets see if we can get it working on a full core image (or remote debugging which should be equivalent), and then move on to the additional challenges with minidumps.

Lei, thanks for the suggestion.  The images to indeed have debuglinks as relative paths (eg. libX11.so.6 has a debuglink to "libX11.so.6.3.0.debug).  If I copy all the image binaries to a new location (I was mounting my image read-only), and then extract the debug files next to it, then things look a LOT better.  GDB now actually loads the debug symbols for most of the shared libraries:

(cr) ((6759411...)) rbyers@rbyers-chrome ~/trunk/src/scripts $ armv7a-cros-linux-gnueabi-gdb /build/image-b1045/opt/google/chrome/chrome      

Unable to lstat() logging checkpoint parent directory: No such file or directory.

GNU gdb (Gentoo 7.2 vanilla) 7.2-gg15

Copyright (C) 2010 Free Software Foundation, Inc.

License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

Type "show copying" and "show warranty" for licensing/warranty details.

This GDB was configured as "--host=x86_64-pc-linux-gnu --target=armv7a-cros-linux-gnueabi".

<http://bugs.gentoo.org/>

Hey, I'm GDB 7.x.  Check me out!  http://wiki/Main/Gdb7x

BFD: /build/image-b1045/opt/google/chrome/chrome: warning: sh_link not set for section `.ARM.exidx'

BFD: /build/image-b1045/opt/google/chrome/chrome: warning: sh_link not set for section `.ARM.exidx'

Reading symbols from /build/image-b1045/opt/google/chrome/chrome...BFD: /build/image-b1045/opt/google/chrome/chrome.debug: warning: sh_link not set for section `.ARM.exidx'

Reading symbols from /build/image-b1045/opt/google/chrome/chrome.debug...done.

done.

(gdb) set sysroot /build/image-b1045 

(gdb) target remote 172.23.176.80:1234

Remote debugging using 172.23.176.80:1234

0x40c9ec76 in poll () from /build/image-b1045/lib/libc.so.6

(gdb) info share

From        To          Syms Read   Shared Object Library

0x401785a8  0x401ca7f0  Yes         /build/image-b1045/usr/lib/libX11.so.6

0x40008920  0x400090e8  Yes (*)     /build/image-b1045/lib/libdl.so.2

0x400852e8  0x40088d7c  Yes         /build/image-b1045/usr/lib/libXrender.so.1

...

But I still can't get good stack traces:

(gdb) bt

#0  0x40c9ec76 in poll () from /build/image-b1045/lib/libc.so.6

warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

#1  0x000000a8 in ?? (warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

) at /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/ext/new_allocator.h:86

warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

#2  0x000000a8 in ?? (warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)

) at /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/ext/new_allocator.h:86

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Any ideas?

Thanks,

   Rick

On Fri, Sep 16, 2011 at 2:30 AM, Lei Zhang <the...@chromium.org> wrote:

Does the chrome for CrOS binary have a .gnu_debuglink? If so, you can
put chrome.debug in the same directory as chrome and gdb should find
it. If not, you can try adding it: objcopy
--add-gnu-debuglink=chrome.debug chrome

[Chris Masone]

Have you tried setting debug-file-directory appropriately?  http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html#Separate-Debug-Files

[Rick Byers]

Yes, those were the instructions I was following.  When I had sysroot pointing at the stripped images (with debuglinks), and debug-file-directory pointed where I extracted debug.tgz, GDB just appeared not to load the debug symbols (info share showed all libraries as loaded properly but with the (*) indicating debug symbols weren't available).

It's possible I screwed something up with that configuration (I'll try again another time), but extracting them next to the images I'm using works well enough to get them loaded.

Perhaps my strackwalk is getting lost when I hit a frame for which I don't have debug symbols (I know the standard ARM calling convention uses a frame pointer, but perhaps we turn that off in optimized builds of ChromeOS?).  Here's another callstack that is great for awhile but then gets lost.

#0  BrowserInit::LaunchBrowser (this=0xbef21d08, command_line=..., profile=0x4a006108, cur_dir=..., process_startup=true, return_code=0xbef21d18)

    at chrome/browser/ui/browser_init.cc:549

#1  0x00287772 in operator<< <std::char_traits<char> > (profile=0x4a006108, login_host=0x2b4e848)

    at /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/ostream:510

#2  chromeos::LoginUtils::DoBrowserLaunch (profile=0x4a006108, login_host=0x2b4e848) at chrome/browser/chromeos/login/login_utils.cc:1126

#3  0x00281fa4 in _M_dispose (this=0x2b843e8, profile=<value optimized out>)

    at /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/bits/basic_string.h:227

#4  ~basic_string (this=0x2b843e8, profile=<value optimized out>) at /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/bits/basic_string.h:498

#5  chromeos::ExistingUserController::OnProfilePrepared (this=0x2b843e8, profile=<value optimized out>)

    at chrome/browser/chromeos/login/existing_user_controller.cc:419

#6  0x002886ea in chromeos::LoginUtilsImpl::OnProfileCreated (this=0x2b936f8, user_profile=0x4a006108, status=<value optimized out>)

    at chrome/browser/chromeos/login/login_utils.cc:659

#7  0x000d8e30 in ProfileManager::OnProfileCreated (this=<value optimized out>, profile=0x4a006108, success=<value optimized out>)

    at chrome/browser/profiles/profile_manager.cc:407

#8  0x0040bda0 in ProfileImpl::DoFinalInit (this=0x4a006108) at chrome/browser/profiles/profile_impl.cc:460

#9  0x0040c1aa in ProfileImpl::OnPrefsLoaded (this=0x4a006108, success=<value optimized out>) at chrome/browser/profiles/profile_impl.cc:990

#10 0x0040c3e4 in Source (this=0x4a006108, type=<value optimized out>, source=..., details=<value optimized out>) at ./content/common/notification_source.h:46

#11 ProfileImpl::Observe (this=0x4a006108, type=<value optimized out>, source=..., details=<value optimized out>)

    at chrome/browser/profiles/profile_impl.cc:1526

#12 0x00c6812c in AsWeakPtr (this=0xbef23a28, type=4244033, source=..., details=<value optimized out>) at ./base/memory/weak_ptr.h:211

#13 Iterator (this=0xbef23a28, type=4244033, source=..., details=<value optimized out>) at ./base/observer_list.h:88

#14 NotificationService::Notify (this=0xbef23a28, type=4244033, source=..., details=<value optimized out>) at content/common/notification_service.cc:104

#15 0x003f1346 in PrefNotifierImpl::OnInitializationCompleted (this=0xbef21d08, succeeded=112) at chrome/browser/prefs/pref_notifier_impl.cc:79

#16 0x000c6400 in Release (this=0x4a047f10) at ./base/memory/ref_counted.h:95

#17 ~scoped_refptr (this=0x4a047f10) at ./base/memory/ref_counted.h:241

#18 PrefValueStore::CheckInitializationCompleted (this=0x4a047f10) at chrome/browser/prefs/pref_value_store.cc:270

#19 0x000c6412 in PrefValueStore::CheckInitializationCompleted (this=0x0) at chrome/browser/prefs/pref_value_store.cc:276

#20 0xbef23bfc in ?? ()

Cannot access memory at address 0x1

#21 0xbef23bfc in ?? ()

Cannot access memory at address 0x1

Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Rick

[Caroline Tice]

I've been checking on this a little bit (with some help). There seem
to be multiple bugs here.

In the first place, minidump-2 does not appear to be doing a very good
job of translating one format to the
other:

- in the ASCII output of minidump_dump
(/home/satorux/chromeos-debug-info/ascii), we have:

module[30]
MDRawModule
base_of_image = 0x72807000
size_of_image = 0x9f000
...
(code_file) = "/opt/google/chrome/chromeos/libcros.so"
(code_identifier) = "id"
(cv_record).cv_signature = 0x53445352
(cv_record).signature = 75476e6a-ee7a-bb9b-5346-34494f01c1ae
(cv_record).age = 0
(cv_record).pdb_file_name = "libcros.so"
(misc_record) = (null)
(debug_file) = "libcros.so"
(debug_identifier) = "75476E6AEE7ABB9B534634494F01C1AE0"
(version) = ""

But that PATH is not reflected in the core file:

strings -a core2 | grep libcros
/var/lib/breakpad/75476E6A-EE7A-BB9B-5346-34494F01C1AE-libcros.so

That might be the primary cause of GDB no seeing shared libraries.

A second problem seems to be a GDB bug:

gdb -q m//opt/google/chrome/chrome
Reading symbols from
/home/satorux/chromeos-debug-info/m/opt/google/chrome/chrome...done.

WARNING: no debugging symbols found in
/home/satorux/chromeos-debug-info/m/opt/google/chrome/chrome.

A third problem also seems to be a GDB bug:

(gdb) set sysroot m
(gdb) core core
[New Thread 5920]
...

warning: while parsing target library list: no element found

Core was generated by `/opt/google/chrome/chrome
--apps-gallery-title=Web Store --apps-gallery-url=htt'.
#0 0x78ba4780 in ?? () from m/lib/ld-linux.so.2
(gdb)
(gdb) add-symbol-file m/opt/google/chrome/chromeos/libcros.so
0x0001c910+0x72807000
add symbol table from file "m/opt/google/chrome/chromeos/libcros.so" at
.text_addr = 0x72823910
(gdb) bt 2
#0 0x78ba4780 in ?? () from m/lib/ld-linux.so.2
#1 0x7282afa0 in chromeos::CryptohomeSignalFilter(DBusConnection*,
DBusMessage*, void*) ()
(gdb) x/i 0x7282afa0
0x7282afa0
<_ZN8chromeos22CryptohomeSignalFilterEP14DBusConnectionP11DBusMessagePv+176>:
Cannot access memory at address 0x7282afa0

I will try to investigate these further and see if I can find/fix the
problems; it would help if you
could file these officially as bugs.

-- Caroline Tice
cmt...@chromium.org

[Xiaohan Wang]

I also did add-symbol-file and calculate the base address manually.
It works well sometimes but not always. When it works it shows nice
stack traces that helped us solve some issues.

I see the error "Cannot access memory at address 0xXXXXXXXX" all the
time too.

> > /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/bits/basic_ string.h:227

> > #4  ~basic_string (this=0x2b843e8, profile=<value optimized out>) at

> > /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/bits/basic_ string.h:498

> >>  http://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html#Se...

> >>> /usr/lib/gcc/armv7a-cros-linux-gnueabi/4.4.3/include/g++-v4.4.3/ext/new_all ocator.h:86

> >>> warning: (Internal error: pc 0xa7 in read in psymtab, but not in symtab.)
> >>> warning: (Internal error: pc 0xa7 in read in psymtab, but not in
>

> ...
>
> read more »

[Satoru Takabayashi]

On Fri, Sep 16, 2011 at 9:59 AM, Markus Gutschke <mar...@chromium.org> wrote:

On Fri, Sep 16, 2011 at 09:48, Caroline Tice <cmt...@google.com> wrote:

 strings -a core2 | grep libcros
 /var/lib/breakpad/75476E6A-EE7A-BB9B-5346-34494F01C1AE-libcros.so

This is intentional. For a lot of crash dumps, it is highly unlikely that the majority of the user's binaries will be natively installed on the computer of whoever is inspecting the crashes. For instance, it is not at all unlikely, that the developer is running Goobuntu, whereas the user might have been running some version of Fedora.

For Chrome OS, this isn't a problem. We have access to the exact OS image that the user is using.

 

If you plan on looking at crash files, you need to go to the trouble of collecting the correct libraries that the user had on their system. The verbose output option of minidump-2-core might be helpful. All of these collected libraries should then go into /var/lib/breakpad and the filename must include the file's signature.

I had always been planning on writing tools to make this task a little easier. But I have never really gotten around to it. Ideally, we should have script files that can be pointed at the FTP sites for common distributions in order to find all the missing libraries and install them into /var/lib/breakpad. Also, it would be extremely helpful if we could have a server that hosts an authoritative collection of commonly encountered libraries.

Markus

[Chris Masone]

On Fri, Sep 16, 2011 at 11:33 AM, Markus Gutschke <mar...@chromium.org> wrote:

I know that you have access to the libraries that Chrome OS uses. But are you actually doing all your development on a Chrome OS machine? Unless that's what you are doing, the your system libraries would still be different from the libraries that were loaded into Chrome, when the crash happened.

But doesn't the "sysroot" mechanism let you point gdb at (essentially) a root file system with all the appropriate libraries?  We can do that, and that's what satoru was trying above.

 

Markus

[Satoru Takabayashi]

On Fri, Sep 16, 2011 at 11:41 AM, Markus Gutschke <mar...@chromium.org> wrote:

On Fri, Sep 16, 2011 at 11:37, Chris Masone <cma...@chromium.org> wrote:

On Fri, Sep 16, 2011 at 11:33 AM, Markus Gutschke <mar...@chromium.org> wrote:

I know that you have access to the libraries that Chrome OS uses. But are you actually doing all your development on a Chrome OS machine? Unless that's what you are doing, the your system libraries would still be different from the libraries that were loaded into Chrome, when the crash happened.

But doesn't the "sysroot" mechanism let you point gdb at (essentially) a root file system with all the appropriate libraries?  We can do that, and that's what satoru was trying above.

Ah yes, I think that might work. In the majority of cases, where people need to look at crash dumps, having a full Linux distribution that matches the user's system is probably not viable. But I forgot that you guys might actually have a full chroot environment easily accessible.

I wouldn't be opposed to a patch to minidump-2-core that optionally retains the original full paths instead of rewriting file names to point to /var/lib/breakpad. This should probably be behind a command line flag, though. Feel free to send me a codereview.

Markus, I've locally modified minidump-2-core.cc to retain the original full paths, but gdb was still unable to locate shared libraries. For people at Google, all files can be found at /home/satorux/chormeos-debug-info

% strings -a core2 | grep libcros

/var/lib/breakpad/75476E6A-EE7A-BB9B-5346-34494F01C1AE-libcros.so

# core3 contains the original library paths retained.

% strings -a core3 | grep libcros

/opt/google/chrome/chromeos/libcros.so

# 'm' directory contains the whole tree

% ls m/opt/google/chrome/chromeos/libcros.so 

m/opt/google/chrome/chromeos/libcros.so*

% gdb m/opt/google/chrome/chrome

...

(gdb) set sysroot m

(gdb) core core3 

[New Thread 5920]

...

warning: while parsing target library list: no element found

warning: while parsing target library list: no element found

Core was generated by `/opt/google/chrome/chrome --apps-gallery-title=Web Store --apps-gallery-url=htt'.

#0  0x78ba4780 in ?? () from m/lib/ld-linux.so.2

(gdb) info sharedlibrary 

warning: while parsing target library list: no element found

From        To          Syms Read   Shared Object Library

0x78ba4760  0x78bbbf67  Yes (*)     m/lib/ld-linux.so.2

(*): Shared library is missing debugging information.

--- a/src/tools/linux/md2core/minidump-2-core.cc

+++ b/src/tools/linux/md2core/minidump-2-core.cc

@@ -796,13 +796,7 @@ ParseModuleStream(CrashedProcess* crashinfo, MMappedRange range,

             record->signature.data4[4], record->signature.data4[5],

             record->signature.data4[6], record->signature.data4[7]);

     std::string filename = full_file.GetString(rawmodule->module_name_rva);

-    size_t slash = filename.find_last_of('/');

-    std::string basename = slash == std::string::npos ?

-      filename : filename.substr(slash + 1);

-    if (strcmp(guid, "00000000-0000-0000-0000-000000000000")) {

-      crashinfo->signatures[rawmodule->base_of_image] =

-        std::string("/var/lib/breakpad/") + guid + "-" + basename;

-    }

+    crashinfo->signatures[rawmodule->base_of_image] = filename;

 

     if (verbose) {

       fprintf(stderr, "0x%08llX-0x%08llX, ChkSum: 0x%08X, GUID: %s, \"%s\"\n",

Satoru

Markus 

[Satoru Takabayashi]

Found a new thing. The addresses from "bt" in gdb with a core file generated by minidump-2-core, don't match with the stack trace at crosbug.com/20451. Besides, it contains weird addresses like  0x00000001.

(gdb) bt

#0  0x78ba4780 in ?? () from /lib/ld-linux.so.2

#1  0x7282afa0 in ?? ()

#2  0x7fab501c in ?? ()

#3  0x72d2f213 in ?? ()

#4  0x7871c510 in ?? ()

#5  0x7871c900 in ?? ()

#6  0x7874b6f0 in ?? ()

#7  0x731eceaf in ?? ()

#8  0x0000000a in ?? ()

#9  0x00000001 in ?? ()

#10 0x73a78e0d in ?? ()

#11 0x82e678cd in ?? ()

#12 0x72d58659 in ?? ()

#13 0x72d63c28 in ?? ()

#14 0x7fab5168 in ?? ()

#15 0x72cbe249 in ?? ()

#16 0x7871c510 in ?? ()

#17 0x00000001 in ?? ()

#18 0x7fab51b8 in ?? ()

#19 0x7312e92b in ?? ()

#20 0x78756300 in ?? ()

#21 0x00000000 in ?? ()

crosbug.com/20451

0x78ba4780

0x7282af9f [libcros.so - chromeos_cryptohome.cc:1120] chromeos::CryptohomeSignalFilter

0x72d2f212 [libdbus-1.so.3.5.7 - dbus-connection.c:4601] dbus_connection_dispatch

0x72cbe248 [libdbus-glib-1.so.2.1.0 - dbus-gmain.c:101] message_queue_dispatch

0x7312e92a [libglib-2.0.so.0.2600.1 - gmain.c:2149] g_main_context_dispatch

0x7312f0df [libglib-2.0.so.0.2600.1 - gmain.c:2780] g_main_context_iterate

0x7312f28d [libglib-2.0.so.0.2600.1 - gmain.c:2843] g_main_context_iteration

0x74960da0 [chrome - base/message_pump_gtk.cc:41] base::MessagePumpGtk::RunOnce

0x74960230 [chrome - base/message_pump_glib.cc:206] base::MessagePumpGlib::RunWithDispatcher

0x74928e02 [chrome - base/message_loop.cc:438] MessageLoop::RunHandler

0x74928f48 [chrome - base/message_loop.cc:823] MessageLoopForUI::Run

0x740e2841 [chrome - chrome/browser/browser_main.cc:244] ChromeBrowserMainParts::TemporaryContinue

0x761d8f65 [chrome - content/browser/browser_main.cc:311] BrowserMain

0x7487b11a [chrome - content/app/content_main.cc:292] content::ContentMain

0x73da9f38 [chrome - chrome/app/chrome_main.cc:764] ChromeMain

0x73daad93 [chrome - chrome/app/chrome_exe_main_gtk.cc:18] main

0x72a1fa95 [libc-2.10.1.so + 0x00016a95]

0x73da95f0 [chrome + 0x0021b5f0]

0x73daad5f [chrome + 0x0021cd5f]

0x73b7dfff [ld-2.10.1.so + 0x0000efff]

[Satoru Takabayashi]

Markus,

Thank you for the explanation. I don't know if it matters, but unlike other shared libraries, libcros.so is loaded at chrome startup, with dlopen(). Would it confuse minidump-2-core in some way?

0x78ba4780

0x7282af9f [libcros.so - chromeos_cryptohome.cc:1120] chromeos::CryptohomeSignalFilter

0x72d2f212 [libdbus-1.so.3.5.7 - dbus-connection.c:4601] dbus_connection_dispatch

Satoru

On Fri, Sep 16, 2011 at 2:32 PM, Markus Gutschke <mar...@chromium.org> wrote:

Yepp, those are typically the symptoms if libraries didn't load correctly or loaded to the wrong address. In that case, "gdb" can't unwind the stack correctly and you'll see funny little mistakes in the stack until "gdb" happens to resynchronize to the correct stack frame. Unfortunately, instead of reporting an actual error condition, "gdb" just does a best effort and silently computes garbage.

If the core file contains all the data that "gdb" expects, and if all the correct dynamic libraries were loaded, you should no longer see this problem.

It's been a good while since I looked into the debugging API for dynamically loaded code, and unfortunately almost all of it is completely undocumented. The current code in minidump-2-core is the result of reading through kernel, libc, dynamic loader, and gdb sources. When I last tried it, it seemed to fix all of the problems that we previously had with our core files; but it is quite possible that something has regressed since.

Markus