linux-gate.so in Linux crash dumps

[Bruce Dawson]

We get a decent number of crash reports where the crash location is linux-gate.so (@0x424, @0x425, @0x430). I understand that linux-gate.so is used to make kernel calls, but that isn't particularly helpful. What I really want is some way to decode what kernel call is being made. I'm not even sure where that information is stored -- parameters in the registers perhaps?

 

Has anybody dug in to these crashes to figure out how to analyze them more effectively? We've also had problems in the past where we don't get call stacks on linux-gate.so crashes but that seems to go in waves, and currently we are getting call stacks.

 

On the three that I'm looking at right now there is one where the caller is abort() calling raise(), another that looks like it is a call to delete (although delete isn't on the call stack), and another that looks like some sort of threading function triggered the crash.

 

I'm focusing my analysis on Ubuntu.

 

Any thoughts or help?

 

[Ted Mielczarek]

On 1/15/2013 12:50 PM, Bruce Dawson wrote:
> We get a decent number of crash reports where the crash location is
> linux-gate.so (@0x424, @0x425, @0x430). I understand that
> linux-gate.so is used to make kernel calls, but that isn't
> particularly helpful. What I really want is some way to decode what
> kernel call is being made. I'm not even sure where that information is
> stored -- parameters in the registers perhaps?
>

I don't have a real answer for you here, but if you haven't noticed
(it's not obvious), there are some linux-gate.so symbol files in the
source tree:
http://code.google.com/p/google-breakpad/source/browse/#svn%2Ftrunk%2Fsrc%2Fclient%2Flinux%2Fdata

-Ted

[Bruce Dawson]

Hey, that's perfect! I took linux-gate-intel.sym, edited it to put in the right module identifier, and then copied it to our breakpad symbol share:

 

cp linux-gate-intel.sym /mnt/symstoresymbols/linux-gate.so/0FBA9CB5B605952100008ED189E95DC30/linux-gate.so.sym

 

Note the use of the build ID and the renaming of the file. Now when I reprocess my dump files I get nicer symbol names on the top function (__kernel_vsyscall) and the call stacks are different -- better I hope.

 

I have no idea what this line means:

 

    STACK WIN 4 400 200 3 3 0 0 0 0 0 1

 

but I assume that that is some metadata to improve stack walking. Anyway, thanks!

[Asher Baker]

On Tue, Jan 15, 2013 at 7:22 PM, Bruce Dawson <bruce....@gmail.com> wrote:
> I have no idea what this line means:
>
> STACK WIN 4 400 200 3 3 0 0 0 0 0 1
>
> but I assume that that is some metadata to improve stack walking. Anyway,
> thanks!

There is a little bit of information about it here:
https://code.google.com/p/google-breakpad/wiki/LinuxSystemCalls

[Bruce Dawson]

I used this shell script (it has some dependencies on the location of our breakpad symbol store) to publish linux-get-intel.sym to a half-dozen locations based on the debug identifiers I've seen. I thought I'd share it in case anybody else finds it useful.

 

# Measure the length of the argument
argLength=$(echo $1 | wc -m)
# Subtract one for the include white space?
argLengh=$(expr $argLength - 1)
if [ 34 = $argLength ]; then
 mkdir /mnt/symstoresymbols/linux-gate.so/$1
 sed s/4FBDA58B5A1DF5A379E3CF19A235EA090/$1/ <linux-gate-intel.sym >/mnt/symstoresymbols/linux-gate.so/$1/linux-gate.so.sym
else
 echo Error! Argument length is $argLength. Debug identifiers are always
 echo Error! 34 characters. Please try again.
fi

[Mark Final]

Sorry to resurrect this, but I'm just about to look at the linux-gate symbols, and have a couple of questions.

The linux-gate-intel.sym in the Breakpad source has a module id of 4FBDA58B5A1DF5A379E3CF19A235EA090, which apparently is not static.

Bruce kindly supplied a shell script to show what he did with this - the argument to which appears to be a new module id.

Question 1) where does this new module id come from? linux-gate.so is a virtual DSO from my understanding, so doesn't exist on disk, so I cannot run dump_sym on it. I've seen it referenced in a number of minidumps generated, and this does have a different module id in. Does it vary per distro? Is it a case of chicken-and-egg, that I need a minidump to see the new module id, before I can create a symbol share to inspect the minidump properly? I'm guessing that this is the case with reference to "publish ... to a half-dozen locations based on the debug identifiers I've seen".

Question 2) is it important to have both linux-gate-intel.sym and linux-gate-amd.sym in a symbols share, in case software is run on either CPU vendor?

Question 3) I'm currently in Ubuntu 10 (kernel 2.6.32). When I run ldd on my executable, it reports linux-vdso rather than linux-gate, which are apparently the same thing (kernel versions differing, http://blogs.igalia.com/aperez/2009/01/13/the-story-of-linux-gatevdsoso/). Is the different name going to cause a problem inspecting symbols?

Thanks if anyone can offer any advice!

Mark

--
You received this message because you are subscribed to the Google Groups "google-breakpad-discuss" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-breakpad-discuss/-/h44xoFXCaVMJ.

To post to this group, send email to google-brea...@googlegroups.com.
To unsubscribe from this group, send email to google-breakpad-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-breakpad-discuss?hl=en.

--

Mark Final
Lead Dev Ops Engineer
The Foundry, 6th Floor, The Communications Building,
48 Leicester Square, London, UK, WC2H 7LT
Tel: +44(0)20 7968 6828 - Fax: +44(0)20 7930 8906

Web: www.thefoundry.co.uk
Email: mark....@thefoundry.co.uk  

The Foundry Visionmongers Ltd.
Registered in England and Wales No: 4642027

[Ted Mielczarek]

On 4/4/2013 8:15 AM, Mark Final wrote:
> Sorry to resurrect this, but I'm just about to look at the linux-gate
> symbols, and have a couple of questions.
>
> The linux-gate-intel.sym in the Breakpad source has a module id of
> 4FBDA58B5A1DF5A379E3CF19A235EA090, which apparently is not static.
> Bruce kindly supplied a shell script to show what he did with this -
> the argument to which appears to be a new module id.
>
> Question 1) where does this new module id come from? linux-gate.so is
> a virtual DSO from my understanding, so doesn't exist on disk, so I
> cannot run dump_sym on it. I've seen it referenced in a number of
> minidumps generated, and this does have a different module id in. Does
> it vary per distro? Is it a case of chicken-and-egg, that I need a
> minidump to see the new module id, before I can create a symbol share
> to inspect the minidump properly? I'm guessing that this is the case
> with reference to "publish ... to a half-dozen locations based on the
> debug identifiers I've seen".

I suspect Bruce is simply taking his existing set of crash reports,
pulling out the debug IDs for linux-gate.so from them, and then moving
the symbol files to match those IDs.

The module ID varies because the module ID we're generating is just a
hash of the first page of the virtual module's .text section, so if the
code changes then the ID will change.

> Question 2) is it important to have both linux-gate-intel.sym and
> linux-gate-amd.sym in a symbols share, in case software is run on
> either CPU vendor?

Probably, I'm not completely sure what the difference is here.

> Question 3) I'm currently in Ubuntu 10 (kernel 2.6.32). When I run ldd
> on my executable, it reports linux-vdso rather than linux-gate, which
> are apparently the same thing (kernel versions differing,
> http://blogs.igalia.com/aperez/2009/01/13/the-story-of-linux-gatevdsoso/).
> Is the different name going to cause a problem inspecting symbols?
>

Nope, Breakpad finds linux-gate by referring to an entry in
/proc/self/auxv, not by name[1], and always writes "linux-gate.so"[2] as
the module name for it.

It strikes me that Breakpad could do better in this case, and have some
baked-in knowledge of how to unwind out of this module. I'm not sure how
hard that would be to implement in the general sense.

-Ted

1.
http://code.google.com/p/google-breakpad/source/browse/trunk/src/client/linux/minidump_writer/linux_dumper.cc#168
2.
http://code.google.com/p/google-breakpad/source/browse/trunk/src/client/linux/minidump_writer/linux_dumper.h#65