Truncated stacks

[Benjamin Smedberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hey y'all, I'm having issues with truncated breakpad stacks, and wondered if
I could get some suggestions how to debug:

In Firefox I wrote a little tool that will corrupt memory and cause
semi-random crashes. When I get the breakpad reports back, the main thread
(which is also the crashing thread) never has more than about three frames
listed. See, for instance http://benjamin.smedbergs.us/tests/short-stack

Note that the stack for all the non-main/crashing threads are complete. Only
the stack for the main thread is truncated.

At first mento suggested looking at the particular functions involved, to
see if breakpad isn't calculating the return address for particular
functions. But since the crashes appear in many different functions, I don't
think this is the case. I think that probably the stack is being truncated
or miscalculated somewhere along the way.

I'm trying to debug, but I probably don't know what I'm looking for.

In MinidumpProcessor::ProcessResult

p *threads->GetThreadAtIndex(0)
... thread_id = 2692 ... memory = {data_size = 4824}

Is threads->GetThreadAtIndex(0) the "Thread 0" listed in the report?
requesting_thread_id is 2692, so this appears to be the crashing/main thread.

p *threads->GetThreadAtIndex(4)
... memory = {data_size = 13224}

This thread is a worker thread which should have a short stack, so I'm
wondering why it has a data_size so large, which looks like it would be
about right for the main thread.

For reference, this is a client library from this week, and a server library
from late last week.

If you'd like to take a look, I've uploaded
http://benjamin.smedbergs.us/tests/short-stack.dump

And, though I doubt it will be helpful,
http://benjamin.smedbergs.us/tests/firefox.sym (46MB)

- --BDS

- --

Benjamin Smedberg
Platform Guru
Mozilla Corporation
benj...@smedbergs.us
http://benjamin.smedbergs.us/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQF/QSSwGp5sTYNkRAoPqAKDpQjEG6gIui08N2BIXYY6+62ln1QCfRaSU
lTG0n2Udv93AesEAPqiYPj4=
=aBiR
-----END PGP SIGNATURE-----

[Benjamin Smedberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Benjamin Smedberg wrote:

> At first mento suggested looking at the particular functions involved, to
> see if breakpad isn't calculating the return address for particular
> functions. But since the crashes appear in many different functions, I don't
> think this is the case. I think that probably the stack is being truncated
> or miscalculated somewhere along the way.

Or something else is going on. I've debugged it down to
StackwalkerX86::GetCallerFrame:

last_frame_info->program string is non-empty and we use it. It is:

"$T2 $esp .cbLocals + .cbSavedRegs + = $T0 .raSearchStart = $eip $T0 ^ =
$esp $T0 4 + ="

This string doesn't ever set $ebp, so when we evaluate it, we end up with
dictionary_validity.find("$ebp") == dictionary_validity.end()

- --BDS

- --

Benjamin Smedberg
Platform Guru
Mozilla Corporation
benj...@smedbergs.us
http://benjamin.smedbergs.us/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQJrsSSwGp5sTYNkRAiw5AJ0Svcmc77hK14AF+YqtbbN3MlVy/QCbBkcb
F0dPHXQxR5Trpausks/IXWE=
=Zd1h
-----END PGP SIGNATURE-----

[Mark Mentovai]

Ah, good catch. Can you walk further if you get rid of that %ebp
check at stackwalker_x86.cc:291? Strictly speaking, we don't need to
treat it as required as we do with the other two.

Mark

[Benjamin Smedberg]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Benjamin Smedberg wrote:

> "$T2 $esp .cbLocals + .cbSavedRegs + = $T0 .raSearchStart = $eip $T0 ^ =
> $esp $T0 4 + ="
>
> This string doesn't ever set $ebp, so when we evaluate it, we end up with
> dictionary_validity.find("$ebp") == dictionary_validity.end()

I've filed issue 165 and attached a potential patch that works for me.
Mento, could you review and commit if acceptable? You added the overzealous
check in revision 42.

- --BDS

- --

Benjamin Smedberg
Platform Guru
Mozilla Corporation
benj...@smedbergs.us
http://benjamin.smedbergs.us/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGQLGESSwGp5sTYNkRAhRuAJ4tnUeeevFFnKhEXsRNMSAw21Nj1ACghV9X
m3hqPOhHDG3BnkgYaeZUi3o=
=TXdi
-----END PGP SIGNATURE-----

[Mark Mentovai]

That looks good to me. I also adjusted another program string that
reset %ebp to itself just to pass the check that's no longer there,
and fixed a couple of comments to match the new code. Thanks for the
bug report and the debugging. Checked in at [163].

Mark

On 5/8/07, Benjamin Smedberg <benj...@smedbergs.us> wrote:
>