[chromium-dev] Linker crash on Snow Leopard

[Kenneth Russell]

Is anyone seeing a linker crash while building Chromium
Framework.framework on Snow Leopard? I'm running XCode 3.2.2 on Mac OS
X 10.6.3. I thought at first that the changes I'm working on were
provoking it, but at this point I'm pretty sure that top of tree
Chromium has the problem. (I'm also syncing to top of tree WebKit.)

mmentovai gave me a patch to get Apple's ld64-97.2 to build, but, once
built, this (32-bit) ld does not crash in either Debug or Release
mode.

I'll start trying to figure out what's causing the crash. There was a
similar crash a few months ago related to the use of anonymous
namespaces.

-Ken

--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Kenneth Russell]

On Wed, May 12, 2010 at 4:33 PM, Kenneth Russell <k...@chromium.org> wrote:
> Is anyone seeing a linker crash while building Chromium
> Framework.framework on Snow Leopard? I'm running XCode 3.2.2 on Mac OS
> X 10.6.3. I thought at first that the changes I'm working on were
> provoking it, but at this point I'm pretty sure that top of tree
> Chromium has the problem. (I'm also syncing to top of tree WebKit.)
>
> mmentovai gave me a patch to get Apple's ld64-97.2 to build, but, once
> built, this (32-bit) ld does not crash in either Debug or Release
> mode.
>
> I'll start trying to figure out what's causing the crash. There was a
> similar crash a few months ago related to the use of anonymous
> namespaces.

OK, after syncing another workspace to top-of-tree I'm not seeing this
linker crash any more, even after applying my changes. I'll keep the
old workspace around in case we want to try to debug the crash
further, and keep my fingers crossed that it doesn't happen again.

[Robert Sesek]

On Wed, May 12, 2010 at 8:14 PM, Kenneth Russell <k...@chromium.org> wrote:

On Wed, May 12, 2010 at 4:33 PM, Kenneth Russell <k...@chromium.org> wrote:
> Is anyone seeing a linker crash while building Chromium
> Framework.framework on Snow Leopard? I'm running XCode 3.2.2 on Mac OS
> X 10.6.3. I thought at first that the changes I'm working on were
> provoking it, but at this point I'm pretty sure that top of tree
> Chromium has the problem. (I'm also syncing to top of tree WebKit.)
>
> mmentovai gave me a patch to get Apple's ld64-97.2 to build, but, once
> built, this (32-bit) ld does not crash in either Debug or Release
> mode.
>
> I'll start trying to figure out what's causing the crash. There was a
> similar crash a few months ago related to the use of anonymous
> namespaces.

OK, after syncing another workspace to top-of-tree I'm not seeing this
linker crash any more, even after applying my changes. I'll keep the
old workspace around in case we want to try to debug the crash
further, and keep my fingers crossed that it doesn't happen again.

-Ken

Rohit and I had seen ld segfault before, and we created http://code.google.com/p/chromium/issues/detail?id=40845 for it.  The issue went away on its own, though.  But it'd be good to understand why it happens.

rsesek / @chromium.org 

[Mark Mentovai]

We can build our own copies of the 10.6/Xcode 3.2 ld64 now, so if this
happens again, please let me know.

Mark

Robert Sesek wrote:
> On Wed, May 12, 2010 at 8:14 PM, Kenneth Russell <k...@chromium.org> wrote:
>>
>> On Wed, May 12, 2010 at 4:33 PM, Kenneth Russell <k...@chromium.org> wrote:
>> > Is anyone seeing a linker crash while building Chromium
>> > Framework.framework on Snow Leopard? I'm running XCode 3.2.2 on Mac OS
>> > X 10.6.3. I thought at first that the changes I'm working on were
>> > provoking it, but at this point I'm pretty sure that top of tree
>> > Chromium has the problem. (I'm also syncing to top of tree WebKit.)
>> >
>> > mmentovai gave me a patch to get Apple's ld64-97.2 to build, but, once
>> > built, this (32-bit) ld does not crash in either Debug or Release
>> > mode.
>> >
>> > I'll start trying to figure out what's causing the crash. There was a
>> > similar crash a few months ago related to the use of anonymous
>> > namespaces.
>>
>> OK, after syncing another workspace to top-of-tree I'm not seeing this
>> linker crash any more, even after applying my changes. I'll keep the
>> old workspace around in case we want to try to debug the crash
>> further, and keep my fingers crossed that it doesn't happen again.
>>
>> -Ken
>
> Rohit and I had seen ld segfault before, and we created
> http://code.google.com/p/chromium/issues/detail?id=40845 for it.  The issue
> went away on its own, though.  But it'd be good to understand why it
> happens.

[Kenneth Russell]

Got the crash again with a debug build of the x86_64 ld via your patch
(thanks). Here's the breakdown and the bug:

(gdb) where
#0 0x00007fff8084f730 in strcpy ()
#1 0x00000001000274c6 in canonicalizeAnonymousName
(inSymbol=0x1856ecf6d "__ZN5gles212_GLOBAL__N_116g_gl_context_keyE",
outSymbol=0x7fff5fbfada0 "__ZN5gles-") at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:2045
#2 0x000000010002777f in Linker::findAtom (this=0x7fff5fbfb2e0,
orderedSymbol=@0x101007e00) at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:2070
#3 0x0000000100028879 in Linker::sortAtoms (this=0x7fff5fbfb2e0) at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:2293
#4 0x000000010002fa3a in Linker::link (this=0x7fff5fbfb2e0) at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:837
#5 0x000000010003022e in main (argc=143, argv=0x7fff5fbfbc08) at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:4283
(gdb) up
#1 0x00000001000274c6 in canonicalizeAnonymousName
(inSymbol=0x1856ecf6d "__ZN5gles212_GLOBAL__N_116g_gl_context_keyE",
outSymbol=0x7fff5fbfada0 "__ZN5gles-") at
/Users/kbr/Apple/ld64-97.2/src/ld/ld.cpp:2045
2045 strcpy(&outSymbol[startLen+1], globEndPtr);
(gdb) p startLen
$1 = 9
(gdb) p globEndPtr
$2 = 0x1856ed04d <Address 0x1856ed04d out of bounds>
(gdb) p globPtr
$3 = 0x1856ecf75 "s212_GLOBAL__N_116g_gl_context_keyE"
(gdb) p length
$4 = 212
(gdb) p inSymbol
$5 = 0x1856ecf6d "__ZN5gles212_GLOBAL__N_116g_gl_context_keyE"
(gdb) p endptr
$6 = 0x1856ecf79 "_GLOBAL__N_116g_gl_context_keyE"

The source code (src/gpu/command_buffer/client/gles2_lib.cc) looks like this:

namespace gles2 {
namespace {
gpu::ThreadLocalKey g_gl_context_key;
} // namespace anonymous
...
}

The bug is in gcc's name mangling. The "2" at the end of "gles2" is
being confused with the length of a portion of the mangled string,
because the identifier is in an anonymous namespace.

I have no idea why this doesn't core dump the 32-bit ld, but
experience showed that it doesn't.

The workaround from our standpoint is to take this identifier out of
the anonymous namespace; I'll do that now. Any idea how to fix this
properly?

Thanks,

-Ken

[Mark Mentovai]

Kenneth Russell wrote:
> Got the crash again with a debug build of the x86_64 ld via your patch
> (thanks). Here's the breakdown and the bug:

[...]

> The bug is in gcc's name mangling. The "2" at the end of "gles2" is
> being confused with the length of a portion of the mangled string,
> because the identifier is in an anonymous namespace.

To be clear, the bug is in Apple ld, not in gcc. There’s nothing wrong
with your mangled name, __ZN5gles212_GLOBAL__N_116g_gl_context_keyE:

C++ decorated name
nested
length 5, gles2
length 12, _GLOBAL__N_1 (anonymous namespace)
length 16, g_gl_context_key
end

or

gles2::(anonymous namespace)::g_gl_context_key

The bug is positively in ld64’s canonicalizeAnonymousName, which takes
a fatal shortcut. Instead of parsing a symbol as a C++ decorated name,
it does a simple string search for “_GLOBAL__N_” and then looks at
everything before it that’s a digit.

This bug can also be triggered by hand-crafting names that contain
_GLOBAL__N_ - such names are, of course, invalid because C++ reserves
names with two consecutive underscores, but there’s nothing preventing
the programmer from writing these names, and from potentially causing
all sorts of mischief. usesAnonymousNamespace has a similar bug but
its negative impact is more limited.

For that matter, the code is entirely ignorant of nested anonymous
namespaces. (I’m not sure how to properly handle these, it requires a
little bit more study.)

> I have no idea why this doesn't core dump the 32-bit ld, but
> experience showed that it doesn't.

Luck. globEndPtr points to memory 212 bytes beyond the first character
of _GLOBAL__N_. In the 64-bit environment and in this very specific
case, as luck would have it, that’s unmapped memory, so you crash.
When you’re running in 32-bit mode in this very specific case, that
pointer references mapped memory and there’s a 0 byte somewhere after.
The function will return garbage and can even corrupt the stack, so
it’s still broken, it just doesn’t crash.

> The workaround from our standpoint is to take this identifier out of
> the anonymous namespace; I'll do that now. Any idea how to fix this
> properly?

Yeah, these functions need to be rewritten. I’ll file a bug with Apple
tomorrow and propose a patch to them. With luck, we can get a fix
rolled into the next Xcode release.

Mark

[Kenneth Russell]

On Thu, May 13, 2010 at 8:04 PM, Mark Mentovai <ma...@chromium.org> wrote:
> Kenneth Russell wrote:
>> Got the crash again with a debug build of the x86_64 ld via your patch
>> (thanks). Here's the breakdown and the bug:
> [...]
>> The bug is in gcc's name mangling. The "2" at the end of "gles2" is
>> being confused with the length of a portion of the mangled string,
>> because the identifier is in an anonymous namespace.
>
> To be clear, the bug is in Apple ld, not in gcc. There’s nothing wrong
> with your mangled name,  __ZN5gles212_GLOBAL__N_116g_gl_context_keyE:
>
> C++ decorated name
> nested
> length 5, gles2
> length 12, _GLOBAL__N_1 (anonymous namespace)
> length 16, g_gl_context_key
> end
>
> or
>
> gles2::(anonymous namespace)::g_gl_context_key
>
> The bug is positively in ld64’s canonicalizeAnonymousName, which takes
> a fatal shortcut. Instead of parsing a symbol as a C++ decorated name,
> it does a simple string search for “_GLOBAL__N_” and then looks at
> everything before it that’s a digit.

Ah, I see. Thanks for pointing out how the mangled name is deconstructed.

> This bug can also be triggered by hand-crafting names that contain
> _GLOBAL__N_ - such names are, of course, invalid because C++ reserves
> names with two consecutive underscores, but there’s nothing preventing
> the programmer from writing these names, and from potentially causing
> all sorts of mischief. usesAnonymousNamespace has a similar bug but
> its negative impact is more limited.
>
> For that matter, the code is entirely ignorant of nested anonymous
> namespaces. (I’m not sure how to properly handle these, it requires a
> little bit more study.)
>
>> I have no idea why this doesn't core dump the 32-bit ld, but
>> experience showed that it doesn't.
>
> Luck. globEndPtr points to memory 212 bytes beyond the first character
> of _GLOBAL__N_. In the 64-bit environment and in this very specific
> case, as luck would have it, that’s unmapped memory, so you crash.
> When you’re running in 32-bit mode in this very specific case, that
> pointer references mapped memory and there’s a 0 byte somewhere after.
> The function will return garbage and can even corrupt the stack, so
> it’s still broken, it just doesn’t crash.

I was more surprised that the link appeared to complete successfully.

>> The workaround from our standpoint is to take this identifier out of
>> the anonymous namespace; I'll do that now. Any idea how to fix this
>> properly?
>
> Yeah, these functions need to be rewritten. I’ll file a bug with Apple
> tomorrow and propose a patch to them. With luck, we can get a fix
> rolled into the next Xcode release.

Thanks a lot for your help and for taking care of this.

-Ken

[Mark Mentovai]

I filed radar 8010332 with Apple. See
http://openradar.appspot.com/radar?id=369401. I’m attaching my test
case and proposed ld patch in case anyone’s interested.

Mark