Crash due to tcmalloc's debugallocation on Windows

[Marshall Greenblatt]

Hi All,

The below crash occurs while loading the Chromium Embedded Framework (CEF) DLL on Windows. It started when I synced past Chromium revision 200327 which enables tcmalloc's debugallocation. If I disable debugallocation by setting disable_debugallocation=1 in GYP then the crash disappears and the application loads normally. Any suggestions on what this crash means and whether it's a bug in my application or in debugallocation?

Thanks,

Marshall

First-chance exception at 0x0f69e8c4 (libcef.dll) in cefclient.exe: 0xC0000005: Access violation reading location 0x0287e2ec.

> libcef.dll!TCMalloc_PageMap1_LazyCommit<20>::get(unsigned int k)  Line 297 + 0x8 bytes C++

  libcef.dll!tcmalloc::PageHeap::GetDescriptor(unsigned int p)  Line 145 C++

  libcef.dll!TCMallocImplementation::GetOwnership(const void * ptr)  Line 805 + 0x10 bytes C++

  libcef.dll!DebugMallocImplementation::GetOwnership(const void * p)  Line 1065 + 0xc bytes C++

  libcef.dll!DebugMallocImplementation::GetAllocatedSize(const void * p)  Line 1055 + 0x13 bytes C++

  libcef.dll!_msize(void * p)  Line 232 + 0x1b bytes C++

  libcef.dll!_onexit_nolock(int (void)* func)  Line 118 + 0x6 bytes C

  libcef.dll!_onexit(int (void)* func)  Line 87 + 0x8 bytes C

  libcef.dll!atexit(void (void)* func)  Line 162 + 0x8 bytes C

  libcef.dll!`anonymous namespace'::`dynamic initializer for 'google_initializer_module_debugallocation''()  Line 1094 + 0x23 bytes C++

  libcef.dll!_cinit(int initFloatingPrecision)  Line 288 + 0x19 bytes C

  libcef.dll!_CRT_INIT(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 132 + 0x19 bytes C

  libcef.dll!__DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 324 + 0x8 bytes C

  libcef.dll!_DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 293 + 0xe bytes C

  ntdll.dll!77bc9950()

[William Chan (陈智昌)]

I'm fairly out of date with our current TCMalloc use, but I didn't know we were using debugallocation on Windows. This stack looks really weird. It appears to be a startup crash, that's somehow invoking atexit()? Where's the code for this initializer? I can't find it in cs.chromium.org. Sounds like a bug in the static initializer.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

[Paweł Hajdan, Jr.]

Marshall, this could be related to how you build CEF in a separate DLL. I think the entire binary should use the same memory allocator for things to stay sane.

Paweł

On Mon, Jun 3, 2013 at 4:02 PM, Marshall Greenblatt <magree...@gmail.com> wrote:

[Marshall Greenblatt]

On Jun 3, 2013, at 7:19 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Marshall, this could be related to how you build CEF in a separate DLL. I think the entire binary should use the same memory allocator for things to stay sane.

Thanks, I'll try building the executable with tcmalloc to see if that resolves the problem. If that's the only solution then I'll likely have to disable debugallocation when building CEF since not all users will be able/willing to use tcmalloc with their applications.

Is debugallocation enabled by default on OS-X? I didn't experience any crashes on that platform even though CEF is loaded from a dylib.

[Vinit]

I am getting something similar behavior while having problem loading chrome.dll, posted it here. https://groups.google.com/a/chromium.org/forum/?fromgroups#!topic/chromium-dev/uF_U3sPjUaQ

[Vinit]

snapshot of my stack for your reference, just in case these two are related. (sorry for the spam but I thought it was kind of hard to find from the discussion link, and I wanted to be specific).

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev
 
 
 

--
Vinit

[chuang zheng]

I have met the same problems, with the latest code

2013/6/4 Vinit <shorte...@gmail.com>

[Marshall Greenblatt]

On Mon, Jun 3, 2013 at 7:10 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

I'm fairly out of date with our current TCMalloc use, but I didn't know we were using debugallocation on Windows. This stack looks really weird. It appears to be a startup crash, that's somehow invoking atexit()? Where's the code for this initializer? I can't find it in cs.chromium.org. Sounds like a bug in the static initializer.

The code is at https://code.google.com/p/chromium/codesearch#chromium/src/third_party/tcmalloc/vendor/src/debugallocation.cc&q=DebugMallocImplementation&sq=package:chromium&type=cs&l=1002

[Daniel Bratell]

Den 2013-06-04 01:02:47 skrev Marshall Greenblatt <magree...@gmail.com>:

> Hi All,
>
> The below crash occurs while loading the Chromium Embedded Framework
> (CEF)
> DLL on Windows. It started when I synced past Chromium revision 200327
> which enables tcmalloc's debugallocation. If I disable debugallocation by
> setting disable_debugallocation=1 in GYP then the crash disappears and
> the
> application loads normally. Any suggestions on what this crash means and
> whether it's a bug in my application or in debugallocation?

I get the same in a normal contentshell.exe (statically built) from trunk
today so I don't think it's you.

I debugged it as far as

REGISTER_MODULE_INITIALIZER(debugallocation, {
// Either we or valgrind will control memory management. We
// register our extension if we're the winner. Otherwise let
// Valgrind use its own malloc (so don't register our extension).
if (!RunningOnValgrind()) {
MallocExtension::Register(&debug_malloc_implementation);
}
});

generating code that calls MallocExtension::Register and that registers an
atexit listener:

062B093F mov ecx,offset google_initializer_module_debugallocation
(0C6EAE1Ch)
062B0944 call GoogleInitializer::GoogleInitializer (5A0590h)
062B0949 push offset `anonymous namespace'::`dynamic atexit
destructor for 'google_initializer_module_debugallocation'' (637BFE0h)
062B094E call atexit (0F03410h)

And atexit wants to look at the heap to see if it needs to resize its
table of functions and then it crashes inside DebugMallocImplementation.

But thanks for the disable_debugallocation=1 suggestion, now I might get
further in my work.

/Daniel

[William Chan (陈智昌)]

So fundamentally what's going on is we aren't initializing the PageHeap (either at all, or in the wrong order). That means we're probably not executing the TCMallocGuard (which will register it). In our very complicated TCMalloc setup code, we leverage allocator_shim.cc to hook _heap_init(), which will invoke the TCMallocGuard.

If people who hit this bug could run their code in the debugger to see if TCMallocGuard is getting executed, that'd narrow down the problem. If it's not getting executed before the crash, then that initialization code is the problem. Of course, I have no idea why 200327 would have changed this.

[Vinit]

I can confirm that TCMallocGuard is being executed in _heap_init which basically news a TCMallocGuard object which will leak for lifetime of the process. Although I do not know where the heap allocation is taking place in the constructor of TCMallocGuard, I saw ThreadCache::InitTSD and ReplaceSystemAlloc (which doesn't do anything) being executed. 

Note : my problem is somewhat different from the OP's. 

--
Vinit

[Marshall Greenblatt]

On Tue, Jun 4, 2013 at 12:53 PM, William Chan (陈智昌) <will...@chromium.org> wrote:

So fundamentally what's going on is we aren't initializing the PageHeap (either at all, or in the wrong order). That means we're probably not executing the TCMallocGuard (which will register it). In our very complicated TCMalloc setup code, we leverage allocator_shim.cc to hook _heap_init(), which will invoke the TCMallocGuard.

If people who hit this bug could run their code in the debugger to see if TCMallocGuard is getting executed, that'd narrow down the problem. If it's not getting executed before the crash, then that initialization code is the problem. Of course, I have no idea why 200327 would have changed this.

The TCMallocGuard constructor is getting executed before the crash.

[Justin Schuh]

[+wfh +jar]

I believe these two have been looking into this.

[Dai Mikurube]

On Tue, Jun 4, 2013 at 8:42 AM, Marshall Greenblatt <magree...@gmail.com> wrote:

On Jun 3, 2013, at 7:19 PM, Paweł Hajdan, Jr. <phajd...@chromium.org> wrote:

Marshall, this could be related to how you build CEF in a separate DLL. I think the entire binary should use the same memory allocator for things to stay sane.

Thanks, I'll try building the executable with tcmalloc to see if that resolves the problem. If that's the only solution then I'll likely have to disable debugallocation when building CEF since not all users will be able/willing to use tcmalloc with their applications.

Is debugallocation enabled by default on OS-X? I didn't experience any crashes on that platform even though CEF is loaded from a dylib.

I'm not sure of debugallocation at all (it's done by Pawel), but I believe tcmalloc is not used in Mac OS.

 

Paweł

On Mon, Jun 3, 2013 at 4:02 PM, Marshall Greenblatt <magree...@gmail.com> wrote:

Hi All,

The below crash occurs while loading the Chromium Embedded Framework (CEF) DLL on Windows. It started when I synced past Chromium revision 200327 which enables tcmalloc's debugallocation. If I disable debugallocation by setting disable_debugallocation=1 in GYP then the crash disappears and the application loads normally. Any suggestions on what this crash means and whether it's a bug in my application or in debugallocation?

Thanks,

Marshall

First-chance exception at 0x0f69e8c4 (libcef.dll) in cefclient.exe: 0xC0000005: Access violation reading location 0x0287e2ec.

> libcef.dll!TCMalloc_PageMap1_LazyCommit<20>::get(unsigned int k)  Line 297 + 0x8 bytes C++

  libcef.dll!tcmalloc::PageHeap::GetDescriptor(unsigned int p)  Line 145 C++

  libcef.dll!TCMallocImplementation::GetOwnership(const void * ptr)  Line 805 + 0x10 bytes C++

  libcef.dll!DebugMallocImplementation::GetOwnership(const void * p)  Line 1065 + 0xc bytes C++

  libcef.dll!DebugMallocImplementation::GetAllocatedSize(const void * p)  Line 1055 + 0x13 bytes C++

  libcef.dll!_msize(void * p)  Line 232 + 0x1b bytes C++

  libcef.dll!_onexit_nolock(int (void)* func)  Line 118 + 0x6 bytes C

  libcef.dll!_onexit(int (void)* func)  Line 87 + 0x8 bytes C

  libcef.dll!atexit(void (void)* func)  Line 162 + 0x8 bytes C

  libcef.dll!`anonymous namespace'::`dynamic initializer for 'google_initializer_module_debugallocation''()  Line 1094 + 0x23 bytes C++

  libcef.dll!_cinit(int initFloatingPrecision)  Line 288 + 0x19 bytes C

  libcef.dll!_CRT_INIT(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 132 + 0x19 bytes C

  libcef.dll!__DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 324 + 0x8 bytes C

  libcef.dll!_DllMainCRTStartup(void * hDllHandle, unsigned long dwReason, void * lpreserved)  Line 293 + 0xe bytes C

  ntdll.dll!77bc9950()

--

Dai MIKURUBE

   dmik...@chromium.org

[Will Harris]

I'm also seeing this issue, and am working with jar@ to try and diagnose it.  I'm tracking the issue here:

https://code.google.com/p/chromium/issues/detail?id=246933

Will

--