About heap memory allocation in chromium browser

[Pengfei Sun]

Hi guys,

I wonder whether TCMALLOC contributes all heap memory allocation of chrome? Tcmalloc overrides malloc, realloc, calloc, new, free and delete, right? If I want to override heap memory allocation functions, I just need to override or change TCMALLCO functions?

I override malloc and new by LD_PRELOAD my modified version. It just injects a few heap memory allocation.

Thank you very much!

Best Regards,

Pengfei

[Peter Kasting]

On Thu, Nov 5, 2015 at 10:26 AM, Pengfei Sun <shaot...@gmail.com> wrote:

I wonder whether TCMALLOC contributes all heap memory allocation of chrome?

No.  See e.g. PartitionAlloc for Blink.

PK 

[Elliott Sprehn]

Chromium uses several memory allocators in different situations, it also varies by platform.

What are you trying to do?

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Pengfei Sun]

Hi Elliott,  I work on linux platform. I want to do heap memory analysis. So I want to add the identifiable information into each heap memory allocation by overriding heap memory allocation functions.

[Pengfei Sun]

I check heap profiling in memory-infra. In this document, they mention Chrome doesn’t allocate memory only through malloc (most pieces are moving away from that). Lot of custom allocators: PartitionAlloc in Blink, the new blink GC / Oilpan, custom allocators for graphics. They also have one document for a more complete survey. But I don't have permission to view this document.

[Elliott Sprehn]

+primiano

Indeed, you might want to talk to someone on memory-infra about what you're trying to accomplish.

[Pengfei Sun]

Yes. I do want. Thanks Elliott.

[Primiano Tucci]

Hello!

Short version

• Chrome uses a wide variety of (often home-brewed) allocators.
  
• Intercepting malloc/free is going to tell you just a small part of the truth
  
• We are building chrome-specific tools to get this data, be patient :) A bleeding edge version is here
  

Longer version

So the situation is this. Chrome uses a wide variety of allocators, depending on the target OS and process (browser vs renderer) you get a different story.

Background context

Most of the native allocations in the renderer (read: the one not coming from the JavaScript world) today happen via PartitionAlloc. Imagine that as a drop-in replacement for malloc. However, blink these days is gradually migrating to a GC world called BlinkGC/Oilpan (you can get more context about that searching for it in blink-dev). The amount of allocations made in the renderer via TCMalloc (or any other system allocator) is marginal.

In the browser process, the situation instead is a bit different and really OS dependent. To the best of my knowledge today:

On Android: we rely on the system allocator provided by Bionic's libc (the Android framework c library) which can be either dlmalloc or jemalloc, depending on the BSP config. See this older post of mine for more details.

On Linux Desktop: we use our own copy of TCMalloc.

On Mac: we use the system allocator, which happens to be TCMalloc (but not 100% the same version we have in Linux).

On Windows: we use mostly WinHeap.

Heap profiling in Chrome

We have an initiative called memory-infra aimed at creating memory profiling infrastructure to debug memory usage inside of Chrome. You can read about that in this wiki page.

Very recently we started working on a heap profiler (as linked in the doc above) to get details about allocations in Chrome.

As you can imagine form the context above, it's tricky due to the number of allocators involved.

In yesterday's canary, we released a first version of the heap profiler, which is able to give details and cluster PartitionAlloc allocations by pseudo stack traces. You can read more in this mail thread and find instructions in this wiki page. I didn't announce widely as it is a very bleeding edge feature (it literally landed yesterday).

More things are going to happen on this profiler side, namely: having a way to cluster allocations by object-type; being able to profile allocations also on the browser side.

About LD_PRELOAD

I think I know precisely what you are trying to do. I did it myself here for this Android memory profiler tool.

It is very tricky and the quick answer is that, due to the reasons explained above, it does not give a complete answer for Chrome. Intercepting malloc/free will not tell you anything at all about:  the renderer process (because we use our own allocators); the browser process on Linux or Windows (where we don't use system malloc)

On top of that, is very hard to get it right (expect calls to malloc before the runtime linking is completed) and you have better chances to use existing tools (gperftools, Mac Instruments, Visual studio 2015 profiler, Android memory inspector) rather than writing your own. Even if you do a perfect job, you will still miss most of the allocations in chrome.
Without mentioning the graphics pipeline, which is a completely different story (see this wiki page)

So, my best suggestion would be: wait for the heap profiler in chrome://tracing to happen or try one of the existing tools, don't invent your own. 

The tricky part is that is very hard to create heap profiling tools that scale with Chrome's size. During loading / rendering we reach peaks of 400 K allocations per second. Be prepared to scale with that size. Most of the tools I tried choke after profiling Chrome for more than a bunch of seconds. 

I hope this is enough as a first answer, but if you want to know more just ask :) 

[Pengfei Sun]

Hi Primiano,

Thanks for the whole detail about chrome. I have learned many from your answer. I focus on linux platform now. According to your answer, there are PartitionAlloc (render), BlinkGC/Oilpan(blink), TCMalloc(render marginal and browser process), and malloc for linux platform. If I just intercept them, do you think I can get all heap memory allocation without GPU memory. Or do you think there is still other home-brewed allocators.

For existing tools, gperftools is only for TCMalloc, which means I cannot get PartitionAlloc and Oilpan. I really look forward to your heap profiling tools.

Thank you very much!

Best Regards,

Pengfei

[Daniel Bratell]

On Fri, 06 Nov 2015 04:43:15 +0100, Pengfei Sun <shaot...@gmail.com> wrote:

Hi Primiano,

Thanks for the whole detail about chrome. I have learned many from your answer. I focus on linux platform now. According to your answer, there are PartitionAlloc (render), BlinkGC/Oilpan(blink), TCMalloc(render marginal and browser process), and malloc for linux platform. If I just intercept them, do you think I can get all heap memory allocation without GPU memory. Or do you think there is still other home-brewed allocators.

For existing tools, gperftools is only for TCMalloc, which means I cannot get PartitionAlloc and Oilpan. I really look forward to your heap profiling tools.

One thing that might help you a bit if you don't want to wait is to set the gyp variable build_for_tool=memcheck and then rebuild the gyp files. That will make many of the custom allocators use standard malloc/free instead and you can use some of the common tools to study memory usage.

GYP_DEFINES="build_for_tool=memcheck profiling=1" build/gyp_chromium

# Run ninja

...

# Now you have a binary that uses standard malloc/free more than otherwise.

Longer term, memory-infra is the plan for understanding memory usage regardless of platform.

/Daniel

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Pengfei Sun]

Hi Daniel,

I apply your solution to my last compiled chrome. When I run ninja, it show me there is no work to do.

sun@sun:~/project/chromium/src$ ninja -C out/Release.gn chrome chrome_sandbox

ninja: Entering directory `out/Release.gn'

ninja: no work to do.

Do you think where is the problem?

Thank you very much!

Best Regards,

Pengfei

[Pengfei Sun]

I build the 32-bit chrome in 64-bit Ubuntu machine by sysroot. Do you think it is the problem that ninja is no work to do?

[Torne (Richard Coles)]

If you're building with gn then setting GYP_DEFINES and running gyp won't do anything, you need to find and set the equivalent gn parameter instead.

[Daniel Bratell]

On Fri, 06 Nov 2015 22:01:24 +0100, Torne (Richard Coles) <to...@chromium.org> wrote:

If you're building with gn then setting GYP_DEFINES and running gyp won't do anything, you need to find and set the equivalent gn parameter instead.

Yes, sorry about that. My recipe was for building with gyp, but I think you will be able to find something similar in the gn build system.