devtools profiler memory versus task manager memory

[marco.l...@gmail.com]

Hi,

I am currently investigating memory usage of my web application. I have done the necessary research on the memory usage and when looking at the developer tools timeline I have a nice sawtooth when repeatedly executing a action. So far so good, however when I monitor the memory usage at OS level the memory usage has grows. 

I would expect to see the same sawtooth, or at least that he memory stabilizes after the first couple of runs. Are my expectations incorrect or am I missing something ? How should I further investigate this issue ? 

Regards,

Marco

[Alexei Filippov]

Hi Marco,

There are several memory usage classes shown in DevTools and Chrome.

Heap profiler always show only memory occupied by live JavaScript objects.

Timeline shows JS heap used by all allocated objects. That includes both live and garbage that is not yet collected. That's why you see the saw graph -- objects are being allocated so the graph goes up and eventually GC kicks in and frees unused objects aka garbage.

When V8 allocates memory for JS heap from the OS it does not do the allocation for each object but rather acquires it in big chunks. So it usually has some extra memory that is committed from the OS but isn't used by any heap objects yet. That's the total heap memory you can see in the Chrome task manager.

If your application's memory usage does not grow so shouldn't do the total heap. It should be within several dozens percents above the peaks of the saw. Do you experience something more than that?

Alexei

25 июля 2013 г. 17:18 пользователь <marco.l...@gmail.com> написал:

--
You received this message because you are subscribed to the Google Groups "Google Chrome Developer Tools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-chrome-develo...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Marco Laponder]

Alexei,

Thanks for your response, it is more clear to me now. I understand that the memory is the Chrome task manager is higher than in the timeline heap. But what 'worries' me is is that the max in the timeline hardly grows on large repeated runs, but the memory on the Chrome task manager keeps hitting higher values.

What I did is start my application and repeat an action 20 times. I used this as my baseline  (to make sure we did reach all the code, cache were filled etc). What I was expecting is when I did another run of the same action another 100 times, the max in the Chrome task manager would not increase, however this seems to be increasing from 35MB upto an 60MB, while the timeline javascript heap memory graph did not exceed the 19MB. When I did another run of 100 after this, the memory reaches max of even 80MB. 

WHen I trigger manually the GC by the garbage collect in the timeline, the memory in the task manager is after the first 20 runs 20MB, after 100 additional runs 48MB en after another 100 additional runs 66MB, while the js memory drops to around 4MB. I have attached some screenshots which might make things even more clear.

It looks like some memory is getting lost outside the javascript heap, or am I jumping to conclusions ? How would you propose to further investigate this problem ? I am relatively new with the memory tools in Chrome, so any tips are appreciated. 

Kind regards,

Marco

2013/7/25 Alexei Filippov <al...@google.com>

[Yury Semikhatsky]

Hi Marco,

On Fri, Jul 26, 2013 at 10:55 AM, Marco Laponder <marco.l...@gmail.com> wrote:

Alexei,

Thanks for your response, it is more clear to me now. I understand that the memory is the Chrome task manager is higher than in the timeline heap. But what 'worries' me is is that the max in the timeline hardly grows on large repeated runs, but the memory on the Chrome task manager keeps hitting higher values.

What I did is start my application and repeat an action 20 times. I used this as my baseline  (to make sure we did reach all the code, cache were filled etc). What I was expecting is when I did another run of the same action another 100 times, the max in the Chrome task manager would not increase, however this seems to be increasing from 35MB upto an 60MB, while the timeline javascript heap memory graph did not exceed the 19MB. When I did another run of 100 after this, the memory reaches max of even 80MB. 

WHen I trigger manually the GC by the garbage collect in the timeline, the memory in the task manager is after the first 20 runs 20MB, after 100 additional runs 48MB en after another 100 additional runs 66MB, while the js memory drops to around 4MB. I have attached some screenshots which might make things even more clear.

It looks like some memory is getting lost outside the javascript heap, or am I jumping to conclusions ? How would you propose to further investigate this problem ? I am relatively new with the memory tools in Chrome, so any tips are appreciated. 

From what you described it sounds like the memory leak is outside of JS heap. One of the possible reasons might be that you keep a pointer to some DOM node in your script and the node prevents a detached DOM tree from being garbage collected. If that's the case the number of DOM nodes in the Timeline Memory view should keep growing and not drop after GC. What do you see on the DOM counters graph?

Yury

[Marco Laponder]

Yuri,

The DOM nodes were my first suspects but the DOM node count and the event listeners count were behaving properly in my opinion. I have attached a screenshot of the timeline where you can see the DOM nodecount and event listener count. I have the raw timeline data saved, so if you want to take a look at that let me know.

Any other suggestions what I can further investigate to find the cause of the leak ?

Kind regards,

Marco

2013/7/26 Yury Semikhatsky <yu...@chromium.org>

[Alexei Filippov]

Hi Marco,

Looks like there's something besides DOM nodes leaking in the native (non JS heap) memory. Is it possible to get a link to the page to take a closer look?

Thanks,

Alexei

2013/7/26 Marco Laponder <marco.l...@gmail.com>

[Marco Laponder]

Alexei,

This page is now only available from out intranet, so I need some time to set this up (or try to make a isolated case, but is it from a large webapp so that would take some time). Is there any way I can inspect/log/profile non JS memory ?

Otherwise I will try to make the page available for you in the meanwhile and will get back to you as soon as possible.

Kind regards,

Marco

2013/7/26 Alexei Filippov <al...@google.com>

[Alexei Filippov]

Marco,

You can try an experimental Native Memory Instrumentation feature present in Chrome 28 (stable) and Chrome 29 (beta) channels.

To enable it you have to run chrome with --enable-devtools-experiments flag, then open DevTools and in its options you should be able to see

Experiments pane. Place a check on native memory instrumentation option and restart DevTools.

Now you should get couple more items on the Profiles panel. Please try to collect Native Memory Distribution profile when the renderer process takes too much memory.

The profile could give you a hint what component is responsible for the memory growth.

Also you can try to collect native memory snapshot and see a detailed view on native objects occupying memory.

Hope it help,

Alexei

[Marco Laponder]

Hi Alexei,

Thanks for the tips, I did some native memory profiling with Chrome 28.0.1500.71 (I couldn't seem to find in in the Canary build I am using). 

I applied the same method as before, repeat an action 20 times for an initial baseline, then repeat the same action 100 times. I create a timeline of the repeat 100 times, and the graph seems to indicates a memory leak, with the largest portion in 'Other' see also the attached image 'nativemem-timineline.png'. 

I also made a heap dump before and after the 100 run, and did a compare of those heap dumps and sorted on #delta trying to find an increase in specific objects, however the largest number there is 0, so I cannot get more information from the compare (see native-heap-compare.png).

So the increase seems to be in 'Other' but not sure how to dive deeper into this and what this exactly means. Do you have any other suggestions ? 

Best regards,

Marco

[Alexei Filippov]

Hi Marco,

Other means that unfortunately the memory chunks does not fall into one of the instrumented categories, so it is something that has not been instrumented.

Comparing heaps won't help here because it operates over instrumented parts only, but these are not increasing.

As next step you could try Deep Memory Profiler [1], but it requires building a special version of Chrome.

+Dai Mikubube 

Dai,

Could you please help Marco with building the appropriate version of Chrome, or perhaps you already have one you can just share. Thank you.

Regards,

Alexei

[1] http://www.chromium.org/developers/deep-memory-profiler

2013/7/29 Marco Laponder <marco.l...@gmail.com>

[Dai Mikurube]

Alexei, Marco,

Build instruction is in the page. What information do you and he need? http://www.chromium.org/developers/deep-memory-profiler

I'm actually not very sure how to get a specific version, especially the official Chrome builds. But, when you get the source code, I believe following the above instruction with the code will work.

Note that DMP records memory allocation log on memory. The memory for DMP itself is categorized as "mmap-profiler" which is usually excluded from the result graph. You can add it explicitly.

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

Hi Dai,

This is all new territory to me, I never build Chrome from source before, so forgive me any stupid questions. So what I should do is :

- get the source code as described in http://dev.chromium.org/developers/how-tos/get-the-code

- set GYP_DEFINES as described in   http://www.chromium.org/developers/deep-memory-profiler

- build chrome (I am on a Mac OS X 10.8, no problem there ? )

- ... and try to get some heap dumps and work from there to try to explain the memory leak (probably need some support from you there as well...)

Am I on the right track ? 

Best regards,

Marco

2013/7/29 Dai Mikurube <dmik...@google.com>

[Dai Mikurube]

Hi Marco,

On Mon, Jul 29, 2013 at 10:27 PM, Marco Laponder <marco.l...@gmail.com> wrote:

Hi Dai,

This is all new territory to me, I never build Chrome from source before, so forgive me any stupid questions. So what I should do is :

- get the source code as described in http://dev.chromium.org/developers/how-tos/get-the-code

At first, what kind of build do you want? Exactly Chrome 28.0.1500.71? Some newer one? Latest?

You look like a non-Googler, so I can't give you official builds ( == "Google Chrome", not fully open-sourced "Chromium") for you. To get the source code of "Chromium", the page is the good description.

- set GYP_DEFINES as described in   http://www.chromium.org/developers/deep-memory-profiler

 

Yes.

- build chrome (I am on a Mac OS X 10.8, no problem there ? )

 

But..., DMP doesn't support Mac anyway. As written in http://www.chromium.org/developers/deep-memory-profiler, it supports only Linux, ChromeOS and Android. You may be able to try the same issue on Linux, but I'm not sure.

The reason why we don't support Mac is that we don't use tcmalloc on Chrome for Mac.

If you need more help, could you summary the issue you have? The mail thread is too long to read. :(

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

Hi Dai,

Yes, you are right, the thread has become way too long to read so I'll give a summary.

I am experiencing in our web application a continuous increase of the memory in the Chrome task manager when keeping repeating the same action. So to set a baseline I executed the similar action 20 times, and when executing it another 100 times the memory was rising in the Chrome task manager. First I analysed the javascript heap and DOM node count, but both seem to behave nicely (see attachment 100runs.png). Then I got the advice to use the native memory profiling experiment, this resulted in a timeline where the 'Other memory' part seems to leak (see attachment nativemem-timeline.png). So the next advice was to use DMP to determine what exactly in the other memory was increasing. 

You are right, I am a non-googler, just your average web-developer. I don't think the exact version seems very relevant, I experienced the memory increase on as well the stable, as the Canary build. 

I was suspecting it would be a problem on Mac, so I should try to reproduce the issue on linux, and if it occurs on that system build the DMP enabled chromium on that system ? Is it ok to do this on a virtualbox ? 

If you have any other suggestions except from DMP to find the issue those are welcome too of course. 

Best regards,

Marco

2013/7/30 Dai Mikurube <dmik...@google.com>

[Dai Mikurube]

Thanks for the summary, Marco.

Makes sense. I believe many of such leaks will happen commonly in Linux and Mac. Virtualbox should be fine, but it may be very slow.

I'm also interested in this issue. If it's available, could you give me the reproduction case? I'm generally working on memory issue inside Chrome. If it's a problem inside Chrome, it's my work. :)

We don't have so many options for such a native memory problem at this time.

about:tracing recently added a feature to track all memory allocation (http://crrev.com/213473), but it's only on trunk.

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

I have no isolated sample of the case, it is part of a large webapp and it is hard to isolate it, I could provide a (temporarly) URL (it is not a public available web app) where you can access the page and give you a description how to reproduce it. 

What is the best way to go forward you think ? Create a DMP on Linux, or should I provide the URL (will need some time to setup), and let you take a look first ?

Many thanks anyway for the support you are providing, I would have no clue where to start otherwise ;-)

Best regards,

Marco

2013/7/30 Dai Mikurube <dmik...@google.com>

Thanks for the summary, Marco.

[Dai Mikurube]

I'm not yet sure which (Chrome or your app) issue it is... I'd like to see this if you're ok, but trying it by yourself at the same time may also solve the problem, especially in this case -- we don't have a small reproduction case. Although both may take some time. ;)

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

I'll try to setup DMP, and also try to create a more isolated case first, if this fails I will setup and url with an example to the actual webapp. I'll keep you posted on the progress. Any clues on what to focus on, what can leak 'other memory' ? Or should I just eliminate code to see when the leak disappears ? 

2013/7/30 Dai Mikurube <dmik...@google.com>

[Yury Semikhatsky]

On Tue, Jul 30, 2013 at 2:51 PM, Marco Laponder <marco.l...@gmail.com> wrote:

I'll try to setup DMP, and also try to create a more isolated case first, if this fails I will setup and url with an example to the actual webapp. I'll keep you posted on the progress. Any clues on what to focus on, what can leak 'other memory' ?

Just a wild guess - does your application use Web Workers? It might be JS heap of a worker which is not counted as part of the page's JS heap. I'd also pay attention to HTML5 storage including IndexDB, local storage, Web SQL etc. The growth is quite significant so I'd start looking for some leaking data.

 

Or should I just eliminate code to see when the leak disappears ? 

Bisection is usually a good idea, not sure how hard it would be given the scale of your application.

[Marco Laponder]

Yury,

Thanks for the tips, but we are not using web workers or any HTML5 storage in our webapp. So I guess I need to go for the bisection method, although this will take some time.

2013/7/30 Yury Semikhatsky <yu...@chromium.org>

[Marco Laponder]

Hi Dai,

Not sure if this is the right place to ask the question, let me know if it isn't. I am trying to build chrome on a Ubuntu virtual box according to https://code.google.com/p/chromium/wiki/LinuxBuildInstructions , checkout,sync and build all went fine. However when I try to start chrome:

ubuntu@ubuntu-VirtualBox:~/chromium/src/out/Release$ ./chrome

[29005:29005:0802/123833:FATAL:browser_main_loop.cc(159)] Running without the SUID sandbox! See https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment for more information on developing with the sandbox on.

Aborted (core dumped)

Any idea what the problem might be or how to proceed ?

Regards,

Marco

2013/7/30 Dai Mikurube <dmik...@google.com>

Thanks for the summary, Marco.

Makes sense. I believe many of such leaks will happen commonly in Linux and Mac. Virtualbox should be fine, but it may be very slow.

.....

[Yury Semikhatsky]

You can either follow the instruction at https://code.google.com/p/chromium/wiki/LinuxSUIDSandboxDevelopment  to install developer's version of Chromium sandbox or alternatively can start Chromium without sandbox by adding --no-sandbox command line flag. The latter should be find for the testing purposes.

[Marco Laponder]

Hi,

Still struggling with the deep memory profiling, I have build chrome according to http://www.chromium.org/developers/deep-memory-profiler

and start chrome:

ubuntu@ubuntu-VirtualBox:~/chromium/src/out/Release$ set | grep HEAP

DEEP_HEAP_PROFILE=1

HEAPPROFILE=/tmp/heap/dump/chromedump

HEAP_PROFILE_MMAP=1

ubuntu@ubuntu-VirtualBox:~/chromium/src/out/Release$ mkdir -p /tmp/heap/dump

ubuntu@ubuntu-VirtualBox:~/chromium/src/out/Release$ ./chrome --no-sandbox --enable-memory-benchmarking

[2951:2951:0809/095709:ERROR:renderer_main.cc(217)] Running without renderer sandbox

[2960:2960:0809/095716:ERROR:renderer_main.cc(217)] Running without renderer sandbox

Then I try to make a dump from the devtools with chrome:

chrome.memoryBenchmarking.heapProfilerDump("browser","test1")

but no output appears in the directory. Can someone tell me what I am doing wrong ?

Regards,

Marco

2013/8/2 Yury Semikhatsky <yu...@chromium.org>

[Dai Mikurube]

Hmm, I'm not sure why it doesn't work... Could you see the settings menu, open "Tools" and then tell me whether the menu has "Profiling enabled" or not?

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

I can't find 'profiling enabled'  in the settings | tools (settings is the 'hamburger' icon , right ?). Did I do anything wrong with building chrome maybe ? 

Regards,

Marco

2013/8/9 Dai Mikurube <dmik...@google.com>

[Dai Mikurube]

If you specified 'profiling=1' successfully in GYP_DEFINES, the menu should have "Profiling enabled". Yes, the icon of three horizontal lines.

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

I copied it from the build instructions i thought, something wrong here :

ubuntu@ubuntu-VirtualBox:~$ set | grep GYP
GYP_DEFINES='profiling=1 profiling_full_stack_frames=1 linux_dump_symbols=1'

Regards,

Marco

[Dai Mikurube]

Hmm, I'm not sure why the options are not applied to your build... Also in fact, Chromium should dump something (with very little information) even without these build options.

My guess is that the environment variables are not affecting to building and Chromium appropriately ( == not applied in an appropriate way). But, I'm not aware of your environment precisely. :(

--

Dai MIKURUBE

   dmik...@google.com

[Marco Laponder]

Hm, not sure what to do now, can't really think of anything what can go wrong setting environment variables. Would it help if I made a debug i.s.o. a release build ?

Regards,

Marco

2013/8/13 Dai Mikurube <dmik...@google.com>

[Dai Mikurube]

Chrome dumps something (although the dumps cannot be used for DMP) whenever HEAPPROFILE is set

* both for Release and Debug builds (even for official Google Chrome)

* regardless of whether profiling=1 is specified.

I verified that the current beta-channel deployed Chrome dumps something when HEAPPROFILE is set. If you have no dumped files, the environment variable is not appropriately set for your Chrome process. That's all what I can say for now...

--

Dai MIKURUBE

   dmik...@google.com

[karuna...@gmail.com]

Hi,

Any updates on this? I'm facing similar issue. Haven't tried DMP yet, will give it a try.

--karuna

[Marco Laponder]

Haven't been able to give this much attention for a while. Hope to get to it next week when some deadlines are met :-) I will keep you posted.

Regards,

Marco

2013/9/30 <karuna...@gmail.com>

[bgs...@gmail.com]

I have exactly the same issue in my application.  JS Mem profiler shows normal sawtooth memory usage, with no growth over time of overall max mem usage. However, the chrome task manager shows the mem usage for my app to grow continuously over time.

It seems that the native memory profiler has been removed now?

[michae...@gmail.com]

I know it's been a year since the original post, but I believe I am having exactly the same issue here.  In my case, the repeated action is DOM-touch heavy, perhaps rendering-intensive, and tied to the scroll event of the window.  Here's a JSfiddle that duplicates the behavior.  Open the Task Manager and watch the memory rise indefinitely as you scroll the Results window up and down.

http://jsfiddle.net/Impirator/Mt89Q/5/

I don't know if this mimics the above problems, but even in this microcosm, I see the memory climb with no upper limit and no reset until the tab is closed.  My app has it even worse.  Let me know what's up!

[rickyhu...@gmail.com]

My application has the same issue in chrome and safari but not in the firefox.  Have you tested your app in the firefox or other browsers? Got the same result as in chrome?  My app retrieve some data from the server by AJAX every 2 second and the memory climb up over time in the task manager in chrome. But the heap snapshot and timeline shows no memory increase. I have googled alot and found no answers to this issue on the internet. Do you have any clue up to now?

[famov....@gmail.com]

Same issue here. Memory in timeline heap seems to be fine and Garbage Collector is working ok, however, memory in Android taskt manager grows on repeated runs and it is never released. 

Our application has a lot of ajax requests and high IndexedDB Storage. In Firefox it seems to be working ok. Any idea about this?