Usage of Partition Alloc bloats in whole on Android Content Shell

[Hajime Hoshi]

Hi,

We found that memory usage of partition alloc bloats in whole on Android Content Shell. Here is the open doc: https://docs.google.com/a/chromium.org/document/d/15PEA61T3GmwgHqwmjYPw1GLCyXens3y8TWGdJA90Xf8/edit#

Summary:

On Android content shell, partition alloc usage increased according to the number of loading pages. Partition alloc is used for various purposes like DOM, string, font and other objects, and we found that not a certain part but all parts bloated.

What we want to know:

* If partition alloc itself has a problem, it might be fragmentation. Is there a good way to know how fragmentation happens?

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>

[Chris Evans]

On Mon, Jun 16, 2014 at 11:18 PM, Hajime Hoshi <hajim...@google.com> wrote:

Hi,

We found that memory usage of partition alloc bloats in whole on Android Content Shell. Here is the open doc: https://docs.google.com/a/chromium.org/document/d/15PEA61T3GmwgHqwmjYPw1GLCyXens3y8TWGdJA90Xf8/edit#

How is memory usage measured? PartitionAlloc holds on to address space but returns pages to the system with madvise(). The measure of actually used memory is best captured by what the Linux kernel calls "private dirty" pages. The number of these is exposed via /proc.

Summary:

On Android content shell, partition alloc usage increased according to the number of loading pages. Partition alloc is used for various purposes like DOM, string, font and other objects, and we found that not a certain part but all parts bloated.

What we want to know:

* If partition alloc itself has a problem, it might be fragmentation. Is there a good way to know how fragmentation happens?

In DEBUG builds (or release if you edit / hack PartitionAlloc.cpp), there's a partitionDumpStats() API. Call it on whichever partition you think might be fragmented.

PartitionAlloc is supposed to fragmentation resistant (more so than tcmalloc) but of course we may have gotten it wrong or have a bug :) As we switched from tcmalloc to PartitionAlloc, we did see the page cyclers report a memory saving.

Cheers

Chris

[Dai Mikurube]

Hi Chris,

On Wed, Jun 18, 2014 at 3:11 AM, Chris Evans <cev...@chromium.org> wrote:

On Mon, Jun 16, 2014 at 11:18 PM, Hajime Hoshi <hajim...@google.com> wrote:

Hi,

We found that memory usage of partition alloc bloats in whole on Android Content Shell. Here is the open doc: https://docs.google.com/a/chromium.org/document/d/15PEA61T3GmwgHqwmjYPw1GLCyXens3y8TWGdJA90Xf8/edit#

How is memory usage measured? PartitionAlloc holds on to address space but returns pages to the system with madvise(). The measure of actually used memory is best captured by what the Linux kernel calls "private dirty" pages. The number of these is exposed via /proc.

The result is based on counting "present" pages in /proc/pid/pagemap using dmprof (http://dev.chromium.org/developers/deep-memory-profiler). It is almost the same as RSS.

I also confirmed that madvise(MADV_DONTNEED) decreases all of RSS, Private Dirty and pagemap count just in case. https://gist.github.com/dmikurube/d451bc8951b4e799095a

Summary:

On Android content shell, partition alloc usage increased according to the number of loading pages. Partition alloc is used for various purposes like DOM, string, font and other objects, and we found that not a certain part but all parts bloated.

What we want to know:

* If partition alloc itself has a problem, it might be fragmentation. Is there a good way to know how fragmentation happens?

In DEBUG builds (or release if you edit / hack PartitionAlloc.cpp), there's a partitionDumpStats() API. Call it on whichever partition you think might be fragmented.

PartitionAlloc is supposed to fragmentation resistant (more so than tcmalloc) but of course we may have gotten it wrong or have a bug :) As we switched from tcmalloc to PartitionAlloc, we did see the page cyclers report a memory saving.

Cheers

Chris

 

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>

--

Dai MIKURUBE

   dmik...@chromium.org

[Hajime Hoshi]

Hi Chris,

On Wed, Jun 18, 2014 at 3:11 AM, Chris Evans <cev...@chromium.org> wrote:

Thank you!

IIUC, "Total Resident" refers to the sum bytes of all pages, "Total Live" to the sum of allocated slots, and "Total Freeable" to the sum of (not-returned) free pages. To estimate how much fragment happens, is seeing (Total Resident) - (Total Live) - (Total Freeable) a good way?

Cheers

Chris

 

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

[Chris Evans]

I think your fragmentation percentage is just something like:

(1.0 - (Total Live) / (Total Resident)) * 100%

(Total Freeable) doesn't really factor in because this represents pages that we've totally emptied => no fragmentation.

Cheers

Chris

[Hajime Hoshi]

Thank you for your correction!

According to PartitionAlloc.cpp:965-967 (https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/wtf/PartitionAlloc.cpp&q=partitionDumpStats()&sq=package:chromium&type=cs&l=965-967 ), (Total Resident) seems to include (Total Freeable), so the number of freeable pages seems to affect the result, right? Shouldn't I subtract (Total Live) from (Total Resident), that is:

  (1 - (Total Live) / ((Total Resident) - (Total Live))) ?

 

Cheers

Chris

 

Cheers

Chris

 

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

[Hajime Hoshi]

I meant:

Shouldn't I subtract (Total Freeable) from (Total Resident), that is:

  (1 - (Total Live) / ((Total Resident) - (Total Freeable))) ? 

 

Cheers

Chris

 

Cheers

Chris

 

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

--

Hajime Hoshi <hajim...@google.com>

[Chris Evans]

Agreed :)

Let me know if you notice any strangeness in the stats, or need any additional stats, and I will try and help. Also let me know what levels of fragmentation you find, and how things fare vs. Android system malloc? It may be possible to get more aggressive about defragmenting on the fly.

Cheers

Chris

[Hajime Hoshi]

OK, I'll investigate this. Thank you very much!

--

Hajime Hoshi <hajim...@google.com>

[Hajime Hoshi]

Hi,

I added the results of the new experiments: https://docs.google.com/a/chromium.org/document/d/15PEA61T3GmwgHqwmjYPw1GLCyXens3y8TWGdJA90Xf8/edit

TL;DR

Probably users of Partition Alloc are leaking. Partition Alloc didn’t cause big fragmentation.

Please let us know if you have some questions or concerns.

--

Hajime Hoshi <hajim...@google.com>