Issue 2191 in v8: V8 reports heap memory size bigger than process private memory

[codesite...@google.com]

Status: New
Owner: ----
CC: alex...@chromium.org, yu...@chromium.org, loi...@chromium.org
Labels: Type-Bug Priority-Medium

New issue 2191 by alex...@chromium.org: V8 reports heap memory size bigger
than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

How to reproduce:
1. open the attached html in chromium
2. open the task manager
3. start hitting refresh button multiple times
4. at some point task manager shows js heap size bigger than process
private memory (pls see the screenshot).


Attachments:
jsweirdmemory.png 33.8 KB
arrays.html 382 bytes

[codesite...@google.com]

Updates:
Status: Assigned
Owner: mstar...@chromium.org

Comment #1 on issue 2191 by mstar...@chromium.org: V8 reports heap

memory size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

I'll look into that.

[codesite...@google.com]

Updates:
Cc: da...@chromium.org

Comment #2 on issue 2191 by pfeld...@chromium.org: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

(No comment was entered for this change.)

[codesite...@google.com]

Updates:
Cc: ul...@chromium.org

Comment #3 on issue 2191 by da...@chromium.org: V8 reports heap memory size

[codesite...@google.com]

Comment #4 on issue 2191 by yu...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Any update on this?

[codesite...@google.com]

Comment #5 on issue 2191 by Alexei.F...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Any news? It still seems to be broken as of:
JavaScript V8 3.13.7 (candidate)


Attachments:
Screenshot-Developer Tools - file:---home-alph-www-test-jsheap.html.png
81.7 KB
Screenshot-Task Manager - Chromium.png 30.8 KB

[codesite...@google.com]

Comment #6 on issue 2191 by Alexei.F...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Here's another way to reproduce it. Open the attached html, open the task
manager and start pressing the button until it's reproduced (usually it
showed up after 5 times for me).

Attachments:
jsheap.html 381 bytes

[codesite...@google.com]

Updates:
Cc: erik.corry veg...@chromium.org

Comment #7 on issue 2191 by mstar...@chromium.org: V8 reports heap

memory size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

So I investigated this again and with the new test case I could reliably
reproduce the mentioned discrepancy.

It happens right after we grow the new-space from 16M to 32M. We do this by
committing previously reserved memory using mmap[1]. We only touch the
first few bytes of each 1MB block in this newly allocated region. It seems
the OS doesn't report the committed pages as "private memory" right away,
but only when we first touch all of them

When I enable zapping[2] (i.e. overwriting with garbage) of the whole
region, the reported "private memory" is much higher, makes bigger jumps
and doesn't show the above discrepancy. But V8 only zaps newly committed
memory in debug mode but not in release mode. And I don't think touching
all the memory we commit is the way to go here.

At the moment I don't have a concrete idea how to make those two numbers
comparable, they are coming from two different memory management systems. I
am CCing others, maybe someone else has suggestions.

[1]
http://code.google.com/p/v8/source/browse/branches/3.13/src/platform-linux.cc#673
[2]
http://code.google.com/p/v8/source/browse/branches/3.13/src/spaces.cc#624

[codesite...@google.com]

Comment #8 on issue 2191 by a...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

As one of the solutions you could reserve pages with READ_PROT|WRITE_PROT
and set a SIGSEGV handler to know when the page gets committed.

[codesite...@google.com]

Comment #9 on issue 2191 by veg...@google.com: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

I am convinced that this is working as intended. V8 heap size tells how
much memory it asked OS to commit (a peak possible consumption in a sense).
V8 used heap size tells how much memory contains actually allocated
objects. As far as I see both numbers are correct, it just mismatches a
number that is counts something else entirely. If number of oranges
mismatches number of apples it does not mean that we should start counting
oranges in a different way.

Setting SIGSEGV is not an option. First of all it conflicts with Breakpad
(which afaik installs its own handler) and its an additional layer of
complexity that solves a nonexistent problem.

[codesite...@google.com]

Comment #10 on issue 2191 by yu...@chromium.org: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

The numbers are correct per se but it is wrong to show the amount of the
memory v8 asked OS to commit as part of the render process private memory.
The process private size is an amount of the committed bytes and JavaScript
memory size in the Chromium's task manager is expected to show the amount
of memory actually taken by v8, not the amount it asked OS to commit but
probably not yet committed.

[codesite...@google.com]

Comment #11 on issue 2191 by al...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

I agree with Yury. The user is not interested in amount of reserved memory
(it might be several gigs). The interesting thing is committed memory that
occupies physical resources.

[codesite...@google.com]

Comment #12 on issue 2191 by veg...@google.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

If you are interested in memory consumption you should look at used bytes
(confusingly called "live" in the column) not heap size. We can even
reorder these numbers inside the column to put an emphasis on the right
thing: used matters --- heap size matters less.

And once again: V8 does not report amount of virtual address space it just
reserves. Otherwise it would be _way_ higher. It reports amount of memory
it asked OS to commit. It is true that OS can be lazy in performing actual
commit, but that is something beyond V8's reach and actually is not
important to V8 at all as for V8 purposes this memory looks like
successfully committed and can be used for allocation .

Would you prefer V8 to workaround OSes lazyness by touching every page
inside the region it commits to make numbers of apples and oranges agree? I
can hardly see any reason in that.

[codesite...@google.com]

Comment #13 on issue 2191 by yu...@chromium.org: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

"And once again: V8 does not report amount of virtual address space it just
reserves. Otherwise it would be _way_ higher. It reports amount of memory
it asked OS to commit. It is true that OS can be lazy in performing actual
commit, but that is something beyond V8's reach and actually is not
important to V8 at all as for V8 purposes this memory looks like
successfully committed and can be used for allocation ."

What does this number mean from the user stand point?

The user wants to see how much of the physical resources were actually
eaten by the render process and V8 in it. Showing amount of memory the
process asked OS to commit for v8 and showing amount of virtual address
space would only give you an upper bound of the actually committed memory.
In that sense they would be equally useless in the task manager. From the
current numbers we can only tell that the actual size of the v8 heap is
something between the size of live JavaScript objects and the size reported
to be the total JavaScript size.

"Would you prefer V8 to workaround OSes lazyness by touching every page
inside the region it commits to make numbers of apples and oranges agree? I
can hardly see any reason in that."

At the present time v8 returns a number of oranges for live JS heap and
apples for the whole JS heap. Since Chromium shows amount of committed
memory as the process total size need v8 to return the number of actually
committed bytes, it doesn't necessarily mean that it will touch every page
it asks OS to commit.

[codesite...@google.com]

Comment #14 on issue 2191 by al...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

"If you are interested in memory consumption you should look at used bytes
(confusingly called "live" in the column) not heap size."

Unfortunately this is not enough for our purposes. We need to show all the
physical memory committed for V8 heap. And it might be up to 2x higher than
the size of actually allocated heap objects.

"We can even reorder these numbers inside the column to put an emphasis on
the right thing: used matters --- heap size matters less."

Well, it's up to you. Probably you're right.

"And once again: V8 does not report amount of virtual address space it just
reserves. Otherwise it would be _way_ higher. It reports amount of memory
it asked OS to commit. It is true that OS can be lazy in performing actual
commit, but that is something beyond V8's reach and actually is not
important to V8 at all as for V8 purposes this memory looks like
successfully committed and can be used for allocation ."

Although it's not that important for v8 (or even for task manager) you can
show apples and oranges to the user. But this is important for DevTools
because we're trying to show the user both apples and oranges on the same
chart and even subtracting one from another. Which is obviously wrong.

"Would you prefer V8 to workaround OSes lazyness by touching every page
inside the region it commits to make numbers of apples and oranges agree? I
can hardly see any reason in that."

Of course not! Increasing process physical memory consumption for the sake
of numbers match in the task manager is not an option. :-)

As a bottomline it seems that unfortunately Heap::CommittedMemory is not ok
for our purposes. So I'm implementing Heap::CommittedPhysicalMemory
function that we will call from DevTools.

[codesite...@google.com]

Updates:
Status: Fixed

Comment #15 on issue 2191 by mstar...@chromium.org: V8 reports heap

memory size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Fixed by alph@ in r12625. Thank you for the contribution.

[codesite...@google.com]

Comment #16 on issue 2191 by a...@google.com: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Just to note, it does not change the number in task manager. It just
provides a new statistics value in the API.

[codesite...@google.com]

Updates:
Owner: al...@chromium.org
Cc: mstar...@chromium.org

Comment #17 on issue 2191 by mstar...@chromium.org: V8 reports heap

memory size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

The patch had to be reverted in r12640 due to incompatibilities with the
Chromium sandbox. Running with --no-sandbox works fine, but with the
sandbox every renderer process goes "Aww Snap" instantly. I guess the
mincore() function is not white-listed. So we cannot roll with this patch
until that is resolved.

I reopened this issue to keep track of that. Just ping the issue (or me)
and I will reland the patch once the sandbox problem is resolved.

[codesite...@google.com]

Updates:
Status: Assigned

Comment #18 on issue 2191 by mstar...@chromium.org: V8 reports heap

memory size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

[codesite...@google.com]

Comment #19 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

is it right if the live object size is larger than heap size, see the
attachment

Attachments:
image002.png 4.0 KB
image003.png 3.4 KB

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[codesite...@google.com]

Comment #20 on issue 2191 by a...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

Looks like a separate issue. Could you please share the source so we could
reproduce? Thank you!

[codesite...@google.com]

Comment #21 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

it's a little bit difficult to provide the source as it's inside a big
internal project:( currently what we find is the private bytes we get from
windows process monitor remains stable(sawtooth curve) and the
totalJSHeapSize also keeps a small value, but the usedJSHeapSize keeps
increasing.

do you have any args that I can pass to V8 to dump the trace for you to
analyze?

[codesite...@google.com]

Comment #22 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

it's a little bit difficult to provide the source as it's inside a big
internal project:( currently what we find is the private bytes we get from
windows process monitor remains stable(sawtooth curve) and the
totalJSHeapSize also keeps a small value, but the usedJSHeapSize keeps
increasing.

do you have any args that I can pass to V8 to dump the trace for you to
analyze?

by the way, I'm using the latest chrome.

[codesite...@google.com]

Updates:
Status: Fixed

Comment #23 on issue 2191 by a...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

The original issue has been fixed a while ago.
https://code.google.com/p/v8/source/detail?r=12793

[codesite...@google.com]

Updates:
Cc: -alex...@chromium.org a...@chromium.org

Comment #24 on issue 2191 by a...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

@yangyubin could you please file a separate bug on the issue you're
observing. It seems to be live vs. total-heap, while this issue was about
total-heap vs. total-process. The latter is fixed a while ago.

If you prefer I can do it myself.

[codesite...@google.com]

Comment #25 on issue 2191 by a...@chromium.org: V8 reports heap memory size

bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

You can try running chrome with flags: --js-flags="--trace_gc" --no-sandbox
to get some output from GC to see which of two values is correct and which
is garbled.

[codesite...@google.com]

Comment #26 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

ok，please tell me the bug id and i will post the gc trace

[codesite...@google.com]

Comment #27 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

hello, I ran chrome with "--js-flags="--trace_gc" --no-sandbox", attached
is the trace file and the moment I stop the test.

traceGC contains only 9999 lines as it always fail when I tried to redirect
the output, I can only copy the output directly from the console.

Attachments:
traceGC.PNG 3.2 KB
traceGC.txt 487 KB

[codesite...@google.com]

Comment #28 on issue 2191 by yangyu...@gmail.com: V8 reports heap memory

size bigger than process private memory
http://code.google.com/p/v8/issues/detail?id=2191

I submitted a new issue: https://code.google.com/p/v8/issues/detail?id=2811