Pointer compression seemingly responsible for large memory leak

[clearsc...@gmail.com]

Greetings!

Our project is a V8 embedding library for .NET on Windows. We have a test that spins up a context, runs SunSpider, and destroys the isolate. We run it in a loop to check for memory leaks.

This test showed a dramatic leak with V8 8.0 that wasn't there with 7.9. Our tools found nothing suspicious in the managed and native heaps, but the process's private memory size kept increasing.

Based on some quick research and a little guesswork, we rebuilt with v8_enable_pointer_compression and v8_enable_31bit_smis_on_64bit_arch both set to false. To our surprise this completely eliminated the leak!

We'd like to report this issue to the V8 team, but we have no simple native repro code. Can someone suggest a way to gather some useful diagnostics?

Thanks!

[Yang Guo]

+Ulan Degenbaev +Igor Sheludko 

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/6325eb1a-2a8e-431d-a7ee-5e7f0a41f046%40googlegroups.com.

[Santiago Aboy Solanes]

Quick question: Have you tried with v8_enable_pointer_compression = false and v8_enable_31bit_smis_on_64bit_arch = true?

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAFSTc_iWyBOpP5xEEM6EZk_pLZhyeGoK0WrgM-L8M9%3D6xenyUQ%40mail.gmail.com.

[Igor Sheludko]

Could you please open an issue here: https://bugs.chromium.org/p/v8/issues/

And yet another suggestion. Could you please try it again on today's tip of tree V8 (say 96f6ac9bc6646ae12acad43a1f9ebdaaf7ceea0a)?

Does your memory leak tool give any hints about the source of the memory leak?

How much memory is leaked after 1 iteration? 

--

Igor Sheludko

Software Engineer

ish...@google.com

Google Germany GmbH

Erika-Mann-Str. 33

80636 München

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

    

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[clearsc...@gmail.com]

That particular combination yielded a V8 build that broke a majority of our test suite, including this memory leak test.

We didn't spend much time investigating, but the build utterly broke interception (e.g., bad data in FunctionCallbackInfo), so we abandoned it.

On Friday, February 14, 2020 at 5:13:35 AM UTC-5, Santiago Aboy Solanes wrote:

Quick question: Have you tried with v8_enable_pointer_compression = false and v8_enable_31bit_smis_on_64bit_arch = true?

On Fri, Feb 14, 2020 at 7:54 AM Yang Guo <yan...@chromium.org> wrote:

+Ulan Degenbaev +Igor Sheludko 

On Thu, Feb 13, 2020 at 8:36 PM <clearsc...@gmail.com> wrote:

Greetings!

Our project is a V8 embedding library for .NET on Windows. We have a test that spins up a context, runs SunSpider, and destroys the isolate. We run it in a loop to check for memory leaks.

This test showed a dramatic leak with V8 8.0 that wasn't there with 7.9. Our tools found nothing suspicious in the managed and native heaps, but the process's private memory size kept increasing.

Based on some quick research and a little guesswork, we rebuilt with v8_enable_pointer_compression and v8_enable_31bit_smis_on_64bit_arch both set to false. To our surprise this completely eliminated the leak!

We'd like to report this issue to the V8 team, but we have no simple native repro code. Can someone suggest a way to gather some useful diagnostics?

Thanks!

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/6325eb1a-2a8e-431d-a7ee-5e7f0a41f046%40googlegroups.com.

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

[Santiago Aboy Solanes]

Do you mean that v8_enable_pointer_compression = false and v8_enable_31bit_smis_on_64bit_arch = true "yielded a V8 build that broke a majority of our test suite, including this memory leak test"?

If that's the case, this is not a pointer compression failure.

Pointer Compression needs 31 bit smis, but 31 bit smis doesn't need Pointer Compression.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/2388a332-c093-4d27-9c4b-67bd21a07064%40googlegroups.com.

[clearsc...@gmail.com]

Sorry, here's a clarification. This is what we're seeing with 8.0:

Pointer Compression ON, 31-bit Smis ON -> Memory leak

Pointer Compression ON, 31-bit Smis OFF -> Memory leak

Pointer Compression OFF, 31-bit Smis ON -> Major breakage

Pointer Compression OFF, 31-bit Smis OFF -> All good

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/2388a332-c093-4d27-9c4b-67bd21a07064%40googlegroups.com.

[Leszek Swirski]

That sounds like a build config mismatch, or something similar -- 31-bit Smis shouldn't cause major breakage, but will if e.g. one header has the build flag enabled and another doesn't.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/e5e207c5-880d-4137-8efa-68d1ad4c9d62%40googlegroups.com.

[Santiago Aboy Solanes]

Adding to that, this configuration

  Pointer Compression ON, 31-bit Smis OFF -> Memory leak

is the same as

  Pointer Compression ON, 31-bit Smis ON -> Memory leak

since Pointer Compression forces 31 bit smis on 64 bit archs

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAGRskv8gsisFHaNKGoU6JPzB4HV6Djcp4Y%3D03LjmV6tnAuh-Hg%40mail.gmail.com.

[clearsc...@gmail.com]

We suspected that, but tested it anyway :)

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/e5e207c5-880d-4137-8efa-68d1ad4c9d62%40googlegroups.com.

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAGRskv8gsisFHaNKGoU6JPzB4HV6Djcp4Y%3D03LjmV6tnAuh-Hg%40mail.gmail.com.

[clearsc...@gmail.com]

>That sounds like a build config mismatch, or something similar -- 31-bit Smis shouldn't cause major breakage, but will if e.g. one header has the build flag enabled and another doesn't.

Thanks, Leszek; we'll double-check our findings.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/e5e207c5-880d-4137-8efa-68d1ad4c9d62%40googlegroups.com.

[clearsc...@gmail.com]

Leszek, you were right. The breakage went away once we defined V8_31BIT_SMIS_ON_64BIT_ARCH in our embedder code.

As an aside, it would be great if V8's public headers were automatically aligned with the V8 build. These mismatches can be difficult to diagnose.

Anyway, we're now testing this configuration for the memory leak. Thanks again.

[clearsc...@gmail.com]

Hi Igor,

>Could you please open an issue here: https://bugs.chromium.org/p/v8/issues/

Sure thing; we were just hoping to collect some useful diagnostic info, since we don't have simple repro code.

>Could you please try it again on today's tip of tree V8 (say 96f6ac9bc6646ae12acad43a1f9ebdaaf7ceea0a)?

We're currently working with 8.0, as our releases are paired with stable V8. We haven't yet ported to 8.1, much less 8.2. An initial test build with 8.1 fails immediately due to breaking API changes (e.g., unlike earlier versions, 8.1 demands non-nestable task support).

>Does your memory leak tool give any hints about the source of the memory leak?

No, our tool shows managed and native heap sizes that don't change much after a few iterations, but the process keeps growing in private memory size.

>How much memory is leaked after 1 iteration?

There's a lot of variance, presumably due to the large components involved (V8, .NET Core, our library, SunSpider, etc.). Each test iteration consists of the following:

* Isolate and context creation.

* SunSpider warmup run and 10 benchmark runs.

* Orderly context and Isolate teardown.

* Redundant, forced, synchronous managed GC.

* Private memory size sample.

The memory size samples don't always increase from iteration to iteration, but here's the bottom line:

Pointer Compression OFF, 31-bit Smis OFF -> ~40MB after 2000 iterations

Pointer Compression OFF, 31-bit Smis ON -> ~40MB after 2000 iterations

Pointer Compression ON, 31-bit Smis ON -> ~75MB after 2000 iterations

This is 100% repeatable, and the V8 build flags and their associated #defines are the only moving parts. What additional info can we can dig up for our report?

Thanks!

On Friday, February 14, 2020 at 5:23:28 AM UTC-5, Igor Sheludko wrote:

Could you please open an issue here: https://bugs.chromium.org/p/v8/issues/

And yet another suggestion. Could you please try it again on today's tip of tree V8 (say 96f6ac9bc6646ae12acad43a1f9ebdaaf7ceea0a)?

Does your memory leak tool give any hints about the source of the memory leak?

How much memory is leaked after 1 iteration? 

On Fri, Feb 14, 2020 at 11:13 AM Santiago Aboy Solanes <sol...@google.com> wrote:

Quick question: Have you tried with v8_enable_pointer_compression = false and v8_enable_31bit_smis_on_64bit_arch = true?

On Fri, Feb 14, 2020 at 7:54 AM Yang Guo <yan...@chromium.org> wrote:

+Ulan Degenbaev +Igor Sheludko 

On Thu, Feb 13, 2020 at 8:36 PM <clearsc...@gmail.com> wrote:

Greetings!

Our project is a V8 embedding library for .NET on Windows. We have a test that spins up a context, runs SunSpider, and destroys the isolate. We run it in a loop to check for memory leaks.

This test showed a dramatic leak with V8 8.0 that wasn't there with 7.9. Our tools found nothing suspicious in the managed and native heaps, but the process's private memory size kept increasing.

Based on some quick research and a little guesswork, we rebuilt with v8_enable_pointer_compression and v8_enable_31bit_smis_on_64bit_arch both set to false. To our surprise this completely eliminated the leak!

We'd like to report this issue to the V8 team, but we have no simple native repro code. Can someone suggest a way to gather some useful diagnostics?

Thanks!

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/6325eb1a-2a8e-431d-a7ee-5e7f0a41f046%40googlegroups.com.

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAFSTc_iWyBOpP5xEEM6EZk_pLZhyeGoK0WrgM-L8M9%3D6xenyUQ%40mail.gmail.com.

[clearsc...@gmail.com]

Hmm... Our latest findings are quite baffling:

* Memory usage levels off after several thousand test iterations, so it may not be a true leak. We've tested up to 12000 iterations, and memory usage never grew beyond ~85MB. Graphing a moving average clearly indicated an asymptotic curve. Still, a build without pointer compression levels off at ~40MB after just a few dozen iterations. What could possibly explain such a difference?

* The managed and native heaps, sampled randomly between iterations, remained below 4MB and 9MB respectively throughout the test, with and without pointer compression. It would appear that the culprit is neither managed memory nor part of the standard C/C++ heap.

* Now this one threw us for a loop. We produce different versions of our library for .NET Framework and .NET Core. The differences between them are trivial and mostly compensate for minor platform differences. Both use the same V8 build. We now find that the high memory usage happens only when V8 pointer compression is combined with .NET Core. In other words:

    .NET Framework, V8 pointer compression OFF -> NORMAL memory usage

    .NET Framework, V8 pointer compression ON -> NORMAL memory usage

    .NET Core, V8 pointer compression OFF -> NORMAL memory usage

    .NET Core, V8 pointer compression ON -> HIGH memory usage

How V8's memory usage could be affected by the .NET runtime is anyone's guess. Could this be a case of address space fragmentation? How could we test for that? Could someone describe how V8's memory behavior changes when pointer compression is enabled?

Thanks in advance for any suggestion, clue, or insight anyone can provide.