mmap contention

[Kenton Varda]

Hi v8-dev,

In Cloudflare Workers, where we often have thousands of isolates running across many threads in one process, we're running into issues with contention on the process's mmap_lock / mmap_sem in the Linux kernel. It appears V8 does a very large number of mmap/munmap/mprotect/page fault operations and in the right circumstances this is enough to bog down the lock.

Before diving deep into the code I wanted to check if anyone has any thoughts, off the top of your heads, on how we could reduce the mmap usage in V8. Is there any easy knob to turn? Good places to look for optimizations in the code?

Assuming there isn't an easy answer, one idea suggested to me by a kernel developer friend is: Instead of using mprotect() to swap pages between write and execute mode while JITing, what if the pages were mapped from a memfd in read-execute mode, and writes were performed by writing to a separate buffer and then doing a pwrite() on the memfd? Does that sound worth exploring? (I'm suggesting this with very little understanding of how JIT works in V8; apologies if it's an obviously bad idea.)

Thanks for any thoughts!

-Kenton

[Hannes Payer]

Hi Kenton,

You can turn off the write protection for code memory with --nowrite_protect_code_memory. The security benefits of this feature are quite small.

-Hannes

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/109e3dbe-edbd-4aeb-a30a-24b932939becn%40googlegroups.com.

--

 

Hannes Payer |

V8 |

Google Germany GmbH |

Erika-Mann Str. 33, 80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls Sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde. 

     

This e-mail is confidential. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Clemens Backes]

If you are also executing WebAssembly, then --no-wasm-write-protect-code-memory will further reduce the number of mprotect calls.

We plan to turn off that feature anyway when we enable lazy compilation for Wasm, so you won't lose much if you already disable it now.

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAKEgpyGJyiVCDNn2BN4TPpYqpxZFEGqh-JZb0oURSC2KnsiYAw%40mail.gmail.com.

--

Clemens Backes

Software Engineer

clem...@google.com

Google Germany GmbH

Erika-Mann-Straße 33

80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian   

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Diese E-Mail ist vertraulich. Falls sie diese fälschlicherweise erhalten haben sollten, leiten Sie diese bitte nicht an jemand anderes weiter, löschen Sie alle Kopien und Anhänge davon und lassen Sie mich bitte wissen, dass die E-Mail an die falsche Person gesendet wurde.

[Kenton Varda]

Thanks for the tips!

I did some investigating and found that it doesn't look like JIT is a major driver here. Instead, it seems to be regular old memory allocation and deallocation. I think I can reduce the number of mmap ops considerably with a custom PageAllocator that batches operations and eliminates redundant ones (e.g. freeing some pages and then immediately allocating them again, or freeing pages on isolate shutdown that would be freed together with the whole pointer cage anyway).

We also noticed that freeing pages does two operations: mprotect(kNoAccess) followed by madvise(MADV_DONTNEED). It seems like we can get away with skipping the mprotect operation since V8 won't attempt to access the pages anyway, and the madvise operation is what actually frees the memory. This change alone looks like it may reduce mmap operations by ~30%-40%.

-Kenton

You received this message because you are subscribed to a topic in the Google Groups "v8-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/v8-dev/OvqzZHemtDI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAGO%3DqhCXJLfy9DMJ_Wh1Bcs1AS%3DiBjRKfTr0K60O-%3D6QQpxBmw%40mail.gmail.com.

[Michael Lippautz]

madvise(MADV_DONTNEED) and mprotect(kNoAccess) have different guarantees on what happens if you try to access that memory again (immediately). This is relevant when assuming that processes are hijacked and/or for debugging.

Depending on your security model with running multiple isolates in the same process, such differences can be relevant though.

-Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAJouXQn16VWxhk1%3Dbzg_aSThX1uHmCQHEYTAsBTrWiSAGY199w%40mail.gmail.com.

[Kenton Varda]

Hi Michael,

Yes, this is a trade-off but I think it makes sense for our use case.

- Setting the pages no-access helps discover use-after-free bugs in V8. For our purposes, though, this isn't a good trade-off, as we've never actually seen such a bug on our end, but we are suffering a lot from the mmap contention. So I'm comfortable forgoing that debugging aid.

- For a hijacked process, would this actually be that useful to an attacker? If an attacker can write arbitrary memory, I imagine they would prefer to target pages that are still active, rather than ones that have been freed and zeroed. Or, are you saying there is likely a class of security bugs that involve attackers being able to write to recently-freed addresses? If so that's certainly worth thinking about.

-Kenton

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/CAH%2BmL5CnFw8bZegB2E0iL1Ohc83Zvh6YQ%2Bf8k1Uc8z9aB2oOeA%40mail.gmail.com.