V8 seems to assume amd64 pointers have high-16 set to 0, what if they're not?

[Dan McDonald]

From a different thread, Subject: "Re: [v8-dev] Sandbox js-dispatch-table code's kFreeEntryTag"

On Tuesday, April 8, 2025 at 10:45:55 AM UTC-4 Dan McDonald wrote:

On Tuesday, April 8, 2025 at 2:58:37 AM UTC-4 *oli wrote:

<SNIP!> 

AIX ran into the same issue. There is currently a CL in review to address it in their case: https://chromium-review.googlesource.com/c/v8/v8/+/6320599 .

Maybe you can try that one and comment if it works for illumos. If it doesn't it would be good if you can directly work with them on that CL to find a solution that works for both OSs.

I'm looking at it now, and will have things to say, thank you!

After much spewage in the Gerrit (sorry about that *oli), I've come to the conclusion that the big problem, and it might not just affect the dispatch table, is that V8, or perhaps its users (I went down this rabbit hole because of Node and its under-review import of V8 13.6) seem to assume the upper 16 bits in a 64-bit pointer are always 0.

As I mentioned (somewhat incorrectly) earlier: 0xfffffc7fffe00000 is the highest allowable virtual address in an illumos process. Our stack starts up there, and along with mmap() segments they both work their way down. Our heap starts from much lower and works it way up. There is a Virtual Address Hole of disallowed space between them. The low end of the hole is 0x800000000000 (47bits of non-hole space below it), the high end is 0xffff800000000000. mmap() calls without an address hint can place anywhere outside the VA hole..  Oracle Solaris, like illumos, descended from Sun Solaris => OpenSolaris, likely has this issue as well.

There are two ways to approach this problem.  The first is to have the OS provide a runtime environment where the top 16 bits of virtual address space are always 0.  The second is to figure out how to get V8 to not allocate/mmap/etc.  Our va_hole low boundary of 0x800000000000 (for internal-to-us reasons it'll actually be no allocations above 0x7fffc0000000; i.e. 1GiB shy) should do.

For illumos this moment, providing that runtime environment is a global kernel-start-time tunable. This is unacceptable for a number of reasons, and providing a per-process solution is something we're investigating. I'll note Oracle Solaris has a link-time mapfile to present a low 47-bits space. I'm not sure if illumos will solve it the same way, but when we do we'll let folks know. Meanwhile I do have test environments with this global tunable set and at least with Node I'm getting far enough along where the Node tests are passing/mostly-passing.

I post this letter to ask if there's any mechanisms V8 uses that I can attempt to exploit so an OS-level solution for making sure allocated space by V8 stays within the assumption about the high 16 bits being masked out?  Perhaps in platform-solaris.cc? (I'll ask about splitting Oracle Solaris and illumos (much) later.)

Thanks,

Dan

[Olivier Flückiger]

Hi Dan,

mmap is pretty much a given for V8. I guess the bottleneck you are looking for is base::Allocate or OS::Allocate. It should be possible to set a custom hint or flag in an os specific file or add an ifdef to get the desired constraints.

The fact that we use these 16 bits in the dispatch table is not super fundamental and we might also get rid of that at some point. However, the way you describe the illumos memory layout it seems like it would be good either way to not mmap in the space reserved for the stack.

*oli

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/v8-dev/9088d250-c6f5-4641-bf22-9676344fcf2dn%40googlegroups.com.

[Dan McDonald]

On Tuesday, April 15, 2025 at 4:29:22 AM UTC-4 *oli wrote:

Hi Dan,

mmap is pretty much a given for V8. I guess the bottleneck you are looking for is base::Allocate or OS::Allocate. It should be possible to set a custom hint or flag in an os specific file or add an ifdef to get the desired constraints.

That might be the useful band-aid.  Thank you for that.

 

The fact that we use these 16 bits in the dispatch table is not super fundamental and we might also get rid of that at some point. However, the way you describe the illumos memory layout it seems like it would be good either way to not mmap in the space reserved for the stack.

Said layout of default 64-bit process high-memory (down from the default USERLIMIT address of 0xfffffc7fffe00000 on amd64) is better-described here:

https://github.com/illumos/illumos-gate/blob/435bf86e2c05fe187d507312c9c2995271bf6b58/usr/src/uts/common/os/exec.c#L1940-L2000

We have a "VM hole" where nothing in a user process can be allocated or mapped in between 0x800000000000 (47-bits aka 4-level amd64 paging) and 0xffff7fffffffffff (one byte before high 2^47-bytes).  

mmaps() aren't intruding per se into stack but some seem to be (at least on Node) starting up in that higher-memory range (our stacks are relatively safe, especially the initial-process one). The pmap(1) command, one of our nifty procfs "ptools" in illumos & Solaris, can show the layout of a running process or a coredump. They tend to be large, but I'm seeing anonymous mappings in sub-47-bit space AND in the very high end.

Remember, I'm approaching this from Node's import of 13.6 of V8. Something goes screwy with the default address space layout in an illumos 64-bit process.  I'm still working at it from my end.

[Dan McDonald]

On Tuesday, April 15, 2025 at 4:22:21 PM UTC-4 Dan McDonald wrote:

We have a "VM hole" where nothing in a user process can be allocated or mapped in between 0x800000000000 (47-bits aka 4-level amd64 paging) and 0xffff7fffffffffff (one byte before high 2^47-bytes).  

I meant to say, "VA hole" earlier.  Sorry.

What's really happening is illumos uses the full range of 4-level paging virtual addresses. It does mean that (esp. anonymous) mmap()ed addresses can come from either side of the VA hole.

The AIX Gerrit:  https://chromium-review.googlesource.com/c/v8/v8/+/6320599

solves a similar problem, but only because they're growing beyond a 48-full/47-low bit allowable VA space.  If you look in

https://github.com/danmcd/node/commits/v8-136/

you will see the IBM one, and an illumos followup to cope with sign-extended 48-bit amd64 4-level paging addresses.

I do think the pointer-compression in the js-dispatch needs to account for such address layouts.  Additionally, I know there are facilities in Oracle Solaris at link-time to clip a process's address space to keep assumptions safe.  illumos doesn't have this on a per-process level yet; we probably will address it concurrent with full 5-level amd64 page support.

[Dan McDonald]

And an illumos-only commit in the context of node's v8 (which if not compiled for illumos doesn't affect any non-illumos code):

https://github.com/danmcd/node/commit/26b9a752a3875b563168cc1efdb557a9848f8cf7

[Dan McDonald]

Per the nearly-approved AIX commit conversation ( https://chromium-review.googlesource.com/c/v8/v8/+/6320599 ), I would like to address V8's mishandling of illumos/amd64 VA48 available address space.  The problem is rather straightforward:  illumos amd64 processes have their 48-bit available VA space split into two parts.

Per the original amd64/x64 4-level paging spec: the low 47 bits of address space: 0x0 -> 0x00007fffffffffff are available, AND SO IS the high 47 bits of available address space: 0xffff800000000000 -> 0xffffffffffffffff.  Notwithstanding carve-outs toward the extremes of the 64-bit address space (i.e. not too close to 0 or to 0xffffffffffffffff), memory mappings can come from either of those ranges.

For pointer-compression that shifts 16 bits to the left, the easy thing to do is to check the highest-order compressed-pointer bit, and fill the top 16 with 1s upon decompression.  Decompression is done in CodeStubAssembler::LoadCodeObjectFromJSDispatchTable(), and while I had my first-attempt implementation picked-apart as part of my experiments with Node, the idea is sound.

Also, the choice made in the JS Dispatch Table to mark a free pointer with 0xffff in the top 16 bits will not work in an amd64 address space using all of the available VA space, because half of it lives in address space starting with 0xffff.  A change of marking bits (I used 0xfeed in the first-attempt) and better clearing/checking (using logical-and) solves this.

The assumption of only-low-47-bits of virtual address space runs up against two problems.  The first is expanded available virtual address space beyond 0x0000800000000000.  The aforementioned IBM/AIX changes hint at this possibility: All 48 lower bits are available with a fixed non-zero 16-bit prefix.  The second is that at least for amd64, address space will appear in both the high-end and the low-end of the 64-bit address space.

Already available in some hardware is VA57, which resembles the aforementioned VA48 except that the low available space grows to 0x0 -> 0x007fffffffffffff, and the high available space grows down to cover 0xff80000000000000 -> 0xffffffffffffffff.  Operating systems may offer the entirety of both ends of VA57 to a process.

I would like to help correct this in V8 so its downstreams, especially Node, can work properly in environments that offer full address space to processes.  I'm reading up on https://v8.dev/docs/contribute , and please consider this email my following of, "Ask on V8’s mailing list for guidance".

[Erik Corry]

Seems like this will soon be a problem for Linux too.  My /proc/cpuinfo says:

address sizes : 52 bits physical, 57 bits virtual

so it looks like we can't assume the high 16 bits are zero for much longer.

[Olivier Flückiger]

Yeah, we are not clinging to that design. It really should be a normal compressed pointer. Then we'd have space for a separate mark bit and argument count like on 32 bit architectures. Last I checked there were some technical issues with getting the correct base to uncompress the pointer and it's also kinda performance sensitive. That's why nobody has addressed it so far.

That said, I don't think we have to worry about user space pointers in that range according to https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt .

*oli

--

--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/1ca103a0-8623-430a-83f4-58ee712b329en%40googlegroups.com.

[Erik Corry]

If your hardware supports 5-level page tables and your kernel is reasonably modern you can get addresses today where only the top 8 bits are zero:

The following program prints:

Size 64T, addr = 0x2000000000000
Size 128T, addr = 0xffba0bca5a6000

The high hint to mmap is important - if you don't have that then the kernel is backwards compatible and won't give out high addresses beyond the 48 bit limit.  This is why V8 still works on such hardware, but it limits the amount of virtual memory you can use.

#include <errno.h>
#include <stdio.h>
#include <sys/mman.h>

int main() {
  unsigned long size = 1ULL << 46;
  for (int i = 0; i < 2; i++) {
    void* addr = mmap(
        reinterpret_cast<void*>(1ULL << 49),
        1ULL << 46,
        PROT_NONE,
        MAP_PRIVATE | MAP_ANONYMOUS,
        0,
        0);
    printf("Size %dT, addr = %p\n", (int)(size >> 40), addr);
    if (addr == reinterpret_cast<void*>(-1)) {
      perror("mmap");
      return 1;
    }
    size <<= 1;
  }
  return 0;
}

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAPfE2j2Z%2BZOmz2YH2DUAO9axJcc1XhKECCDjVtTK8gwyHRDESQ%40mail.gmail.com.

[Olivier Flückiger]

Right, if we request heap pages in that area. Are you running into this issue?

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAHZxHpiBxkLHNKXV5_SQUMyCfXC6xt036d0v%3D5BcYwgzscs6sw%40mail.gmail.com.

[Erik Corry]

I'm running out of virtual memory on hardware that supports 5 level page tables, yes.

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAPfE2j3DCA4F0Fi6-_DxEcNs7VKKri1RWHkW_nBJwn6LhA_p4Q%40mail.gmail.com.

[Olivier Flückiger]

I see. If you have a suggestion on what to change for your usecase I am happy to review CLs. I just realized that a compressed pointer would not work for non-compressed builds. So that's likely not the solution. An easy fix would be to just make the dispatch entry bigger. There is already code for 32bit architectures: https://source.chromium.org/chromium/chromium/src/+/main:v8/src/sandbox/js-dispatch-table.h;l=148?q=js-dispatch-table.h&ss=chromium . We don't want to do this for chrome, but a compile time option would be fine.

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAHZxHpjPJazB2Nf%3Dbnoof%2B7K-TsdVVC7HuoiKy8rRqx9NerMWA%40mail.gmail.com.

[Erik Corry]

We use pointer compression, so it might still work for us.

To view this discussion visit https://groups.google.com/d/msgid/v8-dev/CAPfE2j01d885DrU9pEYVqaH2HYjsNbCAUfZVdpiJUmvwd5Tyaw%40mail.gmail.com.