Fixing mkgrokdump for Windows on Arm, MSVC

[richard....@arm.com]

Hi folks, I'm picking up work again on v8:10012. The goal of this work is to regularly build and test V8 for Windows on Arm under Microsoft's Visual C++ Compiler (MSVC), because downstream projects (in particular Node.js) use this compiler.

I've run into a problem with the mkgrokdump utility. Basically, it links with embedded.o (assembled from host_x64/gen/embedded.S), which is generated by the mksnapshot utility. When building with Clang, mksnapshot emits assembly directives compatible with all platforms (and therefore there's no issue). Microsoft's assemblers however don't support this syntax, and there are incompatible differences between the syntax accepted by their i386/x64 assembler and their ARM64 assembler. So mksnapshot emits ARM64 assembly, GN tries to assemble it with the x64 assembler (for linking with mkgrokdump) and and the assembler reports >10000 errors as a result. 

So based on what I think mkgrokdump does (initialize from the snapshot_blob.bin, enumerate heap constants and export them for another scripts) is there a reason why it has to statically link with embedded.o? Or would it be preferable to keep the build setup as it is and change mksnapshot so it can emit either syntax? Are there more possibilities that we should investigate? I'd appreciate any design input you have on the most workable solution. 

Best

Richard

[Jakob Gruber]

On Thu, Feb 13, 2020 at 5:37 PM <richard....@arm.com> wrote:

Hi folks, I'm picking up work again on v8:10012. The goal of this work is to regularly build and test V8 for Windows on Arm under Microsoft's Visual C++ Compiler (MSVC), because downstream projects (in particular Node.js) use this compiler.

I've run into a problem with the mkgrokdump utility. Basically, it links with embedded.o (assembled from host_x64/gen/embedded.S), which is generated by the mksnapshot utility. When building with Clang, mksnapshot emits assembly directives compatible with all platforms (and therefore there's no issue). Microsoft's assemblers however don't support this syntax, and there are incompatible differences between the syntax accepted by their i386/x64 assembler and their ARM64 assembler. So mksnapshot emits ARM64 assembly, GN tries to assemble it with the x64 assembler (for linking with mkgrokdump) and and the assembler reports >10000 errors as a result. 

So based on what I think mkgrokdump does (initialize from the snapshot_blob.bin, enumerate heap constants and export them for another scripts) is there a reason why it has to statically link with embedded.o?

The contents of embedded.o are probably not important since no JS code runs in mkgrokdump. Theoretically one could use embedded-empty.cc, but currently isolate startup would fail on the embedded-blob consistency check.

 

Or would it be preferable to keep the build setup as it is and change mksnapshot so it can emit either syntax?

mksnapshot already has support for multiple output formats on windows, see here and here. The former looks at compile-time defines to determine compiler/target architecture, while the latter takes --target-arch and --target-os flags passed to mksnapshot [0].

 

Are there more possibilities that we should investigate? I'd appreciate any design input you have on the most workable solution. 

How does mkgrokdump work in other cross-compile configs? Isn't it simply compiled for the target platform / if so, why do something different here? And more generally, do you need mkgrokdump at all?

[0] The mix of compile-time and runtime config of the target arch is here for historical reasons, ideally only the runtime flag should be considered. We just haven't needed that option yet.

Best

Richard

--
--
v8-dev mailing list
v8-...@googlegroups.com
http://groups.google.com/group/v8-dev
---
You received this message because you are subscribed to the Google Groups "v8-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/f2e3e89e-ac47-432b-9fb5-00c65350cbfa%40googlegroups.com.

[richard....@arm.com]

OK, following up on this after a few more experiments:

• It does not appear to be very easy to remove the host mkgrokdump from the dependency chain.
• Emitting target-specific assembly via the runtime flag does appear to work and resolve the issue. However, Arm's internal V8 team are concerned that removing all compile-time defines is potentially a large change to the way mksnapshot works and would prefer to resolve the structural dependency issue.
• If the structural issue is resolved and mkgrokdump is statically linked against embedded-empty.cc, various debug / runtime assertions fire when mkgrokdump is invoked. I presume those asserts (including stuff like IsolateIsCompatibleWithEmbeddedBlob) are there for a reason so I need to think a bit more about what the effect of removing them is.

Best

Richard

On Monday, February 17, 2020 at 6:57:59 AM UTC, Jakob Gruber wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to v8-...@googlegroups.com.

[Jakob Gruber]

On Thu, Feb 27, 2020 at 10:20 PM <richard....@arm.com> wrote:

OK, following up on this after a few more experiments:

• It does not appear to be very easy to remove the host mkgrokdump from the dependency chain.
• Emitting target-specific assembly via the runtime flag does appear to work and resolve the issue. However, Arm's internal V8 team are concerned that removing all compile-time defines is potentially a large change to the way mksnapshot works and would prefer to resolve the structural dependency issue.
• If the structural issue is resolved and mkgrokdump is statically linked against embedded-empty.cc, various debug / runtime assertions fire when mkgrokdump is invoked. I presume those asserts (including stuff like IsolateIsCompatibleWithEmbeddedBlob) are there for a reason so I need to think a bit more about what the effect of removing them is.

This assert in particular effectively verifies that the versions of embedded_blob.S and snapshot_blob.bin match. This is important whenever any builtins are executed (because builtins can index into the `builtins_constants_table`, which is part of the initial heap state), but AFAIK that is not the case in mkgrokdump.

 

To unsubscribe from this group and stop receiving emails from it, send an email to v8-dev+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/8eba1c5e-0a25-4f37-95d5-866e3814681a%40googlegroups.com.

[richard....@arm.com]

So now that the build system changes are making their way into V8, the the mkgrokdump issue is the last remaining thing for fixing the build. We now have two viable approaches:

• https://chromium-review.googlesource.com/c/v8/v8/+/1962854 means that mksnapshot can emit dual syntaxes behind the runtime flag.
• https://chromium-review.googlesource.com/c/v8/v8/+/2085174 is a WIP solution for disentangling the snapshot from mkgrokdump. It has a few issues e.g. with GN visibility and figuring out the assert situation.
  

Let me know which one is most acceptable. 

Thanks

Richard

On Monday, March 2, 2020 at 6:24:17 AM UTC, Jakob Gruber wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/v8-dev/8eba1c5e-0a25-4f37-95d5-866e3814681a%40googlegroups.com.