Issue 1788 in webm: How to build Google_VP9 on windows arm64ec using MSVC

[v-yay… via monorail]

Status: Unconfirmed
Owner: ----

New issue 1788 by v-yay...@microsoft.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788

Summary:
We try MSVC to run on x64, it works. But we tried to build for target arm64ec, this project failed to build. How to build Google_VP9 on windows arm64ec using MSVC? Could you please help look at this issue? Thanks in advance.
steps of x64:
1. Open a VS 2019 x64 prompt
2. set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=amd64
3. git clone https://github.com/webmproject/libvpx F:\libvpx
3. cd F:\gitP\webmproject\libvpx\..
4. set PATH=F:\gitP\webmproject\libvpx\..\tools;%PATH%
set PATH=F:\gitP\webmproject\libvpx\..\tools\msys32\usr\bin;%PATH%
set VP9_SRC_PATH=F:\libvpx
set MSYS2_PATH_TYPE=inherit
set VP9_TARGET=x86_64-win64-vs16
set VP9_BUILD_DIR=build_x64
5. .\tools\msys32\usr\bin\sh.exe --login -i -c "/F/gitP/webmproject/libvpx/../tools/build.sh"

steps of arm64ec:
1. Open a VS 2019 x64 prompt
2. set VSCMD_SKIP_SENDTELEMETRY=1 & "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\Tools\VsDevCmd.bat" -host_arch=amd64 -arch=arm64
3. git clone https://github.com/webmproject/libvpx F:\libvpx
3. cd F:\gitP\webmproject\libvpx\..
4. set PATH=F:\gitP\webmproject\libvpx\..\tools;%PATH%
set PATH=F:\gitP\webmproject\libvpx\..\tools\msys32\usr\bin;%PATH%
set VP9_SRC_PATH=F:\libvpx
set MSYS2_PATH_TYPE=inherit
set VP9_TARGET=ARM64EC-win64-vs16
set VP9_BUILD_DIR=build_ARM64EC
5. .\tools\msys32\usr\bin\sh.exe --login -i -c "/F/gitP/webmproject/libvpx/../tools/build.sh"

Error message:
Starting build in F:\libvpx for target ARM64EC-win64-vs16 in build_ARM64EC
Configure build
enabling static_msvcrt
enabling unit_tests
enabling vp8_encoder
enabling vp8_decoder
enabling vp9_encoder
enabling vp9_decoder
Unrecognized toolchain 'ARM64EC-win64-vs16'

Configuration failed. This could reflect a misconfiguration of your
toolchains, improper options selected, or another problem. If you
don't see any useful error messages above, the next step is to look
at the configure error log file (config.log) to determine what
configure was trying to do when it died.
Failed to configure!

--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[v-yay… via monorail]

Comment #1 on issue 1788 by v-yay...@microsoft.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c1

(No comment was entered for this change.)

Attachments:
build.sh 604 bytes

[jz… via monorail]

Comment #2 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c2

> Unrecognized toolchain 'ARM64EC-win64-vs16'
> ../configure --target=$VP9_TARGET --enable-static-msvcrt --enable-unit-tests

To start with, VP9_TARGET should be arm64-win64-vs16. I haven't checked, but using that followed by a build with the arm64 environment should work.

What is 'EC' in ARM64EC?

[jz… via monorail]

Updates:
Labels: -Needs-Feedback
Owner: jz...@google.com
Status: Assigned

Comment #4 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c4

To answer my own question, ARMEC is 'Emulation Compatible' [1]. I was expecting a ARM64 build when I read it. The targets for configure have ARM64 (not EC) in mind.
The ARM64 target isn't building currently. I'll fix that and see if there's anything more to be done for this new target.

[1] https://blogs.windows.com/windowsdeveloper/2021/06/28/announcing-arm64ec-building-native-and-interoperable-apps-for-windows-11-on-arm/

[Git Watcher via monorail]

Comment #5 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c5

The following revision refers to this bug:
https://chromium.googlesource.com/webm/libvpx/+/858a8c611f4c965078485860a6820e2135e6611b

commit 858a8c611f4c965078485860a6820e2135e6611b
Author: James Zern <jz...@google.com>
Date: Wed Feb 01 21:27:06 2023

vp9_diamond_search_sad_neon: use DECLARE_ALIGNED

rather than the gcc specific __attribute__((aligned())); fixes build
targeting ARM64 windows.

Bug: webm:1788
Change-Id: I2210fc215f44d90c1ce9dee9b54888eb1b78c99e

[modify] https://crrev.com/858a8c611f4c965078485860a6820e2135e6611b/vp9/encoder/arm/neon/vp9_diamond_search_sad_neon.c

[jz… via monorail]

Comment #6 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c6

The code will build now, but I think there's more work to be done since Visual Studio's compiler doesn't define __aarch64__, it has _M_ARM64 and _M_ARM64EC in this case [1]. So we'll take the armv7 path in code that's specialized for __aarch64__. In the module in comment #5 this may cause an issue as a pointer address is being stored as a 32-bit integer when it's actually 64-bit.

[1] https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-170

[a… via monorail]

Comment #7 on issue 1788 by a...@amyspark.me: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c7

Hi all, I just found this while testing the Meson wrap I made for libvpx, under MSVC 17.6.0 preview 4.

> Visual Studio's compiler doesn't define __aarch64__, it has _M_ARM64 and _M_ARM64EC in this case

I thought defining __aarch64__ inplace would do the trick, but libyuv also relies on `volatile` and GNU-style inline assembly that's only gatekept by the presence of that macro (and the absence of LIBYUV_DISABLE_NEON). So those bits should be ported out or disabled for the build to succeed.

However, there's one further catch: if you also enable LIBYUV_DISABLE_NEON, you'll get a compiler crash in fdct_partial_neon.c. I'll be reporting this later this weekend.

[a… via monorail]

Comment #8 on issue 1788 by a...@amyspark.me: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c8

Issue reported: https://developercommunity.visualstudio.com/t/Compiler-crash-C1001-when-libvpx-for-arm/10346110

BTW, cherry picking I2210fc215f44d90c1ce9dee9b54888eb1b78c99e onto 1.13.0 without further flag tinkering is enough to make my Meson wrap happy (apart from the issue highlighted by James in #6).

[Git Watcher via monorail]

Comment #10 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c10

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/33aba6ecc16c71b6f380cfd3d24eab11690deb99

commit 33aba6ecc16c71b6f380cfd3d24eab11690deb99
Author: James Zern <jz...@google.com>
Date: Fri Apr 21 01:15:23 2023

configure: add aarch64 to ARCH_LIST

This will allow identifying Windows Visual Studio targets as aarch64;
the Microsoft compiler does not define __aarch64__.

An alternative would be to define this in the code, checking for
_M_ARM64 or _M_ARM64EC. For now we'll use the existing VPX_ARCH_*
system. For compatibility VPX_ARCH_ARM will continue to be defined to 1
in this case.

Bug: webm:1788
Bug: b/277255076
Change-Id: I12e25710891e86f0c7339ba96884c18ed90ba16f

[modify] https://crrev.com/33aba6ecc16c71b6f380cfd3d24eab11690deb99/build/make/configure.sh
[modify] https://crrev.com/33aba6ecc16c71b6f380cfd3d24eab11690deb99/configure

[Git Watcher via monorail]

Comment #9 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c9

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/57b9afa58f849a8165ce3132c21087ae451d862c

commit 57b9afa58f849a8165ce3132c21087ae451d862c
Author: James Zern <jz...@google.com>
Date: Wed May 03 01:37:59 2023

s/__aarch64__/VPX_ARCH_AARCH64/

This allows AArch64 to be correctly detected when building with Visual
Studio (cl.exe) and fixes a crash in vp9_diamond_search_sad_neon.c.
There are still test failures, however.

Microsoft's compiler doesn't define __ARM_FEATURE_*. To use those paths
we may need to rely on _M_ARM64_EXTENSION.

Bug: webm:1788
Bug: b/277255076

Change-Id: I4d26f5f84dbd0cbcd1cdf0d7d932ebcf109febe5

[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/vpx_convolve8_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vp8/encoder/arm/neon/fastquantizeb_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/highbd_avg_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vp9/encoder/arm/neon/vp9_denoiser_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/vpx_convolve8_neon.h
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/transpose_neon.h
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/avg_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vp9/encoder/arm/neon/vp9_quantize_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vp9/encoder/arm/neon/vp9_diamond_search_sad_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/highbd_quantize_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/quantize_neon.c
[modify] https://crrev.com/57b9afa58f849a8165ce3132c21087ae451d862c/vpx_dsp/arm/sum_neon.h

[jz… via monorail]

Comment #11 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c11

As mentioned in the commit message in comment #10, the library shouldn't crash when built with Microsoft's compiler, but many of the tests fail (in part possibly due to poor vectorization, but this still needs investigation).

For now clang-cl is the recommended configuration. It's currently a manual process to switch this and add ARM64EC. You can generate the projects with --target=arm64-win64-vs17, then update the toolset for each project (Configuration > General > Platform Toolset) and finally copy the configuration to a new one for ARM64EC. I have not tried this final step yet though.

[jz… via monorail]

Updates:
Components: libvpx
Labels: Type-Enhancement

Comment #12 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c12

> For now clang-cl is the recommended configuration. It's currently a manual process to switch this and add ARM64EC. ...

Unfortunately ARM64EC doesn't have clang-cl support in Visual Studio 17.5.5. I see an ongoing conversation in llvm development [1].

[1] https://discourse.llvm.org/t/initial-patches-for-arm64ec-windows-11-now-posted/62449/10

[Git Watcher via monorail]

Comment #13 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c13

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/3916e0e1308acd8511269ca4ed6394601f72cc10

commit 3916e0e1308acd8511269ca4ed6394601f72cc10
Author: James Zern <jz...@google.com>
Date: Sat May 06 02:00:08 2023

gen_msvs_vcxproj: add ARM64EC w/VS >= 2022

rather than define new targets, add a platform to the arm64 list as they
share the same configuration.

Bug: webm:1788
Change-Id: Iac020280b1103fb12b559f21439aeff26568fba4

[modify] https://crrev.com/3916e0e1308acd8511269ca4ed6394601f72cc10/build/make/gen_msvs_vcxproj.sh

[Git Watcher via monorail]

Comment #14 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c14

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/3fe13658846564f37399035146132ee2af2b1ba6

commit 3fe13658846564f37399035146132ee2af2b1ba6
Author: James Zern <jz...@google.com>
Date: Fri May 05 23:56:51 2023

configure: add clang-cl vs1[67] arm64 targets

x86 and armv7 are skipped for now as the intrinsics will need different
flags than cl.exe (/arch:... -> -m...).

Bug: webm:1788
Change-Id: I8ca8660a8644cdd84c51cb1f75005e371ba8207d

[modify] https://crrev.com/3fe13658846564f37399035146132ee2af2b1ba6/build/make/gen_msvs_vcxproj.sh
[modify] https://crrev.com/3fe13658846564f37399035146132ee2af2b1ba6/README
[modify] https://crrev.com/3fe13658846564f37399035146132ee2af2b1ba6/configure

[jz… via monorail]

Comment #15 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c15

Now Arm64EC can be targeted directly with --target=arm64-win64-vs17, no need to change the project file. There's also --target=arm64-win64-vs17-clangcl, but as mentioned in comment #12, that's only for Arm64 right now.

[Git Watcher via monorail]

Comment #16 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c16

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/25f2e1ef255e89d5e7357aa2427926776327765a

commit 25f2e1ef255e89d5e7357aa2427926776327765a
Author: James Zern <jz...@google.com>
Date: Tue May 23 22:50:10 2023

vpx_dsp_common.h,clip_pixel: work around VS2022 Arm64 issue

cl.exe targeting AArch64 with optimizations enabled
produces invalid code for clip_pixel() when the return type is uint8_t.
See:
https://developercommunity.visualstudio.com/t/Misoptimization-for-ARM64-in-VS-2022-17/10363361

Bug: b/277255076
Bug: webm:1788
Change-Id: Ia3647698effd34f1cf196cd33fa4a8cab9fa53d6

[modify] https://crrev.com/25f2e1ef255e89d5e7357aa2427926776327765a/vpx_dsp/vpx_dsp_common.h

[Git Watcher via monorail]

Comment #17 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c17

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/95b56ab7df669ec5dd29c283fa5bf6d38c2df5d1

commit 95b56ab7df669ec5dd29c283fa5bf6d38c2df5d1
Author: James Zern <jz...@google.com>
Date: Tue May 23 22:49:29 2023

fdct_partial_neon.c: work around VS2022 Arm64 issue

cl.exe targeting AArch64 with optimizations enabled

will fail with an internal compiler error.
See:
https://developercommunity.visualstudio.com/t/Compiler-crash-C1001-when-building-a-for/10346110

Bug: b/277255076
Bug: webm:1788

Change-Id: I55caf34e910dab47a7775f07280677cdfe606f5b

[modify] https://crrev.com/95b56ab7df669ec5dd29c283fa5bf6d38c2df5d1/vpx_dsp/arm/fdct_partial_neon.c

[Git Watcher via monorail]

Comment #18 on issue 1788 by Git Watcher: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c18

The following revision refers to this bug:

https://chromium.googlesource.com/webm/libvpx/+/62d09a3e94ef2ce0091b6a8e3a298851657c6891

commit 62d09a3e94ef2ce0091b6a8e3a298851657c6891
Author: James Zern <jz...@google.com>
Date: Tue May 23 22:48:10 2023

fdct8x8_test.cc: work around VS2022 Arm64 issue

cl.exe targeting AArch64 with optimizations enabled

produces invalid code in RunExtremalCheck() and RunInvAccuracyCheck().
See:
https://developercommunity.visualstudio.com/t/1770-preview-1:-Misoptimization-for-AR/10369786

Bug: b/277255076
Bug: webm:1788

Change-Id: Id2c60f3948d8f788c78602aea1b5232133415dea

[modify] https://crrev.com/62d09a3e94ef2ce0091b6a8e3a298851657c6891/test/fdct8x8_test.cc

[jz… via monorail]

Updates:
Status: Fixed

Comment #19 on issue 1788 by jz...@google.com: How to build Google_VP9 on windows arm64ec using MSVC
https://bugs.chromium.org/p/webm/issues/detail?id=1788#c19

With workarounds in place, this should be working now with cl.exe. If you run into any issues feel free to reopen this bug or file a new one.