Issue 7905 in v8: generating v8 build files fails on mac os

[reinstei… via monorail]

Status: Assigned
Owner: machenb...@chromium.org
Components: Infrastructure
Type: Bug

New issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905

I followed the instructions from https://github.com/v8/v8/wiki/Building-from-Source

here are my repro steps:

fetch v8

cd v8

gclient sync

tools/dev/v8gen.py x64.debug -vv



terminal Output:

################################################################################
/usr/local/opt/python@2/bin/python2.7 -u tools/mb/mb.py gen -f infra/mb/mb_config.pyl -m developer_default -b x64.debug out.gn/x64.debug

Writing """\
is_debug = true
target_cpu = "x64"
v8_enable_backtrace = true
v8_enable_slow_dchecks = true
v8_optimized_debug = false
""" to /Users/michaelreinstein/wwwroot/v8/out.gn/x64.debug/args.gn.

/Users/michaelreinstein/wwwroot/v8/buildtools/mac/gn gen out.gn/x64.debug --check
-> returned 1
ERROR at dynamically parsed input that //build/config/mac/mac_sdk.gni:84:19 loaded :1:1: Invalid token.
machine_os_build="17F77"
^
I have no idea what this is.
See //build/toolchain/mac/BUILD.gn:14:1: whence it was imported.
import("//build/config/mac/mac_sdk.gni")
^--------------------------------------
See //BUILD.gn:646:1: which caused the file to be included.
action("js2c") {
^---------------
GN gen failed: 1
Traceback (most recent call last):
File "tools/dev/v8gen.py", line 304, in <module>
sys.exit(gen.main())
File "tools/dev/v8gen.py", line 298, in main
return self._options.func()
File "tools/dev/v8gen.py", line 166, in cmd_gen
gn_outdir,
File "tools/dev/v8gen.py", line 208, in _call_cmd
stderr=subprocess.STDOUT,
File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 223, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['/usr/local/opt/python@2/bin/python2.7&#39;, '-u', 'tools/mb/mb.py', 'gen', '-f', 'infra/mb/mb_config.pyl', '-m', 'developer_default', '-b', 'x64.debug', 'out.gn/x64.debug']' returned non-zero exit status 1


it seems that this line in build/config/mac/mac_sdk.gni is the one that fails:

_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

any ideas?



--
You received this message because:
1. The project was configured to send all issue notifications to this address

You may adjust your notification preferences at:
https://bugs.chromium.org/hosting/settings

[reinstei… via monorail]

Comment #1 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c1

Here's the output of running sdk_info.py directly:

λ python build/config/mac/sdk_info.py macosx
machine_os_build="17F77"
sdk_build="17E189"
sdk_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk"
sdk_platform_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform"
sdk_version="10.13"
xcode_build="9F2000"
xcode_version="0941"
xcode_version_int=941

[machenb… via monorail]

Updates:
Cc: dpr...@chromium.org tha...@chromium.org
Owner: ----
Status: Available

Comment #2 on issue 7905 by machenb...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c2

Seems like a mac sdk detection problem specific to the gn build setup? I assume you'd see the same when running gn in chromium on your machine?

[tha… via monorail]

Comment #3 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c3

What's the output of `env`?

[reinstei… via monorail]

Comment #4 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c4

λ env
rvm_bin_path=/Users/michaelreinstein/.rvm/bin
TERM_PROGRAM=Apple_Terminal
GEM_HOME=/Users/michaelreinstein/.rvm/gems/ruby-2.2.1
CCACHE_SLOPPINESS=time_macros
SHELL=/bin/bash
TERM=xterm-256color
IRBRC=/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1/.irbrc
TMPDIR=/var/folders/_5/_ng5y8rx60b72fdbq7vhh0m00000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.rEKqInesNC/Render
TERM_PROGRAM_VERSION=404
OLDPWD=/Users/michaelreinstein/wwwroot/xid
MY_RUBY_HOME=/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1
TERM_SESSION_ID=346F0BD1-C5DF-41D9-98F3-BDBB1EC6D5C7
USER=michaelreinstein
_system_type=Darwin
rvm_path=/Users/michaelreinstein/.rvm
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.BYnOCseQ85/Listeners
CCACHE_CPP2=yes
rvm_prefix=/Users/michaelreinstein
PATH=/Users/michaelreinstein/.cargo/bin:/Users/michaelreinstein/wwwroot/v8/third_party/llvm-build/Release+Asserts/bin:/usr/local/sbin:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1/bin:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1@global/bin:/Users/michaelreinstein/.rvm/rubies/ruby-2.2.1/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/Users/michaelreinstein/.rvm/bin:/Users/michaelreinstein/wwwroot/depot_tools
PWD=/Users/michaelreinstein/wwwroot/v8
LANG=en_US.UTF-8
_system_arch=x86_64
XPC_FLAGS=0x0
PS1=λ
_system_version=10.13
CXX=ccache clang++ -Qunused-arguments
XPC_SERVICE_NAME=0
rvm_version=1.26.11 (latest)
SHLVL=1
HOME=/Users/michaelreinstein
LOGNAME=michaelreinstein
GEM_PATH=/Users/michaelreinstein/.rvm/gems/ruby-2.2.1:/Users/michaelreinstein/.rvm/gems/ruby-2.2.1@global
CC=ccache clang -Qunused-arguments
DISPLAY=/private/tmp/com.apple.launchd.Lm3LlMuAMR/org.macosforge.xquartz:0
RUBY_VERSION=ruby-2.2.1
_system_name=OSX
_=/usr/bin/env

[tha… via monorail]

Updates:
Labels: OS-Mac

Comment #5 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c5

Looks pretty sane to me. This should work, not sure what's wrong. You'll have to debug things yourself :-/

[reinstei… via monorail]

Comment #6 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c6

Alright, well thanks for asserting some things. Even just hearing things don't look obviously wrong is helpful. :)

[reinstei… via monorail]

Comment #7 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c7

I've narrowed down the commands that run to reproduce this error:

λ gn gen out.gn/x64.release --check

ERROR at dynamically parsed input that //build/config/mac/mac_sdk.gni:84:19 loaded :1:1: Invalid token.
machine_os_build="17F77"
^
I have no idea what this is.
See //build/toolchain/mac/BUILD.gn:14:1: whence it was imported.
import("//build/config/mac/mac_sdk.gni")
^--------------------------------------
See //BUILD.gn:646:1: which caused the file to be included.
action("js2c") {
^---------------

Here is line 84 of mac_sdk.gni:84:

_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

script_name is //build/config/mac/sdk_info.py and sdk_info_args is [ "macosx" ]

So this line is calling python build/config/mac/sdk_info.py macosx which prints this to stdout:

machine_os_build="17F77"
sdk_build="17E189"
sdk_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk"
sdk_platform_path="/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform"
sdk_version="10.13"
xcode_build="9F2000"
xcode_version="0941"
xcode_version_int=941

exec_script() with the 3rd argument set to "scope" is supposed to evaluate this string as gn code and then make it available to the return object (_mac_sdk_result in this case.)

what I don't understand is the stuff that sdk_info.py prints out looks like it should be valid gn code, but it is failing in the tokenizing step (this line I think https://chromium.googlesource.com/chromium/src/+/master/tools/gn/tokenizer.cc#405 )

after doing some digging it appears that gn is being run from buildtools/mac/gn Is it possible this binary needs to be rebuilt with a more up-to-date version?

Any help would be greatly appreciated.

[tha… via monorail]

Comment #8 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c8

What's buildtools/mac/gn --version? The binary should autoupdate when you run `gclient sync` (which you did run).

[reinstei… via monorail]

Comment #9 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c9

λ buildtools/mac/gn --version
UNKNOWN

[tha… via monorail]

Comment #10 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c10

(cd buildtools; git log --oneline | head -1)

?

[reinstei… via monorail]

Comment #11 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c11

λ (cd buildtools; git log --oneline | head -1)
0dd5c6f Merge "Revert "Remove libcpp_is_static""

[tha… via monorail]

Comment #12 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c12

If I read

https://gn.googlesource.com/gn/+log/master

and

https://bugs.chromium.org/p/chromium/issues/detail?id=858852#c3

right, then the UNKNOWN means you have the latest revision (we moved gn around which broke --version, and the binary in buildtools doesn't have the fix for that yet; the flag worked fine before the move).


Since bug 859536 isn't fixed yet: To debug more, you can build a local gn with `ninja -C out/builddir`...oh wait, you only have a v8 checkout, not a gn checkout.

Well, then you can clone https://gn.googlesource.com/gn/ and build that and add some debug printfs to see what's going wrong. (Everything you say in comment 7 sounds right to me.)

If you just want to build v8 and don't mind some local lingering hacks, you can alternatively remove the machine_os_build print line from sdk_info.py and replace

https://cs.chromium.org/chromium/src/build/config/mac/mac_sdk.gni?q=_mac_sdk_result&sq=package:chromium&g=0&l=87

with just

machine_os_build="17F77"

right in the gn file and see if that helps any.

[reinstei… via monorail]

Comment #13 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c13

Thanks for the info!


I've modified mac_sdk.gni as such:

#_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

xcode_version = "0941" #_mac_sdk_result.xcode_version
xcode_build = "9F2000" #_mac_sdk_result.xcode_build
machine_os_build = "17F77" #_mac_sdk_result.machine_os_build

basically just hardcoding the 3 used settings, and then it works:

λ tools/dev/v8gen.py x64.release -vv
################################################################################
/usr/local/opt/python@2/bin/python2.7 -u tools/mb/mb.py gen -f infra/mb/mb_config.pyl -m developer_default -b x64.release out.gn/x64.release

Writing """\
is_debug = false
target_cpu = "x64"
""" to /Users/michaelreinstein/wwwroot/v8/out.gn/x64.release/args.gn.

/Users/michaelreinstein/wwwroot/v8/buildtools/mac/gn gen out.gn/x64.release --check
Done. Made 122 targets from 79 files in 311ms

so it definitely is just this one issue with exec_script() not handling being able to parse this into a scope.


> Since bug 859536 isn't fixed yet ... you can clone https://gn.googlesource.com/gn/

> and build that and add some debug printfs to see what's going wrong.

So it sounds like what might be happening is the gn binary that v8 is using resides in buildtools/mac/gn, and this might not be updated to the newest due to changes to how gn gets included with the project. Is this an accurate re-statement of the problem? I'll try building gn from the link you provided and see if that version solves the problem.

> If you just want to build v8 and don't mind some local lingering hacks,

unfortunately I probably can't have local lingering hacks. I'm ultimately trying to build v8 to include in deno which pulls in v8 as an unmodified dependency (https://github.com/ry/deno)

If the manually built version of gn that I'm about to build works, what's the resolution?

[tha… via monorail]

Comment #14 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c14

If the manually-built version of gn at 4cf8d1531 works, then I'd be really confused since my current mental model is that v8 uses the same gn binary as chromium (as of https://chromium.googlesource.com/v8/v8/+/5d4dbb6ee98b18868807830c358c05dcf8f6d219 , which updated buildtools), and that's gn built at that revision.


Hrm, I suppose the standalone gn build just uses whatever clang is installed on the system building gn instead of the hermetic clang the chromium and v8 builds use. Maybe scottmg's Xcode's clang had some compiler bug causing the weirdness you see, and your local clang is different or something?

But let's see if you can still repro with the self-built gn first.

[reinstei… via monorail]

Comment #15 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c15

It fails to build:

λ git clone https://gn.googlesource.com/gn
λ cd gn/
λ git checkout 4cf8d1531
Note: checking out '4cf8d1531'.
HEAD is now at 4cf8d153 Make Mac explicitly target 10.9
λ python build/gen.py
λ ninja -C out
ninja: Entering directory `out'
[3/54] LINK gn
FAILED: gn
ccache clang++ -Qunused-arguments -Wl,-S -flto -mmacosx-version-min=10.9 -o gn tools/gn/gn_main.o -framework AppKit -framework CoreFoundation -framework Foundation -framework Security base.a gn_lib.a
ld: could not process llvm bitcode object file, because /Users/michaelreinstein/wwwroot/v8/third_party/llvm-build/Release+Asserts/lib/libLTO.dylib could not be loaded file 'tools/gn/gn_main.o' for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[12/54] CXX tools/gn/functions_unittest.o
ninja: build stopped: subcommand failed.

[reinstei… via monorail]

Comment #16 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c16

if I add the --debug flag as in:

λ python build/gen.py --debug and then run the remaining commands it works though.

[tha… via monorail]

Updates:
Cc: sco...@chromium.org

Comment #17 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c17

The -flto is fromhttps://gn.googlesource.com/gn/+/1a883b0163f400af82a140df2166559779b51b70 ; you remove 'flto' in build/gen.py in the gn repo (from both cflags and ldflags) to work around that.

Alternatively you can remove third_party/llvm-build/Release+Asserts/bin from your PATH to use local xcode clang instead of chromium clang.

[reinstei… via monorail]

Comment #18 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c18

The changes you mentioned do indeed allow the build of gn to proceed:

modified the lines in gn/build/gen.py:

cflags.extend(['-O3', '-DNDEBUG'])
#cflags.extend(['-O3', '-flto', '-DNDEBUG'])

ldflags.append('-Wl,-S' if is_mac else '-Wl,-strip-all')
#ldflags.append('-flto')


then ran:

λ rm -rf out/

λ python build/gen.py
λ ninja -C out
ninja: Entering directory `out'

[236/236] LINK gn_unittests

then I copy this binary into the v8 project:
cp out/gn ~/wwwroot/v8/buildtools/mac/

I still get the same error when attempting to run gn on the v8 project

λ tools/dev/v8gen.py x64.release -vv

this still fails with the same error:

ERROR at dynamically parsed input that //build/config/mac/mac_sdk.gni:85:19 loaded :1:1: Invalid token.
machine_os_build="17F77"

[tha… via monorail]

Comment #19 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c19

That's good news, now you can add debug printfs :-)

You don't need to copy over the gn binaries, you should be able to run `../../path/to/gn gen out/foobar` while in v8/src to get the same effect.

[reinstei… via monorail]

Comment #20 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c20

If I wanted to run my gn build through gdb is this sufficient to get all the debug symbols in place?

λ python build/gen.py --debug
λ ninja -C out

thank you very much for your help so far by the way!

[tha… via monorail]

Comment #21 on issue 7905 by tha...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c21

I think that should be enough, but I haven't tried it.

(If you run `ninja -C out gn` then it builds just gn and not gn_unittests, which will be a bit faster.)

[reinstei… via monorail]

Comment #22 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c22

cool, I'll work on stepping this through gdb, or if that fails I'll resort to the good ole' fashioned printfs.

> then it builds just gn and not gn_unittests, which will be a bit faster

thanks!

[reinstei… via monorail]

Comment #23 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c23

I think I've figured out what's happening. After running my debug symbol laden copy of gn through gdb, I've found that the input file that gets passed to the tokenizer does has an 8 byte escape sequence at the beginning: \E[?1034h


If I hexdump the output of the python command this can be readily seen:

λ python build/config/mac/sdk_info.py macosx | hexdump -C
00000000 1b 5b 3f 31 30 33 34 68 6d 61 63 68 69 6e 65 5f |.[?1034hmachine_|
00000010 6f 73 5f 62 75 69 6c 64 3d 22 31 37 46 37 37 22 |os_build="17F77"|
00000020 0a 73 64 6b 5f 62 75 69 6c 64 3d 22 31 37 45 31 |.sdk_build="17E1|
00000030 38 39 22 0a 73 64 6b 5f 70 61 74 68 3d 22 2f 41 |89".sdk_path="/A|
00000040 70 70 6c 69 63 61 74 69 6f 6e 73 2f 58 63 6f 64 |pplications/Xcod|
00000050 65 2e 61 70 70 2f 43 6f 6e 74 65 6e 74 73 2f 44 |e.app/Contents/D|
00000060 65 76 65 6c 6f 70 65 72 2f 50 6c 61 74 66 6f 72 |eveloper/Platfor|
00000070 6d 73 2f 4d 61 63 4f 53 58 2e 70 6c 61 74 66 6f |ms/MacOSX.platfo|
00000080 72 6d 2f 44 65 76 65 6c 6f 70 65 72 2f 53 44 4b |rm/Developer/SDK|
00000090 73 2f 4d 61 63 4f 53 58 31 30 2e 31 33 2e 73 64 |s/MacOSX10.13.sd|
000000a0 6b 22 0a 73 64 6b 5f 70 6c 61 74 66 6f 72 6d 5f |k".sdk_platform_|
000000b0 70 61 74 68 3d 22 2f 41 70 70 6c 69 63 61 74 69 |path="/Applicati|
000000c0 6f 6e 73 2f 58 63 6f 64 65 2e 61 70 70 2f 43 6f |ons/Xcode.app/Co|
000000d0 6e 74 65 6e 74 73 2f 44 65 76 65 6c 6f 70 65 72 |ntents/Developer|
000000e0 2f 50 6c 61 74 66 6f 72 6d 73 2f 4d 61 63 4f 53 |/Platforms/MacOS|
000000f0 58 2e 70 6c 61 74 66 6f 72 6d 22 0a 73 64 6b 5f |X.platform".sdk_|
00000100 76 65 72 73 69 6f 6e 3d 22 31 30 2e 31 33 22 0a |version="10.13".|
00000110 78 63 6f 64 65 5f 62 75 69 6c 64 3d 22 39 46 32 |xcode_build="9F2|
00000120 30 30 30 22 0a 78 63 6f 64 65 5f 76 65 72 73 69 |000".xcode_versi|
00000130 6f 6e 3d 22 30 39 34 31 22 0a 78 63 6f 64 65 5f |on="0941".xcode_|
00000140 76 65 72 73 69 6f 6e 5f 69 6e 74 3d 39 34 31 0a |version_int=941.|
00000150


So somehow invoking python is causing this escape sequence to be included in the output. Now the question seems to be, what is the right way to detect this and prevent python from doing it?

[reinstei… via monorail]

Comment #24 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c24

seems to be the relevant python bug: https://bugs.python.org/issue19884

I'd be happy to submit a patch or help with making this more robust for users as this issue was particularly annoying given that it involves invisible control characters.

just let me know how I can help

[machenb… via monorail]

Comment #25 on issue 7905 by machenb...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c25

This seems to be independent of V8. Should we move the issue to Chromium tracker?

[reinstei… via monorail]

Comment #26 on issue 7905 by reinstei...@gmail.com: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c26

I guess it's a question of which project //build/config/mac/mac_sdk.gni is a part of. It contains this line, which is the culprit:

_mac_sdk_result = exec_script(script_name, sdk_info_args, "scope")

This invokes python and attempts to parse the results as a gn scope and return it. If this line is run with python < v3.6 and TERM=xterm this will inject the hidden escape sequence and break the build. given that:

* v8 relies on python 2.x
* having TERM=xterm in the environement is a very common environment setting for macosx

This seems like it's worth guarding against in v8, right? Or perhaps I'm misunderstanding the relationship between the chromium and v8 repos?

[machenb… via monorail]

Comment #27 on issue 7905 by machenb...@chromium.org: generating v8 build files fails on mac os
https://bugs.chromium.org/p/v8/issues/detail?id=7905#c27

V8 inherits that from Chromium. It lives in src/build. So I'll move to different tracker.