Chromium build refactor update

[Brett Wilson]

We’ve decided to move forward with the GN build replacement. This will
affect you! (Hopefully in a good way long-term.)

The high-level plan is that GN will generate GYP files while we
gradually transition more and more of the build. When all targets are
converted, we can remove the intermediate GYP step.

I just checked in a patch that runs GN at the beginning of GYP
execution. You should see a message to this effect when you do gclient
runhooks next time. Currently, this step reads the GN build but no
targets are configured so it generates nothing. A followup patch will
try to generate a GYP file from one target. I somewhat arbitrarily
picked the “third_party/re2” target as the initial step so keep an eye
out for errors there.

Please see the FAQ for more information:
https://code.google.com/p/chromium/wiki/GNFaq

When I get the initial GN-generated target checked in and sticking, it
will be easier for others to work on the GN build. Look for more
communication on how to use the system and work on the build at that
time.

Brett

[Peter Kasting]

On Wed, Dec 4, 2013 at 2:46 PM, Brett Wilson <bre...@chromium.org> wrote:

I just checked in a patch that runs GN at the beginning of GYP
execution. You should see a message to this effect when you do gclient
runhooks next time. Currently, this step reads the GN build but no
targets are configured so it generates nothing. A followup patch will
try to generate a GYP file from one target. I somewhat arbitrarily
picked the “third_party/re2” target as the initial step so keep an eye
out for errors there.

I don't know if others are getting this, but for me gclient runhooks now gives some kind of GN-related error:

http://pastebin.com/LPGQU4Nj

PK

[Shashi Shekhar]

Looks like you might be missing gn.exe or gnpath is incorrect.

There also seems to be a bug, if one has supplement.gypi that does not have a 'variables' defined then the script might fail.

File "/src/build/gyp_chromium", line 86, in GetVarsStringForGN

for v in variables:

TypeError: 'NoneType' object is not iterable

I use the developer recommended flags for Android in supplement.gypi.

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Stephen Chenney]

So what should gnpath be set to? Where should gn.exe be located? How do we check it is correctly synced?

Horrible example of how to roll out a new feature. I can't find anything at all about the chromium setup for this.

Stephen.

[Nico Weber]

On Wed, Dec 4, 2013 at 5:00 PM, Stephen Chenney <sche...@chromium.org> wrote:

So what should gnpath be set to? Where should gn.exe be located? How do we check it is correctly synced?

The idea is that it Just Works and everything is set correctly. If that isn't the case, please file a bug!

In Peter's case, I'm guessing nobody thought of testing with cygwin.

 

I can't find anything at all about the chromium setup for this.

That's because no setup should be necessary in theory :-)

[Stephen Chenney]

On Wed, Dec 4, 2013 at 8:02 PM, Nico Weber <tha...@chromium.org> wrote:

On Wed, Dec 4, 2013 at 5:00 PM, Stephen Chenney <sche...@chromium.org> wrote:

So what should gnpath be set to? Where should gn.exe be located? How do we check it is correctly synced?

The idea is that it Just Works and everything is set correctly. If that isn't the case, please file a bug!

In Peter's case, I'm guessing nobody thought of testing with cygwin.

So could it be rolled out while someone does think of cygwin?

 

I can't find anything at all about the chromium setup for this.

That's because no setup should be necessary in theory :-)

 

Yeah, I get that now.

[Peter Kasting]

On Wed, Dec 4, 2013 at 4:32 PM, Shashi Shekhar <shashi...@chromium.org> wrote:

Looks like you might be missing gn.exe or gnpath is incorrect.

"which gn" gives a valid python script in depot_tools, but running that gives "unknown platform", because sys.platform isn't returning "win32".

Because this isn't the error I get when running gclient runhooks, I suspect that the gn invocation inside there _is_ getting "win32" as the platform -- but then failing to invoke src/tools/gn/bin/win/gn.exe because that file doesn't exist (I only have a gn.exe.sha1).

I don't know why that file doesn't exist.  I don't know whom to file the relevant bug against.  I'm not sure what revision to roll out to fix things.  I tried to IM Brett hours ago but got no response.  I can't say I'm totally thrilled with this situation.

PK

[Michael Nordman]

The change that enabled this is here...

https://codereview.chromium.org/102243005/

... idk if we want to revert of if the few cygwin users want to workaround locally?

--

[Peter Kasting]

On Wed, Dec 4, 2013 at 5:56 PM, Michael Nordman <mich...@google.com> wrote:

The change that enabled this is here...

https://codereview.chromium.org/102243005/

... idk if we want to revert of if the few cygwin users want to workaround locally?

Considering I'm leaving the office soon, I wouldn't revert for my sake.

But it'd be awesome if Brett/Nico had a solution by tomorrow.  Failing that, I guess I could probably do a local revert of this.

PK

[Michael Nordman]

kochi on irc was having problem with it too, but sounds like something different than cygwin for him...

ERROR at the command-line "--gyp_vars":1:493: Expecting assignment or function call.

[Nico Weber]

kochi: Can you file a bug, please?

cygwin users: Can you file a bug too? Can you also try if the following patch helps?

diff --git a/build/gyp_chromium b/build/gyp_chromium

index 64e4ff5..565756f 100755

--- a/build/gyp_chromium

+++ b/build/gyp_chromium

@@ -142,7 +142,7 @@ def RunGN(supplemental_includes):

 

   # The binaries in platform-specific subdirectories in src/tools/gn/bin.

   gnpath = SRC_DIR + '/tools/gn/bin/'

-  if sys.platform == 'win32':

+  if sys.platform in ('win32', 'cygwin'):

     gnpath += 'win/gn.exe'

   elif sys.platform.startswith('linux'):

     # On Linux we have 32-bit and 64-bit versions.

Sorry for breaking folks – it's always surprising how many different build configurations there are :-/ All trybots were green, when landing this for the first time all bots on the main waterfall were green, and by now all bots that we know about are green. So if things are still broken for folks, we rely on you to report the problems you're seeing.

(I believe I fixed http://crbug.com/325989 that Shashi filed.)

[Takayoshi Kochi]

I had an error running build/gyp_chromium on Ubuntu:

________ running '/usr/bin/python src/build/gyp_chromium' in '/p/kochi/chrome'

ERROR at the command-line "--gyp_vars":1:493: Expecting assignment or function call.

rbe_components_path="<(DEPTH)/third_party/webrtc/modules/remote_bitrate_estimator/internal" remoting="0" component="shared_library" libjingle_peerconnection_additional_deps="['<(DEPTH)/third_party/webrtc/internal/internal.gyp:webrtc_internal']" use_aura="1" libjingle_source="source_internal" internal_pdf="1" libjingle_additional_deps="['<(DEPTH)/hideout/libjingle_shim_headers.gyp:libjingle_headers']" use_temporal_layers="1" libyuv_dir="<(DEPTH)/third_party/libyuv" conditions="[['OS=="win" or OS=="mac" or OS=="linux"', {'libpeer_target_type%': 'loadable_module'}, {'libpeer_target_type%': 'static_library'}], ['OS!="ios" and OS!="android"', {'libpeer_allocator_shim': 1}, {'libpeer_allocator_shim': 0}], ['OS=="win"', {'libjingle_peerconnection_additional_deps': ['<(DEPTH)/hideout/libpeerconnection_version.gyp:libpeer_version']}]]" disable_nacl="1" libvpx_source="<(DEPTH)/third_party/libvpx/source/libvpx-internal"

                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ^---------

Generating gyp files from GN...

Error: Command /usr/bin/python src/build/gyp_chromium returned non-zero exit status 1 in /p/kochi/chrome

Looks like this is a different issue from cygwin, and I worked around by commenting out

these 2 lines in build_gyp, aroun line 220:

  #if not RunGN(supplemental_includes):

  #  sys.exit(1)

Maybe this is happening for my environment only, but just FYI in case if you see the

same error.

I don't have time to look into this right now and live with this work around.

http://groups.google.com/a/chromium.org/group/chromium-dev

--
Takayoshi Kochi

[Nico Weber]

Thanks, I can reproduce this. I filed http://crbug.com/326024 for this and I'll try to fix.

[Brett Wilson]

I'll probably need to hardcode cygwin == win32 in
download_from_google_code's platform detection so you get the binary.
That should be easy to do. If you're waiting on this, you can comment
out the RunGN call in build/gyp_chromium to skip.

Brett

[Brett Wilson]

Look like Nico fixed the quote escaping but you were seeing (thanks!).

As an aside, this list of GYP settings is quite complex! Currently,
this won't affect anything since the current landing is basically a
dry run that doesn't generate any actual build files.

But to set expectations, as we convert the build, this logic and
configuration settings will need to be reimplemented (there should be
a good amount of warning). If people are interested in particular
use-cases aside from the basic build settings (component_mode,
chromeos, use_aura, etc.), please be sure to pay attention to
progress, volunteer to help, join the gn-dev mailing list, etc.
Everybody should assume that any configuration not tested on the
waterfall will break at some point simply since we won't know it
exists or how to test it.

Brett

On Wed, Dec 4, 2013 at 6:31 PM, Takayoshi Kochi (河内 隆仁)
<ko...@google.com> wrote:
> I had an error running build/gyp_chromium on Ubuntu:
> ________ running '/usr/bin/python src/build/gyp_chromium' in
> '/p/kochi/chrome'

> ERROR at the command-line "--gyp_vars":1:493: Expecting assignment or
> function call.

> --
> Takayoshi Kochi

[Brett Wilson]

I'll make sure this works by tomorrow.

For background, there are actually 3 places related to this:

First, when you do runhooks, there is a regexp (in the DEPS file) that
determines if you get that file at all. This is to prevent everybody
from having to download every binary type. This is why you aren't
getting the exe in the first place.

Second, there is the script in depot_tools you found. The entire point
of this is to have something on your path that finds the right
executable so you can run it on the command line. This doesn't affect
anything unless you manually type "gn".

Third, there is the code in gyp_chromium. This is what's actually run
when you sync. It does basically the same thing. If it's not printing
"Unknown platform for GN: cygwin" then I guess gyp is running in a
non-cygwin python context but then the problem in #1 will bite you.

Brett

[Brett Wilson]

I believe these patches will fix the cygwin issue:
https://codereview.chromium.org/106243002/
https://codereview.chromium.org/104763003/
Since it's late I'm not going to try to TBR them into the tree and
risk more problems. If anybody with this problem wants to check them,
verification would be great.

Brett

[Peter Kasting]

On Wed, Dec 4, 2013 at 10:16 PM, Brett Wilson <bre...@chromium.org> wrote:

I believe these patches will fix the cygwin issue:
https://codereview.chromium.org/106243002/
https://codereview.chromium.org/104763003/
Since it's late I'm not going to try to TBR them into the tree and
risk more problems. If anybody with this problem wants to check them,
verification would be great.

Thanks for the fast work Brett!  Can't check these out tonight, but if you haven't landed them by the time I get in tomorrow, I'll take a look.

PK 

[Scott Violet]

For some reason the execute bit wasn't set properly on gn.exe for me.
Not sure if it's just a cygwin thing. Changing that wasn't enough
though. Now I get:

ERROR at //build/toolchain/win/BUILD.gn:24:1: Could not execute python.
exec_script("setup_toolchain.py", [ gyp_win_tool_source ], "value")
^----------
I was trying to execute "C:\depot_tools\python_bin\python.exe".

Generating gyp files from GN...
Error: Command /usr/bin/python src/build/gyp_chromium returned

non-zero exit status 1 in /cygdrive/d/src/build2

I believe this is because for cygwin users depot_tools doesn't install
python, instead you use the python from cygwin.

-Scott

[Scott Violet]

This is likely because of Setup::FillPythonPath() in
tools/gn/setup.cc. The OS_WIN section should use python.exe if in
cygwin. I'm not sure how to build gn and verify this though.

-Scott

[Michael Moss]

On Dec 5, 2013 8:57 AM, "Michael Moss" <mm...@google.com> wrote:
>
> I'm not sure about the cygwin issue, buy the python_bin path in the error message is also a problem since that doesn't exist on machines upgraded to 2.7. I think it should be running depot_tools\python.bat instead.

[Brett Wilson]

This was reverted last night. i'll work on the Python path problems on Windows.

Brett

On Thu, Dec 5, 2013 at 8:57 AM, Michael Moss <mm...@google.com> wrote:
> I'm not sure about the cygwin issue, buy the python_bin path in the error
> message is also a problem since that doesn't exist on machines upgraded to
> 2.7. I think it should be running depot_tools\python.bat instead.
>

[Thiago Farina]

On Thu, Dec 5, 2013 at 2:36 PM, Scott Violet <s...@chromium.org> wrote:
> This is likely because of Setup::FillPythonPath() in
> tools/gn/setup.cc. The OS_WIN section should use python.exe if in
> cygwin. I'm not sure how to build gn and verify this though.
>

Scott,

To build it is just like build any other C++ target in chromium.

If you use ninja on Windows you can do:

ninja -C out\Debug gn

and

ninja -C out\Debug gn_unittests

Hope that helps,

Regards,

--
Thiago

[Peter Kasting]

On Thu, Dec 5, 2013 at 8:00 AM, Scott Violet <s...@chromium.org> wrote:

For some reason the execute bit wasn't set properly on gn.exe for me.

Usually that means someone needs to "svn propset svn:executable=* gn.exe".

PK

[Brett Wilson]

The executable is pulled from Google Code (since we don't want to
check binaries into git).

I talked with Scott this morning and I don't know how cygwin deals
with the executable bits. Won't it treat anything with a .exe
extension as executable? I never had problems running random binaries
on cygwin, and I don't know how it could be otherwise.

Does anybody understand how cygwin deals with this stuff?

Brett

[Jiang Jiang]

I found an interesting error on Mac, when I try to invoke gn directly
from ~/chromium/src:

tools/gn/bin/mac/gn gyp --root=/Users/jjgod/chromium/src --args=is_gyp=true

or

tools/gn/bin/mac/gn gyp --args=is_gyp=true

both works:

Done. Wrote 0 targets to 0 GYP files read from 40 GN files in 35ms

But:

tools/gn/bin/mac/gn gyp --root=~/chromium/src --args=is_gyp=true

didn't:

ERROR at //build/toolchain/mac/BUILD.gn:16:1: Script returned non-zero
exit code.
exec_script("setup_toolchain.py", [ gyp_mac_tool_source ], "value")
^----------
Current dir: out/Default/gn_release.tmp/
Command: python build/toolchain/mac/setup_toolchain.py
../../../tools/gyp/pylib/gyp/mac_tool.py
Returned 2.

For some reason the parent path for setup_toolchain.py is not
correctly resolved in this case.

> --
> --
> Chromium Developers mailing list: chromi...@chromium.org
> View archives, change email options, or unsubscribe:
> http://groups.google.com/a/chromium.org/group/chromium-dev
>

> To unsubscribe from this group and stop receiving emails from it, send an email to chromium-dev...@chromium.org.

[Jiang Jiang]

Also, since the output said "Wrote 0 targets to 0 GYP files read from
40 GN files in 35ms" apparently no gyp file is generated, why is that?

[Nico Weber]

On Thu, Dec 5, 2013 at 1:41 PM, Jiang Jiang <jia...@opera.com> wrote:

Also, since the output said "Wrote 0 targets to 0 GYP files read from
40 GN files in 35ms" apparently no gyp file is generated, why is that?

GN isn't used for anything yet, see the first post on this thread. Over time, stuff will gradually move from gyp to gn hopefully.

 

On Thu, Dec 5, 2013 at 10:39 PM, Jiang Jiang <jia...@opera.com> wrote:
> I found an interesting error on Mac, when I try to invoke gn directly
> from ~/chromium/src:
>
> tools/gn/bin/mac/gn gyp --root=/Users/jjgod/chromium/src --args=is_gyp=true
>
> or
>
> tools/gn/bin/mac/gn gyp --args=is_gyp=true
>
> both works:
>
> Done. Wrote 0 targets to 0 GYP files read from 40 GN files in 35ms
>
> But:
>
> tools/gn/bin/mac/gn gyp --root=~/chromium/src --args=is_gyp=true
>
> didn't:

We don't support ~ in command-line args. Passing --user-data-dir=~/foo to chromium doesn't work either. (Background: http://crbug.com/17922). So this is working as intended, I think.

[Jiang Jiang]

On Thu, Dec 5, 2013 at 10:44 PM, Nico Weber <tha...@chromium.org> wrote:
> On Thu, Dec 5, 2013 at 1:41 PM, Jiang Jiang <jia...@opera.com> wrote:
>>
>> Also, since the output said "Wrote 0 targets to 0 GYP files read from
>> 40 GN files in 35ms" apparently no gyp file is generated, why is that?
>
>
> GN isn't used for anything yet, see the first post on this thread. Over
> time, stuff will gradually move from gyp to gn hopefully.

Ah, I misunderstood the first post, thought that “third_party/re2” is
already generating gyp.

- Jiang

[Peter Kasting]

On Thu, Dec 5, 2013 at 12:11 PM, Brett Wilson <bre...@chromium.org> wrote:

On Thu, Dec 5, 2013 at 11:18 AM, Peter Kasting <pkas...@chromium.org> wrote:
> On Thu, Dec 5, 2013 at 8:00 AM, Scott Violet <s...@chromium.org> wrote:
>>
>> For some reason the execute bit wasn't set properly on gn.exe for me.
>
>
> Usually that means someone needs to "svn propset svn:executable=* gn.exe".

The executable is pulled from Google Code (since we don't want to
check binaries into git).

Pulled via svn or git?  Or some other pulling?

If the original file is in svn or git, I believe you can set the correct properties on it as a commit to the original repo.  If you're pulling the file via direct download or something, you can have a post-download step that chmods the file.

I talked with Scott this morning and I don't know how cygwin deals
with the executable bits. Won't it treat anything with a .exe
extension as executable?

No.  It uses the executable bit on the file (like a POSIX system) and not the file extension.  Normally everything Cygwin has no other information about gets permissions based on your UMASK, so pretty much all files get marked as executable, but if you add files or change permissions from inside Cygwin it's possible for them to have the wrong permissions.  For this reason, we try to mark all executable files in the Chromium repo with the svn:executable prop (because that will ensure the right bit gets set upon checkout).

PK

[Brett Wilson]

On Thu, Dec 5, 2013 at 1:48 PM, Peter Kasting <pkas...@chromium.org> wrote:
> On Thu, Dec 5, 2013 at 12:11 PM, Brett Wilson <bre...@chromium.org> wrote:
>>
>> On Thu, Dec 5, 2013 at 11:18 AM, Peter Kasting <pkas...@chromium.org>
>> wrote:
>> > On Thu, Dec 5, 2013 at 8:00 AM, Scott Violet <s...@chromium.org> wrote:
>> >>
>> >> For some reason the execute bit wasn't set properly on gn.exe for me.
>> >
>> >
>> > Usually that means someone needs to "svn propset svn:executable=*
>> > gn.exe".
>>
>> The executable is pulled from Google Code (since we don't want to
>> check binaries into git).
>
>
> Pulled via svn or git? Or some other pulling?

Sorry, I meant Google Storage.

> If the original file is in svn or git, I believe you can set the correct
> properties on it as a commit to the original repo. If you're pulling the
> file via direct download or something, you can have a post-download step
> that chmods the file.
>
>> I talked with Scott this morning and I don't know how cygwin deals
>> with the executable bits. Won't it treat anything with a .exe
>> extension as executable?
>
>
> No. It uses the executable bit on the file (like a POSIX system) and not
> the file extension. Normally everything Cygwin has no other information
> about gets permissions based on your UMASK, so pretty much all files get
> marked as executable, but if you add files or change permissions from inside
> Cygwin it's possible for them to have the wrong permissions. For this
> reason, we try to mark all executable files in the Chromium repo with the
> svn:executable prop (because that will ensure the right bit gets set upon
> checkout).

I'll mark all downloaded files as executable under cygwin in the
Google Storage downloader.

Brett

[Brett Wilson]

I believe I fixed all the Cygwin detection and permissions issues and
it verified it works on sky's setup. There is also a new gn binary for
Windows that everybody should be getting automatically that finds
Python on your path rather than in depot tools, which should fix the
official builder issue (and also some Cygwin setups).

If you use Cygwin and you still have issues, make sure your depot
tools is up-to-date. The most common reason for it not being
up-to-date is if you made a manual change to the code for some reason.

I've also seen some people that explicitly avoid hooks when they sync
for whatever reason, which will cause problems for you (since you
won't get the new binary). Do the regular gclient sync or you'll keep
getting problems into the future.

I'm going to try to reland the patch in a little while, please ping me
if you see any weirdness.

Brett

[Brett Wilson]

This has been on trunk for an hour or two and looks good so far.

Brett