Visual Studio generators for GN are available now

[Tomasz Moniuszko]

Hi,

It was announced on gn-dev list some time ago. Several issues has been fixed since then and some new options added. I think it's time to announce it again, on chromium-dev this time.

With https://crbug.com/305761 it's now possible to use GN to generate Visual Studio project and solution files. 'gn gen' command needs to be run with '--ide=vs' option in order to have VS files generated. Projects are configured for VS 2015 by default but VS 2013 can be forced with '--ide=vs2013' option.

Since 'gen' command prepares arguments for single configuration only (for instance "Win32|Debug"), only that single configuration is written (unlike from GYP). By default projects are generated for all GN binary targets. 'all.sln' solution file is written to root output directory and project files are written to target out directories. It's also possible to generate projects only for selected targets/directories ('--filters' option) or write .sln file under different name ('--sln' option). See 'gn help gen' for more details.

Tomasz

[Peter Kasting]

Dumb question: is there an easy way to make gclient runhooks automatically use GN instead of GYP?

PK

[Sunny Sachanandani]

You can disable automatic running of gyp by using the GYP_CHROMIUM_NO_ACTION env var but that doesn't mean gn will run. Maybe add a custom hook for running gn?

On Fri, Mar 11, 2016, 1:52 AM Peter Kasting <pkas...@chromium.org> wrote:

Dumb question: is there an easy way to make gclient runhooks automatically use GN instead of GYP?

PK

--
--
Chromium Developers mailing list: chromi...@chromium.org
View archives, change email options, or unsubscribe:
http://groups.google.com/a/chromium.org/group/chromium-dev

[Colin Blundell]

Note that at least when you're invoking ninja explicitly at the command line, GN detects when it needs to regenerate the build files and does so before ninja runs, so you don't actually need to run GN explicitly after doing a gclient sync or making a change to a BUILD.gn file. I'm not sure how that interacts with the VS-related stuff being described here.

[Tomasz Moniuszko]

VS projects are regenerated as well when you run 'ninja'. It works for ninja invocation from shell and also when Build command is used from VS. Running 'gn gen' manually again stops this behavior (all IDE-related options need to be specified again if VS projects should be generated).

[Peter Kasting]

On Fri, Mar 11, 2016 at 3:02 AM, Tomasz Moniuszko <tmoni...@opera.com> wrote:

VS projects are regenerated as well when you run 'ninja'. It works for ninja invocation from shell and also when Build command is used from VS.

Wait, so if you try to build while in Visual Studio it might regenerate your build files, which will cause MSVC to prompt you endlessly about reloading the buildfiles?  That seems unhappy.

It would be much better if I could just always regen during the runhooks phase like GYP does today.  There's a reason to generate build files then.

PK 

[Colin Blundell]

It regenerates the buildfiles precisely when they will have changed (e.g., because a BUILD.gn file changed). To me this seems optimal, although I'm not at all familiar with VS so I don't know anything about the MSVC prompting behavior that you're describing.

[Tomasz Moniuszko]

Ninja runs 'gn gen' only if there's a need to regenerate. Also GN doesn't rewrite Visual Studio project/solution files unless the new contents of the file is different. For instance project file changes when you add/remove source files which is expected behavior. When you just modify existing source files and start building/debugging, projects are unchanged and VS doesn't complain about reloading.

[Peter Kasting]

On Fri, Mar 11, 2016 at 3:47 AM, Tomasz Moniuszko <tmoni...@opera.com> wrote:

Ninja runs 'gn gen' only if there's a need to regenerate. Also GN doesn't rewrite Visual Studio project/solution files unless the new contents of the file is different. For instance project file changes when you add/remove source files which is expected behavior.

Yes, and that's precisely what I'm saying is a problem.  My workflow is:

git pull

gclient sync

devenv build/all.sln &

<Build from inside Visual Studio>

This is now broken since basically every time I sync, there will be project file changes needed, but these won't have been made by the time I load Visual Studio.  And you really, REALLY do not want to change these when Visual Studio is already loaded.  It's a good way to have your computer locked up for ten minutes.

I suppose I can write some kind of script that wraps gclient and auto-calls "ninja" (with no args; hopefully that still works) after doing so every time.  But this is less convenient and more error-prone than just having an explicit gc update stage during runhooks, instead of having ninja do it on demand.

Surely it can't be _that_ hard to make gclient/gn/ninja optionally do things this way, can it?  I'd think implementation-wise such a dumb update step would be much simpler than one that tries to intelligently happen on demand.

PK

[Torne (Richard Coles)]

The problem is that gn is designed to work with an arbitrary number of arbitrarily named output directories, rather than a fixed number of default-named ones, and so "just run gn" in gclient sync is tricky - which directories do you run it for?

--

[Peter Kasting]

On Fri, Mar 11, 2016 at 4:48 AM, Torne (Richard Coles) <to...@chromium.org> wrote:

The problem is that gn is designed to work with an arbitrary number of arbitrarily named output directories, rather than a fixed number of default-named ones, and so "just run gn" in gclient sync is tricky - which directories do you run it for?

Introduce an environment variable like GN_RUNHOOKS_DIRS or something.  If the variable is unset, then we do nothing at all during runhooks (the default behavior).  Or it can be set to a comma-delimited list of dirs or configs or something similar, in which case GN gets run for those during runhooks time.

PK

[Daniel Bratell]

On Fri, 11 Mar 2016 13:10:42 +0100, Peter Kasting <pkas...@chromium.org> wrote:

On Fri, Mar 11, 2016 at 3:47 AM, Tomasz Moniuszko <tmoni...@opera.com> wrote:

Ninja runs 'gn gen' only if there's a need to regenerate. Also GN doesn't rewrite Visual Studio project/solution files unless the new contents of the file is different. For instance project file changes when you add/remove source files which is expected behavior.

Yes, and that's precisely what I'm saying is a problem.  My workflow is:

git pull

gclient sync

devenv build/all.sln &

<Build from inside Visual Studio>

This is now broken since basically every time I sync, there will be project file changes needed, but these won't have been made by the time I load Visual Studio.  And you really, REALLY do not want to change these when Visual Studio is already loaded.  It's a good way to have your computer locked up for ten minutes.

It is not nearly as bad as it used to be. Assuming all project files have changed a "yes to all" takes a little but I've not noticed my computer freeze up as it used to do in old versions of Visual Studio. And "no/ignore all" (which I often use) means that the in-memory version of the projects might be a tiny bit out of sync but normally everything works fine anyway.

Unless your devenv is set to reload without asking.

/Daniel

--

/* Opera Software, Linköping, Sweden: CET (UTC+1) */

[Tomasz Moniuszko]

There are more 'gn gen' options that needs to be specified to make projects generation work. '--ide' is just the first one.

We should keep in mind that developer can use '--filters' option. It's really useful because by default all GN targets are converted to VS projects. 2500+ projects make VS low-responding. Ninja respect recently selected filters but if 'gn gen' will be invoked from gclient it all depends on options. Calling 'gn gen --ide=vs out/Debug' will just overwrite VS solution with default one, with all projects included. On the other hand, 'gn gen out/Debug' won't touch VS projects but leave them out of date (also further ninja invocations won't touch VS projects).

It seems that gclient will need to accept additional environment variable for each 'gn gen' option. Or maybe just one GN_RUNHOOKS_OPTIONS variable for all options. For instance GN_RUNHOOKS="--ide=vs --filters="//chrome/*;//base/*" --sln=chrome" (I'm not sure how if inner "" won't break things).

Tomasz

[Dirk Pranke]

I think you've raised an interesting question.

As others have mentioned, by design ninja automatically re-runs GN as needed, and this eliminates a large class of errors people would hit when they would modify a .gyp/.gypi file and then forget to re-run gyp_chromium. We're not going to get rid of this, at least not for the default non-IDE case.

In addition, you have the N different build directories with N sets of options problem that people of mentioned.

As you say, adding a dumb update hook is easy. Even adding a smart update hook is straightforward; I actually had a design worked out in the early days of the GN migration for dealing with this, but Brett convinced me that it was a lot of complexity that almost no one needed, and I think he was right.

However, that was before we had IDE integration, and I don't think either of us thought of the issue you're bringing up. I think both of us, if we even use IDEs at all, likely do most of our builds from the command line and would use the IDE more for debugging and incremental development rather than the "build all" step after a sync, so we'd be less likely to hit the issue you're hitting.

I think, at least for the moment, we're better off doing nothing in runhooks and you should use a wrapper script to play around with things and see how they work. And, we can get more experience with how many people end up using the IDE integration and what sorts of issues they hit. If we see enough issues that we think it makes sense to solve something generically.

How does that sound? We should also file a bug and track further discussion there ...

-- Dirk

--

[Peter Kasting]

On Fri, Mar 11, 2016 at 9:37 AM, Dirk Pranke <dpr...@chromium.org> wrote:

I think, at least for the moment, we're better off doing nothing in runhooks and you should use a wrapper script to play around with things and see how they work. And, we can get more experience with how many people end up using the IDE integration and what sorts of issues they hit. If we see enough issues that we think it makes sense to solve something generically.

How does that sound? We should also file a bug and track further discussion there ...

I definitely don't think you should add a smart/complicated hook.

I do think a dumb hook should be added.  Reading a single environment variable that passes all necessary options to gn would be sufficient.  Wrapper scripts are just enough extra work to be problematic; I've tried to solve various other problems (e.g. good git<->devenv text editor integration) using wrapper scripts and it makes for major hassles every time I set up a new machine, which happens surprisingly frequently.  It's a lot easier to document a single environment variable string on dev.chromium.org for those of us who really, really do not like to do things from CLIs.

I use "build all from IDE after sync" every other day or so.  I never have more than one build directory.  It seems like GN as currently built has been designed for people who work very differently than me, which is fine, but a tiny bit of assistance would be much appreciated :)

PK

[Jeffrey Yasskin]

Are these options stored in the args.gn file? If not, why not? It doesn't seem like you'd want them to reset for a given output directory, any more than the args.gn options.

--

[Dirk Pranke]

I'm happy to provide assistance, but I don't particularly want to provide assistance for only one tiny inflexible situation :). I also would guess that most people have more than one build directory (at the very least, debug + release). Lastly, in my experience, env vars are the worst of all possible ways to configure the build and checkout process, so I'm going to be pretty reluctant to pick only that solution.

-- Dirk

[Dirk Pranke]

They are not currently stored in the args.gn file.

args.gn contains a list of build arguments (the settings for the flags in the build itself), not the list of command line args passed to GN (admittedly, the naming is a bit confusing here).

I don't know what you mean by "reset for output directory", but args.gn is stored in (and specific to) the output directory. You would need the ide options to be specific to a given output directory as well, because the lists of targets and files in the project is determined by the contents of args.gn in addition to the --filters options.

The command line args passed to GN are written into the build.ninja file, so that ninja can re-invoke GN correctly, though.

-- Dirk

[Peter Kasting]

On Fri, Mar 11, 2016 at 11:20 AM, Dirk Pranke <dpr...@chromium.org> wrote:

I'm happy to provide assistance, but I don't particularly want to provide assistance for only one tiny inflexible situation :). I also would guess that most people have more than one build directory (at the very least, debug + release). Lastly, in my experience, env vars are the worst of all possible ways to configure the build and checkout process, so I'm going to be pretty reluctant to pick only that solution.

I am open to any other solution that is not wrapper scripts.  I am strongly opposed to those.

Some kind of ~/.gn/include.gn kind of a file, for example, would also be totally fine.  Pick whatever syntax you like in that file and make it work for people with multiple build directories if need be.  (I don't understand almost anything about GN or ninja, certainly not enough to have the foggiest idea of how one-versus-several build directories impacts options or syntax here.)

At least adding something simple, though, would allow more people than just me to try and use it, which means you'd have a chance of actually getting useful feedback from multiple perspectives.  I don't think you're as likely to get any feedback if there's nothing provided at all.

PK 

[Brett Wilson]

On Fri, Mar 11, 2016 at 11:14 AM, Peter Kasting <pkas...@chromium.org> wrote:

On Fri, Mar 11, 2016 at 9:37 AM, Dirk Pranke <dpr...@chromium.org> wrote:

I think, at least for the moment, we're better off doing nothing in runhooks and you should use a wrapper script to play around with things and see how they work. And, we can get more experience with how many people end up using the IDE integration and what sorts of issues they hit. If we see enough issues that we think it makes sense to solve something generically.

How does that sound? We should also file a bug and track further discussion there ...

I definitely don't think you should add a smart/complicated hook.

I do think a dumb hook should be added.  Reading a single environment variable that passes all necessary options to gn would be sufficient.  Wrapper scripts are just enough extra work to be problematic; I've tried to solve various other problems (e.g. good git<->devenv text editor integration) using wrapper scripts and it makes for major hassles every time I set up a new machine, which happens surprisingly frequently.  It's a lot easier to document a single environment variable string on dev.chromium.org for those of us who really, really do not like to do things from CLIs.

You have a set of three actions you run on a regular basis. You want it to do something else also. Your proposed solution is to add a global hook to DEPS that will run for all Windows users (there's no more granular way to specify hooks) that will execute a script that somebody writes and checks in that looks for a global environment variable you set on your computer that it interprets as a sequence of directories and command line flags such that when you run step 2 in your sequence it automatically run some additional set of commands.

I would suggest that even if this problem was more severe, the solution would not be to add more obscure and unmaintained variables to DEPS but to just write a script to do the sequence of steps you want. This will even be better for yourself selfishly, since the script will be much easier to edit than some environment variable.

Brett

[Jeffrey Yasskin]

I expect `gn gen <dir>` to do a subset of running `ninja -C <dir> target`. If it does, then all's well. If it additionally clears the gn command line flags, why's that the right thing to do?

[Dirk Pranke]

`gn gen <dir>`, when run against an *existing* build directory, re-generates the ninja files using the arguments (if any) currently stored in <dir>/args.gn . It does not re-generate any previously-generated ide files. Re-running ninja will do both, because it captured the entire prior command line. 

It's probably a bug (or omission, or enhancement, or whatever) that these two things are inconsistent, and we should probably persist the --ide/--filter flags somewhere besides just the build.ninja file, or at least provide some other way to recreate things.

-- Dirk

[Jeffrey Yasskin]

+1. That's what I was suggesting. Thanks!

[Peter Kasting]

On Fri, Mar 11, 2016 at 11:28 AM, Brett Wilson <bre...@chromium.org> wrote:

You have a set of three actions you run on a regular basis. You want it to do something else also. Your proposed solution is to add a global hook to DEPS that will run for all Windows users (there's no more granular way to specify hooks) that will execute a script that somebody writes and checks in that looks for a global environment variable you set on your computer that it interprets as a sequence of directories and command line flags such that when you run step 2 in your sequence it automatically run some additional set of commands.

I would suggest that even if this problem was more severe, the solution would not be to add more obscure and unmaintained variables to DEPS but to just write a script to do the sequence of steps you want. This will even be better for yourself selfishly, since the script will be much easier to edit than some environment variable.

Scripts are so much not easier for me to edit than environment variables.  I literally know zero shell scripting.  Or python, or perl, or any other suitable substitute.  I can make this happen, it's just stunningly slow.

Really where I'm coming from is that it used to be true that "gclient sync" was a one-step command that would ensure your checkout was completely ready for use, whether you use a CLI or IDE to do your work.  Conceptually that made a lot of sense.  I'm rather surprised there's so much resistance in principle to the idea of putting in the most bare-bones possible hooks to allow developers to make this still be true if they use an IDE to build.

Yes, every individual developer who does this can instead write scripts, modify their workflow, etc.  And if the overhead of a dumb hook was really so drastic, that tradeoff would probably make more sense to me.  But I feel like your description above goes out of its way to make this sound as difficult and obfuscated as possible, and portray this request as unreasonable.

I mean, maybe I'm wrong and I really am being unreasonable.  I sure as hell am clueless enough about all these tools to set myself up to ask for stupid things.  But based on Dirk's and Tomasz' previous answers I had assumed that this idea was not too complicated.

PK

[Scott Graham]

Peter, you're going to have to recreate the envvar on every machine containing a list of directories to re-gn, so it doesn't seem better.

Running

  ninja -C out\Debug build.ninja

after `gclient sync` will regenerate the .sln before you open VS, which seems not too onerous. Alternatively, do nothing different than you do now, and click "Yes To All" after the first build in VS to reload. That'll be a little slower, but hey, still way faster than waiting for gyp.

Personally, I just replaced "gclient sync" with "gclient sync && ..\dbg", where dbg is something like this batch file and you can stuff your ide args in there. I don't think this can really be construed as "scripting".

d:\src\cr3>type dbg.bat

gn gen //out/Debug "--args=is_debug=true is_component_build=true" --check

(Thanks Tomasz!)

--

[Brett Wilson]

On Fri, Mar 11, 2016 at 11:51 AM, Peter Kasting <pkas...@chromium.org> wrote:

On Fri, Mar 11, 2016 at 11:28 AM, Brett Wilson <bre...@chromium.org> wrote:

You have a set of three actions you run on a regular basis. You want it to do something else also. Your proposed solution is to add a global hook to DEPS that will run for all Windows users (there's no more granular way to specify hooks) that will execute a script that somebody writes and checks in that looks for a global environment variable you set on your computer that it interprets as a sequence of directories and command line flags such that when you run step 2 in your sequence it automatically run some additional set of commands.

I would suggest that even if this problem was more severe, the solution would not be to add more obscure and unmaintained variables to DEPS but to just write a script to do the sequence of steps you want. This will even be better for yourself selfishly, since the script will be much easier to edit than some environment variable.

Scripts are so much not easier for me to edit than environment variables.  I literally know zero shell scripting.  Or python, or perl, or any other suitable substitute.  I can make this happen, it's just stunningly slow.

Huh? You can't write a script to automate your update steps because you don't know Perl or Python? Try copying this into "sync.bat":

  git pull

  gclient sync

  gn gen out/mybuild --ide=vs  [edit -- scottmg's version is better]

  devenv out/mybuild/all.sln

 Really where I'm coming from is that it used to be true that "gclient sync" was a one-step command that would ensure your checkout was completely ready for use, whether you use a CLI or IDE to do your work.  Conceptually that made a lot of sense.  I'm rather surprised there's so much resistance in principle to the idea of putting in the most bare-bones possible hooks to allow developers to make this still be true if they use an IDE to build.

Yes, every individual developer who does this can instead write scripts, modify their workflow, etc.  And if the overhead of a dumb hook was really so drastic, that tradeoff would probably make more sense to me.  But I feel like your description above goes out of its way to make this sound as difficult and obfuscated as possible, and portray this request as unreasonable.

Are the steps I outlined incorrect? In my opinion any solution here will be weirdly obscure for little benefit, and I tried to emphasize the steps involved  in your proposal to make this more obvious.

Your suggestion requires you to set up a necessarily complicated environment variable to specify directories and args on a new computer, which is honestly more difficult than copying the 4 lines of script I gave you (and if you don't want to do anything on a new computer, you don't *have* to, the worst thing that will happen is that you press "Reload" in a dialog box).

We created a huge mess of build because we kept adding one off features for specific use-cases. GYP has 5 different was of specifying GYP_DEFINES, with various ways of interaction and brokenness. Adding one-off features for a minority of people hurts the project in the same way that adding options in Chrome hurts the product (and the build system is very much a "product").

Dirk very reasonably suggested waiting for more IDE users and that you should use a wrapper script to play around with things and see how they work. You did not seem open to playing around with things to see how they work and continued to argue that you were strongly opposed to wrapper scripts. Several people have been using this for a while and not found it to be a serious problem. When more people use this for longer, we can revisit if it's really a horrible problem.

Brett

[Peter Kasting]

Sorry.  I am trying to be helpful rather than get in the way, but I think I am failing.  Somehow I got from "it would be nice if" to something that sounds much more obstinate.  This is certainly not a big enough issue to make you guys add something you really think is unwise or a burden.

I will try to do this manually.  Sorry for the noise.

PK