Supporting multiple commands on Windows easily

[jvp...@g.rit.edu]

As most of the people on this list probably know already, ninja uses CreateProcess on Windows, which means that you can only run one command at a time without spawning a new subshell via "cmd /c". In addition to being inconvenient, this puts a pretty low upper limit on the length of your command(s); it's my understanding that this last issue is why ninja uses CreateProcess now. (However, ninja now supports response files, so that concern may no longer be valid.) This is also apparently one of the reasons why ninja required a special flag for supporting MSVC-style deps: <https://github.com/ninja-build/ninja/pull/721#issuecomment-41292333>.

If CreateProcess is here to stay, one option would be for ninja to split the command line itself, e.g. by calling CommandLineToArgV, finding instances of "&" and "&&", splitting on them, and passing each sub-array of arguments to its own CreateProcess call. This is relatively straightforward (you wouldn't have to parse Windows command lines yourself) and still restores the most important feature that "cmd /c" had: the ability to run multiple commands. There might be some special cases to contend with (I think cmd /c changes how the rest of the line is parsed), but those shouldn't be too hard to address. Of course, you'd still need the cmd /c prefix for shell builtins and variables, but that would be a much less common occurrence.

Does the above sound sane? My main goal is to simplify working with ninja on Windows, especially for cases where it's inconvenient for the meta-build system to track whether "cmd /c" will be necessary. For instance, suppose you have the following:

rule link
  command = gcc $in -o $out $post_build

build foo: link foo.c
build bar: link bar.c
  post_build = && do_something

Only one of these actually requires wrapping with "cmd /c". Currently, the only solution is to always wrap in these cases, but that puts you into the same situation that precipitated removing the "cmd /c" wrapper from Ninja in the first place!

- Jim

[Scott Graham]

On Thu, Feb 4, 2016 at 4:07 PM, <jvp...@g.rit.edu> wrote:

As most of the people on this list probably know already, ninja uses CreateProcess on Windows, which means that you can only run one command at a time without spawning a new subshell via "cmd /c". In addition to being inconvenient, this puts a pretty low upper limit on the length of your command(s); it's my understanding that this last issue is why ninja uses CreateProcess now. (However, ninja now supports response files, so that concern may no longer be valid.) This is also apparently one of the reasons why ninja required a special flag for supporting MSVC-style deps: <https://github.com/ninja-build/ninja/pull/721#issuecomment-41292333>.

If CreateProcess is here to stay, one option would be for ninja to split the command line itself, e.g. by calling CommandLineToArgV, finding instances of "&" and "&&", splitting on them, and passing each sub-array of arguments to its own CreateProcess call. This is relatively straightforward (you wouldn't have to parse Windows command lines yourself)

I don't think this is a good idea. You do need to fully parse to split properly, and you can't do that correctly unless you know how the target process parses its command line (CommandLineToArgvW is one way, but not mandatory).

 

and still restores the most important feature that "cmd /c" had: the ability to run multiple commands. There might be some special cases to contend with (I think cmd /c changes how the rest of the line is parsed), but those shouldn't be too hard to address. Of course, you'd still need the cmd /c prefix for shell builtins and variables, but that would be a much less common occurrence.

Maybe proposing some syntax where there's a list of commands (each provided separately) would be simpler and less error-prone.

 

Does the above sound sane? My main goal is to simplify working with ninja on Windows, especially for cases where it's inconvenient for the meta-build system to track whether "cmd /c" will be necessary.

I think this is the best option though. Ninja prefers that the meta-build system do the work, when that's possible.

 

For instance, suppose you have the following:

rule link
  command = gcc $in -o $out $post_build

build foo: link foo.c
build bar: link bar.c
  post_build = && do_something

Only one of these actually requires wrapping with "cmd /c". Currently, the only solution is to always wrap in these cases, but that puts you into the same situation that precipitated removing the "cmd /c" wrapper from Ninja in the first place!

- Jim

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[jvp...@g.rit.edu]

On Thursday, February 4, 2016 at 6:35:42 PM UTC-6, Scott Graham wrote:

Maybe proposing some syntax where there's a list of commands (each provided separately) would be simpler and less error-prone.

That would work too (and indeed, is how Make recipes are constructed). One possible syntax would be to allow inserting newlines into "command" (possibly via variables, too!), and then just split on newlines. As far as I can tell, there's no actual way for a variable to contain a newline today, so this wouldn't break compatibility, except by adding a new escape sequence to deal with.

 

Does the above sound sane? My main goal is to simplify working with ninja on Windows, especially for cases where it's inconvenient for the meta-build system to track whether "cmd /c" will be necessary.

I think this is the best option though. Ninja prefers that the meta-build system do the work, when that's possible.

The problem with offloading this complexity onto the meta-build system is that this is something that almost all cross-platform meta-build systems have to contend with. There's obviously some criteria by which "things that the meta-build system can do" ends up in ninja, or "deps=msvc" and `ninja -t msvc` wouldn't exist. A sensible way to specify multiple commands for a given rule would make it easier to drop "deps=msvc" in place of a tool that just reformats MSVC's /showIncludes output (although that would probably also require the ability to pipe stderr to the reformatter).

- Jim