Environment variables support in rules

[Daniel Levin]

Currently ninja lacks of environment variables support in rules.

Some tool accept parameters via command line arguments, some via environment variables, some via both.

Expected build.ninja rule format something like:

build myfile: CUSTOM_COMMAND

  COMMAND = mytool -o myfile

  ENVIRONMENT = MYTOOL_SOME_VALUE=foo,MYTOOL_SOME_OTHER_VALUE=bar

Per process environment variables supported on all platforms, so there should be no portability issues.

Specified environment variables should also be hashed into .ninja_log, because semantic is exactly the same as for COMMAND variable.

[Nico Weber]

You can achieve this by saying

rule r

  command = ENV1=asdf ENV2=sdfs mytool -o $out $in

build myfile: r myinput

on posix. On Windows, you can use `ninja -t msvc -e myenv.file` with your desired env vars stored in myenv.file.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Levin]

Nico, thanks for pointing on workarounds.

1. POSIX only, would be good to have portable solution.

2. On Windows that will not work. The trick is we want per process specific environment. There are tools which accept some parameters ONLY via environment variables.

If you want an example look at ninja itself: progress format has been moved to environment variable NINJA_STATUS instead of command line argument. And I am encountering much more build tools with similar behavior.

[Nico Weber]

On Sun, Aug 9, 2015 at 1:39 PM, Daniel Levin <dend...@gmail.com> wrote:

Nico, thanks for pointing on workarounds.

1. POSIX only, would be good to have portable solution.

2. On Windows that will not work. The trick is we want per process specific environment. There are tools which accept some parameters ONLY via environment variables.

Understood. ninja -t msvc -e env.file spawns a subprocess with the environment loaded from env.file.

[Daniel Levin]

Could you please give an example of env.file? `ninja -t list` does not show msvc in tool list. Do I understand correctly, environment will be applied to ALL subprocesses equally, right?

[Daniel Levin]

Okay, msvc tool is visible only on Windows after updating to 1.6. But I still can't understand how to get help for specific tool usage. And what should be format of env.file.

[Evan Martin]

Earlier discussion:
https://groups.google.com/forum/#!topic/ninja-build/J8UIXu7S5pQ
We probably oughta put this in the Ninja docs somewhere.

On Sun, Aug 9, 2015 at 11:21 PM, Daniel Levin <dend...@gmail.com> wrote:
> Okay, msvc tool is visible only on Windows after updating to 1.6. But I
> still can't understand how to get help for specific tool usage.

ninja -t msvc -h

> And what
> should be format of env.file.

The entries are separated by \0:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682653(v=vs.85).aspx

I think you can also use "cmd" as your command executor to set variables:
http://superuser.com/questions/223104/setting-environment-variable-for-just-one-command-in-windows-cmd-exe
(This is more or less what Ninja does on non-Windows, as all commands
are executed within a shell.)

Also, to your original request -- I seem to recall we used to support
environment variables more directly in some way, but then maybe
removed that? Scott probably recalls the reasoning better than I do.

[Nico Weber]

On Sun, Aug 9, 2015 at 2:32 PM, Evan Martin <mar...@danga.com> wrote:

Earlier discussion:
https://groups.google.com/forum/#!topic/ninja-build/J8UIXu7S5pQ
We probably oughta put this in the Ninja docs somewhere.

On Sun, Aug 9, 2015 at 11:21 PM, Daniel Levin <dend...@gmail.com> wrote:
> Okay, msvc tool is visible only on Windows after updating to 1.6. But I
> still can't understand how to get help for specific tool usage.

ninja -t msvc -h

> And what
> should be format of env.file.

The entries are separated by \0:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682653(v=vs.85).aspx

I think you can also use "cmd" as your command executor to set variables:
http://superuser.com/questions/223104/setting-environment-variable-for-just-one-command-in-windows-cmd-exe
(This is more or less what Ninja does on non-Windows, as all commands
are executed within a shell.)

(Or you can have some wrapper script around your command that sets them, that would work cross-platform too.)

 

Also, to your original request -- I seem to recall we used to support
environment variables more directly in some way, but then maybe
removed that?  Scott probably recalls the reasoning better than I do.

I believe we thought about adding more explicit support but then didn't, as wrapper processes (either custom ones, or shell on posix and -t msvc -e on windows) seemed to work well enough.

[Daniel Levin]

Evan, thanks for explanation, so I believe currently per process environment on POSIX/Windows can be sen by customizing command. And that environment will be hashed into log file, which is good.

So the question is whether per process environment is a good idea generally.

If answer is yes (and I vote for it) then it would be good to support it natively in ninja.

1. Does not require much changes, execl should be replaced with execlp, CreateProcess should accept C-string with environment.

2. Perfectly backward compatible. If user does not use per process environment behavior is old.

I can implement this feature.

[Evan Martin]

On Sun, Aug 9, 2015 at 11:32 PM, Evan Martin <mar...@danga.com> wrote:
> Earlier discussion:
> https://groups.google.com/forum/#!topic/ninja-build/J8UIXu7S5pQ
> We probably oughta put this in the Ninja docs somewhere.

https://github.com/martine/ninja/issues/1002

[Daniel Levin]

Nico, that is how I am doing now to overcome this issue: by creating wrappers. And this approach is very painful each time.

[Daniel Levin]

Just another thought to consider.

From my point of view using `ninja -t msvc -e env.file` has disadvantages.

It requires user to explicitly specify env.file from command line and for sure such approach is too error prone because you can accidentally set wrong environment and screw build artifacts.

To overcome this above command line should be put into wrapper, and that is not good idea as well, ninja should not require any wrappers. Needless to say that creating wrappers per build is cumbersome and does not really fix original problem: you can still can invoke wrong build wrapper.

That is why configuration stage exists, to save all build options within build directory so later you can invoke ninja from any shell/IDE. And obtain results depending how you configured this build without side effects.

And custom build variables (per command, rule or globally) is essential part of these build options.

[Scott Graham]

Yeah, we talked about adding support for variables using something like:

  envblock = "SOME=string\0OTHER=string\0\0"

or whatever.

A subtlety of CreateProcess, is that it does not use the supplied environment block to search for the program to be executed, it instead searches in the parent's PATH. -t msvc putenv()s PATH into itself to workaround this, which gyp/GN use to have "cl.exe" find the correct architecture's compiler.

(It's possible the metabuild system could do that another way, but doesn't at the moment, so they'd still need a wrapper subprocess anyway.)

[Daniel Levin]

I understand the problem with CreateProcess. Though it is a bit different to how I see environment is for. From my point of view environment preserves same semantic as command line arguments. It passes parameters to already launched process. Problem with PATH on Windows related more to executable loader and dynamic linker. In any case if ninja can fix this issue with `ninja -t msvc -e env` then it can do the same by saving that env into build scripts at configuration stage.

[Daniel Levin]

So what do you think?

[Fredrik Medley]

I make use of the `ninja -t msvc -e env` to get a clean environment without anything extra that the user may have got.

In my .ninja file, I need to add the env file as input to every single build statement which is quite annoying (even when specified as a short $cc_extra variable). It would be nice to tell that on the rule level.

[Daniel Levin]

Anybody from Ninja developers, what do you think? Can proceed with implementation expecting you will review it?

[Nicolas Desprès]

Hi Daniel,

After reading the whole discussion, it appears to me that using a wrapper is the right way to go here. It sticks to Ninja's philosophy of staying simple and fast.

If you see environment variables as arguments you can pass the values as argument to your wrapper and let it sets them in the environment:

rule foo:

  mywrapper --env1=1 --env2=2 --arg1=3 --arg2=4

mywrapper being something like:

set ENV1=1

set ENV2=2

mycmd 3 4

This should behaves well with Ninja hashing features.

I do not find that generating extra wrappers is more painful than generating the Ninja build script itself since it could be done by your configuration script if you add the feature there rather than in Ninja.

From my point of view, if there is a place where Ninja should handle environment variables, it should be when it expands variables. Something like:

build file.out: myrule file.in

  var = $env{FOO}

or

from env import FOO as ENV_FOO

build file.out: myrule file.in

  var = $ENV_FOO

Cheers,

-Nico

--

Nicolas Desprès

[Daniel Levin]

Nicolas,

I was compelled to use wrappers for a while and can tell now that it is very painful way. Every single tool that might accept environment variable must be wrapped. Every possible variable must be copied into wrapper, taking in account potential clashes with tool arguments, proper extracting variables from those arguments, avoid overriding some variables you don't want to for particular command, etc. When you have multiple projects you need to duplicate wrappers everywhere and fix all your build scripts. Multiply this by incompatible Windows and Unix shells you need to write your wrappers for.

Staying fast. Running extra wrapper (with creating extra shell and parsing arguments) cannot be faster then passing environment into same process.

Staying simple. If you don't use environment variables in Ninja scripts it will remain for you as simple as before. Nor it will run any slower. And obviously this feature is perfectly backward compatible.

Sorry I cannot understand your examples of handling environment variables, how the suppose to specify variable for particular command? Example from Scott above is something I have in my mind.

[Nicolas Desprès]

Daniel,

On Sat, Aug 22, 2015 at 7:59 AM, Daniel Levin <dend...@gmail.com> wrote:

Nicolas,

Daniel, 

I was compelled to use wrappers for a while and can tell now that it is very painful way. Every single tool that might accept environment variable must be wrapped. Every possible variable must be copied into wrapper, taking in account potential clashes with tool arguments, proper extracting variables from those arguments, avoid overriding some variables you don't want to for particular command, etc. When you have multiple projects you need to duplicate wrappers everywhere and fix all your build scripts. Multiply this by incompatible Windows and Unix shells you need to write your wrappers for.

I think you misunderstand my point.

You have two options:

1) add support for environment variables in Ninja (as you suggested)

2) add support for environment variables in your configuration script that generates the build.ninja file (as I suggested)

Ninja's philosophy is to move most of the logic in the configuration script that generates the build.ninja file to keep ninja fast and simple. That's why I think option (2) is better here.

 

Sorry I cannot understand your examples of handling environment variables, how the suppose to specify variable for particular command? Example from Scott above is something I have in my mind.

Sorry, I should have written that in another post with a different title. My example does not address your issue. It is just a proposal to handle environment variables when expanding Ninja's variables.

 

I hope I made my point clearer.

Have a good day,

--

Nicolas Desprès

[Shezan Baig]

Daniel,

I have a branch that does what you want: https://github.com/bulldy80/ninja/commits/remove_t_msvc
I created it when I was trying to remove "-t msvc", but in the end that effort was abandoned.  You can see the discussion about it here: https://groups.google.com/d/msg/ninja-build/H7lMfaEVqzc/DbgSNZfY2fUJ

Feel free to merge that branch into your local build and see if it suits your needs.
Thanks,

-shez-

--

[Daniel Levin]

Shezan,

Thanks for the link, I believe your solution is not exactly what I mean:

1. It requires to create files with env variable content. If you have 1000 commands to run there will be 1000 env files. Command also should have dependency on env file. Instead env should be placed directly into build.ninja or rules.ninja.

2. It does not support hashing.

But it is a good starting point.

[Daniel Levin]

Nicolas, thanks I got your point and believe that high level configuration tool can do that. Though I don't see any reason to make life of configuration tool developers complicated (different platform support, wrapper generation, etc) if it can be implemented on Ninja side. Also configuration tool cannot do that effectively because of redundant extra shell instance. Environment variables is the core feature of operating systems and should be supported on low level. I still can't find any reason not to implement it on Ninja side.

For instance, in CMake we have huge amount of hacks to overcome Makefile based build system to make it work right. Like recursive Makefiles, caching rule hashes. In Ninja generator CMake behaves more or less close to perfect, moving build tasks to Ninja side to avoid hacks. With missing env variables there will be hacks again.