Add busybox/toybox-like toolkit inside ninja

[Dmytro Ivanov]

Long story short: make is awesome to use, but writing portable make files is ... hard. Ninja is like a better make in all aspects, but generating portable ninja files is ... hard. For example: try adding a portable file copy from one location to another. Or try doing portable sed, awk, grep ... So every ninja-generation tool is on their own in this, cmake for example has "cmake -E copy", while others rely on "switch-casing" supported platforms.

To ensure ninja status of better make, I'm proposing adding an existing portable toolkit to ninja, so tool developers could target it when writing their generator. This should significantly simplify life for tool devs, and this should also simplify life for users - as now having single ninja executable is enough for getting stuff done.

So, ideas? opinions? go / no-go ? :)

[Nico Weber]

Can't generators that want this just bundle busybox itself?

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lee Winter]

It is my intention to add TCL and LuA as cross platform shells. TCL
because it is very comprehensive and even includes a GUI tool kit.
Lua because it it the smallest/fastest shell that I know of.

There are three ways to implement such shells. The simplest
implementation does not affect Ninja at all. The Ninja generator can
just prefix the build rules with the name of the shell. Since the
names do not vary across platforms the only downside is that the
prefix approach takes more CPU, memory, and time than the native
shell.

Another approach is to create the shell as a multi-process demon that
accepts text streams and executes them. The demon approach is a
performance improvement over the native shell because it eliminates
the process launch overhead of each invocation of the shell. But it
requires that Ninja use a stream to handle command invocation.

Probably the optimal approach is to integrate the shells within Ninja.
The integrated approach replaces a process launch or inter-process
message passing with a subroutine call, so it is much faster than any
external approach. But it is also much more work.

But none of that that will happen until at least Q2 of 2018.

Hopefully,

Lee Winter
Center Tuftonborough*, New Hampshire
United States of America (It's alive!)

* not to be confused with Tuftonborough Center.

[rym...@gmail.com]

Well, Lua's kind of verbose...

Has anyone here considered an actual shell, like rc? It's more lightweight than e.g. bash and better suited for actual scripting...

--
Ryan (ライアン)
Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else
http://refi64.com

--  

You received this message because you are subscribed to the Google Groups "ninja-build" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

[Ben Boeckel]

On Sun, Oct 15, 2017 at 10:39:37 -0400, rym...@gmail.com wrote:
> Well, Lua's kind of verbose...

Lua's standard library also leaves things to be desired in places (for
example, filesystem support is basically an external package). Ninja
could require/expect certain ones to be available, but you start losing
ground there quickly if you want Ninja to be a one-stop-tool for build
bits.

I agree with the other message in the thread that generators should
either expect a set of tools to exist (like Autotools) or provide their
own set (like CMake does).

> Has anyone here considered an actual shell, like rc? It's more lightweight
> than e.g. bash and better suited for actual scripting...

What would a shell accomplish? A shell doesn't come with a set of tools
like grep/sed/awk/etc., it just provides a way to fork/exec them in
sequence.

--Ben

[Konstantin Tokarev]

16.10.2017, 18:35, "Ben Boeckel" <ben.b...@kitware.com>:

> On Sun, Oct 15, 2017 at 10:39:37 -0400, rym...@gmail.com wrote:
>>  Well, Lua's kind of verbose...
>
> Lua's standard library also leaves things to be desired in places (for
> example, filesystem support is basically an external package). Ninja
> could require/expect certain ones to be available, but you start losing
> ground there quickly if you want Ninja to be a one-stop-tool for build
> bits.

FWIW, Premake provides Lua with all batteries needed for typical build tasks

https://premake.github.io/

>
> I agree with the other message in the thread that generators should
> either expect a set of tools to exist (like Autotools) or provide their
> own set (like CMake does).
>
>>  Has anyone here considered an actual shell, like rc? It's more lightweight
>>  than e.g. bash and better suited for actual scripting...
>
> What would a shell accomplish? A shell doesn't come with a set of tools
> like grep/sed/awk/etc., it just provides a way to fork/exec them in
> sequence.
>
> --Ben
>

> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
Regards,
Konstantin

[Dmytro Ivanov]

I think this is a chicken&egg problem, as Nico pointed out - if generator wants to bundle something, they already can do it. But the same is true in opposite - if generator don't want to use ninja, they will not :)

I'm just proposing to include more batteries into ninja so it looks like a better choice in comparison to make. But maybe the problem doesn't exist in a first place.

[Ciprian Dorin Craciun]

On Sun, Oct 15, 2017 at 5:28 PM, Lee Winter <lee.j.i...@gmail.com> wrote:
> It is my intention to add TCL and LuA as cross platform shells. TCL
> because it is very comprehensive and even includes a GUI tool kit.
> Lua because it it the smallest/fastest shell that I know of.

I tend to partially agree with Lee on this topic, namely that I think
a much more problematic issue with Ninja generators is the actual
command generation part, and in fact it is the first stumbling block
into writing correct and portable build scripts. Currently the
`ninja` manual says that the commands are actually just thrown to `sh
-c '...'`.

Thus the generator writer has not only to target the "portable" shell
subset, but also to get correct the shell escaping and quoting and
syntax. (And being a Linux user, I don't even take into account the
differences between *NIX / Windows worlds. Also being a *NIX user, I
think that choosing `$` as an escape character was the worst possible
choice, as `sh` scripts are full of `$`...)




However, and here is where I only partially agree with this topic, we
don't actually need a full-blown shell. From my experience with Ninja
--- I've written a generator in Chibi Scheme, and a handful of narrow
domain generators -- all we actually need inside commands is:

* the majority of the time just a command with some arguments,
something that could be easily run by Ninja itself;

* some of the times a sequence of commands, and if one fails just
stop; (i.e. `cmd1 && cmd2 && cmd3`;) this is again something that
Ninja can easily handle;

* in rare cases a sequence of commands, and if one fails just stop and
run some cleanup; (i.e. `{ cmd1 && cmd2 && cmd3 ; } || { cmd4 && cmd5
; }`); again this could be handled by Ninja itself;

* in even rarer cases -- because such an usage is quite complex and
could be split into multiple build rules -- one requires more involved
"code", like scripts; this case I think would be better handled by
just taking out that code and putting it into an "executable" (which
can easily be a script); however even in this case Ninja could handle
it by using a special syntax, for example by replacing `command` with
`interpreter = python -s $script` and `script = ...`, where `$script`
is replaced by the path of a temporary file holding the contents of
the script (or in case of Linux even a pipe `/dev/fd/99` with a large
buffer, to eliminate all the file-system touching); (and it would be
wonderful if one can write `script = ...`, or even `command = ...` as
a multi-line construct without having to resort to `... $` syntax;)

* and finally a way to change the environment variables of the
launched processes; (for example a special syntax `env:VARIABLE =
VALUE`);

* (and just for completion, and the rare occasions where such a
feature would be needed, perhaps throw in some "process limits and
control", starting with `ulimit`, `nice`, `chrt`, `ionice`, and
`taskset`...) :)




As Lee mentioned there are multiple ways to implement such -- or any
-- shell, but the two variants that make the most sense are
(especially in *NIX):

* embedded into Ninja itself, executing each command instead of `sh`;
(optionally `fork`-ing itself;)

* executing another tool (perhaps part of the Ninja package), just
like it does with `sh` now, but with a much clearer "protocol" than
`sh` has when it comes to executing sequences of commands; (this tool
could even be usable outside of Ninja;)

Ciprian.




P.S.: Minor rant related with this: if Ninja is not meant as a
direct "language" for humans to write build scripts in, but as a
target for "generators", then why don't we use a syntax that is easily
parseable and already supported in almost any language known to man,
like for example JSON?

How is this related to all of this? If one would to design a
"protocol" between Ninja and the "shell", why can't we use JSON as the
syntax between the two, like in:

~~~~
{
"version" : 1,
"commands": [
["do", "this"],
["do", "that"]
],
"cleanup" : [
["rm", "this"],
["rm", "that"]
],
"environment" : {
"PATH" : "something"
}
}
~~~~

[Konstantin Tokarev]

17.10.2017, 11:00, "Ciprian Dorin Craciun" <ciprian...@gmail.com>:

I would argue that readability is important feature, and JSON version would be
much less readable.

Also, with current syntax it's possible to copy at least fragments of commands
directly to shell, doing this for JSON would be much more work.

That said, it would be useful to have built-in no-fork commands for such basic
operations as "cd", "cp", "rm", and running command with specific environment
variables in a cross-platform manner.

>
> How is this related to all of this? If one would to design a
> "protocol" between Ninja and the "shell", why can't we use JSON as the
> syntax between the two, like in:
>
> ~~~~
> {
>     "version" : 1,
>     "commands": [
>         ["do", "this"],
>         ["do", "that"]
>     ],
>     "cleanup" : [
>         ["rm", "this"],
>         ["rm", "that"]
>     ],
>     "environment" : {
>         "PATH" : "something"
>     }
> }
> ~~~~
>

[Ben Boeckel]

On Tue, Oct 17, 2017 at 10:59:41 +0300, Ciprian Dorin Craciun wrote:
> * and finally a way to change the environment variables of the
> launched processes; (for example a special syntax `env:VARIABLE =
> VALUE`);

Note that this gets into questions like "should PATH be overwritten"?
Appended? Prepended? What's the separator character (if any)? For Ninja
itself, environment manipulation is best left to a wrapper script IMO.

--Ben

[Nico Weber]

On Mon, Oct 16, 2017 at 4:13 PM, Dmytro Ivanov <jimon...@gmail.com> wrote:

I think this is a chicken&egg problem, as Nico pointed out - if generator wants to bundle something, they already can do it. But the same is true in opposite - if generator don't want to use ninja, they will not :)

I'm just proposing to include more batteries into ninja so it looks like a better choice in comparison to make. But maybe the problem doesn't exist in a first place.

I think providing a tool that contains lots of batteries can make sense in this space, but I also think Ninja is not that tool. The general guideline is "if a generator can do it, let the generator do it", so that ninja can stay small and fast. https://ninja-build.org/manual.html#_philosophical_overview has some more words on this :-)

 

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[Ciprian Dorin Craciun]

On Tue, Oct 17, 2017 at 2:09 PM, Konstantin Tokarev <ann...@yandex.ru> wrote:
>> P.S.: Minor rant related with this: if Ninja is not meant as a
>> direct "language" for humans to write build scripts in, but as a
>> target for "generators", then why don't we use a syntax that is easily
>> parseable and already supported in almost any language known to man,
>> like for example JSON?
>
> I would argue that readability is important feature, and JSON version would be
> much less readable.
>
> Also, with current syntax it's possible to copy at least fragments of commands
> directly to shell, doing this for JSON would be much more work.

I'm afraid that "in real use" (i.e. Ninja scripts generated by an
actual generator) the "readability" of said commands is not that
good... For starters the commands are not in the `build do out : in`
section, but rather in the `rule do` section, therefore in order to
get the actual command one would need to copy-paste the command in the
`rule do` section, and then interpolate at least the `$in` and `$out`
commands, and then to also un-escape any `$` tokens, and concatenate
multi-line commands...

On a second note, no matter what the shell language might be, there is
always the `ninja -n -v` (that could issue a similar command to the
shell, like `sh -n -v`) that will show the actual commands, no matter
how the command description language looks like.

> That said, it would be useful to have built-in no-fork commands for such basic
> operations as "cd", "cp", "rm", and running command with specific environment
> variables in a cross-platform manner.

Where does one stop with including "basic" operations? For example
the `cp` variant should it have a recursive option? How about a do /
don't copy also the attributes? Should it cross symlinks? Should it
cross mount points? (And any of these features I would say are
basic.)

I think the open-source community would be better served if one
develops such a `busybox` / `toybox` alternative, say `fsbox`, that
only implements file-system operations in a cross-platform manner, as
an independent tool, thus allowing its usage for other purposes.

(`cd` should be part of the command environment modification, just
like `export`, thus I don't think it falls under this category.)




To take this idea further, when it comes to build systems I think
there are a set of complementary tools, that each does one thing (or
set of tightly related things) best:

* a build system configurator that generates the Ninja script; (there
could be many, depending on the context;)

* a dependency solver, change detector and execution coordinator -- ninja;

* the "shell" we are speaking about, that allows chaining multiple
commands, and take care of redirections, environment variables, etc.;

* a set of tools that do the actual stuff -- `busybox` / `toybox` for
file-system related operations, `curl` for network interactions, etc.;

One could use any of these in isolation (especially the last two), and
replace any of the layers without impacting the others. Moreover the
only layer that has to do with portability is the last one (i.e. the
tools), and to some less extent the `shell`;

Ciprian.

[Konstantin Tokarev]

17.10.2017, 19:38, "Ciprian Dorin Craciun" <ciprian...@gmail.com>:

Thinking on it further, I also came to conclusion that only "cd" and "export"
belong here.

Looking at what CMake generator produces, looks like 2 features could be
used by it immediately to reduce number of forks:

1. Built-in cd
2. Support for running "cmd1 && cmd2 && ..." without invoking shell, just by
splitting commands by &&, and their contents by space (the latter is needed
on *nix only, AFAIU).

>
> To take this idea further, when it comes to build systems I think
> there are a set of complementary tools, that each does one thing (or
> set of tightly related things) best:
>
> * a build system configurator that generates the Ninja script; (there
> could be many, depending on the context;)
>
> * a dependency solver, change detector and execution coordinator -- ninja;
>
> * the "shell" we are speaking about, that allows chaining multiple
> commands, and take care of redirections, environment variables, etc.;
>
> * a set of tools that do the actual stuff -- `busybox` / `toybox` for
> file-system related operations, `curl` for network interactions, etc.;
>
> One could use any of these in isolation (especially the last two), and
> replace any of the layers without impacting the others. Moreover the
> only layer that has to do with portability is the last one (i.e. the
> tools), and to some less extent the `shell`;
>
> Ciprian.
>

[Dmytro Ivanov]

Maybe what we need is a way to pass commands to other tool without shell/forks? like cgi in web servers. Then there will be a possibility to create a shell/toolkit/batteries inside other tool and make it a counterpart to ninja.

> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
Regards,
Konstantin

--
You received this message because you are subscribed to a topic in the Google Groups "ninja-build" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ninja-build/gs2eMFdEnKc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ninja-build+unsubscribe@googlegroups.com.

[Ciprian Dorin Craciun]

On Tue, Oct 17, 2017 at 8:05 PM, Dmytro Ivanov <jimon...@gmail.com> wrote:
> Maybe what we need is a way to pass commands to other tool without
> shell/forks? like cgi in web servers. Then there will be a possibility to
> create a shell/toolkit/batteries inside other tool and make it a counterpart
> to ninja.

Why this emphasis on reducing the number of forks? Has anyone
benchmarked this and found out that it takes 10% of the overall build
process?

I'm asking because my feeling (thus no benchmarking on my side) is
that 99.99% of the time is spent in `gcc`-like tools that crunch the
code (i.e. CPU bound), or in I/O bound tasks.

Ciprian.