samurai: ninja implementation in C

[Michael Forney]

Hi all,

I wrote a ninja-compatible tool in C. It was partly motivated by the
recent surge of projects switching from autotools to meson (avoiding
the build-time dependency on C++).

https://github.com/michaelforney/samurai

The primary focus was simplicity and portability across POSIX systems.
It currently weighs in at 2823 lines according to sloccount.

samurai should be compatible with ninja files through 1.8. I consider
the project mostly finished, but I will continue tracking ninja
development for compatibility.

I thought I'd announce here in case anyone was interested.

Any feedback is welcome!

[Neil Mitchell]

Hi Michael,

Interesting. I myself also wrote a Ninja compatible evaluation (not
fully 1.8 yet) in Haskell, details are at http://shakebuild.com/ninja

One thing that might be interesting is
https://github.com/ndmitchell/ninjasmith - a tool I wrote that
randomly generates Ninja files and then checks ninja vs shake. Adding
support for samurai should be trivial, and might help you shake out
some bugs before users do. I wrote this code, and discovered a few
incompatibilities which I fixed, but it quickly got to things which I
think are flat out Ninja bugs (reported in the bug tracker, but no
interest in changing to fix them), or places where it's excessively
hard for Shake to match Ninja in the corners. If you're more
aggressively aiming for zero differences it might be rather handy.

Thanks, Neil

> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Evan Martin]

I poked around in the code and it looks super awesome!

Some questions:

- Do you have any tests?  How did you evaluate correctness?

- What are your thoughts on Windows?

- Are there any major pieces you decided to skip?  I noticed you implemented pools and the deps database etc.  Sorry if the docs there were weak, it was intended as an internal detail.

- Was there any part that was particular trouble?

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[Evan Martin]

Also, we probably ought to make a wiki page that points interested users towards other tools like yours and shake.  I created a page, I think it's world-editable: https://github.com/ninja-build/ninja/wiki/Ninja-implementations .

[Evan Martin]

One more question: what sort of systems can compile C but not C++?

On Wed, Oct 11, 2017 at 2:06 AM, Michael Forney <mfo...@mforney.org> wrote:

[Michael Forney]

On 2017-10-11, Evan Martin <evan....@gmail.com> wrote:
> I poked around in the code and it looks super awesome!

Thanks!

> Some questions:
> - Do you have any tests? How did you evaluate correctness?

No, just careful review and manual checks specific to whatever part
I'm working on.

I did run ninja's test cases for CanonicalizePath through my
implementation to make sure I got the behavior right.

> - What are your thoughts on Windows?

I have no interest in Windows, but if someone wanted to contribute
support and it wasn't invasive, I would probably accept it.

> - Are there any major pieces you decided to skip? I noticed you
> implemented pools and the deps database etc. Sorry if the docs there were
> weak, it was intended as an internal detail.

As far as language features, I think I implemented everything except MSVC deps.

Yeah, I know the deps and log formats are internal details, but I
wanted to match the same format so that ninja and samurai agreed on
what needed to be rebuilt (rather than each one trying to rebuild
everything).

In terms of CLI features, I only implemented the clean subtool and
don't currently support NINJA_STATUS.

> - Was there any part that was particular trouble?

The deps and build log were a little troublesome as you suspected, but
nothing too bad. Implementing restat and order-only dependencies was
also a bit tricky to get right.

[Michael Forney]

On 2017-10-11, Neil Mitchell <ndmit...@gmail.com> wrote:
> Interesting. I myself also wrote a Ninja compatible evaluation (not
> fully 1.8 yet) in Haskell, details are at http://shakebuild.com/ninja
>
> One thing that might be interesting is
> https://github.com/ndmitchell/ninjasmith - a tool I wrote that
> randomly generates Ninja files and then checks ninja vs shake. Adding
> support for samurai should be trivial, and might help you shake out
> some bugs before users do. I wrote this code, and discovered a few
> incompatibilities which I fixed, but it quickly got to things which I
> think are flat out Ninja bugs (reported in the bug tracker, but no
> interest in changing to fix them), or places where it's excessively
> hard for Shake to match Ninja in the corners. If you're more
> aggressively aiming for zero differences it might be rather handy.

Neat. I'll have to take a look at that at some point. I'm curious
about what it might find.

[Peter Williams]

Seeing as this thread has garnered some interest, I'll toss out the
idea that I think it'd be very interesting to see Ninja implemented in
the Rust language -- I think the main benefit would be that Rust has a
really well put-together packaging system that would make it pretty
trivial to turn Ninja into a reusable library. Also, if you ask me,
Rust is just a really really excellent language. (Like C/C++ it
compiles down to runtime-free static binaries.)

Timing-wise, this fall the Rust folks are working on ways they can have
their packaging tool, Cargo, integrate better with external build
systems [1, 2]. They're interested in teaching Cargo to export its
internal dependency graph, and I think it'd be totally reasonable to
aim to make the build.ninja format the way that graph is serialized. I
think this thread shows that you can think of the build.ninja format as
a simple protocol that can be supported by a variety of tools.

Peter

[1] https://blog.rust-lang.org/2017/09/18/impl-future-for-rust.html
[2] https://paper.dropbox.com/doc/Cargo-build-system-integration-1sqRG8
uyCqxv9EfoS8cco

[Evan Martin]

I think if I were to start down that path I'd reconsider some of the design choices.  For example, we could avoid a lot of shell quoting issues if we used arrays instead of strings as our primitive for command lines, and (following shake) there's interesting things you can do if you let build rules also modify the graph.  And then at that point I guess I'd wonder why you needed a Rust library to do that thing in the first place.

I guess maybe another way of saying this is that if I were looking to make a build library I'd recommend starting from shake (which is itself already a general-purpose build library and implements ninja support atop that generalized functionality).

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.

To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build+unsubscribe@googlegroups.com.

[Konstantin Tokarev]

12.10.2017, 21:12, "Evan Martin" <evan....@gmail.com>:

> I think if I were to start down that path I'd reconsider some of the design choices.  For example, we could avoid a lot of shell quoting issues if we used arrays instead of strings as our primitive for command lines, and (following shake) there's interesting things you can do if you let build rules also modify the graph.

IMO, nice thing about Ninja files is that you can read them without mental "parsing" of arrayish syntax, and also can copy-paste parts of command lines directly to shell

>> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google Groups "ninja-build" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

-- 
Regards,
Konstantin

[Evan Martin]

On Wed, Oct 11, 2017 at 11:55 PM, Michael Forney <mfo...@mforney.org> wrote:

> - What are your thoughts on Windows?

I have no interest in Windows, but if someone wanted to contribute
support and it wasn't invasive, I would probably accept it.

I had been thinking that maybe if you ignored Windows you would be able to simplify the implementation further, because some of our design decisions (primarily the deps database) were motivated by platform limitations on Windows.  But now that I'm thinking back on it, maybe the deps database is a better design than sprinkling the .d files even on POSIX platforms, so maybe it's all right.  I do know we have some gnarly hacks around forward-vs-back slashes in paths.

 

> - Was there any part that was particular trouble?

The deps and build log were a little troublesome as you suspected, but
nothing too bad. Implementing restat and order-only dependencies was
also a bit tricky to get right.

Makes sense.  Those two (among many) corners of the design that I'm kind of unhappy with, in that I feel like I didn't start from a clear conceptual model of what we were trying to achieve.  Maybe the next person who makes a Ninja-like tool will improve on it.  ;)

[Tiago Gonçalves]

On Thursday, 12 October 2017 19:12:20 UTC+1, Evan Martin wrote:

I think if I were to start down that path I'd reconsider some of the design choices.  For example, we could avoid a lot of shell quoting issues if we used arrays instead of strings as our primitive for command lines, and (following shake) there's interesting things you can do if you let build rules also modify the graph.  And then at that point I guess I'd wonder why you needed a Rust library to do that thing in the first place.

I think that in windows you can't convert from array to string with 100% certainty without knowing what executable you are calling.

When you call an executable it receives a single string with all the arguments and a libc? function splits the string into the 'int main(argc, argv)' arguments. This function has known bugs and the executable can get the original string and process it using other algorithms. So you need to know how an executable parses that string so you known how to convert the array to it.
 

[Evan Martin]

Yes, command lines on Windows are terrifying, but everything there feels a bit terrifying to me.  ;)

Many years ago I wrote some of the code in Chrome related to this!  I wrote a longer blog post about it too: http://neugierig.org/software/blog/2011/10/quoting.html .

Sorry, I didn't mean to derail this thread into a discussion about redesigning Ninja.  I like Rust too but the rewrite-it-in-Rust thing, even as a fan of Rust, can be a bit too much for me.

--

[Peter Williams]

On Thu, 2017-10-12 at 12:20 -0700, Evan Martin wrote:

Yes, command lines on Windows are terrifying, but everything there feels a bit terrifying to me.  ;)

Many years ago I wrote some of the code in Chrome related to this!  I wrote a longer blog post about it too: http://neugierig.org/software/blog/2011/10/quoting.html .

Sorry, I didn't mean to derail this thread into a discussion about redesigning Ninja.  I like Rust too but the rewrite-it-in-Rust thing, even as a fan of Rust, can be a bit too much for me.

Indeed. I do think there's a bit of a time-limited opportunity here, though, with the fact that the Rust folks are currently thinking about how to integrate their package manager with other build systems. I'm not familiar with shake though so maybe it's a better fit for interchange-type things.

Peter