release: ninja v120715

[Evan Martin]

I've tagged a new release of Ninja, version 120715.

== Release notes

As compared to the last release:

- Many performance improvements, particularly on Mac OS X (Nico Weber).
Nico also changed the build log format, making it much smaller. The
build log must be loaded as part of every build, and for large
projects like Chrome Nico's change reduced the file size from
megabytes to kilobytes.

- Various Windows fixes (Scott Graham, Brandon Philips, others).

- New build file variable: $in_newline is the same as $in but with
newlines instead of spaces between files. This is kind of a hack,
necessary to work around a bug in MSVC's linker and response files
(Scott Graham).

- A now-fixed logic bug related to the "restat" attribute caused
spurious builds (Petr Wulf).

== Getting this release

To build this release, clone the repository as follows:
git clone git://github.com/martine/ninja.git
cd ninja
git checkout release-120715
or grab a tarball from https://github.com/martine/ninja/tags .

Once you have the code, run ./bootstrap.py to build and then ./ninja
-V will print the version number.

See
https://github.com/martine/ninja/downloads
for binary downloads.
(There are none there yet. If you think binaries make sense for your
platform, please help me out.)

If you've written packaging support scripts (e.g. Debian, Homebrew, or
Windows batch files) and would like to check them in to the Ninja
repository, send me a pull request.

See the home page for more about Ninja:
http://martine.github.com/ninja/

[Claus Klein]

Please fix!

/cygdrive/c/downloads/martine-ninja-50af448
$ ./bootstrap.py
Building ninja manually...
src/util.cc: In function 'double GetLoadAverage_unix()':
src/util.cc:306:28: error: 'getloadavg' was not declared in this scope
#######################################################

KLEIN_CL@WD002072 /c/downloads/martine-ninja-50af448
$ ./bootstrap.py
Building ninja manually...
src\subprocess-win32.cc: In member function 'void
SubprocessSet::Clear()':
src\subprocess-win32.cc:275: error: 'GetProcessId' was not declared in
this scope
####################################################################

This release is not well tested :-((

And there was a pull request to fix this!

Claus

[Nicolas Desprès]

On Mon, Jul 16, 2012 at 12:57 PM, Claus Klein <claus...@arcormail.de> wrote:
> Please fix!
>
> /cygdrive/c/downloads/martine-ninja-50af448
> $ ./bootstrap.py
> Building ninja manually...
> src/util.cc: In function 'double GetLoadAverage_unix()':
> src/util.cc:306:28: error: 'getloadavg' was not declared in this scope
> #######################################################
>
> KLEIN_CL@WD002072 /c/downloads/martine-ninja-50af448
> $ ./bootstrap.py
> Building ninja manually...
> src\subprocess-win32.cc: In member function 'void
> SubprocessSet::Clear()':
> src\subprocess-win32.cc:275: error: 'GetProcessId' was not declared in
> this scope
> ####################################################################
>
> This release is not well tested :-((

The Ninja project does not have enough resource to test on every
supported platform.

>
> And there was a pull request to fix this!

Can you point out the pull request in question please to help debugging?

Nico

--
Nicolas Desprès

[Claus Klein]

There was more than one!

https://github.com/martine/ninja/pull/339

I am not longer willing to contribute to a project which ignores all the requests to help!

Claus

[Alexander Usov]

Another build issue on linux -- pull request #363

One simple way to do some pre-release testing -- announce upcoming release on the list, so that people can test it in their own environment & fish out some of the glitches.

[Claus Klein]

On 16.07.2012, at 19:50, Alexander Usov wrote:

Another build issue on linux -- pull request #363

but it works without this include?

[Evan Martin » ninja #59](https://buildhive.cloudbees.com/job/martine/job/ninja/59/) SUCCESS

One simple way to do some pre-release testing -- announce upcoming release on the list, so that people can test it in their own environment & fish out some of the glitches.

Yes, we need at least 3 branches: release, next, develop!

see https://groups.google.com/forum/#!topic/ninja-build/TZ8c4-mhPL0 again and read it carefully, it save time!

Claus

[Scott Graham]

On Mon, Jul 16, 2012 at 2:40 PM, Claus Klein <claus...@arcormail.de> wrote:

> Yes, we need at least 3 branches: release, next, develop!
>
> see https://groups.google.com/forum/#!topic/ninja-build/TZ8c4-mhPL0 again and read it carefully, it save time!

"I lack the interest or time to maintain multiple branches of ninja."
seems pretty clear to me.

I view the announcement of tagging as a request to test, personally.
If it doesn't work, then we can shrug (or sigh), fix it, and tag a new
one.

[Claus Klein]

It is hard to separate feature development from bug-fixing while code-frees before a release with only one branch!

It is easy to create 2 additional branches: a stable (for fixing) and a release for tested releases only.

It would be much more easy to merge a pull request in a development branch as right now in the master( release branch ).

That is a normal SW development process. Many projects work so.

Claus

[Yohann Coppel]

You can also have your own "stable" fork and merge any change you like.

[Evan Martin]

On Mon, Jul 16, 2012 at 10:31 AM, Claus Klein <claus...@arcormail.de> wrote:
> There was more than one!
>
> https://github.com/martine/ninja/pull/339
>
> I am not longer willing to contribute to a project which ignores all the
> requests to help!

Hey Claus,

I really appreciate your enthusiasm for helping but:
1) I would prefer to merge changes with clean histories, or single patches
2) I won't merge changes that comment out code. Looking at the diff
of this branch I see stuff like
"+//XXX see "subprocess.h" #include <windows.h>"
I think Nico commented on one of your other changes to that effect.

More broadly, Ninja is in a state where it works for many people who
rely on it continuing to work, and I don't have a lot of time to hack
on it, so I am reluctant to merge changes that aren't self-evidently
correct (or even small changes like introducing C++-style casts in
places where they don't improve code quality or readability).

With that said, there are many changes I see in your diff that would
be welcome if they were small self-contained changes.

[Thiago Farina]

On Tue, Jul 17, 2012 at 12:43 AM, Evan Martin <mar...@danga.com> wrote:
> (or even small changes like introducing C++-style casts in
> places where they don't improve code quality or readability).
>

Could you give us an example of a C++ style cast the improves code
quality and/or readability in your opinion?

--
Thiago

[Claus Klein]

Evan,

it was as small patch https://github.com/ClausKlein/ninja/commit/8fb082c14be3e6afe6c90ffe2cf677ef8577e54f

You started a discussion about patch (code style) instead to fix the real problems:

But at least it should compile and work!

I have to no time to split patches to small peaces, It was indented as a hint.

I will merge you development to my ninja fork.

There will be a stable branch and my releases from time to time.

You may merge or fix your code yourself.

Claus

[Evan Martin]

On Mon, Jul 16, 2012 at 9:42 PM, Claus Klein <claus...@arcormail.de> wrote:
> I have to no time to split patches to small peaces, It was indented as a
> hint.
>
> I will merge you development to my ninja fork.
> There will be a stable branch and my releases from time to time.
>
> You may merge or fix your code yourself.

Ok, thank you for providing the hints! Hopefully they will be useful
for other people who want to work mingw as well.

[Alexander Usov]

On Monday, 16 July 2012 22:40:30 UTC+1, Claus Klein wrote:

On 16.07.2012, at 19:50, Alexander Usov wrote:

Another build issue on linux -- pull request #363

but it works without this include?

[Evan Martin » ninja #59](https://buildhive.cloudbees.com/job/martine/job/ninja/59/) SUCCESS

I think it's a change in the new glibc (2.16.0 in my case).

From what I understand in the newer glibc versions they cleaned up by-includes, and you need to include <sys/resource.h> manually if you want to have 'struct rlimit'.

Yes, we need at least 3 branches: release, next, develop!

see https://groups.google.com/forum/#!topic/ninja-build/TZ8c4-mhPL0 again and read it carefully, it save time!

I have seen it and read it carefully.

What I'm suggesting is something different -- I for example don't actively track ninja HEAD, but if I had seen an email saying "we are going to tag release next week",  I would definitely try to compile it & run unit tests on the few systems I have around.

[Richard Wackerbarth]

Evan,

I run a number of automated builds each night. In addition to my own requirements, I run all of the FreeBSD and some other ninja-based builds for the CMake project http://open.cdash.org/index.php?project=CMake

I just updated those builds to use your 120715 release on ninja and that has gotten us closer to being able to accept ninja for use on all of those platforms.

One of the things that I have tried to do is to base some of those nightly builds on the HEAD of your master branch so that we could see if using a newer version affected our test suite.

In doing so, I (later) found that I had made a mistake and was not using the version of ninja that I had intended. Tracking it down was made more difficult because of your version designation. (All versions of the HEAD branch display "git"). It would be quite helpful if you could change that so that different builds are distinguishable by referencing the version number provided by `ninja -V`

A number of people have addressed ways to automatically generate version numbers from the git repository (for example, http://eatmyrandom.blogspot.com/2010/06/automate-version-numbering-using-git.h )

Using make, cmake, etc. it is easy to trigger a script that determines if a version header file is out of date and rewrite it when it is. That change then propagates through the dependency tree.

In attempting to use ninja, I haven't discovered any way to duplicate this behavior without "wrapping the ninja build" in another build system.  From my perspective, having a mechanism to specify either sequential build phases within one .ninja file, or having some way in addition to filesystem timestamps to specify that the output of a rule is "out-of-date" would be useful.

Did I miss some capability in your design that addresses these issues?

Richard

[Evan Martin]

On Tue, Jul 17, 2012 at 8:09 AM, Richard Wackerbarth <ric...@nfsnet.org> wrote:
> A number of people have addressed ways to automatically generate version
> numbers from the git repository (for example,
> http://eatmyrandom.blogspot.com/2010/06/automate-version-numbering-using-git.h
> )
>
> Using make, cmake, etc. it is easy to trigger a script that determines if a
> version header file is out of date and rewrite it when it is. That change
> then propagates through the dependency tree.
>
> In attempting to use ninja, I haven't discovered any way to duplicate this
> behavior without "wrapping the ninja build" in another build system. From

Coincidentally, we're having a discussion about how to do this right here:
https://groups.google.com/d/topic/ninja-build/txeosqGMc6k/discussion

It wouldn't be too hard to make Ninja include this information but it
ends up being yet more complexity -- for example, we need to figure
out what to do if git isn't available. (This is a plausible problem
on Windows.)

> my perspective, having a mechanism to specify either sequential build phases
> within one .ninja file,

Could you elaborate on what you mean by this?

> or having some way in addition to filesystem
> timestamps to specify that the output of a rule is "out-of-date" would be
> useful.

What other information would you like to key off of for judging
whether a rule is out of date?
Usually you can encode such information using a stamp file in the
filesystem. It's not as pretty as some sort of native support, but
it's also rather standard practice for build systems and it keeps the
number of moving parts down.

[Richard Wackerbarth]

On Jul 17, 2012, at 10:55 AM, Evan Martin wrote:

> On Tue, Jul 17, 2012 at 8:09 AM, Richard Wackerbarth <ric...@nfsnet.org> wrote:
>> A number of people have addressed ways to automatically generate version
>> numbers from the git repository (for example,
>> http://eatmyrandom.blogspot.com/2010/06/automate-version-numbering-using-git.h
>> )
>>
>> Using make, cmake, etc. it is easy to trigger a script that determines if a
>> version header file is out of date and rewrite it when it is. That change
>> then propagates through the dependency tree.
>>
>> In attempting to use ninja, I haven't discovered any way to duplicate this
>> behavior without "wrapping the ninja build" in another build system. From
>
> Coincidentally, we're having a discussion about how to do this right here:
> https://groups.google.com/d/topic/ninja-build/txeosqGMc6k/discussion

Interesting discussion. I will read/join in.

>
> It wouldn't be too hard to make Ninja include this information but it
> ends up being yet more complexity -- for example, we need to figure
> out what to do if git isn't available. (This is a plausible problem
> on Windows.)

The typical solution is to have a default file that does not get overwritten if a recognized scm is not available.

>> my perspective, having a mechanism to specify either sequential build phases
>> within one .ninja file,
>
> Could you elaborate on what you mean by this?

Basically, have a structured dependency scheme.

There are some rules for phase1, some for phase2, ...
In addition to the "normal" dependency rules, phase2 depends on the completion of phase1 but its other dependencies are not computed until all of phase1 has completed. Thus phase1 MIGHT alter the timestamps that affect phase2. Or it might not. Much better than the phony build scheme that always assumes that things will change.

Fundamentally, this is that same as a script that does `build phase1; build phase2; build phase3` but has all of the information in a single file.

>> or having some way in addition to filesystem
>> timestamps to specify that the output of a rule is "out-of-date" would be
>> useful.
>
> What other information would you like to key off of for judging
> whether a rule is out of date?

In general, it needs to be a script. In particular, one thing that often comes up is that an intermediate file is generated, but its content does not actually change and, as a result, the dependency chain can be broken.

> Usually you can encode such information using a stamp file in the
> filesystem. It's not as pretty as some sort of native support, but
> it's also rather standard practice for build systems and it keeps the
> number of moving parts down.

The difference in this and the scm posthook that Nico Weber describes is that posthook is a push mechanism whereas I am suggesting a pull mechanism.

Richard

[Evan Martin]

On Tue, Jul 17, 2012 at 9:35 AM, Richard Wackerbarth <ric...@nfsnet.org> wrote:
>> What other information would you like to key off of for judging
>> whether a rule is out of date?
>
> In general, it needs to be a script. In particular, one thing that often comes up is that an intermediate file is generated, but its content does not actually change and, as a result, the dependency chain can be broken.

Ah, I think this and some of your above problem can be addressed by
the 'restat' rule attribute:
http://martine.github.com/ninja/manual.html#_rule_variables


For multi-stage builds in Chrome, the pattern we frequently use is a
"stamp" file to mark completion of some important action. For
example, imagine a script that generates 50 header files from IDL:

build header-files.stamp foo1.h foo2.h: run-idl-gen foo1.idl foo2.idl
build next-phase/a.o: cc a.cc || header-files.stamp
build next-phase/b.o: cc b.cc || header-files.stamp

Note that a.o/b.o don't list which header files they use -- this
relies on depfiles (header dependencies as emitted by the compiler) to
remain correct in the presence of modifications to the IDL files.

[Richard Wackerbarth]

On Jul 17, 2012, at 11:42 AM, Evan Martin wrote:

> On Tue, Jul 17, 2012 at 9:35 AM, Richard Wackerbarth <ric...@nfsnet.org> wrote:
>>> What other information would you like to key off of for judging
>>> whether a rule is out of date?
>>
>> In general, it needs to be a script. In particular, one thing that often comes up is that an intermediate file is generated, but its content does not actually change and, as a result, the dependency chain can be broken.
> Ah, I think this and some of your above problem can be addressed by
> the 'restat' rule attribute:
> http://martine.github.com/ninja/manual.html#_rule_variables

The combination of restat and custom command scripts in the rules generates (at least most of) the capability I was seeking.

> For multi-stage builds in Chrome, the pattern we frequently use is a
> "stamp" file to mark completion of some important action. For
> example, imagine a script that generates 50 header files from IDL:
>
> build header-files.stamp foo1.h foo2.h: run-idl-gen foo1.idl foo2.idl
> build next-phase/a.o: cc a.cc || header-files.stamp
> build next-phase/b.o: cc b.cc || header-files.stamp

This seems awfully verbose. But I guess that you are pushing that off onto the .ninja file generator.

[Richard Wackerbarth]

On Jul 17, 2012, at 10:55 AM, Evan Martin wrote:

> On Tue, Jul 17, 2012 at 8:09 AM, Richard Wackerbarth <ric...@nfsnet.org> wrote:
>> A number of people have addressed ways to automatically generate version
>> numbers from the git repository

> It wouldn't be too hard to make Ninja include this information but it
> ends up being yet more complexity -- for example, we need to figure
> out what to do if git isn't available. (This is a plausible problem
> on Windows.)

Elaborating --

Generate header and/or body files that contain the version number in whatever text formats you require by means of a script that queries the scm and reformats the content.

For example, in ninja.cc, replace the version line with

#include "version.h"

and have a generated file, version.h which contains a single line such as (or get fancier if you wish)

const char* kVersion = "12.07.15";

Define a script that generates a temporary file from information provided by the scm and formats it in a similar fashion. For example,

const char* kVersion = "12.07.15.1.gdee5";

If the scm is not present/active, the script simply terminates successfully.

If this temporary file differs, in content, from the previous version.h stored, copy the newly generated version in as a replacement for the previous version.

In ninja, use the restat option to break the dependency chain to avoid unnecessary rebuilding when the version has not been changed.


Richard

[Nico Weber]

On Sun, Jul 15, 2012 at 1:52 PM, Evan Martin <mar...@danga.com> wrote:
> I've tagged a new release of Ninja, version 120715.

Thanks for the release! I pushed this release to chromium's bots and
developers a little over a week ago, and haven't heard of any issues
yet. After a week of heavy bot use, this probably means that it works
pretty well.

> See
> https://github.com/martine/ninja/downloads
> for binary downloads.
> (There are none there yet. If you think binaries make sense for your
> platform, please help me out.)

A mac binary built from this release is at
http://src.chromium.org/viewvc/chrome/trunk/tools/depot_tools/ninja-mac?view=log
. Please copy it to the downloads folder :-)

Nico