Sample vala+ninja project

[Emmanuel Oga]

Hi,

I created a sample vala project built with ninja. You can check it out on github:

https://github.com/EmmanuelOga/valaninja

as comparison, here is the same thing built with tup:

https://github.com/EmmanuelOga/valatup

To be fair, I'm using a full blown programming language to build the ninja configuration file, while the tup project uses just Tupfiles and a very small shell script to do the same thing. But tup has the limitation that all the build artifacts must be output to the same directory where the source files are, something I don't like very much.

Let me know if you see any easy improvements to be done. 

[Evan Martin]

On Sat, Feb 18, 2012 at 3:11 PM, Emmanuel Oga <emman...@gmail.com> wrote:
> I created a sample vala project built with ninja. You can check it out on
> github:
>
> https://github.com/EmmanuelOga/valaninja

Cool! I don't know a whole lot about vala, but I took a look at
https://github.com/EmmanuelOga/valaninja/blob/master/build/vala.ninja

It appears you generate a rule per source file (?). I'm not sure if
it's required by valac, but you could express equivalent to what's in
that build file with something like the following. The difference is
you can pass the "vapi" value into the rule by defining its value per
build statement.

rule valac
description = valac compilation to .c files
restat = true
command = valac --pkg gtk+-3.0 -C $in -d _obj $
--use-fast-vapi=$vapi

build _obj/src/main.c: valac src/main.vala | $
_obj/src/main.vapi $
_obj/src/windows/sync_sample_1.vapi
vapi =_obj/src/windows/sync_sample_1.vapi

build _obj/src/windows/sync_sample_1.c: valac src/windows/sync_sample_1.vala | $
_obj/src/main.vapi $
_obj/src/windows/sync_sample_1.vapi
vapi = _obj/src/main.vapi

Additionally, if you mean for all .vapi files to be implicit inputs to
all valac calls, rather than listing out all .vapi files as implicit
inputs to each valac, you can represent it with a stamp file as
follows.

First make vapi.stamp get rebuilt if any vapi file changes:

rule stamp
command = touch $out
build _obj/vapi.stamp: _obj/src/main.vapi _obj/src/windows/sync_sample_1.vapi

Then pass _obj/vapi.stamp as an implicit input to each valac "build"
line, like this.

build _obj/src/windows/sync_sample_1.c: valac src/windows/sync_sample_1.vala | $
_obj/vapi.stamp
vapi = _obj/src/main.vapi

Finally, your link line looks like:
build _obj/main: ccbin _obj/src/main.o
_obj/src/windows/sync_sample_1.o | _obj/src/main.o
_obj/src/windows/sync_sample_1.o

You can drop the text from "|" onward.

None of these are likely to have much of a performance impact either
way unless your project gets huge, but otherwise they all are slightly
more efficient. And less code is always nice.

>
> as comparison, here is the same thing built with tup:
>
> https://github.com/EmmanuelOga/valatup

Ah, here I saw the link to some vapi-related docs.
https://live.gnome.org/Vala/Documentation/ParallelBuilds says:

It says:
"Note that fast-vapi files are generated solely from the given source
file -- no other source files or even system packages are consulted.
Also note that the timestamp of the output file is only modified in
the case that the contents actually change."
That latter sentence is exactly what the "restat" attribute is for in Ninja.

[Emmanuel Oga]

Using restat = true and a variable for the vapis I could create a single rule for the .vala to .c conversion, which is great.

https://github.com/EmmanuelOga/valaninja/blob/80b98eb7a2c0746795d7c77375d52917e9d0824e/build/vala.ninja

Thanks for the advice! 

[Emmanuel Oga]

So talking with Mike from the tup project:

https://groups.google.com/d/topic/tup-users/ge_4pgDNji4/discussion

He made a fair point: with ninja you need to rebuild the config file whenever the src files change, unless I'm missing something?

That's ok with me, I tried to implement this scheme:

* generate a manifest of files with a script which only overwrites the manifest if the files actually changed

* have ninja re build the ninja configuration if the manifest changes

* run ninja again with the generated ninja file

There are two problems that I probably solved in the wrong way: how to make ninja always run the manifest command? My solution was to touch a file on ninja and remove it inside the manifest generator.

The second problem was that it seems like you cannot include or subninja a file that you are generating while running the build. I think that's ok, but maybe there is a better way to do this than:

ninja > /dev/null; ninja -f build/vala.ninja

Anyway, here is what I came up with:

https://github.com/EmmanuelOga/valaninja/commit/d3237da7ce135d80a8dfce667c5901bb3017ceb8

 

[Evan Martin]

On Mon, Feb 20, 2012 at 12:40 PM, Emmanuel Oga <emman...@gmail.com> wrote:
> So talking with Mike from the tup project:
>
> https://groups.google.com/d/topic/tup-users/ge_4pgDNji4/discussion
>
> He made a fair point: with ninja you need to rebuild the config file
> whenever the src files change, unless I'm missing something?

It appears the change he made was to rename a source file.
In that case, yes, you must rebuild the config file.

His principle, that there must only be one command to build a project,
is a good one. However it's incompatible in the limit with Ninja's
goal of performance: Ninja will never support wildcard rules because
it would mean Ninja would need to scan directories to run.

For what it's worth, this is also true of other build systems like
autotools. I agree it's inconvenient though.

> That's ok with me, I tried to implement this scheme:
>
> * generate a manifest of files with a script which only overwrites the
> manifest if the files actually changed
> * have ninja re build the ninja configuration if the manifest changes
> * run ninja again with the generated ninja file
>
> There are two problems that I probably solved in the wrong way: how to make
> ninja always run the manifest command? My solution was to touch a file on
> ninja and remove it inside the manifest generator.
>
> The second problem was that it seems like you cannot include or subninja a
> file that you are generating while running the build. I think that's ok, but
> maybe there is a better way to do this than:
>
> ninja > /dev/null; ninja -f build/vala.ninja

Try setting the "generator" attribute on the rules that generate the
build files. That will cause ninja to reboot the build when it
regenerates the build files. I'm not sure if you'll get into an
infinite loop though.

> Anyway, here is what I came up with:
>
> https://github.com/EmmanuelOga/valaninja/commit/d3237da7ce135d80a8dfce667c5901bb3017ceb8

I admit I haven't given much thought to how to make Ninja always
regenerate the build files, because it's not really the point of
Ninja. I am philosophically opposed to "always-run" build commands
because they impact build performance. With that said, there are
probably ways to make it work (like the one you found).

If you really want to run that Ruby script every time you build, Ninja
is no longer buying you too much. You might as well have that Ruby
script run the valac commands itself. Ninja is mostly useful for
making a specific case -- a project with a huge number of files --
rebuild quickly.

PS: rather than "ninja -f build/vala.ninja", you might consider
generating different paths in your build files to make "ninja -C
build" work. That also matches how autotools works (where you run
"make" from within the build directory).

[Evan Jones]

On Feb 20, 2012, at 16:50 , Evan Martin wrote:
> His principle, that there must only be one command to build a project,
> is a good one. However it's incompatible in the limit with Ninja's
> goal of performance: Ninja will never support wildcard rules because
> it would mean Ninja would need to scan directories to run.

I'm curious: is the extra cost of the readdir() that bad? I have to imagine that in most cases, it wouldn't be. I guess difference between:

- Stat 1000 files in 100 directories

and

- Stat 1000 files
- Readdir 100 directories

is probably something like 10% of a no-op build, since the readdir is probably a similar cost as the stats?

I'm asking because I've been using ninja in a cross-language project that involves Java, and the Java build model and ninja's build system don't really work very well for incremental builds. I may end up hacking up a tool that does the readdirs() on every build, in order to allow people to just drop new .java source code in the right directories.

Evan

--
http://evanjones.ca/

[Evan Martin]

On Mon, Feb 20, 2012 at 5:23 PM, Evan Jones <ev...@csail.mit.edu> wrote:
> On Feb 20, 2012, at 16:50 , Evan Martin wrote:
>> His principle, that there must only be one command to build a project,
>> is a good one.  However it's incompatible in the limit with Ninja's
>> goal of performance: Ninja will never support wildcard rules because
>> it would mean Ninja would need to scan directories to run.
>
> I'm curious: is the extra cost of the readdir() that bad? I have to imagine that in most cases, it wouldn't be. I guess difference between:
>
> - Stat 1000 files in 100 directories
>
> and
>
> - Stat 1000 files
> - Readdir 100 directories

Thanks for asking! You make a very good point. Perhaps this could be
made to work...

I worry about needing to go down the route of the mini-programming
language you see in other build systems, where you can say "for each
input X, I need the basename of X, plus some other string, plus the
filename suffix of X", and once you start producing basenames and
suffixes you end up wishing you had a real programming language.

But, with that said, I could see making some sort of "directory node"
concept where you could say that foo/bar.ninja depends on foo/*.cc,
instructing Ninja to re-run the build file generator when the set of
inputs change. Much like what Emmanual is seeking.

> I'm asking because I've been using ninja in a cross-language project that involves Java, and the Java build model and ninja's build system don't really work very well for incremental builds. I may end up hacking up a tool that does the readdirs() on every build, in order to allow people to just drop new .java source code in the right directories.

Effectively, you'd need to run "ls -R > new-manifest; diff
old-manifest new-manifest" on each run. You might just as well put
that in a shell script around Ninja for now.

[Nicolas Desprès]

On Tue, Feb 21, 2012 at 6:53 AM, Evan Martin <mar...@danga.com> wrote:

[...] 

But, with that said, I could see making some sort of "directory node"
concept where you could say that foo/bar.ninja depends on foo/*.cc,
instructing Ninja to re-run the build file generator when the set of
inputs change.  Much like what Emmanual is seeking.

Nobody write .ninja files by hand. Only generators do, so why do you consider wildcard necessary? The less feature, the better :-)

 

-Nico

[Emmanuel Oga]

This is just seeking a way to know _when_ to _regenerate_ the ninja config file automatically.

Right now, if you add or delete a file you need to _remember_ to re run the configure script. In general, having a single command you need to run even if something changes on your build is a good thing.

Something like this could work:

rule build_config

  command = ./configure

  manifest = src/**/*.vala

build build.ninja: build_config $build_config.manifest

Where the manifest configuration on the rule is a man 7 glob, and $build_config.manifest is a ninja generated variable that represents the manifest for that rule.

ninja would generate the manifest as my little script does, and build build.ninja whenever that manifest changes.

Does that sound too hideous? :p

[Andrew Wagner]

On Sunday, February 19, 2012 12:11:42 AM UTC+1, Emmanuel Oga wrote:

as comparison, here is the same thing built with tup:

https://github.com/EmmanuelOga/valatup

...

But tup has the limitation that all the build artifacts must be output to the same directory where the source files are, something I don't like very much.

Let me know if you see any easy improvements to be done. 

Recent versions of tup have lifted this restriction. You still have to explicitly list the outputs in the rule, but tup can now generate outputs in subdirectories (and even make the subdirectories and manage the .gitignore files...).   Parent directories might work too, although I haven't tried it yet personally.