Supporting Java compilation in ninja
Elazar Leibovich
source files in a certain package in order to compile it. It must
compile all those files together.
For example, if I have a.java and b.java, I'll have to write
# javac a.java b.java
# find -name \*.class
a.class
b.class
However, it produce as many class files as source files in this phase.
Is there a way to support such dependency tracking in ninja now? (Why
not use Java native tools? Since on a huge C++ project, with little
Java interop classes, it adds too much overhead).
The best solution I could think of, is, compiling everything to a
single jar, and never let ninja know there were classes. However if
you actually need the class files, I see no solution.
Evan Martin
> Java's compiler has an interesting feature. It requires all Java's
> source files in a certain package in order to compile it. It must
> compile all those files together.
>
> For example, if I have a.java and b.java, I'll have to write
>
> # javac a.java b.java
>
> # find -name \*.class
> a.class
> b.class
>
> However, it produce as many class files as source files in this phase.
> Is there a way to support such dependency tracking in ninja now?
Can you elaborate on the problem? It seems what you've described is a
build file that has a.java and b.java as inputs to a build line that
produces a.class and b.class.
(I'm sorry for not being too familar with Java, so you might need to
explain seemingly-obvious details to me.)
Elazar Leibovich
Can you elaborate on the problem? It seems what you've described is a
build file that has a.java and b.java as inputs to a build line that
produces a.class and b.class.
(I'm sorry for not being too familar with Java, so you might need to
explain seemingly-obvious details to me.)
Evan Martin
> The basic problem is, that ninja assumes each command have a single output
> file,
Ah, here is the problem. This isn't correct. You can list multiple
files before the colon in a build statement.
> however the javac command have multiple ouput files. A .class file for
> each input file.
> We could do
> build a.class: javac a.java b.java
> but then ninja wouldn't know about b.class.
> We could do
> build a.class: javac a.java b.java
> build b.class: javac a.java b.java
build a.class b.class: javac a.java b.java
By the way, the code involving multiple outputs isn't exercised as
frequently, so there may be bugs in it. If you encounter any
surprising semantics with it, please ask whether the behavior is
intended before worrying too much about whether it makes sense.
Evan Jones
> Java must compile all java files in the current directory at once! It cannot compile just a.java, and then just b.java. It must compile both simultaneously.
Its been a while since I've fought with Java compiles, but I don't recall this being true. I just tested it really quickly, and it *does* allow you to just compile one .java file in a directory (or at least it did in my tiny test). Is there some additional options or something that I need, in order for javac to have this behaviour?
HOWEVER: javac does (by default) implicitly compile any dependencies. I created a class One, and a class Two. Class Two uses One. If I compile One, it compiles that single class. If I compile Two, it compiles both:
Yamnuska:~ ej$ javac Two.java
Yamnuska:~ ej$ ls *.class
One.class Two.class
This probably makes writing ninja files tricky, as you ideally need to get this dependency information, and looking at javac -help doesn't make it clear that there is any easy way to do that (maybe by parsing the output from -verbose? but -verbose is very verbose, so that seems unfortunate).
Worse: My understanding is that javac performs MUCH better if you give it all the .java files to compile at once, as it then only loads and parses each file once. This would be tricky to do with Ninja, at least not without writing a bunch of additional support code.
My conclusion: you can probably use ninja to build java code correctly, but it probably will involve writing some additional tools / scripts to do it well.
Evan
Evan Martin
> My conclusion: you can probably use ninja to build java code correctly, but it probably will involve writing some additional tools / scripts to do it well.
Well, using this:
https://github.com/martine/ninja/blob/master/misc/ninja_syntax.py
It'd be something like this:
import ninja_syntax
n = ninja_syntax.Writer(open('build.ninja', 'w'))
n.rule('javac', command='javac $in', description='JAVAC $in')
def javac(*basenames):
return n.build([b + '.class' for b in basenames], 'javac', [b +
'.java' for b in basenames])
That would then let you write the rest of your build with statements like:
classfiles = javac('Foo', 'Bar', 'Baz')
exe = ...some other function that build executables...(classfiles)
Where Foo/Bar/Baz are the names of your source .java files.
Doesn't seem so bad to me. I guess javac is likely doing some of the
same work as ninja again to determine which files to actually build,
but javac's problem is much simpler and smaller than ninja's and that
info is likely to be in cache anyway, so it shouldn't cost much.
Elazar Leibovich
On Nov 6, 2011, at 14:00 , Elazar Leibovich wrote:Its been a while since I've fought with Java compiles, but I don't recall this being true. I just tested it really quickly, and it *does* allow you to just compile one .java file in a directory (or at least it did in my tiny test). Is there some additional options or something that I need, in order for javac to have this behaviour?
> Java must compile all java files in the current directory at once! It cannot compile just a.java, and then just b.java. It must compile both simultaneously.
HOWEVER: javac does (by default) implicitly compile any dependencies. I created a class One, and a class Two. Class Two uses One. If I compile One, it compiles that single class. If I compile Two, it compiles both:
My conclusion: you can probably use ninja to build java code correctly, but it probably will involve writing some additional tools / scripts to do it well.
Rachel Blum
Elazar Leibovich
That's happening because your test setup has the wrong file names :)
Evan Jones
> That would then let you write the rest of your build with statements like:
> classfiles = javac('Foo', 'Bar', 'Baz')
> exe = ...some other function that build executables...(classfiles)
> Where Foo/Bar/Baz are the names of your source .java files.
If I understand your proposal correctly here, this would generate a single invocation of javac with all .java files on the command line, right? That's fine for full builds, but not ideal for incremental rebuilds. My understanding is that ideally for incremental Java rebuilds, you want a two-pass system:
a) Traverse the dependency graph to determine which .class files are out of date (identically to what would happen for C++)
b) Issue a single javac command line with the out-of-date .java files.
(optional): Parallelize (b) by splitting the list (intelligently?) into pieces according to the number of available CPUs. However, javac is fast enough that I haven't needed to do this on any of my projects yet.
Ninja's design makes (a) relatively easy, but (b) is still hard.
You can also run javac as a server, which would make it more efficient with Ninja's typical "build one file at a time" mode.
I may actually be working with Java again in the next week or two, so I might end up trying to get ninja to play nice with Java.
Evan
Evan Martin
> If I understand your proposal correctly here, this would generate a single invocation of javac with all .java files on the command line, right? That's fine for full builds, but not ideal for incremental rebuilds. My understanding is that ideally for incremental Java rebuilds, you want a two-pass system:
>
> a) Traverse the dependency graph to determine which .class files are out of date (identically to what would happen for C++)
>
> b) Issue a single javac command line with the out-of-date .java files.
Hm, interesting. I suppose we could expose an $in_dirty that is the
subset of $in that we think needs rebuilding...
...oh, but wait, it's the *output* files that are dirty. So we'd need
a way to map from those back to the corresponding input names. I
really don't want to go down the path of a weak programming language.
Heh, something like
javac `echo $in_dirty | sed -e s/.class/.java/`
might work, though not on Windows.
> (optional): Parallelize (b) by splitting the list (intelligently?) into pieces according to the number of available CPUs. However, javac is fast enough that I haven't needed to do this on any of my projects yet.
This one is definitely harder. It sorta reminds me of the problem of
splitting the list of input files when the command line is too long.
I wonder if there's a way to provide both in some nice simple way.
> You can also run javac as a server, which would make it more efficient with Ninja's typical "build one file at a time" mode.
>
>
> I may actually be working with Java again in the next week or two, so I might end up trying to get ninja to play nice with Java.
How much does it matter, in the end? Does javac not do a "compare
output file time to input file time and only build if necessary"
decision? I wonder if another way around all of this is to use a
javac wrapper that does the above computations for you...
to_rebuild = []
for dst in sys.argv[1:]:
src = os.path.splitext(dst)[0] + '.class'
if os.path.getmtime(src) > os.path.getmtime(dst):
to_rebuild.push(src)
subprocess.check_call(['javac'] + to_rebuild)
However, getting it to parallelize well would require exposing some
Ninja internals. My first guess would be to examine or implement the
make jobserver API1], but I would hope there was something more simple
available. Like maybe Ninja could support a helper command emitting a
list of command lines that need to be run and then it would enqueue
those into the existing system of running them in parallel.