Rule with unknowable collection of output files

[Miles Steele]

Hi,

I'm trying to use tup in a partially-java project. My issue is that javac will produce multiple .class per .java input and I have no way of knowing in advance what exactly their name will be.

The input files for the problematic rule are:

    build-tup/src/grammars/DecafParser.java

    build-tup/src/grammars/DecafScanner.java 

The output files definitely include: (notice 'src' is not in the path this time, sorry about the confusing dir structure)

    build-tup/grammars/DecafParser.java

    build-tup/grammars/DecafScanner.java 

The other output files are:

    tup error: File '/build-tup/grammars/DecafScanner.class' was written to, but is not in .tup/db. You probably should specify it as an output

    tup error: File '/build-tup/grammars/DecafScanner$DFA9.class' was written to, but is not in .tup/db. You probably should specify it as an output

    tup error: File '/build-tup/grammars/DecafParser.class' was written to, but is not in .tup/db. You probably should specify it as an output

    tup error: File '/build-tup/grammars/DecafParser$str_literal_return.class' was written to, but is not in .tup/db. You probably should specify it as an output

    ... and many more!

How can I instruct tup to use build-tup/grammars as the output from the tup rule. Or, failing that, could tup consider only DecafScanner.class and DecafParser.class and ignore the rest of build-tup/grammars/*.class? Or am I thinking about this wrong and is there another more tup-friendly solution?

I currently get this error:

    * 1) javac -cp build-tup:vendor/antlr.jar -d build-tup build-tup/src/grammars/DecafParser.java build-tup/src/grammars/DecafScanner.java                                                                                                       

     *** tup errors ***

    tup error: Directory '/home/miles/Documents/15th_Grade/6.035/damaj/build-tup/grammars' was created, but not subsequently removed. Only temporary directories can be created by commands.

     *** Command ID=654 ran successfully, but tup failed to save the dependencies.

This is my Tupfile so far:

    # scanner .g -> .java

    : src/grammars/DecafScanner.g \

        |> java -cp vendor/antlr.jar org.antlr.Tool -o build-tup -debug %f \

        |> build-tup/src/grammars/DecafScanner.java build-tup/DecafScanner.tokens

    # parser .g -> .java

    : src/grammars/DecafParser.g | build-tup/DecafScanner.tokens \

        |> java -cp vendor/antlr.jar org.antlr.Tool -o build-tup -debug %f \

        |> build-tup/src/grammars/DecafParser.java build-tup/DecafParser.tokens

    # antlr .java -> .class

    : build-tup/src/grammars/*.java \

        |> javac -cp build-tup:vendor/antlr.jar -d build-tup %f \

        |>

Thanks,

- Miles

[Freddie Chopin]

The thing you ask for is not currently possible with tup - all output
files have to be stated explicitly and there's nothing you can do to
change that.

If the files created are not needed, you can delete them in the same tup
rule and it will work fine. Sth like:

: input.ext |> compile %f %o && rm -rf
<files-or-directories-you-don't-need> |> output.ext

One tup rule is allowed to create any number of files with random names
as long as all that are _NOT_ stated as output are removed when the
execution ends.

Regards,
FCh

[Neil Shepperd]

Hi,

My approach for building java projects with tup is to work in terms of jar files bundling collections of classes. It's not perfect, since it somewhat reduces the granularity of the dependency resolution, but it avoids a lot of fussing about. The scripts I use are uploaded at https://github.com/nshepperd/tup-java-tools - hopefully they haven't bitrot yet... There's some android scripts as well, which you can ignore. But basically instead of calling javac directly from tup, I have a python script which calls javac then wraps all compilation products into a jar file. This way you get a single known output file for each command.

For example, given a set of three java sources {A,B,C}.java, where A and B are independent but C could import A and B, I would do:

(with !javac = |> $(TOOLS_HOME)/javac.py -o %o %f |>)
: src/A.java |> !javac |> src/A.jar
: src/B.java |> !javac |> src/B.jar
: src/C.java src/A.jar src/B.jar |> !javac |> src/C.jar

This results in three jar files, containing the classes output from A.java, B.java, C.java respectively. You could then combine the products into one file (analogous to a static library) with mashjar.py:

(!mashjar = |> $(TOOLS_HOME)/mashjar.py -o %o %f |>)
: src/*.jar |> !mashjar |> everything.jar

Alternatively, you could compile everything at once:

: src/*.java |> !javac |> everything.jar

This is the less granular but easier approach.

These scripts all support a "-e entry.Point" option for defining the entry point to an executable jarfile, but other than that they're qute dumb and could do with extension -- for example, currently there's no way to supply compiler flags like "-debug". Also if you try to compile without having javac and fastjar both available in your PATH I think these scripts will just fail mysteriously. But this at least gives you the idea of how you can handle this.

Hope this helps-

Regards,

Neil

[Spooge]

On 11/02/2014 02:56 PM, Miles Steele wrote:
> Or, failing that, could tup consider only DecafScanner.class and
> DecafParser.class and ignore the rest of build-tup/grammars/*.class?

IIRC you can create directories as long as you delete them as part of
the same command. You could use that to work around this:

1. Create a tmp directory
2. Output all .class files in there
3. Copy the ones you need out
4. Delete the tmp directory

That should be simple enough to abstract with some variables or
something, but if not I think you could encapsulate it easily enough in
a Lua function.

[Miles Steele]

Ok, I see. Thanks for those very clear explanations and workarounds. I'll try some of them when I get the time to play with this again. It's too bad tup doesn't support marking a directory as completely untracked.