Re: [tup] Will tup ever support writing output to a different directory?

[Mike Shal]

Hi Matt,

On Wed, Jan 16, 2013 at 3:34 PM, Matt Oswald <mos...@gmail.com> wrote:
> I find it counter intuitive that a Tupfile lives with the output, rather
> than the source. If I'm mimicking a more Windows-centric (Visual Studio)
> build, I've got to create my Tupfile in an intermediate folder and another
> in an output folder. I'd find it a lot easier to have one Tupfile in the
> source folder that builds the source into another folder, then links that
> source into a final executable in yet another folder.
>
> Will this scenario ever be directly supported?

Personally I find it easier to figure out what's going on with the
Tupfile in the same directory - want to know how files are created
here? Just open the Tupfile!

But, I understand that there are cases where outputting files to
different directories can be useful (re: the threads on supporting
install, probably putting a Tupfile in /usr/bin won't be a good
solution). It's not impossible to do so, but the primary difficulty
lies in how tup parses directories, when it creates nodes in the
database, and how wildcards are handled. I think a different (possibly
backwards incompatible) approach would be needed, but I don't know for
sure.

For example, consider this setup:

lib/Tupfile:
: |> ... |> foo.o
: |> ... |> bar.o
: *.o |> link |> lib.a
: |> ... |> not_included.o

other/Tupfile:
: |> ... |> ../lib/other.o

Tup may parse lib/Tupfile first, and so when it goes to resolve the
*.o in the link line, it will be looking at a combination of the
file-system (which probably just has foo.c and bar.c), as well as the
outputs of rules in previous lines (foo.o and bar.o). Note that
not_included.o is not included in the library, since it is listed
after the link line. This is true even if not_included.o has already
been compiled and is in the file-system.

Now, if you have other/Tupfile that is also generating a .o file in
lib/, where should it be generated? One option is to put them all at
the 'end', so other.o is like not_included.o and doesn't get linked
in. But, that makes it difficult to actually put things together
(which is presumably what you'd want to do). Another option is to put
them all first, so *.o resolves all objects coming from other
Tupfiles. This could work if the *.o resolution was moved out of the
parser and put into the updater when the command is dispatched.
However, I don't know of a way to easily exclude not_included.o if
this approach is taken.

If instead we change the meaning of *.o to mean 'any object in the
directory' rather than 'any object created until this point in the
Tupfile', then this example would link foo.o, bar.o, not_included.o,
and ../lib/other.o, which is probably the most sensible in this case.
However, this breaks the shell-script like top-down readability of
Tupfiles. It would also break things like:

: foreach *.c |> gcc ... |> %B.o
: *.o |> link ... |> file_generator
: file_generator |> ./file_generator > %o |> generated_file.c

Since this would then create a circular dependency (generated_file.c
-> generated_file.o -> file_generator -> generated_file.c)

What do you think?
-Mike

[Tal Hadad]

> Hi Matt,

>
> On Wed, Jan 16, 2013 at 3:34 PM, Matt Oswald <mos...@gmail.com> wrote:
> > I find it counter intuitive that a Tupfile lives with the output, rather
> > than the source. If I'm mimicking a more Windows-centric (Visual Studio)
> > build, I've got to create my Tupfile in an intermediate folder and another
> > in an output folder. I'd find it a lot easier to have one Tupfile in the
> > source folder that builds the source into another folder, then links that
> > source into a final executable in yet another folder.
> >
> > Will this scenario ever be directly supported?
>
> Personally I find it easier to figure out what's going on with the
> Tupfile in the same directory - want to know how files are created
> here? Just open the Tupfile!
>
> But, I understand that there are cases where outputting files to
> different directories can be useful (re: the threads on supporting
> install, probably putting a Tupfile in /usr/bin won't be a good
> solution). It's not impossible to do so, but the primary difficulty
> lies in how tup parses directories, when it creates nodes in the
> database, and how wildcards are handled. I think a different (possibly
> backwards incompatible) approach would be needed, but I don't know for
> sure.

I want to give another example for this:
If a user have in /src this dir tree:

/src
   utils
   ui
   input
      keyboard
      mouse
      touch
      joystick
   output
      display
      audio
   backend
       windows
       unix
In each dir, there are .C and .H files.
Some users want to center all the .C|>CC|>.O to one tupfile!
They(and I personally) don't want to create individual tup files for each directory and write all the .C files over again.
It's more logical(to some users) to have all the C files listed in $FOO_SRC variable like automake style! Then, the user can use the useful foreach keyword.
The only way to achieve this in tup, is giving up to the fact that all generated .O files will be in the main /src dir, which is very inconvenient.

I think that Tup should calculate all this circularity, even if it complicated.
Devs use auto-tools because they let them do complicated things more easy.

And about the circular dependency, I don't think it's a real problem- Tup can track this and print an error about this.
This means Tup should read all Tupfiles first, build a valid build tree, and built it if no error found.

Yours
Tal

[Freddie Chopin]

W dniu środa, 30 stycznia 2013 09:48:45 UTC+1 użytkownik Tal napisał:

Some users want to center all the .C|>CC|>.O to one tupfile!

What's the problem?

SRCS = utils/*.c ui/*.c input/keyboard/*.c ...

foreach SRCS |> CC |>

 

They(and I personally) don't want to create individual tup files for each directory and write all the .C files over again.

In case you have tupfile in each directory each of them looks EXACTLY THE SAME, as all they have is a wildcard-rule:

foreach *.c |> CC |>

There may be differences only if the process of compiling one folder of files is in fact different from compiling another one...

 

It's more logical(to some users) to have all the C files listed in $FOO_SRC variable like automake style! Then, the user can use the useful foreach keyword.
The only way to achieve this in tup, is giving up to the fact that all generated .O files will be in the main /src dir, which is very inconvenient.

First of all, you could have some output folder specificaly for *.o files and have your tupfile there, but the second solution is using variants (which unfortunately don't work on Windows for now).

 

I think that Tup should calculate all this circularity, even if it complicated.

I wonder how exactly would you do that...

The *.a thing is a minor inconvenience and there's no other way in Tup - as Mike said there would be no way to "remove" the object from the archive in case the rule (or the source) was removed, which defeats the main purpose of Tup - being correct.

 

Devs use auto-tools because they let them do complicated things more easy

But tup is a replacement for make, not auto-tools...

 

And about the circular dependency, I don't think it's a real problem- Tup can track this and print an error about this.
This means Tup should read all Tupfiles first, build a valid build tree, and built it if no error found.

It was explained somewhere before - Tup is fast, because it DOES NOT scan all the directories on each run. With your approach Tup would not be so fast.

Currently tup assumes that one "folder" is governed only by "one" tupfile. So if one tupfile changes tup has to check only one folder. If tup want's to build out.exe and it knows that it depends only on 1/file.o and 2/file.o it has to check only these two Tupfiles in 1/ and 2/, as no other tupfile can mess with files in these two folders. With your approach a change of one tupfile would result in re-parsing of ALL tupfiles that exist, COMPLETE rebuild (or at least verification) of build tree and so on. Sure it doesn't make a difference in the example you gave, but now imagine that there are no 10 folders, but 100000 folders - how long would a pointless scan through all the dirs take?

1. Tup is _build_ system

2. Main goals of tup are speed and corectness

With 1. in mind you will not seek for a way to do "install" with it. With 2. in mind you have to understand that each of these goals has potential downsides and limitations, as nothing is free.

To have all the non-goals of tup just use regular Makefile for them - make install, make doc, make whatever just acts as before, but make all has just one command - "tup upd". I'm using a setup like this, I've replaced only the actual build of make with tup, but make still does all the things that don't fit tup model very well - in my project that's creating and checking for a proper "context" (moving of folders around, symlinks and stuff like that) before each build (you could say that's it is something between bootstrap and configure step). It works pretty well (;

4\/3!!

[Oren Ben-Kiki]

There is a category of use cases - "generate a mirror hierarchy containing intermediate files, then combine all these intermediate files to a final target" - which (I think) today isn't supported "well".

Specifically, I have a documentation tool that take as sources every `src/.../file.whatever` and generates as output `chunks/src/.../file.whatever.html`. The "chunks" are then all combined together to generate a final unified everything-included-in-it `combined.html` file.

"make Clean" in this case means `rm -rf chunks`. The target directories don't even exist before building the documentation.

Ideally what I'd like to say is something like `foreach src/** |> generate-chunk %f |> chunks/%f` and then `chunks/** |> combine-chunks |> combined.html` (assuming "**" means "any possibly nested sub-paths"). 

This wouldn't work today because:

(1) outputs must be in the same directory as the `Tupfile`. If I understand correctly, the workaround is to manually create a `chunks/...` directory for each `src/...` directory, and populate it with a (one/two liner?) `Tupfile`. Ouch,

(2) globs don't have the full power of `find`. I guess the workaround would be to manually list all the `src` directories in the `combine-chunks` line, probably generating it using a script. This isn't too bad, especially if the Lua integration comes through. But it seems somewhat arbitrary.

(3) the `chunks` directories will not be automatically cleaned when they should (e.g. if I rename `src/old` to `src/new`, an empty `chunks/src/old` directory will remain on the disk). I don't know of any workaround for this..., except possibly manually running a `remove empty directories` command. Luckily `git` will just ignore directories that don't contain any tracked files, and of course I can add `chunks` to my global `.gitignore`, so these directories are "mostly harmless".

I assume one would face similar problem if trying to generate `.class` files in a filesystem hierarchy which is separate from the hierarchy containing the `.java` files, and then pack them into a `.jar` file (this isn't the standard practice but it makes sense in some contexts). It is possible to invoke `javac` on one `.java` file at a time, if one so chooses :-)

This category of use cases is a bit more "structured" than simply "generating some files in an arbitrary directory", so it may be easier to solve. E.g., if `tup` somehow kept track of "generated directories", each of which had (say) a single "source directory", probably this wouldn't impact the basic logic of the system. Maybe.

[lst...@gmail.com]

Though I don't have a proposal as of yet, I just thought I'd chime in to indicate that this was the second issue I ran into (after trying to capture shell output) trying to convert some of my Makefiles to Tupfiles.

One pattern that we use in our current system: we produce several binaries that ship as part of a package. Each binary is built within its project's directory, then stripped and placed in the package directory. Subsequently, the package directory gets zipped up and made available for download.

Another simpler pattern (already mentioned here): given a project in the directory "myproj", we often keep source files in "myproj/src". It would be ideal to be able to place a generated file into "myproj/src" along with the other existing source files.

I'll try to think about this a bit more - increased flexibility for output locations would be very convenient, if possible.

Cheers,

Liam