Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Limitations and Irritations of Tup

1,309 views
Skip to first unread message

ad...@computerquip.com

unread,
Jan 29, 2013, 12:38:23 AM1/29/13
to tup-...@googlegroups.com
So far, I've enjoyed the use of tup. However, I've come across some rather rough scenarios that I think should be addressed, in order of priority:

1. Activating commands that might output in multiple directories is really unclean. 
Let's take make for example. Adding tup dependencies on the generated files for make are an absolute pain the ass, especially if they output to multiple directories. 
So, for now, I've learned to just bootstrap makefiles since using tup to do so does nothing for me. While tup could potentially check for files generated by makefiles, it can't currently. 

If one tool generates into multiple directories which directory Tupfile should call the tool that gives the output, especially if that tool can only be called once?
What I would like is to allow Tupfiles to depend on other Tupfiles to make sure a tool generated the correct files. This would mean only 
one Tupfile has to actually call the tool and then can depend on other Tupfiles to make sure the files generated. For instance, a dependency file might look like:

dependency boost 
check libboost_system.so

Where "dependency boost" declares that file a dependency of whatever makes boost. A dependent file might look like

depend boost
: |> ./bootstrap |> ./b2

Where "depend boost" will cause all dependency files to check for the generated files after all rules have been executed.

2. An individual rule re-generating the same file doesn't work. Take the following example:

: foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |> libts3qClient.so

This gives the following error: 

tup error: Unable to create a unique link from 42929 to 432 because the destination is already linked to by node 433.
 - [433] bin/[ar -r libts3qClient.so ../src/ts3qClient.o]
 - [42929] bin/[ar -r libts3qClient.so ../src/parser/cqLex.o]
 - [432] bin/libts3qClient.so
tup error: You may have multiple commands trying to create file 'libts3qClient.so'
tup error: Error parsing Tupfile line 5
  Line was: ': foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |> libts3qClient.so'
 [                                                                              ETA~=3s  Remaining=3380                                                                                ]   0%
 *** tup: 1 job failed.


I believe the expected functionality is that the last command to generate or modify the file is the only version of the file that should matter but should obviously fail if any command fails. 

3. Macros aren't very useful outside of reducing redundancy.
I would like to be able to assign to a variable and as such, use macros within macros, or next to other macros. 
Why am I still having to manually insert a "-I" before every include folder? Isn't this a sort of redundancy itself (and error prone)? Can't this be avoided through tup?

4. Why is a newline required
The last thing I'm worried about when building my project rules are whether or not each Tupfile has a newline at the end. 
I know for a fact parsing doesn't have to demand a newline to realize the end of a file. So why? It's my biggest irritation ever. 

Andrew Wagner

unread,
Jan 29, 2013, 3:26:28 AM1/29/13
to tup-...@googlegroups.com
On Tue, Jan 29, 2013 at 6:38 AM, <ad...@computerquip.com> wrote:
> 2. An individual rule re-generating the same file doesn't work. Take the
> following example:
>
> : foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |> libts3qClient.so
>
> This gives the following error:
>
> tup error: Unable to create a unique link from 42929 to 432 because the
> destination is already linked to by node 433.
> - [433] bin/[ar -r libts3qClient.so ../src/ts3qClient.o]
> - [42929] bin/[ar -r libts3qClient.so ../src/parser/cqLex.o]
> - [432] bin/libts3qClient.so
> tup error: You may have multiple commands trying to create file
> 'libts3qClient.so'
> tup error: Error parsing Tupfile line 5
> Line was: ': foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |>
> libts3qClient.so'
> [
> ETA~=3s Remaining=3380
> ] 0%
> *** tup: 1 job failed.
>
>
> I believe the expected functionality is that the last command to generate or
> modify the file is the only version of the file that should matter but
> should obviously fail if any command fails.

This use case with ar has been addressed in previous threads. I
didn't see an example with globbing, but ar can certainly merge
multiple .o files at a time. Some restrictions are good for the
sanity they enforce in your build configuration; my understanding is
that this "files can't depend on themselves" rule is one of them.

> 3. Macros aren't very useful outside of reducing redundancy.
> I would like to be able to assign to a variable and as such, use macros
> within macros, or next to other macros.
> Why am I still having to manually insert a "-I" before every include folder?
> Isn't this a sort of redundancy itself (and error prone)? Can't this be
> avoided through tup?

One of the reasons I personally find CMake infuriating to use is that
every little trivial thing is hidden behind macros. That's how they
achieve handling of all the little stupid platform dependent special
cases, but the cost is that many (I would say most) CMake users have
no idea what is going on under the hood and they just resort to trial
and error until things work.

> 4. Why is a newline required
> The last thing I'm worried about when building my project rules are whether
> or not each Tupfile has a newline at the end.
> I know for a fact parsing doesn't have to demand a newline to realize the
> end of a file. So why? It's my biggest irritation ever.

Including the newline at the end of every file allows your file to
work with unix utilities like "cat" more cleanly. This is a
tradition/convention thing. At least adding the newline doesn't hurt
anything, unlike the OS incompatibilities over what bytes represent a
line ending.

Ben Boeckel

unread,
Jan 29, 2013, 12:23:56 PM1/29/13
to tup-...@googlegroups.com
On Tue, Jan 29, 2013 at 09:26:28 +0100, Andrew Wagner wrote:
> On Tue, Jan 29, 2013 at 6:38 AM, <ad...@computerquip.com> wrote:
> > 3. Macros aren't very useful outside of reducing redundancy.
> > I would like to be able to assign to a variable and as such, use macros
> > within macros, or next to other macros.
> > Why am I still having to manually insert a "-I" before every include folder?
> > Isn't this a sort of redundancy itself (and error prone)? Can't this be
> > avoided through tup?
>
> One of the reasons I personally find CMake infuriating to use is that
> every little trivial thing is hidden behind macros. That's how they
> achieve handling of all the little stupid platform dependent special
> cases, but the cost is that many (I would say most) CMake users have
> no idea what is going on under the hood and they just resort to trial
> and error until things work.

Telling CMake to generate verbose Makefiles usually helps this problem
(or just doing make VERBOSE=1) to see what CMake thought you meant. Yes,
lots of platforms have silly little incompatabilities, but CMake also
supports quite a wide range of tools (from Sun's compiler to AIX
machines to MSVC) without the developer needing to know that rpath
support on platform X with compiler Y needs flags A, B, and C while Z
just needs D. For a fairly large project, the only platform-dependent
things I have in the build system are:

- some install path differences between Windows and not Windows;
- a section for adding some GCC flags versus MSVC flags;
- setting the extension for Python libraries to pyd on Windows (rather
than ${CMAKE_SHARED_MODULE_SUFFIX});
- path list separator characters (';' versus ':') (and configured
headers used to get the default paths for Windows because ';' is not
valid on the command line...even in "/DFOO=bar;baz"); and
- telling Boost to not do its auto-linking on Windows.

I don't need to care that MSVC uses /D rather than -D for defines, that
Windows needs .lib and .dll files generated for libraries, that *nix
platforms like to have a soversion while Windows does not, that the
build tree works best with rpaths in the libraries (but they should be
stripped when installed), and executing a process during the configure
stage (to extract git hashes, determine where NumPy is installed and its
headers live, etc.) is the same on Windows and other platforms.

I admit that CMake isn't perfect, but the things that it does for its
users in the face of obscure platforms is fairly substantial. Not to
mention that I can make the people who use Visual Studio happy while
being able to not have to touch a single vcxproj file[1].

It's nice that Tup is trying to make the build system cleaner and more
rigorous, but I believe that Tup might do well to try and hide these
ugly differences from the user as much as possible. The real world is
not pretty, especially when the goal of "cross platform" is involved
(which is fairly important in build systems).

--Ben

[1]I tend to use cmake --build on Windows, so I don't use VS more than I
need to check that things look as they should there (such as the list of
targets, that the target names aren't too long, and other issues that
have cropped up).

Mike Shal

unread,
Jan 29, 2013, 3:13:10 PM1/29/13
to tup-...@googlegroups.com
Hi admin,

On Tue, Jan 29, 2013 at 12:38 AM, <ad...@computerquip.com> wrote:
> So far, I've enjoyed the use of tup. However, I've come across some rather
> rough scenarios that I think should be addressed, in order of priority:
>
> 1. Activating commands that might output in multiple directories is really
> unclean.
> Let's take make for example. Adding tup dependencies on the generated files
> for make are an absolute pain the ass, especially if they output to multiple
> directories.
> So, for now, I've learned to just bootstrap makefiles since using tup to do
> so does nothing for me. While tup could potentially check for files
> generated by makefiles, it can't currently.

If you are trying to combine make and tup, it is probably easiest to
have a master script that does 'cd make-project; make; cd tup-project;
tup upd' or something. Although tup could in theory run make and know
what files it writes to, it is not trivial to do so without explicitly
listing the outputs before-hand. Writing files to other directories
also makes parsing / wildcarding difficult, as discussed in
https://groups.google.com/forum/?fromgroups=#!topic/tup-users/P2bDYj3MqdU

>
> If one tool generates into multiple directories which directory Tupfile
> should call the tool that gives the output, especially if that tool can only
> be called once?
> What I would like is to allow Tupfiles to depend on other Tupfiles to make
> sure a tool generated the correct files. This would mean only
> one Tupfile has to actually call the tool and then can depend on other
> Tupfiles to make sure the files generated. For instance, a dependency file
> might look like:
>
> dependency boost
> check libboost_system.so
>
> Where "dependency boost" declares that file a dependency of whatever makes
> boost. A dependent file might look like
>
> depend boost
> : |> ./bootstrap |> ./b2
>
> Where "depend boost" will cause all dependency files to check for the
> generated files after all rules have been executed.

That is an interesting suggestion, and may help with the wildcarding
issue for parsing. I don't think it will help with unspecified
outputs, though. Additionally, running make from tup (if I understand
correctly what you're trying to do) is unlikely to work how you'd want
- tup removes all outputs before re-running a sub-process. So if tup
were running make, it would remove all outputs, causing make to
completely rebuild everything. Since make is untrustworthy, that's
what it would have to do anyway.

>
> 2. An individual rule re-generating the same file doesn't work. Take the
> following example:
>
> : foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |> libts3qClient.so
>
> This gives the following error:
>
> tup error: Unable to create a unique link from 42929 to 432 because the
> destination is already linked to by node 433.
> - [433] bin/[ar -r libts3qClient.so ../src/ts3qClient.o]
> - [42929] bin/[ar -r libts3qClient.so ../src/parser/cqLex.o]
> - [432] bin/libts3qClient.so
> tup error: You may have multiple commands trying to create file
> 'libts3qClient.so'
> tup error: Error parsing Tupfile line 5
> Line was: ': foreach $(SRC)/*.o $(SRC)/parser/*.o |> ar -r %o %f |>
> libts3qClient.so'
> [
> ETA~=3s Remaining=3380
> ] 0%
> *** tup: 1 job failed.
>
>
> I believe the expected functionality is that the last command to generate or
> modify the file is the only version of the file that should matter but
> should obviously fail if any command fails.

No, this doesn't work, nor should it. Suppose you have 3 objects: 1.o,
2.o, and 3.o. What commands are you running to generate the library?

ar -r lib.a 1.o
ar -r lib.a 2.o
ar -r lib.a 3.o

If you add a symbol to 2.o, what commands would you expect to run to
generate a new library? Just 'ar -r lib.a 2.o'? What happens if you
instead remove 2.o? How are you removing it from the archive?

Just remove the foreach here and create the archive once with all the
files you want to put in it.

Also, unrelated to tup, I think creating an archive (using 'ar') with
an extension of a shared library ('.so') might confuse things, but I
don't have any direct evidence to support that.

>
> 3. Macros aren't very useful outside of reducing redundancy.
> I would like to be able to assign to a variable and as such, use macros
> within macros, or next to other macros.
> Why am I still having to manually insert a "-I" before every include folder?
> Isn't this a sort of redundancy itself (and error prone)? Can't this be
> avoided through tup?

Have you tried Rendaw's lua support at all? Does that help address any
of your concerns with the parser?

>
> 4. Why is a newline required
> The last thing I'm worried about when building my project rules are whether
> or not each Tupfile has a newline at the end.
> I know for a fact parsing doesn't have to demand a newline to realize the
> end of a file. So why? It's my biggest irritation ever.

Sorry, I had no idea this was a major issue. Please pull the latest -
it shouldn't complain about that anymore.

Thanks for the feedback,
-Mike

Andrew Wagner

unread,
Nov 19, 2013, 9:17:05 AM11/19/13
to tup-...@googlegroups.com
We been happily using tup in my group for a year now (Yaaay! Thanks
Mike!). By FAR the most painful restriction of tup is the need for
ugly hacks to handle rules that output a tree of files. We have been
able to survive by tricking tup with ugly hacks like shell scripts
that generate trees of symlinks and by building some targets outside
of the tup hierarchy and copying (just some) of their results back in,
but this is not pleasant to set up, and is error prone. These are not
exotic isolated use cases; pretty much anything involving
meta-programming, including very mainstream/standard things like
google protocol buffers, or compilers that output files in
subdirectories (i.e. haskell/ghc) are a huge pain to deal with in a
tup based project.

I have been a bit out of the tup loop; have any new techniques for
dealing with this arisen in the community, or have any new tup
features been added for it?

I know the "targets live next to their tupfile" assumption is deeply
ingrained in tup, but pretty much ~any implementation that is provided
with tup is bound to be better than the ugly hacks we (and presumably
many others) are resorting to. I feel like there has to be some
(perhaps equally strong) assumptions that could be made to make this
feasible to implement in tup, i.e. "for targets that generate a tree
of files, you can only have one tuprule in one Tupfile in that
directory, you have to use xyz special syntax, there can be no
subdirectories with tupfiles, and you should expect the rule to be
re-run frequently if targets in other directories depend on any of the
generated files, and warnings will be thrown whenever the set of
generated files changes"

Thanks a lot!
Andrew


On Tue, Jan 29, 2013 at 6:38 AM, <ad...@computerquip.com> wrote:
> --
> --
> tup-users mailing list
> email: tup-...@googlegroups.com
> unsubscribe: tup-users+...@googlegroups.com
> options: http://groups.google.com/group/tup-users?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "tup-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tup-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

Andrew Wagner

unread,
Nov 22, 2013, 9:59:50 AM11/22/13
to tup-...@googlegroups.com
That last message can be disregarded; in recent versions of tup,
outputs need not be next to their tupfiles.

venkatakrishnarao ks

unread,
Jun 8, 2014, 6:00:13 AM6/8/14
to tup-...@googlegroups.com
Have I got it right that the User manual is then out of date? It still states:

You must create a file called "Tupfile" anywhere in the tup hierarchy that you want to create an output file based on the input files. The input files can be anywhere else in the tup hierarchy, but the output file(s) must be written in the same directory as the Tupfile.

Iain Merrick

unread,
Sep 25, 2014, 10:53:20 AM9/25/14
to tup-...@googlegroups.com
Hi all,

Sorry to resurrect an ancient thread but this question still looks relevant! I'm considering switching a biggish project (Android app with lots of generated data+code) over to tup. Reading the manual and skimming this mailing list, the difficulty of generating a mirror hierarchy is potentially a big problem. Apart from that I'd love to use tup as it looks terrific.


That last message can be disregarded; in recent versions of tup,
outputs need not be next to their tupfiles.

Have I got it right that the User manual is then out of date? It still states:

You must create a file called "Tupfile" anywhere in the tup hierarchy that you want to create an output file based on the input files. The input files can be anywhere else in the tup hierarchy, but the output file(s) must be written in the same directory as the Tupfile.

Same question! Is it now OK to output to a different directory? If so, are there any restrictions? Any gotchas to be aware of?

Cheers,

Iain

Freddie Chopin

unread,
Sep 25, 2014, 1:13:41 PM9/25/14
to tup-...@googlegroups.com
On 09/25/2014 04:53 PM, Iain Merrick wrote:
> Same question! Is it now OK to output to a different directory? If so,
> are there any restrictions? Any gotchas to be aware of?

It works like a charm now (; I've been using this feature ever since it
was available and reported some bugs, but all of them are fixed now.
Moreover - with lua parser you can have output files in an identical
tree as sources, but this whole tree can be in a subdirectory of the
project! So if you have folders with sources:
a/*.c
a/a1/a2/*.c
b/*.c
c/*.c
d/*.c
d/d1/*.c
tup is able to compile all of the sources into relevant folders of this
hierarchy:
output/a/*.o
output/a/a1/a2/*.o
output/b/*.o
output/c/*.o
output/d/*.o
output/d/d1/*.o
(the name "output" is just an example - it can be anything)

You can check this project of mine, which has tupfiles written in lua
and they do exactly that:
https://github.com/DISTORTEC/distortos

In case of tupfiles written in "original tup parser" it's a bit harder
and I guess such build as above wouldn't be possible (without manually
stating outputs for each subfolder of the sources)... I usually just
output all files into one subfolder, with no hierarchy (all of them go
into this folder), so the only limitation is that I cannot have two
object files with the same name (probably a bad idea anyway to have two
source files with the same name). It would be probably easy to improve
this hierarchy - you could easily output to a subfolder of this "output"
folder, with the subfolder named as the last directory of the sources.
Taking the above example, it would look like that:
output/a/*.o
output/a2/*.o
output/b/*.o
output/c/*.o
output/d/*.o
output/d1/*.o

Ask me anything about that, as I've excercised this ability of tup a lot (;

Regards,
FCh
Reply all
Reply to author
Forward
0 new messages