tup required a "clean build" (bug?)

[Ervin Oro]

Let's say that I want all inc* directories in project to be automatically in the include path. In my Makefile I used to have the following:

CFLAGS = $(shell find "$(CURDIR)" -iname inc* -exec find {} -type d \; | sed -e "s/^/-I/")

I am not at all sure, how to do that with tup, but I came up with the following solution:

CFLAGS = '\\$(find "' .. tup.getcwd() .. '" -iname inc* -exec find {} -type d \\; | sed -e "s/^/-I/")'

CFLAGS then gets included in the compilation command. I would assume that tup then instruments the find executable, and notices that it reads the content of all directories, therefore detecting all directories as dependencies? This does not seem to be the case however: adding new directories (named inc) does not cause re-compilation of sources. 

When I add a new directory named inc, such that it appears first in the find output, and create a new .h file there, named such that it would be included in some existing source, tup claims that everything is up to date (while the output actually doesn't include the new header), but does compile with this new header after I manually delete previous output files.

As the manual claims that "Should you find otherwise, you've likely found a bug in tup (not your Tupfiles), in which case you should notify the mailing list.", so here I am. Two questions: is this a bug, and how to best achieve my desired result with tup? Note that the solution that I came up with executes the find commands again for each compiled source, which is not really optimal, how could I have it so that it gets called at most once per tup update? (My previous make solution of course required make clean, but then again, it had support for make clean.)

[Bernhard]

Hi Ervin,

I only have an answer to your first question: no, I don't think this is a bug in tup. From http://gittup.org/tup/manual.html

Note that you may see a syntax using back-ticks when setting variables, such as:

CFLAGS += `pkg-config fuse --cflags`

Tup does not do any special processing for back-ticks, so the pkg-config command is not actually executed when the variable is set in this example. Instead, this is passed verbatim to any place that uses it. Therefore if a command later references $(CFLAGS), it will contain the string `pkg-config fuse --cflags`, so it will be parsed by the shell.

I have emphasised the important part for you. This behaviour is different from make! Unfortunately I don't have an idea how to solve your problem. May be the Lua scripting feature of tup is of help?

Kind regards, Bernhard.

[Ervin Oro]

Hi Bernhard,

Thank you for the answer. However, that's not actually what I think to be the bug, I am aware that in that situation, tup does not evaluate the subcommand immideately, and instead simply passes it to the shell later. This is suboptimal, as it causes the command to be executed again every time, and I would be happy to learn of better ways, but I acknowledge that currently it works as intended. (Lua scripts aren't allowed to access os.execute, and io.popen is theoretically available, but in my testing it doesn't seem to have permissions to access the file system.)

What makes me think that there might be a bug in tup is the fact that, in this situation, tup claims everything to be up-to-date, while actually it is not, and therefore manually doing a "clean build" gives different result than just doing a build would, which I think is not supposed to happen with tup.

[Bernhard]

Hi Ervin,

I think you misunderstand tup's design. tup never evaluates anything between back-ticks. That is shell syntax, to tup it is just another string without meaning. The issue you are dealing with is the fact, that tup never learns the results from your find operation. It just calls the shell to process the whole command string, but it does not get back any results from subshells. Therefore it never knows that these results may change and you want it to rebuild your files.

If you want to learn a little bit of what tup memorises between invocations, I suggest that you inspect tup's SQLite database (stored in the .tup/db file). I use the cross-platform "DB Browser for SQLite" https://sqlitebrowser.org/ for this. Open the database file and inspect the "node" table. Filter for (type == 1). You will find all command strings, that tup has or will execute during a run. But you will not find the results of your find operation anywhere. That's just not how it works.

Other than explaining why your use case does not work like that I can't help you any further...

Kind regards, Bernhard.