order of rules matters sometimes

[Todd Doucet]

The documentation gives the following Tupfile in the section describing how to handle a generated header file:

: |> sh gen_triangle.sh > %o |> triangle.h

: foreach *.c | triangle.h |> gcc -Wall -c %f -o %o |> %B.o

: *.o |> gcc %f -o %o |> hello

That works fine in the example, but if you switch the order of the first two lines, yielding:

: foreach *.c | triangle.h |> gcc -Wall -c %f -o %o |> %B.o

: |> sh gen_triangle.sh > %o |> triangle.h

: *.o |> gcc %f -o %o |> hello

then tup generates an error on the first rule, saying that the file triangle.h does not exist.  Of course it does not exist, it is generated.  Apparently it needs to know how before it is willing to accept an order-only dependency on that file.

It took me a while to reduce the problems I was having to this, perhaps in part because the documentation is silent on the order dependency.  It is odd that there is one and it took me a long time to realize this was the problem.

(In my case, I build an executable that in turn is run by a rule to generate a header, which in turn is used to build the final executable.  I must have all three in the right order or it does not work.

Perhaps a note about order dependency would be useful to others.  Even better, if possible simply make the rule order not relevant.

[Layus]

Yep, I also got bitten by this one :-).

There is some discussion about this in the ill-named issue on github https://github.com/gittup/tup/issues/247.

Basically, it is not considered a pressing issue, and not really a bug.

I guess a pull request on the doc would be trivial. Would you feel like doing it  ?

You can do it directly in github (https://github.com/gittup/tup/edit/master/tup.1) or send your updated/new text here and I would happily make the PR.

Regards,

-- Layus.

--
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tup-users/2ea8d4ad-bb85-4ae7-8334-33d2439de175%40googlegroups.com.

[Todd Doucet]

Thank you for the reply—I read the thread you cited.

I thought about how I would phrase the change to the doc, and interestingly I cannot think of what I would say that is both simple and true.

For example, it is not correct to say (as I did in the post for clarity) that tup must know how to make the file if it does not already exist.   Because the file can exist and it would still be an error if it is “scheduled for deletion”.  Further, there could very well be additional cases that I am not aware of.

In my view, there seems to be no fundamental (mathematical or logical) reason for the rules to be required to be in a particular order.  They are required to be in the order that the program expects them to be in, and I just do not know enough about how tup internals work to be able to describe its expectations.

In my view, the biggest reason to make the rule order not matter (if that is indeed possible) is that it gets both the user and the documentation writer out of the business of describing these limitations on order (because there would be none).  It is a powerfully good reason to have the rules be “declarative”, and in my view far more persuasive than the specific use case contemplated in the thread you cited.

I acknowledge, however, that it might be impractical to make the rule order not matter, for reasons of code inertia or even related to performance.  What I am suggesting, though, is that if it is practical to remove the hard-to-describe limitations, doing so would be really good.

Thanks,

Todd

[Todd Doucet]

Looking into some more, I guess tup has the idea that the rules generally are executed in order and that is a design decision that goes beyond order-only dependencies.

For example, even without any order-only dependencies, the following two Tupfiles would do different things (only the order of the lines is swapped):

First:

: foreach *.cpp |> g++ -Wall -c %f -o %o |> %B.o
: *.o |> g++ %f -o %o |> hello

Second:

: *.o |> g++ %f -o %o |> hello
: foreach *.cpp |> g++ -Wall -c %f -o %o |> %B.o

The first one would build the target “hello”, but the second one would not.

So I gather that one thinks of a Tupfile as a kind of build script with annotations and wildcards and similar, really designed to be executed in sequence.  That might be in the docs even; I am quite new with tup.

If that is right, then my suggestion to make the order-only dependency not matter probably would not make much sense.  Order is supposed to matter, for normal everyday things, if I understand it right. 

--Todd

[Guillaume Maudoux (Layus)]

My understanding was that order does not necessarily need to matter as rules are indeed declarative.

But, as your examples shows, it is non trivial to make tup able to understand the rules in any order.

You basically require two passes, or a way to encode glob patterns in the dependency tree. Globs can only range over files in the current directory, so that helps a bit. Now, encoding globs in the dependency tree would be funny becaus you could theoretically glob in other directories...

Tup dependency on rules order is not a fatality, but requires non-trivial patches to work.

-- G.

[Todd Doucet]

I find it helps me a lot to think of Tup rules as simply command generators for a script, which is executed in sequence from beginning to end (at least on a fresh build).  Thinking of it that way removed all the puzzlement I was experiencing by thinking of the rules as make-style declarative rules.

Tup will of course omit parts of the script that do not need to be re-run, etc, and it will do more than that: it will remove stale artifacts and in fact make a marvelous guarantee.

When I write a Tup script, I imagine a clean build and I just write a script that will build my project from scratch.  The wildcards and other features make that script compact and easy-to-understand as a build-from-scratch script.

Then, critically, Tup will make this guarantee: when I run the script subsequently, it will build my project and the result of a successful build will be exactly the same as if I had built it from scratch the first time.  Including deleting any stale artifacts, and even taking into account that I may have edited the Tupfile (that part is really marvelous).

Thinking about it that way, it makes little sense to permute the “rules” in the Tupfile, because I would not expect a script to work properly if the lines in the script were re-ordered.

I think I am much closer to thinking about Tup correctly now than I was originally, and the reason is that I have been using Make for decades and was implicitly assuming that Tup was more similar to make than it is.

Having fun.

On Monday, September 9, 2019 at 10:45:12 AM UTC-4, Guillaume Maudoux (Layus) wrote:

My understanding was that order does not necessarily need to matter as rules are indeed declarative.

But, as your examples shows, it is non trivial to make tup able to understand the rules in any order.

You basically require two passes, or a way to encode glob patterns in the dependency tree. Globs can only range over files in the current directory, so that helps a bit. Now, encoding globs in the dependency tree would be funny becaus you could theoretically glob in other directories...

Tup dependency on rules order is not a fatality, but requires non-trivial patches to work.

-- G.

On 8/09/19 20:57, Todd Doucet wrote:

Looking into some more, I guess tup has the idea that the rules generally are executed in order and that is a design decision that goes beyond order-only dependencies.

For example, even without any order-only dependencies, the following two Tupfiles would do different things (only the order of the lines is swapped):

First:

: foreach *.cpp |> g++ -Wall -c %f -o %o |> %B.o
: *.o |> g++ %f -o %o |> hello

Second:

: *.o |> g++ %f -o %o |> hello
: foreach *.cpp |> g++ -Wall -c %f -o %o |> %B.o

The first one would build the target “hello”, but the second one would not.

So I gather that one thinks of a Tupfile as a kind of build script with annotations and wildcards and similar, really designed to be executed in sequence.  That might be in the docs even; I am quite new with tup.

If that is right, then my suggestion to make the order-only dependency not matter probably would not make much sense.  Order is supposed to matter, for normal everyday things, if I understand it right. 

--Todd

unsubscribe: tup-...@googlegroups.com

options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tup-...@googlegroups.com.

[Dequan Zhang]

I thought Tup 

1) will collect all the possible dependency implicitly and explicitly, including commands and files and their interactions. 

2) tell users of missing dependency and help different developers without the background information to manage a large codebase dependency

3) Thus that I can have fearless parallel build and consistent artifacts state for every same commit.

But reading your feedback shows me that Tup is not that intelligent yet, as least for rules parsing. 

Is the statement correct? 

unsubscribe: tup-users+...@googlegroups.com

options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tup-users/8a4601db-4da4-4182-bbd5-751f45938b50%40googlegroups.com.

[Todd Doucet]

I am probably not the person to offer authoritative answers on 1, 2 and 3.   But for what it is worth I think you are correct in #1 and #3.  I just do not understand exactly what you mean by #2.

Nothing I wrote should be understood as a criticism of how Tup works.  I shared the way I look at what it does in case it would be useful to others coming from a background similar to mine.

On Tuesday, September 10, 2019 at 8:08:48 PM UTC-4, Robin wrote:

I thought Tup 

1) will collect all the possible dependency implicitly and explicitly, including commands and files and their interactions. 

2) tell users of missing dependency and help different developers without the background information to manage a large codebase dependency

3) Thus that I can have fearless parallel build and consistent artifacts state for every same commit.

But reading your feedback shows me that Tup is not that intelligent yet, as least for rules parsing. 

Is the statement correct? 

To view this discussion on the web visit https://groups.google.com/d/msgid/tup-users/8a4601db-4da4-4182-bbd5-751f45938b50%40googlegroups.com.