Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

glob operation on database

29 views
Skip to first unread message

Peter Jaspers

unread,
Sep 29, 2023, 4:07:12 PM9/29/23
to tup-users
Hi,

 
From the EXAMPLE section in the tup manual:
The foreach :-rule will generate a command to compile each file. First tup will parse the input section, and use the glob operation on the database since a '*' is present. This glob matches foo.c and bar.c. 

To my understanding, for this to work, tup must populate the database with a file node for each file found in the directory before it starts parsing the tupfile. Or does tup only create nodes for files that match the glob pattern? In the latter case why then perform a glob operation on the database?



Peter
 




Jameson York

unread,
Sep 29, 2023, 11:20:37 PM9/29/23
to tup-...@googlegroups.com
I'm just another tup user, but I think I can answer your question.

The key thing to understand is that performing a build, with 'tup', needn't involve parsing any Tupfiles at all. These files must of course be parsed the *first* time. But from that point on, unless they change or the listed entries of their local directory change, tup can continue updating the outputs of that directory based on updates to the existing inputs using nothing but the database, and, as you observe, the cached initial result of the globbing.

For larger builds, this distinction of re-parsing vs. re-performing build steps becomes obvious from the console output. Re-parsing is the first pass, indicated by a yellow progress bar, whereas the second pass--the build proper--is indicated with a second green progess bar. Incremental builds will for the most part only include the second pass, since fuse can reliably inform tup that no re-parse is required.

I hope that's helpful/accurate, it's just what I've gathered from waiting on a many incremental builds over the last few years.

--
--
tup-users mailing list
email: tup-...@googlegroups.com
unsubscribe: tup-users+...@googlegroups.com
options: http://groups.google.com/group/tup-users?hl=en
---
You received this message because you are subscribed to the Google Groups "tup-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tup-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tup-users/02686f7a-d99f-4102-aaf9-5dff08ffe750n%40googlegroups.com.

Peter Jaspers

unread,
Oct 1, 2023, 5:19:08 AM10/1/23
to tup-users

Thanks for explaining the incremental behavior of tup. 
My question however is on the doubt/trouble I have in understanding these three phrases in the EXAMPLE of the tup manual:
The foreach :-rule will generate a command to compile each file. 
First tup will parse the input section, and use the glob operation on the database since a '*' is present. 
This glob matches foo.c and bar.c. 

This is my interpretation of these three phrases:

If the input section of the rule contains a glob on some directory D {
    DirNode DN = database.findNode(D)
    if (DN == null) DN = database.createDirectoryNode(D)
    //Cache the directory content in the tup database
    foreach direntry F in D { 
        FileNode FN = database.findNode(F.name)
        if (FN == null) database.createFileNode(F.name)
    }
    // evaluate the glob on the file nodes in the database
    foreach filenode FN in DN: if (glob matches FN.name) inputNodes += FN
} else {
    foreach file F in the input section {
        FileNode FN = database.findNode(F.name)
        if (FN == null) database.createFileNode(F.name) 
         inputNodes   += FN
    }
}
// ...continue parsing the output section and cmd sections of the rule

Tup will only re-parse the tupfile when directory D has changed since the previous build and/or when the tupfile itself has changed.

Another interpretation is:

If the input section of the rule contains a glob on some directory D {
    GlobNode DN = database.findNode(D, globPattern)
    if (DN == null) DN = database.createGlobNode(D)
    //Cache the glob result in the tup database
    foreach direntry F in D { 
        if (glob matches F) {
            FileNode FN = database.findNode(F)
            if (FN == null) database.createFileNode(F)
             inputNodes  += FN
        }
    }
} else {
    foreach file F in the input section { 
        FileNode FN = database.findNode(F)
        if (FN == null) database.createFileNode(F)
        inputFileNames += FN.name
    }
}

This second interpretation however does not reflect the "...and use the glob operation on the database... 
because it does not apply the glob to the (file nodes in the) database but directly to the directory entries.

My question: which interpretation best matches tup's implementation? Or is tup acting in yet another way?

Why do I want to understand this ?
Consider a directory that contains a large numbers files of which only few files match the glob and
of which only few files will ever be used as input by the build.
For such a directory the first interpretation will result in many not-used file nodes in the tup database, making the
database larger and possibly slower.


Op zaterdag 30 september 2023 om 05:20:37 UTC+2 schreef jameso...@gmail.com:
Reply all
Reply to author
Forward
0 new messages