Augmenting FileName- (FileQ/FileA-) based rules

[alex.f...@gmail.com]

Hi,

I've been using shake to replace an existing recursive-make system (which is well overdue for a rewrite) in a relatively large monorepo.

I have a working version that uses filepaths, but I'd really rather be able to define rules against objects that have a little more information than just the filepath. For instance, I'd like to tag the filepath with the type of file (as a typical union data type, probably) and define a UserRule creation function that can match on the tagged type rather than using information embedded in the file name (such as the file extension). Basically, I'd like to create a more explicit rule-matching interface wherein user rules may be defined against explicitly-defined types rather than using pattern matching on file names (which can be stored with arbitrary paths).

As shake is right now, it seems like the only way to do this would be to create a builtin rule against my own data type that holds the file path and the extra information. The problem with this is that the file BuiltinLint and BuiltinRun are not exposed, or at least a lot of what they do isn't exposed, so I immediately lose a lot of the functionality those provide for files. I could simply copy-paste the functionality, but if there's a more elegant way of handling this sort of use-case, I'd like to hear it.

Thanks,
Alex

[Neil Mitchell]

Hi Alex,

What extra information are you thinking of including? Remember that
any information will have to be included in everyone who asks for the
file - e.g. if you want to include MIME-type of the file, then if the
user wants to build a specific file you'll need to do:

$ myShakeProgram MyFile.txt#content/text

You'll also have to have every `need` pass this extra information,
e.g. need [File2 "MyFile.txt" "content/text"]. What if two users ask
for "MyFile.txt" with different MIME types? Or is it that you more
want to enrich a file given its FilePath alone? If that's what you are
after then defining enrich :: FilePath -> MyADT and then just
redefining %> as something that matches on ADT shouldn't be too hard.

Thanks, Neil

[alex.f...@gmail.com]

Neil,

I should clarify: I'm trying to make the internal/compile-time interface different, not the user-facing side. The way our configuration works, you specify a build type (i.e. static lib, dynamic lib, etc) and input configuration, and the output(s) that are generated given a build type and input configuration pair specify what to build.

What I'd like is to make the programmer interface easier so that where the intermediate components are stored (of which there are many) can be abstracted away. It would be nice if more information about the intermediate components could be passed to other consuming rules, rather than just relying on the file paths. In this way, they don't need to care about what the file path looks like, but instead can provide rules based on the augmented intermediate information. It would be more precise in my opinion to not just convert a more descriptive type to a file pattern and feed that to the normal file rules.

It sounds like what I want may necessitate making a custom RuleResult. But as it is, I may be trying to abstract it away too much.

-Alex

[Neil Mitchell]

Hi Alex,

Is there a bijection between filenames and this configuration? If so,
Shake can see the rules as filepaths, and you can define some simple
sugar functions on top that parse/display in your helper functions. If
there isn't a bijection between files and you configuration variables,
I think you'll run into problems, in debugging the intermediate
results if nothing else. You'll probably find that using the native
filepath stuff will be fastest, as it's well optimised, and your
parsing functions will only be invoked if it also has work to do (and
then the work will usually dominate).

If you want to push the pattern/config language down further a custom
RuleResult would be the right thing to do. It's not nearly as easy,
but everything should be exposed - the FilePath native stuff isn't
meant to be privileged in any way - just a handy built-in RuleResult.

Thanks, Neil

[alex.f...@gmail.com]

Neil,

There could potentially be a bijection, but not one I'd be comfortable with, as the configuration can be involved. But as it is, I realized that I didn't really need the FilePath as an input/key: for intermediate results, the filepath is actually the value that is yielded from updating the item keyed with the configuration. I've taken the approach of making a custom RuleResult. So far it seems like it will work nicely. A very important goal of this system is reproducible and correct builds, which also includes rebuilding correctly when flags for particular targets have changed. All build outputs are keyed based on the configuration flags and the binaries used to create them. The goal is to allow someone to specify flags in the configuration files or on the command line that affect certain outputs and have only those rebuilt. It's a large codebase, so build time is important. But not at the cost of correctness!

Shake has made this sort of behavior very easy to implement, and I have you to thank for that.

-Alex