wildcard for targets on the command line

[Kris Malfettone]

All,
I was wondering if it would be possible to add the ability to specify wildchars on the command line to ninja?  For instance:

ninja foo* 
would build all targets starting with foo.

I haven't seen this feature requested before but it would help me out greatly.  In my project I have a large number of targets (~50k) and I often want to build a subset of them.  The target names follow a regular pattern and would be easily specified via wildchars.  Currently, I filter them with grep and pass them back to ninja:

ninja $(ninja -t targets | grep ^foo | sed 's/:.*//')

However, this is not ideal for several reasons:

1. Each invocation of ninja takes a non-trivial amount of time to start due to the large number of targets.  So this approach at least doubles the time it takes for ninja start.
   
2. I either have to wrap this with script or type all that each time

It seems that simple wildchars would suffice and shouldn't be difficult to implement.  Regex would be fine as well but IMO overkill.  I have tried other approaches in the past such as:

• Having CMake generate phony targets to group / build multiple targets
• Very annoying to do in CMake (opinion)
• Only allows for the predefined groupings
  
• I believe this should be a feature of the build system and not the generator.
• A patch (from Nicolas I believe) that allowed me to group via subdirs and build that way
• Only works if subdir grouping is appropriate
• Again, only allows predefined groupings
• Need to run a patched version of ninja

Also, if we were worried about this slowing down people that don't need this feature it could be opted into via a flag:

ninja -w foo*
or
ninja -r ^foo

 

Any thoughts?  Has this been discussed before? I couldn't find anything in my searches.

Thanks,

Kris

[Scott Graham]

Perhaps you could generate a phony grouping target that depends on the commonly used subsets.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Malfettone, Kris]

I have tried done that in the past.  However, IMO the problems with that approach are:

·         Very annoying to do in CMake (opinion)

·         Only allows for the predefined groupings

·         I believe this should be a feature of the build system not the generator.  My CMakeLists.txt are complicated enough without having to try and embed tricks to help make specifying what to build easier J

---

IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

[Nico Weber]

How many different prefixes do you use? If it's just a small set, if the generator had a "phony target = glob_targets(foo*)" thing, that would address your use case.

Out of interest, why do you have so many unrelated targets?

[Malfettone, Kris]

The project is a framework that contains a large number utilities for handling different formats of data.  There are about 300 different data formats and for each we ship a set of ~15 “applications” for.  These “applications” are each tailored to each of the formats but ultimately perform the same functions.  Some of the applications serve as examples/reference some are useful utilities.  The code ends up being structured into well-defined folder structures according to the format it is intended for. 

 

For instance, let’s say I have version 1 of a format named “A.B “ (the naming isn’t important other than its hierarchal):
`-- A

    `-- B

        `-- v1

            |-- examples

            |   |-- example1

            |   |-- example2

            |   `-- example3

            `-- utils

                |-- util1

                |-- util2

                `-- util3

 

Each example / util below would be a different target named something like:
A.B.v1.examples.example1

If you are a developer working on one of these formats you might want to build all the related tools for it:
ninja A.B.v1.*

However, if you are someone developing a new tool for all of them you might want to build only that tool for all the formats:
ninja *utils.newutil

 

There are more complications than this but I think it illustrates the use case.  I have in the past setup CMake to generate these phony grouping targets but it gets very ugly very fast.  I have since removed the functionality from our CMakeLists.txt in an effort to keep them more clear and concise (not working by the way). 

 

However, to me this doesn’t feel like a job of the generator.  This feels like an appropriate feature for the build system itself.  Particularly because it is not specifying “how to build” something just the users desire of “what to build”.  In addition, a feature like this seems like you can get a lot of “bang for your buck”.  Would only require a small function to pattern match against the known list of targets and now I can satisfy all of the intended use cases.

 

I can try and describe the setup better if this wasn’t clear enough.

[Nico Weber]

Thanks for explaining! I think it's a reasonable feature request. Things I'm worried about:

* Just saying `ninja foo*` won't work on posix as the shell will expand `foo*` before handing it to ninja. So you'd have to write `ninja 'foo*'`, and people would likely get this wrong often and get confused.

* People will probably have different expectations for the globbing syntax, so there'll probably many feature requests on tweaking the globs (add real regexs, etc)

For the invocation time issue, you could write the output of `ninja -t targets | grep ^foo | sed 's/:.*//'` to a file once after running your generator and the run `ninja $(cat targets)` to build. That could be wrapped with some script to make it a bit nicer to use, but I agree it's still a bit clunky.

[Richard Geary]

I don't have an opinion on whether this feature is a good idea to include or not, but as a factual note :

> * Just saying `ninja foo*` won't work on posix as the shell will expand `foo*` before handing it to ninja. So you'd have to write `ninja 'foo*'`, and people would likely get this wrong often and get confused.

Linux passes the * to the executable without expanding it if there is no file/dir which matches foo*. Plenty of modern linux tools support * wildcards via this, including yum & apt-get.

[Konstantin Tokarev]

20.07.2014, 03:40, "Richard Geary" <richar...@gmail.com>:

> I don't have an opinion on whether this feature is a good idea to include or not, but as a factual note :
>
>> * Just saying `ninja foo*` won't work on posix as the shell will expand `foo*` before handing it to ninja. So you'd have to write `ninja 'foo*'`, and people would likely get this wrong often and get confused.
>
> Linux passes the * to the executable without expanding it if there is no file/dir which matches foo*. Plenty of modern linux tools support * wildcards via this, including yum & apt-get.

Linux does not expand globs, it's a behavior of shell (bash in this case).


--
Regards,
Konstantin

[Malfettone, Kris]

Cool thanks. 

 

Again, whatever globbing scheme you feel is appropriate is fine with me: glob, regex, anything really.

 

I also must say that I am continuously impressed by ninja.  Both in by the quality of the software and the community supporting it.  Thanks again for all the effort put into this project.

 

-Kris

[Dirk Pranke]

I'm not aware of any other build tool that supports this sort of glob-expansion, which suggests that this is not a commonly-needed feature. Please correct me if I'm uninformed. I don't think it's an inherently bad idea, just a little unorthodox.

Perhaps an alternative would be to modify the generator to also generate a list of targets somewhere (i.e., so that cmake generates a out/Release/ninja_targets.txt or some such) so that you can do the glob faster than having to start ninja itself? Or create a target that can be run to generate this once to speed up subsequent invocations? I don't know how feasible those ideas would be for you, but might be a better solution than what you have now, and wouldn't require mods to ninja itself ...

-- Dirk

[Malfettone, Kris]

I also cannot think of another command line build tool that does this globbing.  Most are done in the following way:
tool [target [ target2 [target3] … ]]
with no target running a default “all” target

 

All GUI build systems that I remember also had a way to build multiple targets.  Typically selecting checkboxes in some modal dialog.  It’s been a while since I have used any of them so I may be wrong here.  However, I find this similar to the above rather than a glob.

 

I do feel all the “modify your generator” arguments fall into the same category of answers though.  Basically, cluttering up your “how to build my target” definitions with “what I want to build at this moment” definitions.  As I said earlier, I can do this via the generator and have in the past.  However, I think this is solving it in the wrong location.

 

Inherently what we are talking about here is whether or not the user should have to type out all the targets they want to build or whether the build tool itself should allow some short hand.  I believe most people already think “some” sort of short hand is a good idea.  For instance, make’s “build from here down” behavior or the default “all” target.   Many people use that in make and have come to expect that from other build tools.  I think the reason globbing has not been seen before is the fact that most projects boil down to “one or two targets that have many dependencies” rather than “many independent targets”.  If your project either fits into the “one or two targets with many dependencies” category or is small enough that the “all” target is fast enough you wouldn’t need to specify many targets on the command line or this feature.  Another interesting effect in my opinion is that most project layouts are now crafted with the “make” style of organization to leverage the “build from here down” behavior.  Was this a result of the build systems limitations or are the build systems features reflective of the one true organizational pattern?  I do not know.

 

 

-Kris

[Dirk Pranke]

On Mon, Jul 21, 2014 at 1:36 PM, Malfettone, Kris <Kris.Ma...@sig.com> wrote:

I also cannot think of another command line build tool that does this globbing.  Most are done in the following way:
tool [target [ target2 [target3] … ]]
with no target running a default “all” target

 

All GUI build systems that I remember also had a way to build multiple targets.  Typically selecting checkboxes in some modal dialog.  It’s been a while since I have used any of them so I may be wrong here.  However, I find this similar to the above rather than a glob.

 

I do feel all the “modify your generator” arguments fall into the same category of answers though.  Basically, cluttering up your “how to build my target” definitions with “what I want to build at this moment” definitions.  As I said earlier, I can do this via the generator and have in the past.  However, I think this is solving it in the wrong location.

Maybe you misunderstood my suggestion; I was suggesting only modifying the generator to make it possible to do a faster glob expansion using an auxiliary script. The actual determination of what to build would still happen via a wrapper script that called out to ninja (and yes, that is similar to your initial script).

 

I do not disagree with the rest of your analysis, but I'm unsure that what you want to do is common enough to warrant adding the feature to ninja itself. That's more up to Nico than me, though :).

-- Dirk

[Richard O'Grady]

It's not exactly what you asked for, but I use a bash tab-completion script to get the targets list and then Alt-* to expand the ones that match.

--

[Chris Hopman]

I've used a build system that allows you to specify to build all targets under a directory with something like

//repo3/some/sub/dir/...