Wildcards could be made to work reasonably

[Joonas Saarinen]

The small problems mentioned in the FAQ related to wildcards could be solved. I know that this topic has been wrestled before, but it really is something that has a high demand. Always manually adding new files to the build file feels like an unnecessary extra step, and it would be cool if it all worked automatically.

1) The performance issue: Meson could only start globbing if the string contains wildcards. If no wildcards are found, it would just read the filenames as usual. This would retain good performance for non-wildcard situations, but also allow wildcards for those that want to use them.

2) Ninja not supporting wildcards: Meson could expand the filenames for Ninja.

What do you think?

Joonas

[Jussi Pakkanen]

On Tue, Feb 6, 2018 at 5:07 PM, Joonas Saarinen
<joonas....@gmail.com> wrote:

> 2) Ninja not supporting wildcards: Meson could expand the filenames for
> Ninja.

This is the exact problem. If one were to have wildcard support, then
it would mean that adding a new source file (that matches the glob)
and running Ninja should automatically and reliably pick up the new
file. Anything else is a bad UX.

Ninja does not support globs and thus it can not know that file that
has appeared somewhere would mean that it needs to reconfigure
everything. The only way to do this would be to have Ninja call into
some program to parse the original globs. Not only does Ninja not have
this functionality it would mean that every build invocation would
need to reparse and reglob everything. That slows down the common case
where the project configuration does not change.

[Joonas Saarinen]

On Tuesday, 6 February 2018 20:54:04 UTC+2, Jussi Pakkanen wrote:

This is the exact problem. If one were to have wildcard support, then
it would mean that adding a new source file (that matches the glob)
and running Ninja should automatically and reliably pick up the new
file. Anything else is a bad UX.

Ah, you're certainly right, that problem would still remain.

Joonas

[Mark Maker]

Hi,

I understand that file system based globs are a no-go for the build system. They are also arguably a bad idea, as mistakenly adding a new file (such as a quick "Save as" backup copy of another file) happens all-too easy.

But everybody is using a versioning system nowadays, and I wondered, if their file index was a possible dependency.

The following is numbered for easy referencing:

1. git (for instances) has the .git/index file that registers files, recursively, including those only staged (see Refs at end).
2. Other versioning software, at least those with a well-known file based index, should work in a similar way. But let's face it, even just covering git would go a long way.
3. Assuming Ninja could be told to include a dependency from the  .git/index file, a globbing check operation could be triggered whenever it changes.
4. Note: with an unchanged index file, this would still be a NOP in any rebuild. A check operation would only be triggered after someone checked in or added files (etc.). Most importantly it would never run for those that just modify existing files during routine build cycles.
5. But even if the index changes, it does not yet mean a full reconfigure. The globbing check operation could do
      git ls-files --stage
   which lists all files that are officially added to the repo, recursively. The command also seems very fast, compared to recursive file system ls[*].
6. I assume Meson already has git well integrated for its automatic subproject support, so I guess this should be feasible, technically.
7. The output could be filtered to just look at relevant portions (i.e. just the file names), then be hashed, and the hash written into a glob-trigger artifact, but only if the hash changed!
8. Finally that glob-trigger artifact file in turn could be added to the Ninja dependencies to reconfigure the build, i.e., to finally perform any new globbing.
9. All this could be made to be "pay as you go", i.e. only generated in projects with globs.
   
10. It could be made conditional on whether the .git/index file exists, so any external/automated "tarball" building system (no files are ever dynamically added or removed) would not incur a dependency from git.
    
11. Finally, I just wanted to express my opinion, that not having globs is kind of like the elephant in the room for Meson, where everything else is declarative and auto-discovered so nicely! It would be so cool to have this! :-)

Caveats:

1. Arguably the build system should be agnostic of the versioning system, i.e. just downloading a tarball should work the same way as checking out with git. But on the other hand, the same could be said for git subprojects etc. where Meson equally engages with the versioning system. Plus see point 10.
2. One could argue that people git add new files only at the end, builds should already work before. That could be a valid point. Frankly I don't know what the "universally accepted git workflow" is.
   

Thanks!

_Mark

Refs:

https://github.com/git/git/blob/master/Documentation/technical/index-format.txt

https://mincong.io/2018/04/28/git-index/

[*] just from a quick test: 35 times faster with a cold cache on high end SSDs/medium-size repo)

[Eli Schwartz]

On 6/9/22 4:47 AM, Mark Maker wrote:
> Hi,
>
> I understand that file system based globs are a no-go for the build system.
> They are also arguably a bad idea, as mistakenly adding a new file (such as
> a quick "Save as" backup copy of another file) happens all-too easy.
>
> But everybody is using a versioning system nowadays, and I wondered, if
> their file index was a possible dependency.
>
> The following is numbered for easy referencing:
>

> 1. git (for instances) has the .git/index file that registers files,

> recursively, including those only staged (see Refs at end).

> 2. Other versioning software, at least those with a well-known file

> based index, should work in a similar way. But let's face it, even just
> covering git would go a long way.

> 3. Assuming Ninja could be told to include a dependency from the .git/index
> file, a globbing *check operation* could be triggered whenever it
> changes.
> 4. Note: with an unchanged index file, this would still be a NOP in any
> rebuild. A *check operation* would* only* be triggered after someone

> checked in or added files (etc.). Most importantly it would never run for
> those that just modify existing files during routine build cycles.

Actually, .git/index changes whenever any git status changes, including
but not limited to modifying existing files.


> 5. But even if the index changes, it does not yet mean a full
> reconfigure. The globbing *check operation* could do

> git ls-files --stage
> which lists all files that are officially added to the repo,
> recursively. The command also seems very fast, compared to recursive file
> system ls[*].

> 6. I assume Meson already has git well integrated for its automatic
> subproject support
> <https://mesonbuild.com/Release-notes-for-0-40-0.html#automatic-initialization-of-subprojects-that-are-git-submodules>,

> so I guess this should be feasible, technically.

This is only used for one-time source discovery, and has a fallback mode
in that if you're not in a git repo Meson needs you to provide the same
url that would normally be in .gitmodules, inside a wrap file, so that
Meson can clone it into an extracted tarball too.


> 7. The output could be filtered to just look at relevant portions (i.e.
> just the file names), then be hashed, and the hash written into a *glob-trigger
> artifact**, but only if the hash changed!*
> 8. Finally that *glob-trigger artifact* file in turn could be added to
> the Ninja dependencies to* reconfigure the build*, i.e., to finally
> perform any new globbing.
> 9. All this could be made to be "pay as you go", i.e. only generated in
> projects with globs.
> 10. It could be made conditional on whether the .git/index file exists,

> so any external/automated "tarball" building system (no files are ever
> dynamically added or removed) would not incur a dependency from git.

And if you download the project as a tarball, what *does* it do?
Completely fail to generate the build in the first place as it cannot
run git-ls-files during `meson setup`? Do filesystem globs, then fail if
you apply a patch to the extracted tarball?


> 11. Finally, I just wanted to express my opinion, that not having globs

> is kind of like the elephant in the room for Meson, where everything else

> is *declarative* and *auto-discovered* so nicely! It would be so cool to
> have this! :-)
>
> Caveats:
>
> 1. Arguably the build system should be agnostic of the versioning

> system, i.e. just downloading a tarball should work the same way as
> checking out with git. But on the other hand, the same could be said for
> git subprojects etc. where Meson equally engages with the versioning
> system. Plus see point 10.

See my reply to point 6. :)


> 2. One could argue that people git add new files only at the end, builds

> should already work before. That could be a valid point. Frankly I don't
> know what the "universally accepted git workflow" is.

It doesn't need to be a universally accepted git workflow, there are
certainly people doing it. As a random data point, I do it that way.


--
Eli Schwartz

[mark maker]

Hi Eli,

Thanks for looking into that.

> Actually, .git/index changes whenever any git status changes, including but not limited to modifying existing files.

I was talking about just modifying files in the file system, between build cycles, a developer at work, wanting fast turn-around times.

Regarding index changes through git commands: yes, it'll not only change when files are added/deleted, but for many other reasons too. That's why the separate check operation is needed (my point 5) to separate the relevant file list changes from all other changes. If no file list changes are detected, the step would still be very fast, i.e. it does not require a reconfigure.

> This is only used for one-time source discovery...

What I meant is that Meson already has means to detect the presence of a git repo, and to to find the git tool and execute some commands etc., i.e., the (optional) functional dependency from git would not be a completely new thing.

> And if you download the project as a tarball, what *does* it do? ... Do filesystem globs ...?

Yes, it would fall back to file system globs, assuming the balled-up files are "pristine". It would assume that such configures were "one-shot".

> ... then fail if you apply a patch to the extracted tarball?

I admit my knowledge of such build systems in the wild is very limited. 😇 Still I think it is reasonable to assume some "sane" behaviour from them.

If you patch after the configure, then yes, it would fail. But...

If patches are unavoidable, they must be applied before the setup/configure, which makes sense anyways, as the patch could contain changes to the meson.build files too (without globbing this would always be the case when files are added/deleted), and one would want to avoid to configure twice.

If even incremental patches over time are unavoidable, you can always "meson reconfigure" explicitly in the build pipeline. But IMHO such an incremental patch system would be insane not to use the versioning system directly. One can bootstrap it with git clone --depth 1 instead of tarballs and it'll just work from there (and also never miss an incremental patch, which would be constant headache otherwise).

Anyways, I was under the impression that such automated external builds were realistically "meson dist" based, i.e. "all or nothing" propositions, and most probably built in fresh builddirs, or even fresh containers, nowadays.

All interactive editing and hand-patching would have to be done inside the authoritative versioning system context. IMHO, that's a reasonable assumption for a project and a build-system.

I'm taking one of the welcome design principles of Meson for granted here: "we optimize for the needs of today, and not from '80s". 😉

_Mark