No pre-build event support? How to capture `git describe --dirty`?

[Brendan McDonnell]

I have an embedded C project that builds with CMake, Ninja, and gcc-arm-none-eabi.

I want to capture the output (string) of a git describe --dirty ... command as (1) a string variable in my program, and (2) in the generated executable filename. I think the sensible way to do this would be with a pre-build event, which ninja does not appear to support.

Without the --dirty argument, I have it working using this git_describe CMake function. But that is only run by CMake at configure time (and it hard codes the output strings in build.ninja). So for example, if I just add the --dirty arg in my current setup, then CMake configure (on a clean git working tree), then modify a source file (making the worktree dirty), then build again, it does not re-run CMake configure, and does not re-run the git describe command, and so does not detect the dirty.

Questions:

1. Am I correct that ninja does not support pre-build events?
2. Is it possible to use ninja to reliably do what I'm describing here? On every build, without requiring the user to manually trigger a CMake configure before triggering the build.

Ideally, I think, I'd want to be able to make ninja execute and check the output of my preferred git describe command on every build, and trigger the dependent targets to rebuild if it changes.

---

Originally posted on GitHub (#2500). I was advised to post it here.

[David Turner]

On Tue, Oct 1, 2024 at 4:26 PM 'Brendan McDonnell' via ninja-build <ninja...@googlegroups.com> wrote:

I have an embedded C project that builds with CMake, Ninja, and gcc-arm-none-eabi.

I want to capture the output (string) of a git describe --dirty ... command as (1) a string variable in my program, and (2) in the generated executable filename. I think the sensible way to do this would be with a pre-build event, which ninja does not appear to support.

Without the --dirty argument, I have it working using this git_describe CMake function. But that is only run by CMake at configure time (and it hard codes the output strings in build.ninja). So for example, if I just add the --dirty arg in my current setup, then CMake configure (on a clean git working tree), then modify a source file (making the worktree dirty), then build again, it does not re-run CMake configure, and does not re-run the git describe command, and so does not detect the dirty.

Questions:

1. Am I correct that ninja does not support pre-build events?

Yes, absolutely. And this is not a goal.
 

1. Is it possible to use ninja to reliably do what I'm describing here? On every build, without requiring the user to manually trigger a CMake configure before triggering the build.

Ideally, I think, I'd want to be able to make ninja execute and check the output of my preferred git describe command on every build, and trigger the dependent targets to rebuild if it changes.

It is possible to have a Ninja build action that depends on your `.git/HEAD` and/or `.git/index` and invokes `git describe --dirty ...`, saving the value into an output file, using the `restat = 1` feature to avoid rebuilding all dependents if its content do not change.
You can then use the content of that file to generate a header file or anything else that would go into your binary. Everytime Ninja runs, it will check the timestamps of your .git inputs, and re-run the script. and only rebuild things if needed.

Something that would look like:

rule generate_git_dirty

    command = git describe --dirty --always > $out.tmp && if ! cmp --quiet $out $out.tmp; then cp $out.tmp $out; fi

    restat = 1

build git_dirty.txt: generate_git_dirty .git/HEAD

rule generate_git_dirty_header

    command = printf "const char kGitDirty[] = \"%s\";\n" $$(< $in) > $out

build git_dirty.h: generate_git_dirty_header git_dirty.txt

On the other hand, changing the name of the output based on build command is not supported, and generally not desirable.

Hope this helps,

- Digit
 

---

Originally posted on GitHub (#2500). I was advised to post it here.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/c0654782-e69b-46c2-9bdf-f48eb658c2c9n%40googlegroups.com.

[Eli Schwartz]

On 10/1/24 8:20 AM, 'Brendan McDonnell' via ninja-build wrote:
> I have an embedded C project that builds with CMake, Ninja, and
> gcc-arm-none-eabi.
>
> I want to capture the output (string) of a git describe --dirty ... command
> as (1) a string variable in my program, and (2) in the generated executable
> filename. I think the sensible way to do this would be with a pre-build
> event, which ninja does not appear to support.
>
> Without the --dirty argument, I have it working using this git_describe CMake
> function

> <https://github.com/rpavlik/cmake-modules/blob/1b450496c5d11fdcad8b000843d0c516e1eaa59f/GetGitRevisionDescription.cmake#L178>.

> But that is only run by CMake at configure time (and it hard codes the
> output strings in build.ninja). So for example, if I just add the --dirty arg
> in my current setup, then CMake configure (on a clean git working tree),
> then modify a source file (making the worktree dirty), then build again, it
> does not re-run CMake configure, and does not re-run the git describe command,
> and so does not detect the dirty.
>
> Questions:
>

> 1. Am I correct that ninja does not support pre-build events?
> 2. Is it possible to use ninja to reliably do what I'm describing here?

> On every build, without requiring the user to manually trigger a CMake
> configure before triggering the build.
>
> Ideally, I think, I'd want to be able to make ninja execute and check the
> output of my preferred git describe command on every build, and trigger the
> dependent targets to rebuild if it changes.

I would say that what you're describing is *not* a "pre-build event". A
pre-build event would simply run before every build.

What you want is to make sure your targets are built on every build, and
for changes in git describe to invalidate those targets. So, you want
something like https://mesonbuild.com's vcs_tag() function.

It adds a `build_always_stale: true` target, with:

- a template e.g. version.h.in as the input

- a header e.g. version.h as the output

- the command that writes it is a custom `meson --internal vcstagger`
program shipped as part of meson, that:

- takes a command to run (git describe --dirty=+ --always by default)
- compares the results to the existing version.h, if any
- if the git describe versioning has changed, write the new file

If you manually build a target that doesn't utilize the version number
at all (perhaps because that executable doesn't have a --version option,
but other executables do) then version.h isn't a dependency and git
describe is not run.

If you build a target that utilizes the version number, e.g. because the
default "all" target includes all build targets, then version.h is a
*target* that is a dependency of another target, and must be up to date,
but the rule is crafted so it is *never* up to date, hence it always
reruns the build rule.

Since it only modifies version.h when actual changes occur, repeatedly
running `ninja` will not cause you to rebuild tons of files because
version.h keeps on changing. But repeatedly running `ninja` will cause
you to rebuild every time the output of `git describe --dirty` actually
changes.


--
Eli Schwartz

[Ben Boeckel]

On Tue, Oct 01, 2024 at 05:20:39 -0700, 'Brendan McDonnell' via ninja-build wrote:
>
>
> I have an embedded C project that builds with CMake, Ninja, and
> gcc-arm-none-eabi.
>
> I want to capture the output (string) of a git describe --dirty ... command
> as (1) a string variable in my program, and (2) in the generated executable
> filename. I think the sensible way to do this would be with a pre-build
> event, which ninja does not appear to support.
>
> Without the --dirty argument, I have it working using this git_describe CMake
> function

> <https://github.com/rpavlik/cmake-modules/blob/1b450496c5d11fdcad8b000843d0c516e1eaa59f/GetGitRevisionDescription.cmake#L178>.

> But that is only run by CMake at configure time (and it hard codes the
> output strings in build.ninja). So for example, if I just add the --dirty arg
> in my current setup, then CMake configure (on a clean git working tree),
> then modify a source file (making the worktree dirty), then build again, it
> does not re-run CMake configure, and does not re-run the git describe command,
> and so does not detect the dirty.
>
> Questions:
>

> 1. Am I correct that ninja does not support pre-build events?
> 2. Is it possible to use ninja to reliably do what I'm describing here?

> On every build, without requiring the user to manually trigger a CMake
> configure before triggering the build.
>
> Ideally, I think, I'd want to be able to make ninja execute and check the
> output of my preferred git describe command on every build, and trigger the
> dependent targets to rebuild if it changes.

I don't think this should be a "ninja" thing. I implemented most of what
you want here back in 2010 using CMake (renaming the output is not done,
but could be done by also writing some things to a CMake script that is
then `cmake -P`-run as a `POST_LINK` step to make a symlink/copy with
the desired name; install scripts may want to incorporate it as well):

https://github.com/Kitware/sprokit/blob/master/src/sprokit/CMakeLists.txt

The `version.h` is attempted to regen on every build (this is done by
declaring an output that is never made). This uses some infrastructure I
wrote to do `configure_file` during the build (that's the big list of
variables' purpose: those are the variables to "ship" to the build).
These variables are either static (in the case of a build-from-tarball)
or constructed via `execute_process` commands injected into the
build-time script. The backing implementations are here:

https://github.com/Kitware/sprokit/blob/master/conf/sprokit-macro-configure.cmake

--Ben

[Brendan McDonnell]

David, Eli, and Ben,

Thanks for your quick and informative responses. I will dig into your suggestions when I get back onto this task.

David and Eli, I'd welcome any additional pointers about how to implement your suggestions from CMake.

Meanwhile, a meta question for David below.

On Tue, Oct 1, 2024 at 10:42 AM David Turner <di...@google.com> wrote:

...

On the other hand, changing the name of the output based on build command is not supported, and generally not desirable.

Why do you think it's not desirable? Including a version number in the filename of an installer program seems like a pretty common practice, and I generally prefer that as a user. In an embedded context (e.g. using gcc-arm-none-eabi), the executable essentially is the installer, too.

Thanks,

Brendan

[Eli Schwartz]

On 10/14/24 4:24 PM, Brendan McDonnell wrote:
> David, Eli, and Ben,
>
> Thanks for your quick and informative responses. I will dig into your
> suggestions when I get back onto this task.
>
> David and Eli, I'd welcome any additional pointers about how to implement
> your suggestions from CMake.

I really couldn't say. I know what ninja describes as possible, and I
know how meson implements that (because I am a maintainer of meson) and
because your specific use case (generating a `git describe` version.h)
is a distinct function.

I don't really like cmake so unless I actually have to contribute to a
project which does use it (risky, in my experience) I try to avoid
writing cmake code. I know more about how cmake can mess up and how to
fix its messups, than about how it can do niche things.

> Meanwhile, a meta question for David below.
>
>
> On Tue, Oct 1, 2024 at 10:42 AM David Turner <di...@google.com> wrote:
>
>> ...
>>
> On the other hand, changing the name of the output based on build command
>> is not supported, and generally not desirable.
>>
>
> Why do you think it's not desirable? Including a version number in the
> filename of an installer program seems like a pretty common practice, and I
> generally prefer that as a user. In an embedded context (e.g. using
> gcc-arm-none-eabi), the executable essentially is the installer, too.
>
> Thanks,
>
> Brendan

I think the point that David is trying to make is that a build.ninja
file contains a crystalized build plan including a list of build outputs
to create, and the command to create them.

Including a version number in the filename of an installer bundle is
indeed quite common, but also not typically a "build" artifact, i.e. you
do not repeatedly cd to your project workdir, make tweaks, run `ninja`,
and test out those tweaks by running an installer .MSI to install the
software. You just test out the main program entrypoint.

You'd create an installer once, during your release pipeline, and it
would be a packaging step that consumes the build artifacts. For
example, you could create a "run_target" rule (as meson calls them) that
depends on all your build artifacts to make sure they are up to date and
then runs a packaging script. You don't want the packaging script to run
as part of "all", and you don't care if it checks whether it is up to
date -- you only run it interactively. Ninja doesn't know or care what
filename output it creates, and it dynamically calculates the correct
filename for an installer based on git hash.

But the build target isn't

```
ninja myapp-1.2.3-11-gHASH-64bit.msi
```

It is just

```
ninja make-msi
```

For an embedded context I doubt it is significantly different except
that you can also just rename the executable while deploying it. But I
am skeptical of this because usually executables being deployed to an
embedded platform need the embedded platform to know where and how to
execute the program, which is difficult if the firmware doesn't know the
correct filename to use. I'd simply expect the program to report the git
describe version in its startup logs.



--
Eli Schwartz

[David Turner]

On Mon, Oct 14, 2024 at 1:25 PM Brendan McDonnell <bmcdo...@ieee.org> wrote:

David, Eli, and Ben,

Thanks for your quick and informative responses. I will dig into your suggestions when I get back onto this task.

David and Eli, I'd welcome any additional pointers about how to implement your suggestions from CMake.

Meanwhile, a meta question for David below.

On Tue, Oct 1, 2024 at 10:42 AM David Turner <di...@google.com> wrote:

...

On the other hand, changing the name of the output based on build command is not supported, and generally not desirable.

Why do you think it's not desirable? Including a version number in the filename of an installer program seems like a pretty common practice, and I generally prefer that as a user. In an embedded context (e.g. using gcc-arm-none-eabi), the executable essentially is the installer, too.

Note that I said "in general" :-)

If this is a final artifact (i.e. the output of a command that is never used as an input by another command), there is no issue at all. You can even have a final `install` step that copies the files outside of the build directory and adds a version suffix at the same time.

On the other hand, if this is an intermediate target, its name cannot be in the build plan (since it is computed during the build).

This introduces all kind of complexities, for example:

• The file must be generated as one _hidden_ output of a given command. Many things will not work as expected (e.g. tools like `-t clean` or `-t inputs`). And its proper creation will be hard to observe / debug. Dyndeps can help, but many Ninja generators do not support them.
  
  
• Any command that needs to use the versioned file must find its location using a custom mechanism (the path is not available from $in / $out or any other variable expansion). This will be specific to your project and generally not portable / brittle.
  
  
• Unless you use dyndeps, Ninja cannot test the file's timestamp at all.  I predict you may have some quality time debugging broken incremental builds at some point.

Thus it is doable, under certain conditions, but I don't think it's worth the complexity.