rebuild reason on nija deps

[Michael Nazzareno Trimarchi]

Hi all

I have done a project but I notice that re-build is triggered even if
the file is not changed but just edit, restore to original and save.
It's that something that ninja should take in account in dep
calculation?

Michael

[Lee Winter]

Your procedure does not change the file contents, but does change the file timestamp.

-- Lee

-- 

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwmBQeZgfEvkOF8ewCP_tnHWFE30-H2UdS-j7K3mTFJzAQ%40mail.gmail.com.

[Michael Nazzareno Trimarchi]

Hi Lee

and what sense can you see to rebuild it?

Michael

[Michael Nazzareno Trimarchi]

Hi

On Thu, Jun 1, 2023 at 4:49 PM David Turner <di...@google.com> wrote:

>
>
>
> On Thu, Jun 1, 2023 at 4:32 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>
>> Hi Lee
>>
>> and what sense can you see to rebuild it?
>>

> Ninja uses filesystem timestamps to decide when to re-run commands, as this is much simpler and faster than trying to compare the content of each input file.
> When you save the file, even if it is unmodified, you refresh the timestamp to a newer (i.e. greater) value. Ninja will then run any command that uses that file as input on the next invocation (and anything that depends on its outputs as well).
>
> So the behavior you are seeing is working as intended.
>
> There are many reasons to consider that relying on timestamps is a bad idea, but all the alternatives improve correctness at the cost of great complexity. And Ninja is, by design, a simple tool.
>

So it means that it's much faster to compile a c ++ file then
calculate an md5 on the file. I mean do you have an article or paper
that shows that it's a good compromise?

Michael

>> To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwk3Kc3iHwnqp5r1PmR8ioNEkjZ4NU6KOj5V34NFAiiuaA%40mail.gmail.com.

[K. Moon]

It's faster to not compile 5000 C++ files than to hash 5000 files, and discover that they haven't changed. This doesn't require a paper to demonstrate; it should be obvious, but it's also very easy to test.

The only efficient way to implement a content-aware solution requires a custom file system or watcher approach. Timestamps are a simpler solution that works for most use cases.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uw%3DmcaqF5sNpd3B4jV4-gYU2w7X6y7rj%3DxBgK0KsrO37%2BQ%40mail.gmail.com.

[K. Moon]

If you want to implement a wrapper that hashes the C++ file before compiling it, that's also relatively easy to implement, but gets away from keeping the build system simple. It's also not always going to be a win in terms of overall performance.

[Michael Nazzareno Trimarchi]

Hi

On Thu, Jun 1, 2023 at 7:40 PM K. Moon <km...@chromium.org> wrote:
>
> If you want to implement a wrapper that hashes the C++ file before compiling it, that's also relatively easy to implement, but gets away from keeping the build system simple. It's also not always going to be a win in terms of overall performance.
>
> On Thu, Jun 1, 2023 at 10:39 AM K. Moon <km...@chromium.org> wrote:
>>
>> It's faster to not compile 5000 C++ files than to hash 5000 files, and discover that they haven't changed. This doesn't require a paper to demonstrate; it should be obvious, but it's also very easy to test.
>>

time sha256sum `find -name *.cpp` | wc -l
145

real 0m0,021s
user 0m0,020s
sys 0m0,004s

Suppose to get a CI task and you build the code without removing the
build directory and then you checkout a newer version of the software
suppose to be 5000 files c++, and 89% of the files are the same as the
previous build or just git stash your changes, git rebase , git stash
apply and try to build 145 c++ files again

Michael

[Michael Nazzareno Trimarchi]

Hi

On Thu, Jun 1, 2023 at 7:47 PM Kousik Kumar <kous...@gmail.com> wrote:
>
> There was a patch to Ninja to invalidate based on file hash instead of relying only on file modification times - https://github.com/ninja-build/ninja/pull/929. The patch is intended for exactly the kind of use case you are describing.
>

I will take a look at it

thank you
Michael


> Kousik

>
> On Thu, Jun 1, 2023 at 9:23 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>

[K. Moon]

I did a similar experiment on my system, and while Ninja can perform operations in parallel, the SHA-256 sum took on the order of a minute to complete on my system, while it took only a few seconds for Ninja to perform a no-op build.

On Thu, Jun 1, 2023 at 11:04 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

Hi

On Thu, Jun 1, 2023 at 7:47 PM Kousik Kumar <kous...@gmail.com> wrote:
>
> There was a patch to Ninja to invalidate based on file hash instead of relying only on file modification times - https://github.com/ninja-build/ninja/pull/929. The patch is intended for exactly the kind of use case you are describing.
>

I will take a look at it

thank you
Michael


> Kousik
>
> On Thu, Jun 1, 2023 at 9:23 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>
>> Hi all
>>
>> I have done a project but I notice that re-build is triggered even if
>> the file is not changed but just edit, restore to original and save.
>> It's that something that ninja should take in account in dep
>> calculation?
>>
>> Michael
>>
>> --
>> You received this message because you are subscribed to the Google Groups "ninja-build" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwmBQeZgfEvkOF8ewCP_tnHWFE30-H2UdS-j7K3mTFJzAQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwndmRAAjNc4NkfEZpG8_Qq%2BWzN5cisWbLnJ%2BLceTKXZmQ%40mail.gmail.com.

[Ben Boeckel]

On Thu, Jun 01, 2023 at 11:23:41 -0700, K. Moon wrote:
> I did a similar experiment on my system, and while Ninja can perform
> operations in parallel, the SHA-256 sum took on the order of a minute to
> complete on my system, while it took only a few seconds for Ninja to
> perform a no-op build.

Is the sha256sum unconditional or is it gated by a timestamp
shortcircuit as well?

--Ben

[K. Moon]

It probably should be unconditional, since the point is to detect content changes (or not) not reflected in the timestamp. A conditional approach would be equivalent to wrapping the compiler driver, since you could rely on the build system to handle gating on timestamp.

[Michael Nazzareno Trimarchi]

Il gio 1 giu 2023, 21:53 K. Moon <km...@chromium.org> ha scritto:

It probably should be unconditional, since the point is to detect content changes (or not) not reflected in the timestamp. A conditional approach would be equivalent to wrapping the compiler driver, since you could rely on the build system to handle gating on timestamp.

Well you can not analyze file where timestamp is not changed and calculate sha256 when is needed. 

Michael 

[K. Moon]

There's a whole space of trade-offs here, depending on your particular needs; I think the point being that there isn't one solution that works for everyone.

[Brian Osman]

Exactly. Every build system ever written has to weigh these tradeoffs. Some focus on simplicity, which tends to favor purely operating on timestamps. Others lean into content-aware build rules. Once you go down that road ... you need to store the previous hashes somewhere. Of course, caching is complex and adds more layers of failure, so it's entirely justifiable to skip that step. Having worked with (and written) build systems that operate both ways ... I find the marginal benefits of content-aware rebuilding to be vastly outweighed by the downsides, except in extremely specific circumstances. (And as others have mentioned - you can often get some of the benefits with another layer above/below a tool like ninja).

--

You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CACwGi-7os1smaLZK9rjuTmoo83Ehe-d--hprbLD3xVjV9ahG9w%40mail.gmail.com.

[Daniel Moody]

I thought this was a good write up of some of the issue with mtime: https://apenwarr.ca/log/20181113

That said I think mtime is okay enough for most use cases, but it would be nice if ninja made it optional

 

{ name     : "Daniel Moody",
  title    : "Sen Server Development Platform Eng",
  phone    : "321-506-6255",
  location : "Cedar Rapids, IA",
  twitter  : "@MongoDB",
  facebook : "MongoDB" }

On Thu, Jun 1, 2023 at 1:04 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

Hi

On Thu, Jun 1, 2023 at 7:47 PM Kousik Kumar <kous...@gmail.com> wrote:
>
> There was a patch to Ninja to invalidate based on file hash instead of relying only on file modification times - https://github.com/ninja-build/ninja/pull/929. The patch is intended for exactly the kind of use case you are describing.
>

I will take a look at it

thank you
Michael


> Kousik
>
> On Thu, Jun 1, 2023 at 9:23 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>
>> Hi all
>>
>> I have done a project but I notice that re-build is triggered even if
>> the file is not changed but just edit, restore to original and save.
>> It's that something that ninja should take in account in dep
>> calculation?
>>
>> Michael
>>
>> --
>> You received this message because you are subscribed to the Google Groups "ninja-build" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwmBQeZgfEvkOF8ewCP_tnHWFE30-H2UdS-j7K3mTFJzAQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwndmRAAjNc4NkfEZpG8_Qq%2BWzN5cisWbLnJ%2BLceTKXZmQ%40mail.gmail.com.

[Kousik Kumar]

There was a patch to Ninja to invalidate based on file hash instead of relying only on file modification times - https://github.com/ninja-build/ninja/pull/929. The patch is intended for exactly the kind of use case you are describing.

Kousik

On Thu, Jun 1, 2023 at 9:23 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

[David Turner]

On Thu, Jun 1, 2023 at 4:56 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

Hi

On Thu, Jun 1, 2023 at 4:49 PM David Turner <di...@google.com> wrote:
>
>
>
> On Thu, Jun 1, 2023 at 4:32 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>
>> Hi Lee
>>
>> and what sense can you see to rebuild it?
>>
> Ninja uses filesystem timestamps to decide when to re-run commands, as this is much simpler and faster than trying to compare the content of each input file.
> When you save the file, even if it is unmodified, you refresh the timestamp to a newer (i.e. greater) value. Ninja will then run any command that uses that file as input on the next invocation (and anything that depends on its outputs as well).
>
> So the behavior you are seeing is working as intended.
>
> There are many reasons to consider that relying on timestamps is a bad idea, but all the alternatives improve correctness at the cost of great complexity. And Ninja is, by design, a simple tool.
>

So it means that it's much faster to compile a c ++ file then
calculate an md5 on the file. I mean do you have an article or paper
that shows that it's a good compromise?

No, it means that it is much faster to lookup one file's timestamp than computing its content hash.

Here's some discussion about this https://softwareengineering.stackexchange.com/questions/319359/why-incremental-builds-in-make-dont-use-hashing-algorithms though you will find plenty of other references on the subject.

[David Turner]

On Thu, Jun 1, 2023 at 4:32 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

Hi Lee

and what sense can you see to rebuild it?

Ninja uses filesystem timestamps to decide when to re-run commands, as this is much simpler and faster than trying to compare the content of each input file.
When you save the file, even if it is unmodified, you refresh the timestamp to a newer (i.e. greater) value. Ninja will then run any command that uses that file as input on the next invocation (and anything that depends on its outputs as well).

So the behavior you are seeing is working as intended.

There are many reasons to consider that relying on timestamps is a bad idea, but all the alternatives improve correctness at the cost of great complexity. And Ninja is, by design, a simple tool.




 

Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwk3Kc3iHwnqp5r1PmR8ioNEkjZ4NU6KOj5V34NFAiiuaA%40mail.gmail.com.

[Mathias Stearn]

On Thu, Jun 1, 2023 at 4:56 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:

Hi

On Thu, Jun 1, 2023 at 4:49 PM David Turner <di...@google.com> wrote:
>
>
>
> On Thu, Jun 1, 2023 at 4:32 PM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>>
>> Hi Lee
>>
>> and what sense can you see to rebuild it?
>>
> Ninja uses filesystem timestamps to decide when to re-run commands, as this is much simpler and faster than trying to compare the content of each input file.
> When you save the file, even if it is unmodified, you refresh the timestamp to a newer (i.e. greater) value. Ninja will then run any command that uses that file as input on the next invocation (and anything that depends on its outputs as well).
>
> So the behavior you are seeing is working as intended.
>
> There are many reasons to consider that relying on timestamps is a bad idea, but all the alternatives improve correctness at the cost of great complexity. And Ninja is, by design, a simple tool.
>

So it means that it's much faster to compile a c ++ file then
calculate an md5 on the file. I mean do you have an article or paper
that shows that it's a good compromise?

Have you considered using something like https://ccache.dev/? It will basically do exactly what you want, but even better because it also handles cases like git stash after running a compile. I basically always combine ninja + ccache and they work very well together.

 

 

Michael


>
>
>
>
>>
>> Michael
>>
>> On Thu, Jun 1, 2023 at 3:30 PM Lee Winter <lee.j.i...@gmail.com> wrote:
>> >
>> > Your procedure does not change the file contents, but does change the file timestamp.
>> >
>> > -- Lee
>> > --
>> >
>> > On Thu, Jun 1, 2023 at 9:23 AM Michael Nazzareno Trimarchi <mic...@amarulasolutions.com> wrote:
>> >>
>> >> Hi all
>> >>
>> >> I have done a project but I notice that re-build is triggered even if
>> >> the file is not changed but just edit, restore to original and save.
>> >> It's that something that ninja should take in account in dep
>> >> calculation?
>> >>
>> >> Michael
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google Groups "ninja-build" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
>> >> To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwmBQeZgfEvkOF8ewCP_tnHWFE30-H2UdS-j7K3mTFJzAQ%40mail.gmail.com.
>>
>> --
>> You received this message because you are subscribed to the Google Groups "ninja-build" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uwk3Kc3iHwnqp5r1PmR8ioNEkjZ4NU6KOj5V34NFAiiuaA%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "ninja-build" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ninja-build...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/ninja-build/CAOf5uw%3DmcaqF5sNpd3B4jV4-gYU2w7X6y7rj%3DxBgK0KsrO37%2BQ%40mail.gmail.com.