Possible issue with reclient and ninja rebuild decisions

[Yngve N. Pettersen]

Hi,

A number of our developers have been having problems the past several
months with constant rebuilding of hundreds or thousands of ninja targets,
even if NO files have been modified.

I haven't seen this myself, but I do full rebuilds fairly often due to
rebases.

My suspicion is that this is caused by a source file (probably a header
file "foo.h") having been modified so that TimeStamp("foo.h") is newer than
TimeStamp("file.obj") (file.cc includes the h file), but in such a way that
Compile("file.cc") result in the exact same binary object file as the
previous build action did.

My guess is Reclient does not write the duplicate result to disk, but
neither does it *touch* the existing object file (which is apparently what
Make does), so that ninja can know that the object file is no longer dirty.

The result, if that is what happens, is that "file.obj" is always dirty.

The only real way to "solve" the problem is to delete the out dir, and
start a full rebuild.

--
Sincerely,
Yngve N. Pettersen
Vivaldi Technologies AS

[Michael Savigny]

Hey Yngve, here's a few thoughts on what might be the problem.

Rewrapper has a flag to define the behaviour of mtimes on action outputs that haven't changed:

preserve_unchanged_output_mtime: "Boolean indicating whether or not to preserve mtimes of unchanged outputs when they are downloaded. Default is false."

The default is to not preserve the mtime, and I don't think that the chromium build changes this setting.

There is also the ninja 'restat' feature, which also affects what gets build/rebuilt in these kind of situations (https://ninja-build.org/manual.html)

The symptoms you are describing sound very similar to cases where there is a non-deterministic file being generated as part of the build that is a dependency for those actions getting rebuilt.  A common case of this problem is having a generated header that contains things like build timestamps or the commit hash of the build.

For an example in chromium, check the gn argument 'use_dummy_lastchange' (https://source.chromium.org/chromium/chromium/src/+/main:build/util/lastchange.gni).  If this is true, then the generated file doesn't contain the real time and commit hash, it just has constant fake values to prevent unnecessary invalidations in the build.

If fiddling around with the reclient preserve_mtime and ninja restat settings don't fix the problem, have a look and see if there is a common dependency for the actions being rebuilt all the time that could be the problem.

Let me know how it goes!

Mike

--
You received this message because you are subscribed to the Google Groups "reclient-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reclient-user...@chromium.org.
To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/1737041135248.1739417960.2176925284%40vivaldi.com.

[Yngve N. Pettersen]

Hi,

Will check that. However, there are no changes to that flag in our configuration, at least. I also see that it is NOT documented in the reclient docs at bazel github, I had to go into the source to find it.

One case we did narrow down is reported upstream (via a chromium dev) as https://issues.chromium.org/issues/382387670 , in part because the GN config added test targets that was unnecessary for us (we have now added an override to remove that dep in our code), but which cause several hundred target rebuilds every build some of our devs.

A problem with using ninja -d explain is that the current ninja version outputs a very large wall of text for a project like chromium. (This is supposed to have been fixed or made more readable in the next ninja version, but that is not yet released)

The most recent cases seems to have started yesterday after we did a minor chromium bump for 132 two days ago, and one dev reports that the reported number of dirty items is 20000+ , although it eventually goes down to 2000ish items.

To view this discussion visit https://groups.google.com/a/chromium.org/d/msgid/reclient-users/CA%2BgbkXugsjVoRopNxsxRkDGdg03Oz-dcXCxovPoD4jPV6jMDow%40mail.gmail.com.

[Michael Savigny]

Follow up question.  The actions that are being invalidated and rerun, when they are run through reclient are they coming back as remote cache hits or are they being executed?