[boost] Windows Regression Runners - Failing on Develop

[Tom Kent via Boost]

Starting sometime between 3am and 5pm on September 11, 2023, all the
windows regression runners started failing on the develop branch.

Master branch runners on the same VM are still doing fine (
https://www.boost.org/development/tests/master/developer/summary.html).

The output doesn't have anything jump out at me. Nor were there any commits
to the develop branch in that window that seem suspicious to me.

The regression run output ends with:
...found 1 target...
# Searching for "process_jam_log.exe" in
"C:\teeks99-09\run\boost_regression\stage\bin"...
Traceback (most recent call last):
File "run.py", line 71, in <module>
runner(root)
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 286, in
__init__
self.main()
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 623, in
main
getattr(self,action_m)()
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 580, in
command_regression
self.command_setup()
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 351, in
command_setup
self.build_if_needed(self.process_jam_log,self.pjl_toolset)
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 714, in
build_if_needed
tool[ 'build_path' ] = self.tool_path( tool )
File "C:\teeks99-09\run\boost_regression_src\regression.py", line 739, in
tool_path
, '\n'.join( [ name_or_spec[ 'path' ], build_dir ] )
Exception: Cannot find "process_jam_log.exe" in any of the following
locations:
C:\teeks99-09\run\boost_regression\stage\bin\process_jam_log.exe
C:\teeks99-09\run\boost_regression\stage\bin

With the full output available here:
https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf

Any ideas on this would be much appreciated.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Peter Dimov via Boost]

Tom Kent wrote:
> Starting sometime between 3am and 5pm on September 11, 2023, all the
> windows regression runners started failing on the develop branch.
>
> Master branch runners on the same VM are still doing fine (
> https://www.boost.org/development/tests/master/developer/summary.html)
> .
>
> The output doesn't have anything jump out at me. Nor were there any
> commits
> to the develop branch in that window that seem suspicious to me.
>
> The regression run output ends with:
> ...found 1 target...

[...]

> With the full output available here:
> https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
>
> Any ideas on this would be much appreciated.

According to the log, the script tries to build process_jam_log.exe using
msvc-10.0.

Since process_jam_log depends on Boost.Filesystem, which now requires
C++11, the build silently fails.

[Andrey Semashev via Boost]

On 9/28/23 02:06, Peter Dimov via Boost wrote:
> Tom Kent wrote:
>> Starting sometime between 3am and 5pm on September 11, 2023, all the
>> windows regression runners started failing on the develop branch.
>>
>> Master branch runners on the same VM are still doing fine (
>> https://www.boost.org/development/tests/master/developer/summary.html)
>> .
>>
>> The output doesn't have anything jump out at me. Nor were there any
>> commits
>> to the develop branch in that window that seem suspicious to me.
>>
>> The regression run output ends with:
>> ...found 1 target...
>
> [...]
>
>> With the full output available here:
>> https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
>>
>> Any ideas on this would be much appreciated.
>
> According to the log, the script tries to build process_jam_log.exe using
> msvc-10.0.
>
> Since process_jam_log depends on Boost.Filesystem, which now requires
> C++11, the build silently fails.

Which brings me back to the idea[1] that Boost.Build should probably
show a warning when an explicitly requested target is not built because
of unsatisfied requirements.

Is this something we can (or should) do on Boost level or should we
request a Boost.Build enhancement?

[1]: https://github.com/boostorg/auto_index/issues/4#issuecomment-1718949433

[Peter Dimov via Boost]

Tom Kent wrote:
> Any ideas on how I can convince that build to use msvc-14.3 while keeping
> msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking
> logic at play that I might be able to control? I'd like to keep still testing with
> that version of visual studio a bit longer, thus I don't want to completely get rid
> of it.

Easiest is probably just to put 14.3 first (in user-config).

[Tom Kent via Boost]

On Wed, Sep 27, 2023 at 6:06 PM Peter Dimov via Boost <bo...@lists.boost.org>
wrote:

> Tom Kent wrote:
> > Starting sometime between 3am and 5pm on September 11, 2023, all the
> > windows regression runners started failing on the develop branch.
> >
> > Master branch runners on the same VM are still doing fine (
> > https://www.boost.org/development/tests/master/developer/summary.html)
> > .
> >
> > The output doesn't have anything jump out at me. Nor were there any
> > commits
> > to the develop branch in that window that seem suspicious to me.
> >
> > The regression run output ends with:
> > ...found 1 target...
>
> [...]
>
> > With the full output available here:
> > https://gist.github.com/teeks99/bb4230f6a1efdd7af11bd93341c950bf
> >
> > Any ideas on this would be much appreciated.
>
> According to the log, the script tries to build process_jam_log.exe using
> msvc-10.0.
>
> Since process_jam_log depends on Boost.Filesystem, which now requires
> C++11, the build silently fails.
>

Any ideas on how I can convince that build to use msvc-14.3 while keeping

msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking
logic at play that I might be able to control? I'd like to keep still
testing with that version of visual studio a bit longer, thus I don't want
to completely get rid of it.

Thanks,
Tom

[Tom Kent via Boost]

On Wed, Sep 27, 2023 at 7:12 PM Peter Dimov <pdi...@gmail.com> wrote:

> Tom Kent wrote:
> > Any ideas on how I can convince that build to use msvc-14.3 while keeping
> > msvc-10.0 in my user-config.jam? I'm guessing there is some auto-picking
> > logic at play that I might be able to control? I'd like to keep still
> testing with
> > that version of visual studio a bit longer, thus I don't want to
> completely get rid
> > of it.
>
> Easiest is probably just to put 14.3 first (in user-config).
>

That did make msvc-14.3 the build tool (b2 documentation on user-config.jam
could be improved to say order matters), however it didn't solve the
problem.

New output here:
https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774

The end:
[3]
msvc-14.3/release/build-no/link-static/python-2.7/threadapi-win32/threading-multi/visibility-hidden

Tom

[Peter Dimov via Boost]

Tom Kent wrote:
> On Wed, Sep 27, 2023 at 7:12 PM Peter Dimov <pdi...@gmail.com
> <mailto:pdi...@gmail.com> > wrote:
>
>
> Tom Kent wrote:
> > Any ideas on how I can convince that build to use msvc-14.3 while
> keeping
> > msvc-10.0 in my user-config.jam? I'm guessing there is some auto-
> picking
> > logic at play that I might be able to control? I'd like to keep still testing
> with
> > that version of visual studio a bit longer, thus I don't want to
> completely get rid
> > of it.
>
> Easiest is probably just to put 14.3 first (in user-config).
>
>
>
> That did make msvc-14.3 the build tool (b2 documentation on user-config.jam
> could be improved to say order matters), however it didn't solve the problem.
>
> New output here:
> https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774
>
> The end:
>
> [3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi-
> win32/threading-multi/visibility-hidden
> ...found 1 target...

That's odd. Even the "has BCrypt API" check fails, and it should succeed.

What does config.log in the binary directory contain?

[Peter Dimov via Boost]

Andrey Semashev wrote:
> > According to the log, the script tries to build process_jam_log.exe using
> > msvc-10.0.
> >
> > Since process_jam_log depends on Boost.Filesystem, which now requires
> > C++11, the build silently fails.
>
> Which brings me back to the idea[1] that Boost.Build should probably
> show a warning when an explicitly requested target is not built because
> of unsatisfied requirements.

I grow increasingly convinced that the way these requirements work
(add <build>no if the requirement isn't met) is no longer what we need.

It's much preferable to have the compile error instead.

In other words, the libraries adding requirements of this kind is in my
opinion not a feature. It creates more problems than it solves - if it even
solves any.

[René Ferdinand Rivera Morell via Boost]

On Thu, Sep 28, 2023 at 11:40 AM Peter Dimov via Boost
<bo...@lists.boost.org> wrote:
>
> Andrey Semashev wrote:
> > > According to the log, the script tries to build process_jam_log.exe using
> > > msvc-10.0.
> > >
> > > Since process_jam_log depends on Boost.Filesystem, which now requires
> > > C++11, the build silently fails.
> >
> > Which brings me back to the idea[1] that Boost.Build should probably
> > show a warning when an explicitly requested target is not built because
> > of unsatisfied requirements.
>
> I grow increasingly convinced that the way these requirements work
> (add <build>no if the requirement isn't met) is no longer what we need.
>
> It's much preferable to have the compile error instead.
>
> In other words, the libraries adding requirements of this kind is in my
> opinion not a feature. It creates more problems than it solves - if it even
> solves any.

+1

And I remember making that point multiple times in the past.

--
-- René Ferdinand Rivera Morell
-- Don't Assume Anything -- No Supone Nada
-- Robot Dreams - http://robot-dreams.net

[Tom Kent via Boost]

I'm not familiar enough with that to make much sense out of it, but it is
here:
https://gist.github.com/teeks99/a59fde8c9fbb9208b3a5dac24a97d4ef

Tom

[Peter Dimov via Boost]

Tom Kent wrote:
> > New output here:
> >
> https://gist.github.com/teeks99/4b0bf5fea31c972be8f3d79fd71bf774
> >
> > The end:
> >
> > [3] msvc-14.3/release/build-no/link-static/python-2.7/threadapi-
> > win32/threading-multi/visibility-hidden
> > ...found 1 target...
>
> That's odd. Even the "has BCrypt API" check fails, and it should succeed.
>
> What does config.log in the binary directory contain?
>
>
>
> I'm not familiar enough with that to make much sense out of it, but it is here:
> https://gist.github.com/teeks99/a59fde8c9fbb9208b3a5dac24a97d4ef

Well the reason all of Filesystem's configure checks fail, even those that ought
to succeed, is

C:\teeks99-09\run\boost_root\libs\filesystem\src\platform_config.hpp(74): fatal error C1083: Cannot open include file: 'boost/winapi/config.hpp': No such file or directory

The reason 'boost/winapi/config.hpp' isn't found is because the corresponding
header symlink in $BOOST_ROOT/boost isn't created, for some reason.

I can reproduce the failure when I delete my boost/ subdirectory, and when I
recreate it with `b2 headers`, I can un-reproduce it.

b2 is supposed to create these header symlinks as it goes, though; the explicit
`b2 headers` is generally unnecessary. Something must have changed to throw
it off, I'm not sure what.

[Steven Watanabe via Boost]

AMDG

On 9/28/23 19:59, Peter Dimov via Boost wrote:
>
> C:\teeks99-09\run\boost_root\libs\filesystem\src\platform_config.hpp(74): fatal error C1083: Cannot open include file: 'boost/winapi/config.hpp': No such file or directory
>
> The reason 'boost/winapi/config.hpp' isn't found is because the corresponding
> header symlink in $BOOST_ROOT/boost isn't created, for some reason.
>
> I can reproduce the failure when I delete my boost/ subdirectory, and when I
> recreate it with `b2 headers`, I can un-reproduce it.
>
> b2 is supposed to create these header symlinks as it goes, though; the explicit
> `b2 headers` is generally unnecessary. Something must have changed to throw
> it off, I'm not sure what.
>

When b2 fails to create a symlink it's usually because the header is
only included through a macro, and the #include scanner misses it.
The fact that entire directories are symlinked also masks most
issues. If the directory is getting split because multiple
libraries contain headers in the same directory, then that can
cause new issues to appear.

In Christ,
Steven Watanabe

[Peter Dimov via Boost]

Steven Watanabe wrote:
> When b2 fails to create a symlink it's usually because the header is
> only included through a macro, and the #include scanner misses it.

That was my first thought, but it's a normal include:

https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a894a06726/src/platform_config.hpp#L74

It might have to do something with the fact that platform_config.hpp
is in src/; not sure how this gets into the include path.

https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a894a06726/config/has_bcrypt.cpp#L9

But boost/winapi/config.hpp is also included directly from the config
check source:

https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a894a06726/config/has_bcrypt.cpp#L17C11-L17C34

so the include scanner must be able to see it anyway.

boost/winapi is a junction/symlink for me, so it's not the split either.

[Peter Dimov via Boost]

> Steven Watanabe wrote:
> > When b2 fails to create a symlink it's usually because the header is
> > only included through a macro, and the #include scanner misses it.
>
> That was my first thought, but it's a normal include:
>
> https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be
> d7280a894a06726/src/platform_config.hpp#L74
>
> It might have to do something with the fact that platform_config.hpp
> is in src/; not sure how this gets into the include path.

Here:

https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a894a06726/config/Jamfile.v2#L40

> https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be

> d7280a894a06726/config/has_bcrypt.cpp#L9
>
> But boost/winapi/config.hpp is also included directly from the config
> check source:
>
> https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193be
> d7280a894a06726/config/has_bcrypt.cpp#L17C11-L17C34
>
> so the include scanner must be able to see it anyway.
>
> boost/winapi is a junction/symlink for me, so it's not the split either.

Further experimentation reveals that the header links are correctly created
when I try to build the has_bcrypt exe directly:

C:\boost-git\develop>rd/s/q boost

C:\boost-git\develop>b2 libs/filesystem/config//has_bcrypt
Performing configuration checks

- default address-model : 64-bit (cached) [1]
- default architecture : x86 (cached) [1]
- symlinks supported : yes (cached)

[1] msvc-14.3
...found 40 targets...
...updating 7 targets...
mklink-or-dir boost
mklink-or-dir boost\predef
symbolic link created for boost\predef <<===>> ..\libs\predef\include\boost\predef
mklink-or-dir boost\winapi
symbolic link created for boost\winapi <<===>> ..\libs\winapi\include\boost\winapi
mklink-or-dir boost\filesystem
symbolic link created for boost\filesystem <<===>> ..\libs\filesystem\include\boost\filesystem
link.mklink boost\config.hpp
symbolic link created for boost\config.hpp <<===>> ..\libs\config\include\boost\config.hpp
mklink-or-dir boost\system
symbolic link created for boost\system <<===>> ..\libs\system\include\boost\system
mklink-or-dir boost\detail
mklink-or-dir boost\config
symbolic link created for boost\config <<===>> ..\libs\config\include\boost\config
link.mklink boost\detail\workaround.hpp
symbolic link created for boost\detail\workaround.hpp <<===>> ..\..\libs\config\include\boost\detail\workaround.hpp
...patience...
link.mklink boost\version.hpp
symbolic link created for boost\version.hpp <<===>> ..\libs\config\include\boost\version.hpp
compile-c-c++ bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.obj
has_bcrypt.cpp
msvc.link bin.v2\libs\filesystem\config\msvc-14.3\debug\threading-multi\has_bcrypt.exe
...updated 13 targets...

So it's only when it's being built by configure.builds:

https://github.com/boostorg/filesystem/blob/0d58357ab9eeaa6ad064193bed7280a894a06726/build/Jamfile.v2#L39

are the headers not being linked.

[Tom Kent via Boost]

Sometime this morning, this seems to have migrated to master as well as
develop.

There was a bit of conversation about symlinks, but I'm not sure that those
are ever being made on this machine, as windows has some weird rules about
what users can make them. Failing that, does it make copies?

If anyone needs access to this environment (i.e. have ideas but not a
windows environment), I can give remote desktop credentials for a test
machine.

Tom

[Andrey Semashev via Boost]

Yes, it should copy the headers if symlinks cannot be created. But, if
Peter is correct, the problem is not that it can't create the symlinks.
The problem is that Boost.Build doesn't track headers required by
configure checks.

I think the regression testing scripts should be running `b2 headers`
before building anything. Automatic creation of the unified include tree
has never worked reliably, so `b2 headers` is a mandatory step after git
checkout.

IMHO, the automatic creation of symlinks should be removed, as it's not
useful and just wastes build time.