Issue 982 in include-what-you-use: ipp files

42 views
Skip to first unread message

notifi...@include-what-you-use.org

unread,
Dec 16, 2021, 6:18:23 AM12/16/21
to include-wh...@googlegroups.com
New issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Hi,

I'm trying to understand what are the correct pragmas to use to setup ipp files (where template implementation is split out into a separate header file to make the main definition easier to read). I'm using the clang_10 branch I can't find a combination that checks all files correctly for the below setup. The below combination of pragmas is crashing with the error `/local/1/home/source_code/include-what-you-use/iwyu_output.h:295: Assertion failed: desired_includes_have_been_calculated_ && "Must calculate desired includes before calling desired_includes()"` I'd expect to be able to setup iwyu so that it would group the header files (and any cpp file) together and validate them as a unit.

```cpp
// a.hpp
template<typename T> void Func1(T & t);
void Func2();
#include "a_ipp.hpp"
```
```cpp
// a_ipp.hpp
// IWYU pragma: private, include "a.hpp"
#include "a.hpp" // IWYU pragma: associated
template<typename T> void Func1(T & t) { /*implementation */ }
```
```cpp
//a.cpp
#include "a.hpp"
void Func2() { /* implementation */ }
```



notifi...@include-what-you-use.org

unread,
Dec 19, 2021, 1:52:00 PM12/19/21
to include-wh...@googlegroups.com
Comment #0 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

> #include "a.hpp" // IWYU pragma: associated

This looks suspicious. "Associated" has a very specific meaning in IWYU: the header that contains the public interface of a .cpp file, and which makes sense to analyze together with its .cpp file. For example:

```
// a.hpp
#include <string>
std::string foo();

// a.cpp
#include "a.hpp"

std::string foo() {
return "foooooooo";
}
```
If a.hpp and a.cpp were not analyzed as a unit (by virtue of being "associated"), it would be reasonable to require that a.cpp includes `<string>`. But since these two are closely related, it makes sense to cheat a little and say that a.hpp can be responsible for `<string>` even in the context of a.cpp.

So I wonder if the assertion disappears if you remove the `IWYU pragma: associated` comment.

And it might also help to skip the include of a.hpp in a_ipp.hpp, to avoid the circular include there. You could replace it with some kind of preprocessor assertion, e.g.
```
// a.hpp
template<typename T> void Func1(T & t);
// ...
#define DEFINING_FUNC1
#include "a_ipp.hpp"
#undef DEFINING_FUNC1

// a_ipp.hpp
// IWYU pragma: private, include "a.hpp"
#ifndef DEFINING_FUNC1
#error "Listen up! Include a.hpp instead."
#endif

notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 2:01:05 AM12/20/21
to include-wh...@googlegroups.com
Comment #1 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

The idea behind using associated in the ipp file is that the ipp is meant to be the equivalent of a cpp file (except for templated code) and therefore if the ipp requires an include that is already included from the header then it shouldn't need to include it (same rules as an associated cpp file). Is that use case not supported by the `IWYU pragma: associated` comment?


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 2:43:51 AM12/20/21
to include-wh...@googlegroups.com
Comment #2 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Do you ever run IWYU on only the ipp file?


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 2:49:01 AM12/20/21
to include-wh...@googlegroups.com
Comment #3 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

We want to check the ipp files yes as we'd like to make them correct and ensure they're only using what they need to help speed up compile times.


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 3:01:06 AM12/20/21
to include-wh...@googlegroups.com
Comment #4 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

I see, thanks. Then the reasoning should hold, though I'm not exactly sure what happens when ipp is included in cpp and the `IWYU pragma: associated` shows up midstream there.

Quite a bit of template specialization fixes have gone in after `clang_10`, so it might be interesting to play around with a later version in a separate environment if it's hard to move past version 10 in your primary.

The assertion you're seeing also had a very specific trigger; if a file was first included using one path, and then skipped (detected to already be included) using another path. Are you invoking IWYU with absolute paths to source files?


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 3:35:09 AM12/20/21
to include-wh...@googlegroups.com
Comment #5 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

We're using the iwyu_tool and all arguments are absolute paths with the exception of --check_also which has absolute path with a wildcard.


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 3:53:20 AM12/20/21
to include-wh...@googlegroups.com
Comment #6 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Could you try tweaking the compilation database to use relative paths instead? Mostly to confirm if the assertion disappears and what frightful things might happen instead. Thanks!


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 5:24:19 AM12/20/21
to include-wh...@googlegroups.com
Comment #7 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982


I've been testing for a bit now and it's an intermittent error :/, is there any known race conditions or threading issues in clang_10? Using CMake and it's inbuilt IWYU runner if I touch all source in our library it will trigger then assertion 99% of the time and the build fails. If I then run the build again (without forcing a touch/rebuild) then it's 50/50 whether the assertion triggers or the build is now able to build the TU that previously failed.


notifi...@include-what-you-use.org

unread,
Dec 20, 2021, 1:37:07 PM12/20/21
to include-wh...@googlegroups.com
Comment #8 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Interesting... I don't think the Clang libraries do any threading behind the scenes (but I haven't followed Clang development closely for a while, so things could have changed).

I wonder if it's some kind of memory corruption issue; would be interesting to see the result with an ASAN-instrumented `include-what-you-use` binary. But I guess you're not building from source?

As for the original example you posted, I get very different behavior:
```
$ include-what-you-use --version
include-what-you-use 0.14 (git:aa7c663) based on clang version 10.0.0-4ubuntu1

$ include-what-you-use tests/982/a.cpp
In file included from tests/982/a.cpp:2:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
tests/982/a.hpp:4:10: error: #include nested too deeply
#include "a_ipp.hpp"
^
In file included from tests/982/a.cpp:2:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
In file included from tests/982/a_ipp.hpp:4:
In file included from tests/982/a.hpp:4:
tests/982/a_ipp.hpp:5:27: error: redefinition of 'Func1'
template<typename T> void Func1(T & t) { /*implementation */ }
^
tests/982/a_ipp.hpp:5:27: note: previous definition is here
template<typename T> void Func1(T & t) { /*implementation */ }
^
```
.. the preprocessor goes into a loop here until it gives up.

If I add include guards (`#pragma once`), it only complains that `a.hpp` doesn't need `a_ipp.hpp`, which can be worked around with a `// IWYU pragma: keep`.

So I wonder what's going on in your environment...


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 1:25:13 AM12/21/21
to include-wh...@googlegroups.com
Comment #9 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

I am building from source but I don't see an IWYU CMake option to enable sanitized builds and I don't see an ASAN build in the ci workflow to copy, what's the best way to enable?


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 1:26:18 AM12/21/21
to include-wh...@googlegroups.com
Comment #10 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

I am building from source but I don't see an IWYU CMake option to enable sanitized builds and I don't see an ASAN build in the ci workflow to copy, what's the best way to enable?

Apologies the code I added was just a motivating pseudo code - I'll get you a real world minimal example shortly.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 1:26:40 AM12/21/21
to include-wh...@googlegroups.com
Comment #10 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

I am building from source but I don't see an IWYU CMake option to enable sanitized builds and I don't see an ASAN build in the ci workflow to copy, what's the best way to enable?

Apologies the code I added was just a motivating pseudo code - I'll get you a real world minimal example.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 1:38:54 AM12/21/21
to include-wh...@googlegroups.com
Comment #10 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

You can simply add `-fsanitize=address` to the compile and link flags.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 1:39:35 AM12/21/21
to include-wh...@googlegroups.com
Comment #11 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

You can simply add `-fsanitize=address` to the compile and link flags. Edit: oh, assuming you're building with Clang. Otherwise override C++ compiler too.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 2:38:53 AM12/21/21
to include-wh...@googlegroups.com
Comment #11 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Get the following ASAN error every time the assertion is hit `iwyu.cc:330:48: runtime error: member call on null pointer of type 'struct ASTNode'`


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 3:01:07 AM12/21/21
to include-wh...@googlegroups.com
Comment #12 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Ah, sounds like you're hitting https://github.com/include-what-you-use/include-what-you-use/issues/767. I wonder if there's a correlation, though.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 3:12:32 AM12/21/21
to include-wh...@googlegroups.com
Comment #13 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Yea looks to be just correlation. I cherry-picked [#767](https://github.com/include-what-you-use/include-what-you-use/issues/767) into clang_10 and rebuilt and still get the assertion but no ASAN error.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 3:15:37 AM12/21/21
to include-wh...@googlegroups.com
Comment #14 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Yea looks to be unrelated. I cherry-picked [#767](https://github.com/include-what-you-use/include-what-you-use/issues/767) into clang_10 and rebuilt and still get the assertion but no ASAN error.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 3:16:19 AM12/21/21
to include-wh...@googlegroups.com
Comment #14 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Yea looks to be unrelated. I cherry-picked [#767](https://github.com/include-what-you-use/include-what-you-use/issues/767) into clang_10 and rebuilt and the ASAN error goes away but the the assertion is still getting hit.


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 8:19:59 AM12/21/21
to include-wh...@googlegroups.com
Comment #14 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

I've had no luck creating a minimal example to reproduce.



I've been trying to reduce the source code of the error case as much as possible to try and get an example from that direction but I'm getting very unusual results. I've paired down the error to a single TU. When I remove an unused dependency (so that it's include folders are no longer in the command line) the error disappears and is not reproducible! Including the include files from [soci](https://github.com/SOCI/soci) with the MYSQL compilation option enabled causes the error to occur in our code 100% of the time and removing soci removes the error... I think it’s unlikely that there’s actually an issue in soci and it’s more than likely just another way some sort of race condition is rearing its head. Is there anything more that can be done to try and debug this?



#0 include_what_you_use::IwyuFileInfo::desired_includes (this=0x6150002f9ea8) at /local/1/home/corcoand/source_code/include-what-you-use/iwyu_output.h:299

#1 0x0000000000e1c45e in include_what_you_use::IwyuFileInfo::AssociatedDesiredIncludes (this=0x615000319a28) at /local/1/home/corcoand/source_code/include-what-you-use/iwyu_output.h:323

#2 0x0000000000e10102 in include_what_you_use::IwyuFileInfo::CalculateAndReportIwyuViolations (this=0x615000319a28) at /local/1/home/corcoand/source_code/include-what-you-use/iwyu_output.cc:2144

#3 0x0000000000497da7 in include_what_you_use::IwyuAstConsumer::HandleTranslationUnit (this=0x614000002840, context=...) at /local/1/home/corcoand/source_code/include-what-you-use/iwyu.cc:3663

#4 0x00007fffefedef69 in clang::ParseAST(clang::Sema&, bool, bool) () from /opt/rh/llvm-toolset-10.0/root/usr/lib64/libclang-cpp.so.10

#5 0x00007ffff182bec9 in clang::FrontendAction::Execute() () from /opt/rh/llvm-toolset-10.0/root/usr/lib64/libclang-cpp.so.10

#6 0x00007ffff17e5bab in clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) () from /opt/rh/llvm-toolset-10.0/root/usr/lib64/libclang-cpp.so.10

#7 0x0000000000425c21 in main (argc=51, argv=0x7fffffffce78) at /local/1/home/corcoand/source_code/include-what-you-use/iwyu.cc:4128


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 12:28:01 PM12/21/21
to include-wh...@googlegroups.com
Comment #15 on issue 982 by andrewkcorcoran: ipp files
```


notifi...@include-what-you-use.org

unread,
Dec 21, 2021, 12:37:12 PM12/21/21
to include-wh...@googlegroups.com
Comment #16 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Hmm. So now that you're building from source, have you also cherry-picked https://github.com/include-what-you-use/include-what-you-use/commit/d9658485d05e07f5dc74a4e393b871a3d75d6143? If not, does it help to do so, or are we hitting some other quirk with soci?

I'd like to think there are no race conditions as such in IWYU, as there should be no multithreading, but I'm always prepared for a surprise :-)



notifi...@include-what-you-use.org

unread,
Dec 22, 2021, 12:37:40 AM12/22/21
to include-wh...@googlegroups.com
Comment #17 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

It turns out d965848 is already in the clang_10 branch. So looks like this issue (while resulting in the same assertion being hit) is different than #738.



It's possible it's not a race condition. I've seen similar non-deterministic behaviour when code assumed that an unordered container had defined ordering. e.g. looping through unordered_maps and expecting ordering to be the same across multiple maps with the same elements.


notifi...@include-what-you-use.org

unread,
Dec 22, 2021, 2:01:45 AM12/22/21
to include-wh...@googlegroups.com
Comment #18 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Oh, I wouldn't have guessed clang 10 and that fix were in the same time period. The commit message for 738 mentions parallel structures, one with quoted-includes and one with FileEntry pointers. I wonder if this might be a similar problem, that AddGlobToReport... isn't called for all files.

Maybe printf debugging might give more information here.


notifi...@include-what-you-use.org

unread,
Dec 22, 2021, 6:46:12 AM12/22/21
to include-wh...@googlegroups.com
Comment #18 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982


The non-determinism is coming from the fact that the files to check are stored as a set of pointers. The ordering of the pointers is somewhat random and it looks like if the ipp file is ordered before the hpp file then the assertion is hit.
https://github.com/include-what-you-use/include-what-you-use/blob/master/iwyu.cc#L3656



notifi...@include-what-you-use.org

unread,
Dec 22, 2021, 11:57:29 AM12/22/21
to include-wh...@googlegroups.com
Comment #19 on issue 982 by kimgr: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Interesting!

It's not 100% clear to me what source code is triggering this, but you have something like the repro example above, and also involving soci? I think the assumption mentioned in the `NOTE` here might be the root cause: https://github.com/include-what-you-use/include-what-you-use/blob/master/iwyu_output.cc#L1604.

There seems to be an implicit dependency between the `AssociatedDesiredIncludes` call in `CalculateAndReportIwyuViolations` and the finalization that happens in `CalculateIwyuViolations`. Or maybe `CalculateIwyuViolations` just needs to be called first to finalize the associated-ness.

I need to think more about why order matters here, but this is a good start.




notifi...@include-what-you-use.org

unread,
Dec 23, 2021, 1:50:42 AM12/23/21
to include-wh...@googlegroups.com
Comment #20 on issue 982 by andrewkcorcoran: ipp files
https://github.com/include-what-you-use/include-what-you-use/issues/982

Unfortunately I haven't been able to get an minimal example working. The more I strip out the more deterministic the ordering becomes and the less likely the assertion is hit.


Reply all
Reply to author
Forward
0 new messages