Issue 1235 in include-what-you-use: New IWYU pragma for extension point patterns
2 views
Skip to first unread message
notifi...@include-what-you-use.org
Apr 12, 2023, 12:21:10 PM4/12/23
to include-wh...@googlegroups.com
New issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
# IWYU always_keep pragma proposal
## Background
In clang-tooling space there is a new focus around getting more automated & safe include management, similar to IWYU with a new “use” policy focused around spelling of symbols (hence also called as include-what-you-spell, IWYS). See [clang-include-cleaner](https://github.com/llvm/llvm-project/tree/main/clang-tools-extra/include-cleaner/)
## Problem
It’s common in C++ to have extension points for types to enable their usage in certain generic algorithms, e.g. one common example could look like:
foo.h:
```cpp
struct Foo { … };
```
foo_ext.h:
```cpp
#include “foo.h”
bool isLess(Foo&, Foo&);
```
sort.h:
```cpp
template<typename Range> void sort(Range) { /* uses isLess(T&,T&) */ }
```
application.cc:
```cpp
#include “foo.h” // provides vector<Foo>& getStats();
#include “sort.h”
#include “foo_ext.h” // unused, there’s no "direct" usage of isLess(Foo&,Foo&)
void app() {
auto &f = getStats(); // returns a vector<Foo>&.
sort(f);
}
```
Here, removing `foo_ext.h` will prevent the code from compiling, so we need to avoid suggesting that. However no symbol from the file is "used" in our usual sense.
## Current ways to tackle such usages
### Analyze template instantiations and mark the usage of such extension points
This is close to the current IWYU tool and powerful, but doesn’t align with IWYS’s definition of “use”. It complicates the implementation drastically and is impossible in certain tools (e.g. clangd, which doesn’t keep template bodies from headers around). Moreover, deducing intent fully is still error-prone. Most of these arguments and complexity is also mentioned in [https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYUIsDifficult.md#who-is-responsible-for-dependent-template-types](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYUIsDifficult.md#who-is-responsible-for-dependent-template-types). So having all this complexity and still getting certain cases wrong goes against our automation goals.
### Restrict such patterns
We can call such patterns IWYU violations (and they are), and forbid people from using them. But this is really hard to enforce in practice. Most of the common alternatives require:
* extensions to go with type, which is not always possible (e.g. type might be generated, provider of the extension point and type might be completely different sets of parties, etc.)
* introduce these symbols at linking time via registries or forward declaring them.
All of these are drastic changes to the way people write C++ code today, hence it’ll make adoption unlikely on existing codebases and new code because people still need to use existing libraries that violate these new style guidelines.
### Use existing IWYU pragmas
IWYU export and IWYU keep pragmas are relevant in this case.
In theory an IWYU export pragma inside foo_ext.h on forward declaration or the type could help in certain cases. But:
* from IWYS perspective, the above application doesn’t really have a dependency on type Foo as it’s never spelled. This gives the library provider the ability to change their APIs easily (e.g. return type of getStats could change from Foo to Bar without breaking the application). So this doesn’t work in all cases.
* this allows the application to include foo_ext.h and not foo.h, even when Foo is used directly
IWYU keep pragma on the inclusion of foo_ext.h definitely works, but it pushes too much work to many end-users with completely obscure reasons. Hence creates a hard to understand model of C++ include management.
## Proposal, a new IWYU pragma
Considering these alternatives, we’d like to introduce a new pragma, IWYU pragma: always_keep (bikeshedding welcome), to be used on such extension headers and disable deletion of any include of a header that has an always_keep pragma.
This has the benefit of:
* centralizing the IWYU pragma management, only the extension provider needs to introduce a single pragma.
* no possibility of breaking the builds or semantics of a program.
* making it more obvious when an extension provider is working against IWYU principles and have them explicitly consider alternatives.
But it comes at the cost of less fine grained analysis. The include needs to be cleaned up by the application manually even if all the usages in code go away. We believe this cost is acceptable, since the suggested pragma also makes such headers easy to find and clean up centrally.
## Follow up work
We’d like to get this mentioned in [https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md) documentation as an official one and help introducing it into the current IWYU tool.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
# IWYU always_keep pragma proposal
## Background
In clang-tooling space there is a new focus around getting more automated & safe include management, similar to IWYU with a new “use” policy focused around spelling of symbols (hence also called as include-what-you-spell, IWYS). See [clang-include-cleaner](https://github.com/llvm/llvm-project/tree/main/clang-tools-extra/include-cleaner/)
## Problem
It’s common in C++ to have extension points for types to enable their usage in certain generic algorithms, e.g. one common example could look like:
foo.h:
```cpp
struct Foo { … };
```
foo_ext.h:
```cpp
#include “foo.h”
bool isLess(Foo&, Foo&);
```
sort.h:
```cpp
template<typename Range> void sort(Range) { /* uses isLess(T&,T&) */ }
```
application.cc:
```cpp
#include “foo.h” // provides vector<Foo>& getStats();
#include “sort.h”
#include “foo_ext.h” // unused, there’s no "direct" usage of isLess(Foo&,Foo&)
void app() {
auto &f = getStats(); // returns a vector<Foo>&.
sort(f);
}
```
Here, removing `foo_ext.h` will prevent the code from compiling, so we need to avoid suggesting that. However no symbol from the file is "used" in our usual sense.
## Current ways to tackle such usages
### Analyze template instantiations and mark the usage of such extension points
This is close to the current IWYU tool and powerful, but doesn’t align with IWYS’s definition of “use”. It complicates the implementation drastically and is impossible in certain tools (e.g. clangd, which doesn’t keep template bodies from headers around). Moreover, deducing intent fully is still error-prone. Most of these arguments and complexity is also mentioned in [https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYUIsDifficult.md#who-is-responsible-for-dependent-template-types](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYUIsDifficult.md#who-is-responsible-for-dependent-template-types). So having all this complexity and still getting certain cases wrong goes against our automation goals.
### Restrict such patterns
We can call such patterns IWYU violations (and they are), and forbid people from using them. But this is really hard to enforce in practice. Most of the common alternatives require:
* extensions to go with type, which is not always possible (e.g. type might be generated, provider of the extension point and type might be completely different sets of parties, etc.)
* introduce these symbols at linking time via registries or forward declaring them.
All of these are drastic changes to the way people write C++ code today, hence it’ll make adoption unlikely on existing codebases and new code because people still need to use existing libraries that violate these new style guidelines.
### Use existing IWYU pragmas
IWYU export and IWYU keep pragmas are relevant in this case.
In theory an IWYU export pragma inside foo_ext.h on forward declaration or the type could help in certain cases. But:
* from IWYS perspective, the above application doesn’t really have a dependency on type Foo as it’s never spelled. This gives the library provider the ability to change their APIs easily (e.g. return type of getStats could change from Foo to Bar without breaking the application). So this doesn’t work in all cases.
* this allows the application to include foo_ext.h and not foo.h, even when Foo is used directly
IWYU keep pragma on the inclusion of foo_ext.h definitely works, but it pushes too much work to many end-users with completely obscure reasons. Hence creates a hard to understand model of C++ include management.
## Proposal, a new IWYU pragma
Considering these alternatives, we’d like to introduce a new pragma, IWYU pragma: always_keep (bikeshedding welcome), to be used on such extension headers and disable deletion of any include of a header that has an always_keep pragma.
This has the benefit of:
* centralizing the IWYU pragma management, only the extension provider needs to introduce a single pragma.
* no possibility of breaking the builds or semantics of a program.
* making it more obvious when an extension provider is working against IWYU principles and have them explicitly consider alternatives.
But it comes at the cost of less fine grained analysis. The include needs to be cleaned up by the application manually even if all the usages in code go away. We believe this cost is acceptable, since the suggested pragma also makes such headers easy to find and clean up centrally.
## Follow up work
We’d like to get this mentioned in [https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md](https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/IWYUPragmas.md) documentation as an official one and help introducing it into the current IWYU tool.
notifi...@include-what-you-use.org
Apr 13, 2023, 1:35:44 AM4/13/23
to include-wh...@googlegroups.com
Comment #2 on issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
glad to hear that you liked the idea!
as for the naming, in the proposal I stick to `always_keep` as it is similar to existing `IWYU pragma: keep` (just feels like a natural generalization of the existing pragma).
one other alternative pointed out during our offline discussions is `IWYU pragma: no_remove`, similar to `no_include`. but this one felt "heavier" as despite sharing the `no_` prefix, it actually introduces a new "concept" that doesn't sound related to `keep pragma` at first.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
glad to hear that you liked the idea!
as for the naming, in the proposal I stick to `always_keep` as it is similar to existing `IWYU pragma: keep` (just feels like a natural generalization of the existing pragma).
one other alternative pointed out during our offline discussions is `IWYU pragma: no_remove`, similar to `no_include`. but this one felt "heavier" as despite sharing the `no_` prefix, it actually introduces a new "concept" that doesn't sound related to `keep pragma` at first.
notifi...@include-what-you-use.org
Apr 13, 2023, 2:36:18 PM4/13/23
to include-wh...@googlegroups.com
Comment #3 on issue 1235 by kimgr: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
There's something that feels off about how `always_keep` sounds like a stronger version of `keep`, which isn't really true. It's just a matter of directionality, isn't it?
The most closely related (in terms of where they're used) seem to be `private` and `friend`, but they don't really share anything that could form a category or convention.
I was toying with `preserve`, but that's really just a synonym of `keep`, so it doesn't cut it.
I actually like `no_remove`, but it's a shame that its "siblings" `no_include` and `no_forward_declare` are both used at point of includer, as opposed to this one which marks a header itself. On the other hand I'm not sure the `no_` prefix implies anything about directionality.
`IWYU pragma: sacred`? :smiley:
https://github.com/include-what-you-use/include-what-you-use/issues/1235
There's something that feels off about how `always_keep` sounds like a stronger version of `keep`, which isn't really true. It's just a matter of directionality, isn't it?
The most closely related (in terms of where they're used) seem to be `private` and `friend`, but they don't really share anything that could form a category or convention.
I was toying with `preserve`, but that's really just a synonym of `keep`, so it doesn't cut it.
I actually like `no_remove`, but it's a shame that its "siblings" `no_include` and `no_forward_declare` are both used at point of includer, as opposed to this one which marks a header itself. On the other hand I'm not sure the `no_` prefix implies anything about directionality.
`IWYU pragma: sacred`? :smiley:
notifi...@include-what-you-use.org
Apr 15, 2023, 7:50:39 AM4/15/23
to include-wh...@googlegroups.com
Comment #4 on issue 1235 by kimgr: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
One thing that occurred to me -- I'm not sure if I like it -- is we could just use `IWYU pragma: keep` on a separate line to indicate that an included header should always be kept;
```
/* current.cc */
#include "blah.h" // IWYU pragma: keep
class Foo; // IWYU pragma: keep
#include "suggested.h"
/* suggested.h */
// IWYU pragma: keep
class Bar {};
```
On the whole I think it's pretty clear, but it's a new kind of two-sided construct.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
One thing that occurred to me -- I'm not sure if I like it -- is we could just use `IWYU pragma: keep` on a separate line to indicate that an included header should always be kept;
```
/* current.cc */
#include "blah.h" // IWYU pragma: keep
class Foo; // IWYU pragma: keep
#include "suggested.h"
/* suggested.h */
// IWYU pragma: keep
class Bar {};
```
On the whole I think it's pretty clear, but it's a new kind of two-sided construct.
notifi...@include-what-you-use.org
Apr 19, 2023, 6:09:22 AM4/19/23
to include-wh...@googlegroups.com
Comment #5 on issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
sorry for the lat reply.
> There's something that feels off about how always_keep sounds like a stronger version of keep, which isn't really true. It's just a matter of directionality, isn't it?
right, it's mostly to indicate that an include of this file should be kept always.
> IWYU pragma: sacred
or "scared" :P
> we could just use IWYU pragma: keep on a separate line
this sounds like a backward incompatible change and I think it's likely to trigger coincidental behaviours in most of the existing tools, e.g. clang has a mechanism to attach comments to decls and if it precedes a declaration it'll get attached to that. so i'd rather not "overload" the meaning of an existing pragma.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
sorry for the lat reply.
> There's something that feels off about how always_keep sounds like a stronger version of keep, which isn't really true. It's just a matter of directionality, isn't it?
right, it's mostly to indicate that an include of this file should be kept always.
> IWYU pragma: sacred
or "scared" :P
> we could just use IWYU pragma: keep on a separate line
notifi...@include-what-you-use.org
Apr 21, 2023, 1:55:20 PM4/21/23
to include-wh...@googlegroups.com
Comment #6 on issue 1235 by kimgr: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
> clang has a mechanism to attach comments to decls and if it precedes a declaration it'll get attached to that.
Ah, yeah, I can see that. IWYU does its own blunt parsing to find end-of-line comments, so for us it wouldn't be problematic.
`IWYU pragma: sticky`...?
https://github.com/include-what-you-use/include-what-you-use/issues/1235
> clang has a mechanism to attach comments to decls and if it precedes a declaration it'll get attached to that.
`IWYU pragma: sticky`...?
notifi...@include-what-you-use.org
Apr 24, 2023, 3:31:30 AM4/24/23
to include-wh...@googlegroups.com
Comment #7 on issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
`sticky` also conveys the same semantics (at least in my mind) but TBH I find it a little bit vauge/informal.
Can you mention any particular concerns/confusions you have with `always_keep`? So that I can also think about alternatives that'll avoid those. If it's purely for taste, I am fine with both `sticky` and `always_keep` but as I mentioned above, I think `sticky` is quite informal compared to all the other pragmas we have, so i'd actually lean towards `always_keep` for the sake of consistency.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
`sticky` also conveys the same semantics (at least in my mind) but TBH I find it a little bit vauge/informal.
Can you mention any particular concerns/confusions you have with `always_keep`? So that I can also think about alternatives that'll avoid those. If it's purely for taste, I am fine with both `sticky` and `always_keep` but as I mentioned above, I think `sticky` is quite informal compared to all the other pragmas we have, so i'd actually lean towards `always_keep` for the sake of consistency.
notifi...@include-what-you-use.org
Apr 26, 2023, 3:36:14 PM4/26/23
to include-wh...@googlegroups.com
Comment #8 on issue 1235 by kimgr: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
My main concern with `always_keep` is I think it's semantically identical to `keep`. I always want to keep an `#include` line marked with `IWYU pragma: keep`, so why does `always_` imply that it's a passive force?
As a user when I'm looking to use a pragma to keep an include (or a whole header), what's the distinguishing quality between `always_keep` and `keep` that makes me think "that's the one!"?
I hesitate to suggest it, because it's similarly subtle, but `IWYU pragma: kept`?
https://github.com/include-what-you-use/include-what-you-use/issues/1235
My main concern with `always_keep` is I think it's semantically identical to `keep`. I always want to keep an `#include` line marked with `IWYU pragma: keep`, so why does `always_` imply that it's a passive force?
As a user when I'm looking to use a pragma to keep an include (or a whole header), what's the distinguishing quality between `always_keep` and `keep` that makes me think "that's the one!"?
I hesitate to suggest it, because it's similarly subtle, but `IWYU pragma: kept`?
notifi...@include-what-you-use.org
May 1, 2023, 6:49:32 AM5/1/23
to include-wh...@googlegroups.com
Comment #9 on issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
> My main concern with always_keep is I think it's semantically identical to keep. I always want to keep an #include line marked with IWYU pragma: keep, so why does always_ imply that it's a passive force?
Yeah I agree that the disambiguation there is not straightforward, but I am also failing to find better alternatives. I definitely feel like having `keep` on it's own line or some other form to attach it to the file, rather than some declaration/include inside the file would be the best. But as mentioned before, I think that's just likely to cause too much trouble with existing tools to hinder adoption.
Having "always_" as a prefix helps me (and also the tools) that this is a keep pragma that should apply "always", which tells me it's for the file rather than something in the file.
We can try some more specific names for the problem we're trying to solve, e.g. `extension_provider`, but IMO, it puts a whole lot more semantics into the pragma that's likely to be hard to implement in practice and people might use it in cases where it isn't the proper pragma (a header that should always be kept around for some other reason). Hence it feels better to name it in a way that implies the consequences, rather than reasons for having it (similar to rest of the pragmas).
> I hesitate to suggest it, because it's similarly subtle, but IWYU pragma: kept?
As for `kept`, I think it tries to solve the similar issue around having something similar to `keep` but different and I am afraid that it's solving it by violating consistency with rest of the pragmas (rest of the pragmas tell the tool what to do, keep, no_include, private include xxx, export). Also it's too similar to `keep` in spelling hence might actually be hard to notice during reviews etc.
https://github.com/include-what-you-use/include-what-you-use/issues/1235
> My main concern with always_keep is I think it's semantically identical to keep. I always want to keep an #include line marked with IWYU pragma: keep, so why does always_ imply that it's a passive force?
Yeah I agree that the disambiguation there is not straightforward, but I am also failing to find better alternatives. I definitely feel like having `keep` on it's own line or some other form to attach it to the file, rather than some declaration/include inside the file would be the best. But as mentioned before, I think that's just likely to cause too much trouble with existing tools to hinder adoption.
Having "always_" as a prefix helps me (and also the tools) that this is a keep pragma that should apply "always", which tells me it's for the file rather than something in the file.
We can try some more specific names for the problem we're trying to solve, e.g. `extension_provider`, but IMO, it puts a whole lot more semantics into the pragma that's likely to be hard to implement in practice and people might use it in cases where it isn't the proper pragma (a header that should always be kept around for some other reason). Hence it feels better to name it in a way that implies the consequences, rather than reasons for having it (similar to rest of the pragmas).
> I hesitate to suggest it, because it's similarly subtle, but IWYU pragma: kept?
As for `kept`, I think it tries to solve the similar issue around having something similar to `keep` but different and I am afraid that it's solving it by violating consistency with rest of the pragmas (rest of the pragmas tell the tool what to do, keep, no_include, private include xxx, export). Also it's too similar to `keep` in spelling hence might actually be hard to notice during reviews etc.
notifi...@include-what-you-use.org
May 1, 2023, 6:58:34 AM5/1/23
to include-wh...@googlegroups.com
Comment #10 on issue 1235 by kadircet: New IWYU pragma for extension point patterns
https://github.com/include-what-you-use/include-what-you-use/issues/1235
(also sent out a PR to kick off review of documentation wording, not trying to cut off the naming discussions, we can still change it)
https://github.com/include-what-you-use/include-what-you-use/issues/1235
(also sent out a PR to kick off review of documentation wording, not trying to cut off the naming discussions, we can still change it)
Reply all
Reply to author
Forward
0 new messages