Re: Auto-Pimpl with partial

[Matthew Woehlke]

On 2017-08-12 10:25, Nicol Bolas wrote:
> That being said, it should be noted that much of the reason for pimpl will
> be made obsolete once we get modules.

Does "much of the reason" include binary compatibility?

--
Matthew

[Nicol Bolas]

If you mean that one library can change aspects of a class's implementation which are not reflected in its class definition without the consuming code having to be recompiled? The answer is yes.

If you mean that one library can change private definitions of an interface class without the consuming code having to be recompiled? The answer is no.

If you mean that the inevitable recompile from item 2 will be far faster due to not having to recompile the standard library, Boost, and every other thing that wasn't changed just because they're being included in a .cpp file that has to be recompiled? The answer is yes.

So it all really depends on why you used Pimpl in the first place.

[Matthew Woehlke]

On 2017-08-15 15:09, Nicol Bolas wrote:
> On Tuesday, August 15, 2017 at 2:38:06 PM UTC-4, Matthew Woehlke wrote:
>> On 2017-08-12 10:25, Nicol Bolas wrote:
>>> That being said, it should be noted that much of the reason for pimpl
>>> will be made obsolete once we get modules.
>>
>> Does "much of the reason" include binary compatibility?
>
> If you mean that one library can change aspects of a class's implementation
> which are not reflected in its class definition without the consuming code
> having to be recompiled? The answer is yes.
>
> If you mean that one library can change private definitions of an interface
> class without the consuming code having to be recompiled? The answer is no.
>

> If you mean that the inevitable recompile from item 2 will be *far faster*

> due to not having to recompile the standard library, Boost, and every other

> thing that *wasn't* changed just because they're being included in a .cpp

> file that has to be recompiled? The answer is yes.

So... that would be a "no".

Why?

Let's take Qt, for example. Widely used; lots of consumers. Uses PIMPL
to ensure BC. This means that Qt is updated... I can just update Qt, and
(hopefully) nothing breaks.

If Qt wasn't using PIMPL, then I would need to recompile *my entire
desktop environment* (that's on the order of 50-100 libraries and
applications). Actually, it's worse, because *I* am not recompiling
them. The recompile (which may in fact be much faster) happens on some
build farm. The cost *I* pay — which *everyone* pays — is in downloading
and installing all those updates. Modules won't affect those costs.

...which is why Qt uses PIMPL, and will continue to use PIMPL even
post-modules.

I'm not even convinced that modules are a reason to stop using PIMPL, or
other dependency-reducing techniques. Recompiling "much faster" still
doesn't beat "not needed", even in the quantitative sense, never mind
the qualitative difference (especially in the case of libraries with
external consumers)...

All that said, I don't see where the OP's proposal offers significant
any benefit over traditional PIMPL. At best it reduces boilerplate.

--
Matthew

[Nicol Bolas]

On Tuesday, August 15, 2017 at 4:09:27 PM UTC-4, Matthew Woehlke wrote:

On 2017-08-15 15:09, Nicol Bolas wrote:
> On Tuesday, August 15, 2017 at 2:38:06 PM UTC-4, Matthew Woehlke wrote:
>> On 2017-08-12 10:25, Nicol Bolas wrote:
>>> That being said, it should be noted that much of the reason for pimpl
>>> will be made obsolete once we get modules.
>>
>> Does "much of the reason" include binary compatibility?
>
> If you mean that one library can change aspects of a class's implementation
> which are not reflected in its class definition without the consuming code
> having to be recompiled? The answer is yes.
>
> If you mean that one library can change private definitions of an interface
> class without the consuming code having to be recompiled? The answer is no.
>
> If you mean that the inevitable recompile from item 2 will be *far faster*
> due to not having to recompile the standard library, Boost, and every other
> thing that *wasn't* changed just because they're being included in a .cpp
> file that has to be recompiled? The answer is yes.

So... that would be a "no".

Why?

Because C++ doesn't allow that sort of thing.

C++ is a statically typed language. If you have a class, that class has a certain size. It has a certain set of member variables. And you can't change them without recompiling every piece of code that uses it.

And let's not forget: just because a class has private stuff in it doesn't mean that the private stuff has no bearing on the class's interface. Especially with reflection allowing you to reflect into privates easily. Privates are not accessible, but they're not invisible or irrelevant.

The goal of modules is not to change the nature of how your code can interact with an external library. It's to make it so that you don't have to keep recompiling stuff that doesn't change.

Let's take Qt, for example. Widely used; lots of consumers. Uses PIMPL
to ensure BC. This means that Qt is updated... I can just update Qt, and
(hopefully) nothing breaks.

If Qt wasn't using PIMPL, then I would need to recompile *my entire
desktop environment* (that's on the order of 50-100 libraries and
applications). Actually, it's worse, because *I* am not recompiling
them. The recompile (which may in fact be much faster) happens on some
build farm. The cost *I* pay — which *everyone* pays — is in downloading
and installing all those updates. Modules won't affect those costs.

...which is why Qt uses PIMPL, and will continue to use PIMPL even
post-modules.

Pimpl only provides protection from changes to the private interface of a class. You still need a recompile whenever new API functions are added.

I'm not even convinced that modules are a reason to stop using PIMPL, or
other dependency-reducing techniques. Recompiling "much faster" still
doesn't beat "not needed", even in the quantitative sense, never mind
the qualitative difference (especially in the case of libraries with
external consumers)...

Sure. But "much faster" is good enough in many cases; bothering to remove that last 5% of your full-build recompile time may not be worth it. After all, Pimpl isn't exactly free or anything; it has costs. If those costs are not worth getting slightly better compile time, then why do it?

[Thiago Macieira]

On Tuesday, 15 August 2017 13:09:23 PDT Matthew Woehlke wrote:
> ...which is why Qt uses PIMPL, and will continue to use PIMPL even
> post-modules.

Even if modules did solve the problem, we would continue using d pointers
until at least 2025, probably a little more (Qt 7.0).

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Matthew Woehlke]

On 2017-08-15 16:26, Nicol Bolas wrote:
> On Tuesday, August 15, 2017 at 4:09:27 PM UTC-4, Matthew Woehlke wrote:
>> On 2017-08-15 15:09, Nicol Bolas wrote:
>>> On Tuesday, August 15, 2017 at 2:38:06 PM UTC-4, Matthew Woehlke wrote:
>>>> On 2017-08-12 10:25, Nicol Bolas wrote:
>>>>> That being said, it should be noted that much of the reason for pimpl
>>>>> will be made obsolete once we get modules.
>>>>
>>>> Does "much of the reason" include binary compatibility?
>>>

>>> [...]

>>
>> So... that would be a "no".
>>
>> Why?
>
> Because C++ doesn't allow that sort of thing.

This seems like a somewhat pedantic use of "allow". With PIMPL, the
class keeps its private stuff in a separate, opaque class, and keeps
only a pointer to that "private class" as its visible definition. That's
certainly "allowed" in the sense that there is lots of C++ code using
this idiom that can be compiled.

> Pimpl only provides protection from changes to the private interface of a
> class. You still need a recompile whenever new API functions are added.

Uh... no? Adding (non-member or non-virtual member) functions does not
change ABI; it is both SC and BC. In fact, there are lost of ways an API
can mutate without breaking BC; see
https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
for a fairly extensive list.

Yes, many of these things may be *detectable* by consumers that are
specifically attempting to do so, but then, most (decent) libraries also
supply version information somehow.

You are perhaps applying a very pedantic meaning to "need". I (and
pretty much everyone that talks about BC/BIC) define "need" as whether
or not code built against an old version will continue to *function as
intended*.

>> I'm not even convinced that modules are a reason to stop using PIMPL, or
>> other dependency-reducing techniques. Recompiling "much faster" still
>> doesn't beat "not needed", even in the quantitative sense, never mind
>> the qualitative difference (especially in the case of libraries with
>> external consumers)...
>
> Sure. But "much faster" is good enough in many cases; bothering to remove
> that last 5% of your full-build recompile time may not be worth it.

I think we have a different definition of "many".

For stuff that's local to your project? Sure. (Were you really using
PIMPL for that before?) For stuff that's visible to external consumers?
Just the aggravation that you need to rebuild those *at all* may be
reason enough, even if it is really, really fast to rebuild them.

Not all of us are using Google's One True Repository that contains all
software in our "universe" managed under a single build tree.

--
Matthew

[Thiago Macieira]

On Tuesday, 15 August 2017 14:03:17 PDT Matthew Woehlke wrote:
> > Pimpl only provides protection from changes to the private interface of a
> > class. You still need a recompile whenever new API functions are added.
>
> Uh... no? Adding (non-member or non-virtual member) functions does not
> change ABI; it is both SC and BC. In fact, there are lost of ways an API
> can mutate without breaking BC; see
> https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B
> for a fairly extensive list.

Strictly speaking, the standard recognises no such thing as Binary
Compatibility for changed sources. It's ODR violation and your code is ill-
formed no diagnostic required.

Binary compatibility depends on the ABI of each platform and compiler. Knowing
the ABI allows us to violate the ODR still run.

[Izzy Coding]

> All that said, I don't see where the OP's proposal offers significant
> any benefit over traditional PIMPL. At best it reduces boilerplate.

That was my main intent. To reduce boilerplate code as depending on how complicated your class is, using pimpl is even more complicated and easy to get wrong and introduce bugs/leaks.

I did/do not wish to extend this idea further as there are still quite a few other papers in flight that could affect this both for the good and the bad. So for now this was just to post and get feedback to help me design and write a proposal that covers everyone's points here where possible.

[Thiago Macieira]

On Tuesday, 15 August 2017 22:36:25 PDT Izzy Coding wrote:
> > All that said, I don't see where the OP's proposal offers significant
> > any benefit over traditional PIMPL. At best it reduces boilerplate.
>
> That was my main intent. To reduce boilerplate code as depending on how
> complicated your class is, using pimpl is even more complicated and easy to
> get wrong and introduce bugs/leaks.

Actually, no, it's not difficult to use.

The difficult part is keeping BC while updating the public API. Your proposal
does not make that easier, but depending on how it gets into the standard and
the ABIs, it could actually make things harder.

Besides, pimpl has one other advantage: you can derive from the public type.