make_shared and enable_shared_from_this

[Viacheslav Usov]

std::make_shared is an optimization offered by the standard to allocate memory for an object and the shared_ptr 's state together.

std::enable_shared_from_this is a convenience offered by the standard to convert a naked pointer to a shared pointer.

Unless I misread the standard, their use together is not optimal, because the latter would still maintain a weak pointer to itself, plus there is a potential with data races when shared_from_this() is used.

Is there any reason why the standard does not define a template like std::shared_ptr_state, that could be used just like std::enable_shared_from_this is used?

The benefit of std::shared_ptr_state is that a class derived from it would then contain its own shared state, thus dispensing with the need to have a weak pointer to itself, and that, I think, the data race could then be fully eliminated when calling shared_from_this().

Any insight?

Cheers,

V.

[Nicol Bolas]

On Friday, August 5, 2016 at 8:14:38 AM UTC-4, Viacheslav Usov wrote:

std::make_shared is an optimization offered by the standard to allocate memory for an object and the shared_ptr 's state together.

Well that's one reason for it. Another is to deal with C++'s squirrelly expression evaluation ordering rules. Fortunately, C++17 seems to be on track to fix that one.

std::enable_shared_from_this is a convenience offered by the standard to convert a naked pointer to a shared pointer.

It's not a "convenience" since you wouldn't be able to implement it yourself. It is based on machinery that exists directly within `shared_ptr`'s constructor. It has to detect that the type it is claiming ownership of is derived from `enable_shared_from_this`, and then provides that base class with whatever is needed to generate a `shared_ptr` instance.

For you to do it yourself would require two-phase construction: first, you construct the object&shared_ptr, then you create a weak_ptr and stick that in the object somewhere. By having the standard do it for you, the connection happens when the `shared_ptr` is created.
 

Unless I misread the standard, their use together is not optimal, because the latter would still maintain a weak pointer to itself,

It doesn't have to maintain a `weak_ptr` per-se. It simply maintains whatever state is needed; the *guts* of a `weak_ptr`. All it really needs is to bump the weak reference count when the object is first attached and to store a pointer to the shared state. The standard shows a `weak_ptr` being used as a "possible implementation", but it is hardly mandatory.

Yes, `enable_shared_from_this` is not free. But nobody ever claimed that it was.

plus there is a potential with data races when shared_from_this() is used.

A data race with what? Can you provide some code examples of such a race?

Is there any reason why the standard does not define a template like std::shared_ptr_state, that could be used just like std::enable_shared_from_this is used?

Yes, there is: the allocator/deleter.

`shared_ptr`'s constructors that create the shared state get to store a copy of the allocator/deleter. These objects are type-erased relative to the `shared_ptr<T>` itself, but the type erasure doesn't have to require dynamic allocation. Because those types are known at the time when the shared state is constructed, it should be possible to allocate this block within the shared state block.

Which means that the shared state does not have a fixed size. Your `shared_ptr_state` type would have to have a fixed size, so it could only represent the

Now, there are probably ways around that, but they complicate the implementation of such things.

The benefit of std::shared_ptr_state is that a class derived from it would then contain its own shared state, thus dispensing with the need to have a weak pointer to itself, and that, I think, the data race could then be fully eliminated when calling shared_from_this().

Relative to a decent implementation of `enable_shared_from_this`, your suggested type would save you... one pointer of space. And thus, a single indirection on `shared_from_this()` calls. Is that really worth it?

I would say that the primary benefit of what you're talking about gaining the advantages of using `make_shared` to allocate your object (having only one allocation instead of two) without actually calling that function.

[Viacheslav Usov]

On Fri, Aug 5, 2016 at 6:34 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> It's not a "convenience" since you wouldn't be able to implement it yourself.

I can implement it, and I am not willing to argue about the meaning of "convenience".

> It doesn't have to maintain a `weak_ptr` per-se. It simply maintains whatever state is needed; the *guts* of a `weak_ptr`. All it really needs is to bump the weak reference count when the object is first attached and to store a pointer to the shared state. The standard shows a `weak_ptr` being used as a "possible implementation", but it is hardly mandatory.

Hardly mandatory, bot MSVC and GCC both use a weak pointer.

> A data race with what? Can you provide some code examples of such a race?

I may have been mistaken here. I need to think more about it. Ditto for your allocator/deleter explanation.

> Relative to a decent implementation of `enable_shared_from_this`, your suggested type would save you... one pointer of space.

Not one, but two. And CPU cycles spent on promoting weak to shared. 

Cheers,

V.

[Andrey Semashev]

I'd say if you want to optimize then you should take a look at
boost::intrusive_ptr. I wish it was part of the standard library.

[Patrice Roy]

You might want to bring this to SG14, where there's interest in intrusive containers too.

--

--- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to std-discussion+unsubscribe@isocpp.org.
To post to this group, send email to std-dis...@isocpp.org.
Visit this group at https://groups.google.com/a/isocpp.org/group/std-discussion/.

[Miro Knejp]

Am 05.08.2016 um 19:34 schrieb Viacheslav Usov:

On Fri, Aug 5, 2016 at 6:34 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> It's not a "convenience" since you wouldn't be able to implement it yourself.

I can implement it, and I am not willing to argue about the meaning of "convenience".

> It doesn't have to maintain a `weak_ptr` per-se. It simply maintains whatever state is needed; the *guts* of a `weak_ptr`. All it really needs is to bump the weak reference count when the object is first attached and to store a pointer to the shared state. The standard shows a `weak_ptr` being used as a "possible implementation", but it is hardly mandatory.

Hardly mandatory, bot MSVC and GCC both use a weak pointer.

> A data race with what? Can you provide some code examples of such a race?

I may have been mistaken here. I need to think more about it. Ditto for your allocator/deleter explanation.

I wouldn't call it a race, but there is this annoying little thing that you can't call shared_from_this() in the constructor. It makes perfect sense when you think about it. Unfortunately that doesn't stop people from trying it anyway. Every single time I see someone use enable_shared_from_this for the first time it is probably *the* first issue they run into. And when telling them it doens't work you can almost see the disgust in their eyes. Somehow people just assume it *should* work.

I like that if we had access to shared_ptr_state we could basically use shared_ptr intrusively, and having that in the object instead of enable_shared_from_this would also allow the use of shared_from_this() in the constructor.

[Nicol Bolas]

On Friday, August 5, 2016 at 1:34:23 PM UTC-4, Viacheslav Usov wrote:

On Fri, Aug 5, 2016 at 6:34 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> It's not a "convenience" since you wouldn't be able to implement it yourself.

I can implement it, and I am not willing to argue about the meaning of "convenience".

> It doesn't have to maintain a `weak_ptr` per-se. It simply maintains whatever state is needed; the *guts* of a `weak_ptr`. All it really needs is to bump the weak reference count when the object is first attached and to store a pointer to the shared state. The standard shows a `weak_ptr` being used as a "possible implementation", but it is hardly mandatory.

Hardly mandatory, bot MSVC and GCC both use a weak pointer.

Sure, but that's a quality of implementation issue, not something the standard library can mandate.
 

> A data race with what? Can you provide some code examples of such a race?

I may have been mistaken here. I need to think more about it. Ditto for your allocator/deleter explanation.

> Relative to a decent implementation of `enable_shared_from_this`, your suggested type would save you... one pointer of space.

Not one, but two.

The only information `enable_shared_from_this` truly needs is a pointer to the shared state. The fact that some implementations use more space does not change the fact that the only necessary information is one pointer.

If you feel that a standard library implementation could be improved, that's something that should be taken up with your implementation provider(s). File a GCC/MSVC bug on the matter. This is not something the standard can or should fix.
 

And CPU cycles spent on promoting weak to shared. 

Nope; you'd have to do that work anyway. Whether the object holds the actual shared state or just a pointer to the shared state, it must do all of the work that `weak_ptr<T>::lock` must does in order to generate that `shared_ptr<T>`.

You have to atomically check the `use_count` in the shared state and bump it if it's non-zero (a tricky bit of atomic logic to get right). If it was zero, then you get back a `shared_ptr<T>{}`. If not, then the shared object still exists and you can use the shared state and a pointer to `T` to construct the `shared_ptr`.

The only difference between `weak_ptr<T>::lock()` and `shared_from_this()` is that the latter can get the "pointer to `T`" via `static_cast<T*>(`this`), while the former stores the "pointer to `T`" internally. In terms of "CPU cycles spent", they are functionally identical.

[Nicol Bolas]

What exactly does it mean to get a shared pointer from an object who's lifetime is not yet managed by a shared pointer? Does that cause it to become owned by that shared pointer? Can an object transfer ownership of itself, without the calling code necessarily being aware of it?

This sound rather dangerous. It'd probably be better to just work out a proper intrusive pointer type.

[Miro Knejp]

The word "intrusive" does sound dangerous to me. You already decided to label something dangerous even though you only have asked questions. But somehow you already know that it's better to do something else instead. That implies you know the answers already.

[Viacheslav Usov]

On Fri, Aug 5, 2016 at 8:17 PM, Andrey Semashev <andrey....@gmail.com> wrote:

> I'd say if you want to optimize then you should take a look at boost::intrusive_ptr. I wish it was part of the standard library.

I take issue with the fact that one needs to look elsewhere for an optimal implementation.

std::shared_ptr & Co try to cater to many needs, but a prime use case  - no frills new/delete allocation with a reference count embedded into a derived class - is not addressed directly. This is at variance with the C++ tenet that one does not pay for what is not used.

Cheers,

V.

[Viacheslav Usov]

On Fri, Aug 5, 2016 at 9:35 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Sure, but that's a quality of implementation issue, not something the standard library can mandate.

[...]

> Nope; you'd have to do that work anyway. Whether the object holds the actual shared state or just a pointer to the shared state, it must do all of the work that `weak_ptr<T>::lock` must does in order to generate that `shared_ptr<T>`.

So you're saying that using weak_ptr is QoI, while saying all that it implies must be done anyway. I'd suggest you think again.

> You have to atomically check the `use_count`

If the shared state is embedded into the object via shared_ptr_state, there is no need to check anything, because the object is there as long as 'this' is valid.

Cheers,

V.

[Edward Catmur]

The issue is that shared_from_this would have to account for a T constructed on the stack, the heap, by an allocator, as a base class subobject, as a data member, and do something sensible in each case.

It might be possible to make it work in the make_shared case, but I have trouble seeing how to pass the information to the enable_shared_from_this base.

In contrast, intrusive pointers are well understood to solve this problem.

[Edward Catmur]

Don't pay for what you don't use certainly does not mean that every facility should cater for every use case.

[Edward Catmur]

On Saturday, 6 August 2016 09:17:43 UTC+1, Viacheslav Usov wrote:
> On Fri, Aug 5, 2016 at 9:35 PM, Nicol Bolas <jmck...@gmail.com> wrote:
>
>
>
> > Sure, but that's a quality of implementation issue, not something the standard library can mandate.
>
>
> [...]
>
>
> > Nope; you'd have to do that work anyway. Whether the object holds the actual shared state or just a pointer to the shared state, it must do all of the work that `weak_ptr<T>::lock` must does in order to generate that `shared_ptr<T>`.
>
>
> So you're saying that using weak_ptr is QoI, while saying all that it implies must be done anyway. I'd suggest you think again.

The difference is the footprint; the object pointer is unnecessary because the desired value is this.

[Edward Catmur]

What about an object under destruction?

[Viacheslav Usov]

On Sat, Aug 6, 2016 at 2:29 PM, Edward Catmur <e...@catmur.co.uk> wrote:

> Don't pay for what you don't use certainly does not mean that every facility should cater for every use case.

True as it stands, but, as I said, in arguably the simplest use case we pay for all that cruft.

Cheers,

V.

[Viacheslav Usov]

On Sat, Aug 6, 2016 at 2:31 PM, Edward Catmur <e...@catmur.co.uk> wrote:

> The difference is the footprint; the object pointer is unnecessary because the desired value is this.

Yep, that's how he got only ONE pointer saved out of TWO.

But my point is that TWO can be saved, along with all that code dealing with the weak aspect.

Cheers,

V.

[Viacheslav Usov]

On Sat, Aug 6, 2016 at 2:32 PM, Edward Catmur <e...@catmur.co.uk> wrote:

> What about an object under destruction?

Please clarify what you mean.

Cheers,

V.

[Nicol Bolas]

Wait, I just realized something. You cannot embed the shared state in the object being managed. If you did, `weak_ptr` wouldn't be able to function at all.

Weak_ptr works by referencing the shared state for the managed object. And it has to be able to access the shared state even when the object being managed has been destroyed. But if the object being managed contains the shared state, then the shared state will be destroyed right along with the object. The storage may not be, but you would be accessing an object who's lifetime has ended.

The reason `make_shared` works is because the shared state exists outside the object being managed. The shared state is within the same allocation, but they are not in the same object; the shared state is not a subobject of the managed object.

You cannot have weak_ptr work and have the shared state be a subobject of the managed object.

On Saturday, August 6, 2016 at 4:17:43 AM UTC-4, Viacheslav Usov wrote:

On Fri, Aug 5, 2016 at 9:35 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Sure, but that's a quality of implementation issue, not something the standard library can mandate.

[...]

> Nope; you'd have to do that work anyway. Whether the object holds the actual shared state or just a pointer to the shared state, it must do all of the work that `weak_ptr<T>::lock` must does in order to generate that `shared_ptr<T>`.

So you're saying that using weak_ptr is QoI, while saying all that it implies must be done anyway. I'd suggest you think again.

You're right. I was thinking about the case where you try to call `shared_from_this`, while another thread is destroying the last `shared_ptr`. I forgot that, once that `shared_ptr` is destroyed, so too is `T`, and therefore the `enable_shared_from_this` base class subobject itself along with its contents. So you've already hit UB by accessing the value representation of an object who's lifetime has ended.

And therefore, an implementation need not concern itself about what happens then. And even those that use a `weak_ptr` don't gain any safety, since the `weak_ptr` itself has been destroyed by the time they access it.

[Edward Catmur]

If an object with an intrusive shared pointer is being destructed, shared_from_this has to fail as otherwise you have a dangling shared pointer afterwards. But that means that shared_from_this has to know that the object is being destructed, so it has to check the use count.

[Edward Catmur]

On 6 Aug 2016 2:54 p.m., "Nicol Bolas" <jmck...@gmail.com> wrote:
>
> Wait, I just realized something. You cannot embed the shared state in the object being managed. If you did, `weak_ptr` wouldn't be able to function at all.
>
> Weak_ptr works by referencing the shared state for the managed object. And it has to be able to access the shared state even when the object being managed has been destroyed. But if the object being managed contains the shared state, then the shared state will be destroyed right along with the object. The storage may not be, but you would be accessing an object who's lifetime has ended.
>
> The reason `make_shared` works is because the shared state exists outside the object being managed. The shared state is within the same allocation, but they are not in the same object; the shared state is not a subobject of the managed object.
>
> You cannot have weak_ptr work and have the shared state be a subobject of the managed object.
>

You can have the weak state only be a separate allocation; that means that if you never take a weak pointer you could avoid that allocation, and if you don't want weak pointers at all you can avoid the overhead of the pointer to weak state. I have a vague recollection that the boost intrusive shared pointer might do this.

[Miro Knejp]

There may be issues but automatically labeling something "dangerous"
a-priori is what I have an issue with.

Intrusive pointers don't actually "solve" any of those problems. On the
contrary: depending on what the object does when the refcount reaches
zero you can do only a subset of the things you listed as soon as *any*
intrusive_ptr to the object exists that might cause the refcount to
reach zero. It all depends on what assumptions the deleting operation
has on how and where the object was supposed to come into existence.

With shared_ptr_state it is the same, we just give it a name and make it
compatible with shared_ptr. Just instantiating the object as a member
works fine as long as no shared_ptr points to it. Otherwise you run into
exactly the same problems as with any other intrusive pointer if the
object wasn't designed for such a use case. Make shared_ptr_state accept
a custom deleter that only fires when the refcount goes to zero for the
extra kick.

And lastly if your object already has shared_ptr_state it doesn't need
enable_shared_from_this as the former can provide shared_from_this()
already.

[Nicol Bolas]

Boost.IntrusivePtr doesn't seem to have a weak version at all.

And all of `weak_ptr`'s constructors are explicitly `noexcept`. So that nix's the idea of having them dynamically allocating memory.

Even if that weren't the case, such dynamic allocations lead to unpleasant race conditions. What happens if you get two weak_ptrs at the same time? How do you stop that from causing two allocations? What happens if the last shared_ptr and last weak_ptr are destroyed at the same time?

Furthermore, this requires all of weak_ptr's interface to work very differently (internally) based on whether the managed object contains its own shared state or not.

It's not that these cannot be overcome (except for the `noexcept` thing). The question is more why should we do that?

The `make_shared` solution is a great middle ground for `shared_ptr`; the only thing that has to change is how the object is destroyed. `weak_ptr` doesn't have to know that the allocation for the shared state and the managed object are the same. Indeed, `shared_ptr` doesn't have to know either. It's all in the implementation of the destruction of the managed object and the shared state.

It saves allocations, but with some overhead if you keep `weak_ptr`s around after the object is destroyed. That's the best `shared_ptr` should do.

What people seem to want is `intrusive_ptr`. That's wonderful; let's get that done. But let's not try to transform `shared_ptr` into an `intrusive_ptr`. They are two different types for two different use cases with two different needs.

[Nicol Bolas]

They can solve these problems. Whether a particular implementation does or not is to be determined, but they certainly could be made to do so.

For example, you can have a different set of constructors that are used on subobjects than on directly allocated heap objects. One set of constructors says that the instance is legal to be used in intrusive pointers, and the other says that it is not. When you go to get an intrusive_ptr to such an object, the pointer can call an interface that returns whether the instance can be wrapped in an intrusive pointer. If not, an exception is thrown or some other thing happens.

Such a thing can also be used to stop the creation of intrusive pointers when the class's destructor has started as well.

On the
contrary: depending on what the object does when the refcount reaches
zero you can do only a subset of the things you listed as soon as *any*
intrusive_ptr to the object exists that might cause the refcount to
reach zero. It all depends on what assumptions the deleting operation
has on how and where the object was supposed to come into existence.

With shared_ptr_state it is the same, we just give it a name and make it
compatible with shared_ptr. Just instantiating the object as a member
works fine as long as no shared_ptr points to it. Otherwise you run into
exactly the same problems as with any other intrusive pointer if the
object wasn't designed for such a use case. Make shared_ptr_state accept
a custom deleter that only fires when the refcount goes to zero for the
extra kick.

Here's the thing. The weak_ptr issue makes it impossible for `shared_ptr_state<T, D>` to be the exact same type that `shared_ptr` and `make_shared` would normally have allocated. This is because weak_ptr would have to have a separate allocation for the weak reference count, so `shared_ptr_state` wouldn't have a weak reference count.

As such, `shared/weak_ptr` now have to basically have separate class implementations: one for objects that don't inherit from `shared_ptr_state`, and one for types that do. And it can't even use SFINAE/concepts to make that determination, since the `T` is not the managed object type. So instead, it has to be a runtime check, used by every function that touches the managed state.

All to save the overhead of one pointer (on good implementations. Two worst-case). If you need an overhead-less smart pointer, use an intrusive pointer. We should have such a type in the standard. But we should not transform `shared/weak_ptr` into such a type.

[Miro Knejp]

shared_ptr does not have *one* type of shared state but basically an infinite number since it performs type erasure. So in reality there actually *is* a shared_ptr_state<T, D> (among others) but shared_ptr itself only accesses the erased base type. Make the user-accessible shared_ptr_state<T, D> derive from the same base and you can make it compatible.

As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.

[Nicol Bolas]

`shared_ptr` actually has to store and type-erase a `shared_ptr_state<T, D, A>`, with `A` being the allocator used to allocate it the memory for it. I bring that up because that throws another snag in your plan. Namely, that `A` is both (potentially) user-provided and stored within that object. An object which will be destroyed potentially well before the memory is deallocated. And since the allocator is a potentially user-provide class, you can't rely on compiler magic to make it work even though it is destroyed.

Now, you might say that if `A` were user-provided, then clearly the shared state can't be a base class of `T`. So on the user end, if a class derives from this state object, then `A` will never exist. So the actual user-facing shared state object would be the internal shared state, which uses an empty default allocator that would never be used.

So... what should happen if someone calls `alloc_shared<T>` on such a type? If the user calls one of the `shared_ptr` constructors that takes an allocator, then one could conceivably just ignore the allocator, since the storage for the shared state already exists. But with `alloc_shared`, no storage exists for anything yet. `alloc_shared` is required to allocate from that allocator, store the allocator within that storage, and use that stored copy of the allocator to delete it.

Is there some form of type-erasure that could handle this without requiring further allocations? Or would you just make it an error to call `alloc_shared` on such a type?

As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.

Are there any other types, outside of Chapters 18 and a bit of 19, which have such an explicit reliance on compiler magic? Are there any other types where implementing them requires direct subversion of the C++ object and lifetime model? We're not talking about a type that, by its very nature, has to directly talk to the system (threads, IO, etc) or the compiler (type traits, etc). We're talking about a mere smart pointer.

It's one thing to have a type like `tuple`, where a clever compiler intrinsic or something could improve an implementation, but you can still implement it effectively otherwise. It's quite another thing altogether to define a type in such a way where it is functionally impossible to write an implementation that doesn't rely on platform-specific behavior.

[Miro Knejp]

It is trivial to construct a `container<A>` that stores a copy of `A` in a way that does not run its destructor when the enclosing object's destructor runs.

Now, you might say that if `A` were user-provided, then clearly the shared state can't be a base class of `T`. So on the user end, if a class derives from this state object, then `A` will never exist. So the actual user-facing shared state object would be the internal shared state, which uses an empty default allocator that would never be used.

So... what should happen if someone calls `alloc_shared<T>` on such a type? If the user calls one of the `shared_ptr` constructors that takes an allocator, then one could conceivably just ignore the allocator, since the storage for the shared state already exists. But with `alloc_shared`, no storage exists for anything yet. `alloc_shared` is required to allocate from that allocator, store the allocator within that storage, and use that stored copy of the allocator to delete it.

Is there some form of type-erasure that could handle this without requiring further allocations? Or would you just make it an error to call `alloc_shared` on such a type?

If I call `allocate_shared()` I expect it to do what `allocate_shared()` does, and that is to not care about the nature of `T` and just do the normal business it does today. You don't seem to understand that the purpose of having `shared_ptr_state` accessible is to do these things *yourself* as is the point of having intrusive pointers in the first place. So If I decide to derivce my class from `shared_ptr_state<T, D, A>` I explicitly buy into managing its allocation myself and therefore can provide the allocator it needs.

As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.

Are there any other types, outside of Chapters 18 and a bit of 19, which have such an explicit reliance on compiler magic? Are there any other types where implementing them requires direct subversion of the C++ object and lifetime model? We're not talking about a type that, by its very nature, has to directly talk to the system (threads, IO, etc) or the compiler (type traits, etc). We're talking about a mere smart pointer.

It's one thing to have a type like `tuple`, where a clever compiler intrinsic or something could improve an implementation, but you can still implement it effectively otherwise. It's quite another thing altogether to define a type in such a way where it is functionally impossible to write an implementation that doesn't rely on platform-specific behavior.

Honestly I don't care. The standard only describes how types behave, not whether they need to rely on compiler magic to realize their contract. And you would be surprised how many intrinsics are used throughout std implementations in places one doesn't necessarily expect. One doesn't need special rules to disable automatically running the allocator and ref counter destructors. One only needs to make sure the compiler does not remove access to these memory locations or overwrite them with magic values like DEADBEEF in debug builds until they are actually deallocated. Sure, an implementation that cannot do that needs a separate allocation but I would consider that a QoI problem.

[Nevin Liber]

On 6 August 2016 at 14:09, Nicol Bolas <jmck...@gmail.com> wrote:

`shared_ptr` actually has to store and type-erase a `shared_ptr_state<T, D, A>`, with `A` being the allocator used to allocate it the memory for it.

No it does not.  shared_ptr never does any allocations after construction time.

--

 Nevin ":-)" Liber  <mailto:ne...@eviloverlord.com>  +1-847-691-1404

[Nicol Bolas]

On Saturday, August 6, 2016 at 4:02:45 PM UTC-4, Nevin ":-)" Liber wrote:

On 6 August 2016 at 14:09, Nicol Bolas <jmck...@gmail.com> wrote:

`shared_ptr` actually has to store and type-erase a `shared_ptr_state<T, D, A>`, with `A` being the allocator used to allocate it the memory for it.

No it does not.  shared_ptr never does any allocations after construction time.

But it does have to deallocate that memory with the allocator. Therefore, it must store it.

[Edward Catmur]

For example, using an aligned_storage buffer and neglecting to run the destructor on the contained A. Interesting; that could almost be legal.

>>
>> Now, you might say that if `A` were user-provided, then clearly the shared state can't be a base class of `T`. So on the user end, if a class derives from this state object, then `A` will never exist. So the actual user-facing shared state object would be the internal shared state, which uses an empty default allocator that would never be used.
>>
>> So... what should happen if someone calls `alloc_shared<T>` on such a type? If the user calls one of the `shared_ptr` constructors that takes an allocator, then one could conceivably just ignore the allocator, since the storage for the shared state already exists. But with `alloc_shared`, no storage exists for anything yet. `alloc_shared` is required to allocate from that allocator, store the allocator within that storage, and use that stored copy of the allocator to delete it.
>>
>> Is there some form of type-erasure that could handle this without requiring further allocations? Or would you just make it an error to call `alloc_shared` on such a type?
>
> If I call `allocate_shared()` I expect it to do what `allocate_shared()` does, and that is to not care about the nature of `T` and just do the normal business it does today. You don't seem to understand that the purpose of having `shared_ptr_state` accessible is to do these things *yourself* as is the point of having intrusive pointers in the first place. So If I decide to derivce my class from `shared_ptr_state<T, D, A>` I explicitly buy into managing its allocation myself and therefore can provide the allocator it needs.
>
>>
>>> As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.
>>
>>
>> Are there any other types, outside of Chapters 18 and a bit of 19, which have such an explicit reliance on compiler magic? Are there any other types where implementing them requires direct subversion of the C++ object and lifetime model? We're not talking about a type that, by its very nature, has to directly talk to the system (threads, IO, etc) or the compiler (type traits, etc). We're talking about a mere smart pointer.
>>
>> It's one thing to have a type like `tuple`, where a clever compiler intrinsic or something could improve an implementation, but you can still implement it effectively otherwise. It's quite another thing altogether to define a type in such a way where it is functionally impossible to write an implementation that doesn't rely on platform-specific behavior.
>
> Honestly I don't care. The standard only describes how types behave, not whether they need to rely on compiler magic to realize their contract. And you would be surprised how many intrinsics are used throughout std implementations in places one doesn't necessarily expect. One doesn't need special rules to disable automatically running the allocator and ref counter destructors. One only needs to make sure the compiler does not remove access to these memory locations or overwrite them with magic values like DEADBEEF in debug builds until they are actually deallocated. Sure, an implementation that cannot do that needs a separate allocation but I would consider that a QoI problem.
>

> --
>
> ---
> You received this message because you are subscribed to a topic in the Google Groups "ISO C++ Standard - Discussion" group.
> To unsubscribe from this topic, visit https://groups.google.com/a/isocpp.org/d/topic/std-discussion/WF0gyN9g2mQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to std-discussio...@isocpp.org.

[Nicol Bolas]

On Saturday, August 6, 2016 at 3:40:19 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 21:09 schrieb Nicol Bolas:

On Saturday, August 6, 2016 at 12:26:31 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 18:14 schrieb Nicol Bolas:

Here's the thing. The weak_ptr issue makes it impossible for `shared_ptr_state<T, D>` to be the exact same type that `shared_ptr` and `make_shared` would normally have allocated. This is because weak_ptr would have to have a separate allocation for the weak reference count, so `shared_ptr_state` wouldn't have a weak reference count.

As such, `shared/weak_ptr` now have to basically have separate class implementations: one for objects that don't inherit from `shared_ptr_state`, and one for types that do. And it can't even use SFINAE/concepts to make that determination, since the `T` is not the managed object type. So instead, it has to be a runtime check, used by every function that touches the managed state.

All to save the overhead of one pointer (on good implementations. Two worst-case). If you need an overhead-less smart pointer, use an intrusive pointer. We should have such a type in the standard. But we should not transform `shared/weak_ptr` into such a type.

shared_ptr does not have *one* type of shared state but basically an infinite number since it performs type erasure. So in reality there actually *is* a shared_ptr_state<T, D> (among others) but shared_ptr itself only accesses the erased base type. Make the user-accessible shared_ptr_state<T, D> derive from the same base and you can make it compatible.

`shared_ptr` actually has to store and type-erase a `shared_ptr_state<T, D, A>`, with `A` being the allocator used to allocate it the memory for it. I bring that up because that throws another snag in your plan. Namely, that `A` is both (potentially) user-provided and stored within that object. An object which will be destroyed potentially well before the memory is deallocated. And since the allocator is a potentially user-provide class, you can't rely on compiler magic to make it work even though it is destroyed.

It is trivial to construct a `container<A>` that stores a copy of `A` in a way that does not run its destructor when the enclosing object's destructor runs.

Sure, but how do you get the pointer to `A` back? And you have to be able to do that from just a `weak_ptr`.

Now, you might say that if `A` were user-provided, then clearly the shared state can't be a base class of `T`. So on the user end, if a class derives from this state object, then `A` will never exist. So the actual user-facing shared state object would be the internal shared state, which uses an empty default allocator that would never be used.

So... what should happen if someone calls `alloc_shared<T>` on such a type? If the user calls one of the `shared_ptr` constructors that takes an allocator, then one could conceivably just ignore the allocator, since the storage for the shared state already exists. But with `alloc_shared`, no storage exists for anything yet. `alloc_shared` is required to allocate from that allocator, store the allocator within that storage, and use that stored copy of the allocator to delete it.

Is there some form of type-erasure that could handle this without requiring further allocations? Or would you just make it an error to call `alloc_shared` on such a type?

If I call `allocate_shared()` I expect it to do what `allocate_shared()` does, and that is to not care about the nature of `T` and just do the normal business it does today. You don't seem to understand that the purpose of having `shared_ptr_state` accessible is to do these things *yourself* as is the point of having intrusive pointers in the first place. So If I decide to derivce my class from `shared_ptr_state<T, D, A>` I explicitly buy into managing its allocation myself and therefore can provide the allocator it needs.

OK, so you're saying that if you have some type that derives from shared_ptr_state, you want it to fail if the user tries to provide a different deleter or allocator type than the one in the object already. You want `allocate_shared` to fail if you provide a different allocator, and you want the `shared_ptr` constructors to fail if you provide a different deleter/allocator.

You've yet to answer the critical question: why do you need `shared_ptr` to do all of this? If you want to make an `intrusive_ptr` type that's based on some `instrusive_state`, why not propose that? Why do we need to transform `shared_ptr` into an intrusive pointer? Because `enable_shared_from_this` is a pointer or two larger than it absolutely needs to be?

Not only that, such an intrusive_ptr wouldn't even be as good as a real intrusive_ptr. Why? Because in a real intrusive_ptr, it would allow you to manage the state, rather than forcing you to derive from a class and break standard layout. Just look at `boost::intrusive_ptr`. It calls your code; it doesn't force you to use its class and derive from it. If you don't want to use atomic increments/decrements, you don't have to.

Why do you want to settle for this overcomplicated, half-formed intrusive_ptr-that-pretends-its-not instead of using the real thing?

As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.

Are there any other types, outside of Chapters 18 and a bit of 19, which have such an explicit reliance on compiler magic? Are there any other types where implementing them requires direct subversion of the C++ object and lifetime model? We're not talking about a type that, by its very nature, has to directly talk to the system (threads, IO, etc) or the compiler (type traits, etc). We're talking about a mere smart pointer.

It's one thing to have a type like `tuple`, where a clever compiler intrinsic or something could improve an implementation, but you can still implement it effectively otherwise. It's quite another thing altogether to define a type in such a way where it is functionally impossible to write an implementation that doesn't rely on platform-specific behavior.

Honestly I don't care. The standard only describes how types behave, not whether they need to rely on compiler magic to realize their contract. And you would be surprised how many intrinsics are used throughout std implementations in places one doesn't necessarily expect. One doesn't need special rules to disable automatically running the allocator and ref counter destructors. One only needs to make sure the compiler does not remove access to these memory locations or overwrite them with magic values like DEADBEEF in debug builds until they are actually deallocated. Sure, an implementation that cannot do that needs a separate allocation but I would consider that a QoI problem.

It can't be a QoI problem; weak_ptr's constructors are `noexcept`, so they're not allowed to allocate memory. Changing that is a breaking change. Furthermore, even if we were OK with such a change, we would also need to expand `weak_ptr`'s set of constructors, so that you could feed them an allocator to manage any memory they choose to allocate. And God only knows what happens if you pass a different allocator type to different `weak_ptr` constructors.

Not caring about these problems doesn't make them go away.

[Miro Knejp]

Am 06.08.2016 um 23:48 schrieb Nicol Bolas:

On Saturday, August 6, 2016 at 3:40:19 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 21:09 schrieb Nicol Bolas:

On Saturday, August 6, 2016 at 12:26:31 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 18:14 schrieb Nicol Bolas:

Here's the thing. The weak_ptr issue makes it impossible for `shared_ptr_state<T, D>` to be the exact same type that `shared_ptr` and `make_shared` would normally have allocated. This is because weak_ptr would have to have a separate allocation for the weak reference count, so `shared_ptr_state` wouldn't have a weak reference count.

As such, `shared/weak_ptr` now have to basically have separate class implementations: one for objects that don't inherit from `shared_ptr_state`, and one for types that do. And it can't even use SFINAE/concepts to make that determination, since the `T` is not the managed object type. So instead, it has to be a runtime check, used by every function that touches the managed state.

All to save the overhead of one pointer (on good implementations. Two worst-case). If you need an overhead-less smart pointer, use an intrusive pointer. We should have such a type in the standard. But we should not transform `shared/weak_ptr` into such a type.

shared_ptr does not have *one* type of shared state but basically an infinite number since it performs type erasure. So in reality there actually *is* a shared_ptr_state<T, D> (among others) but shared_ptr itself only accesses the erased base type. Make the user-accessible shared_ptr_state<T, D> derive from the same base and you can make it compatible.

`shared_ptr` actually has to store and type-erase a `shared_ptr_state<T, D, A>`, with `A` being the allocator used to allocate it the memory for it. I bring that up because that throws another snag in your plan. Namely, that `A` is both (potentially) user-provided and stored within that object. An object which will be destroyed potentially well before the memory is deallocated. And since the allocator is a potentially user-provide class, you can't rely on compiler magic to make it work even though it is destroyed.

It is trivial to construct a `container<A>` that stores a copy of `A` in a way that does not run its destructor when the enclosing object's destructor runs.

Sure, but how do you get the pointer to `A` back? And you have to be able to do that from just a `weak_ptr`.

Now, you might say that if `A` were user-provided, then clearly the shared state can't be a base class of `T`. So on the user end, if a class derives from this state object, then `A` will never exist. So the actual user-facing shared state object would be the internal shared state, which uses an empty default allocator that would never be used.

So... what should happen if someone calls `alloc_shared<T>` on such a type? If the user calls one of the `shared_ptr` constructors that takes an allocator, then one could conceivably just ignore the allocator, since the storage for the shared state already exists. But with `alloc_shared`, no storage exists for anything yet. `alloc_shared` is required to allocate from that allocator, store the allocator within that storage, and use that stored copy of the allocator to delete it.

Is there some form of type-erasure that could handle this without requiring further allocations? Or would you just make it an error to call `alloc_shared` on such a type?

If I call `allocate_shared()` I expect it to do what `allocate_shared()` does, and that is to not care about the nature of `T` and just do the normal business it does today. You don't seem to understand that the purpose of having `shared_ptr_state` accessible is to do these things *yourself* as is the point of having intrusive pointers in the first place. So If I decide to derivce my class from `shared_ptr_state<T, D, A>` I explicitly buy into managing its allocation myself and therefore can provide the allocator it needs.

OK, so you're saying that if you have some type that derives from shared_ptr_state, you want it to fail if the user tries to provide a different deleter or allocator type than the one in the object already. You want `allocate_shared` to fail if you provide a different allocator, and you want the `shared_ptr` constructors to fail if you provide a different deleter/allocator.

I did not say any of these things. Please quote the part where I say `allocate_shared()` is supposed to fail.

You've yet to answer the critical question: why do you need `shared_ptr` to do all of this? If you want to make an `intrusive_ptr` type that's based on some `instrusive_state`, why not propose that? Why do we need to transform `shared_ptr` into an intrusive pointer? Because `enable_shared_from_this` is a pointer or two larger than it absolutely needs to be?

It's not about making `shared_ptr` intrusive but about having the option for it to be so if the situation calls for it.

`shared_ptr` should not care about how its control block or the object were allocated as long as it conforms to the protocol `shared_ptr` establishes with its shared state. If I create an object that contains the shared state, create a `shared_ptr` form the `shared_ptr_state` and then pass said `shared_ptr` to the rest of the program I expect it to work. Using `shared_ptr_state` implies that one understands they are dealing with a low-level facility that has implications on how the type interacts with other parts of the library. To me it is obvious that if I create the `shared_ptr_state` as part of my object I have to use it as a factory to get the first `shared_ptr` instance out of it. Anything else just doesn't make sense and cannot possibly work. If I create the object and then construct a `shared_ptr` with a pointer I expect it to have a constructor that accepts `shared_ptr_state*` and deals with it appropriately. That behavior can be extrapolated to `make_shared()` and `allocate_shared()` as necessary.

It isn't any different from how a type using intrusive_ptr behaves. If a type is hardcoded to use a certain allocator then guess what, you cannot allocate it in a differnt way or things fall apart when the intrusive ref count goes to zero. You keep bringin up problems that are also problems with intrusive pointers.

Not only that, such an intrusive_ptr wouldn't even be as good as a real intrusive_ptr. Why? Because in a real intrusive_ptr, it would allow you to manage the state, rather than forcing you to derive from a class and break standard layout. Just look at `boost::intrusive_ptr`. It calls your code; it doesn't force you to use its class and derive from it. If you don't want to use atomic increments/decrements, you don't have to.

People already break standard layout with enable_shared_from_this. Put the data you care about having standard layout as member of the actual object and nothing is lost.

Why do you want to settle for this overcomplicated, half-formed intrusive_ptr-that-pretends-its-not instead of using the real thing?

Because this is not about getting an intrusive_ptr it's about having control over shared_ptr's control block management. It'd be great to have a dedicated `intrusive_ptr` but that won't be compatible with `shared_ptr` without having access to a facility like the one proposed here. Obviously if you don't need to be compatible with `shared_ptr` then you won't care about `shared_ptr_state`. Right now I cannot safely create an object on stack/array/subobject/etc and create a `shared_ptr` to it without at least allocating the shared state with a null-allocator or requiring the enclosing scope to be managed by a `shared_ptr` already.

As to the weak_ptr issue: shared_ptr_state<T, D> is part of the implementation and as such it is allowed to do what would otherwise be considered UB, including accessing the weak count member even though the containing object has already ended its lifetime using the appropriate compiler intrinsics/annotations or simply by knowing how the compiler works.

Are there any other types, outside of Chapters 18 and a bit of 19, which have such an explicit reliance on compiler magic? Are there any other types where implementing them requires direct subversion of the C++ object and lifetime model? We're not talking about a type that, by its very nature, has to directly talk to the system (threads, IO, etc) or the compiler (type traits, etc). We're talking about a mere smart pointer.

It's one thing to have a type like `tuple`, where a clever compiler intrinsic or something could improve an implementation, but you can still implement it effectively otherwise. It's quite another thing altogether to define a type in such a way where it is functionally impossible to write an implementation that doesn't rely on platform-specific behavior.

Honestly I don't care. The standard only describes how types behave, not whether they need to rely on compiler magic to realize their contract. And you would be surprised how many intrinsics are used throughout std implementations in places one doesn't necessarily expect. One doesn't need special rules to disable automatically running the allocator and ref counter destructors. One only needs to make sure the compiler does not remove access to these memory locations or overwrite them with magic values like DEADBEEF in debug builds until they are actually deallocated. Sure, an implementation that cannot do that needs a separate allocation but I would consider that a QoI problem.

It can't be a QoI problem; weak_ptr's constructors are `noexcept`, so they're not allowed to allocate memory.

Let `shared_ptr_state` allocate the memory if needed. Nobody except you is suggesting `weak_ptr`c needs to allocate something. So yes, it is a QoI issue.

Changing that is a breaking change. Furthermore, even if we were OK with such a change, we would also need to expand `weak_ptr`'s set of constructors, so that you could feed them an allocator to manage any memory they choose to allocate. And God only knows what happens if you pass a different allocator type to different `weak_ptr` constructors.

Not caring about these problems doesn't make them go away.

And caring about it too much creates situations where "God only knows what happens if".

[Thiago Macieira]

On sábado, 6 de agosto de 2016 10:06:54 PDT Viacheslav Usov wrote:
> std::shared_ptr & Co try to cater to many needs, but a prime use case - no
> frills new/delete allocation with a reference count embedded into a derived
> class - is not addressed directly. This is at variance with the C++ tenet
> that one does not pay for what is not used.

Exactly. Since shared_ptr is meant to be used with weak_ptr, if you carry your
own reference count in your object, you'd be paying for something you don't
want to use. So use a better implementation for your object: that's not
std::shared_ptr.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Nicol Bolas]

On Saturday, August 6, 2016 at 6:34:21 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 23:48 schrieb Nicol Bolas:

On Saturday, August 6, 2016 at 3:40:19 PM UTC-4, Miro Knejp wrote:

Am 06.08.2016 um 21:09 schrieb Nicol Bolas:

You've yet to answer the critical question: why do you need `shared_ptr` to do all of this? If you want to make an `intrusive_ptr` type that's based on some `instrusive_state`, why not propose that? Why do we need to transform `shared_ptr` into an intrusive pointer? Because `enable_shared_from_this` is a pointer or two larger than it absolutely needs to be?

It's not about making `shared_ptr` intrusive but about having the option for it to be so if the situation calls for it.

We already have that option; it's called `make/allocate_shared`. Using that makes the object exactly as intrusive as your suggested code would.

It simply can't be used for an already existing object. But if you "have to use it as a factory to get the first `shared_ptr` instance out of it", then there's no reason why you can't use `make/allocate_shared` to do so.

`shared_ptr` should not care about how its control block or the object were allocated as long as it conforms to the protocol `shared_ptr` establishes with its shared state. If I create an object that contains the shared state, create a `shared_ptr` form the `shared_ptr_state` and then pass said `shared_ptr` to the rest of the program I expect it to work. Using `shared_ptr_state` implies that one understands they are dealing with a low-level facility that has implications on how the type interacts with other parts of the library. To me it is obvious that if I create the `shared_ptr_state` as part of my object I have to use it as a factory to get the first `shared_ptr` instance out of it. Anything else just doesn't make sense and cannot possibly work. If I create the object and then construct a `shared_ptr` with a pointer I expect it to have a constructor that accepts `shared_ptr_state*` and deals with it appropriately.

We now seem to have gone far afield from the basis of the thread.

This thread was originally about having a way to get `enable_shared_from_this` behavior without additional overhead of a pointer(s) to the shared state. You seem to now be talking about a generic "I want to be able to have direct control over the shared state" kind of thing.

Are you asking for the ability to create `shared_ptr_state` arbitrarily, then hand it and a pointer to `T` off to `shared_ptr`'s constructor, and expect it to use the shared state you created?

If so, that's a very different kind of conversation. Granted, it still won't work, but it's a different design.

 

That behavior can be extrapolated to `make_shared()` and `allocate_shared()` as necessary.

No, that behavior makes absolutely no sense for such functions. One of the primary purposes of these functions is that they allocate the memory for the object and the shared state together. By using them, you are choosing to give up control of such allocations.
 

It isn't any different from how a type using intrusive_ptr behaves. If a type is hardcoded to use a certain allocator then guess what, you cannot allocate it in a differnt way or things fall apart when the intrusive ref count goes to zero. You keep bringin up problems that are also problems with intrusive pointers.

Intrusive pointers don't tend to have weak variants, which cuts down on the problems a great deal. Indeed, the existence of `weak_ptr` is basically 99% of the problem with wanting complete control over the shared state.

Why do you want to settle for this overcomplicated, half-formed intrusive_ptr-that-pretends-its-not instead of using the real thing?

Because this is not about getting an intrusive_ptr it's about having control over shared_ptr's control block management. It'd be great to have a dedicated `intrusive_ptr` but that won't be compatible with `shared_ptr` without having access to a facility like the one proposed here. Obviously if you don't need to be compatible with `shared_ptr` then you won't care about `shared_ptr_state`.

Why would you want an `intrusive_ptr` to be compatible with a `shared_ptr`? Is it just for compatibility with other libraries that take `shared_ptr`s?

`shared_ptr` is intended to handle the vast majority of cases. It is not meant to handle all of them, and we shouldn't try to force it to do so.

Right now I cannot safely create an object on stack/array/subobject/etc and create a `shared_ptr` to it without at least allocating the shared state with a null-allocator or requiring the enclosing scope to be managed by a `shared_ptr` already.

So your use case here is that you have some object on the stack. And you want to create a `shared_ptr` to it. And presumably there's some API you're going to pass this pointer to which will claim ownership of the object, but this ownership will not escape the stack that called it.

And you want to avoid "allocating the shared state." So the problem you're trying to solve is the memory allocation for the shared state.

Now, let's take this example to the next issue:

Honestly I don't care. The standard only describes how types behave, not whether they need to rely on compiler magic to realize their contract. And you would be surprised how many intrinsics are used throughout std implementations in places one doesn't necessarily expect. One doesn't need special rules to disable automatically running the allocator and ref counter destructors. One only needs to make sure the compiler does not remove access to these memory locations or overwrite them with magic values like DEADBEEF in debug builds until they are actually deallocated. Sure, an implementation that cannot do that needs a separate allocation but I would consider that a QoI problem.

It can't be a QoI problem; weak_ptr's constructors are `noexcept`, so they're not allowed to allocate memory.

Let `shared_ptr_state` allocate the memory if needed. Nobody except you is suggesting `weak_ptr`c needs to allocate something. So yes, it is a QoI issue.

Your previous example was that you wanted a stack object wrapped in a shared_ptr, and you wanted to do this without having to allocate extra memory. Well, you've now said that you have to do just that.

After all, "if needed" happens the moment the type is wrapped in a `shared_ptr`. It cannot wait to allocate that memory until someone gets a `weak_ptr` to the object, since `weak_ptr`'s constructors are `noexcept`. And therefore neither themselves nor any code they call are allowed to allocate memory.

So "if needed" must be when they are first used with a `shared_ptr`. And therefore, if `weak_ptr`'s implementation requires allocated memory (and it will, under your rules), this memory must be allocated when the `shared_ptr` is first constructed. And thus, your solution doesn't solve the problem you intended.

The core design of `shared/weak_ptr` is built around the shared state object being managed by the internal system and thus being able to outlive the object it manages. `weak_ptr`s simply can't do their job otherwise. This is also why you often don't see intrusive pointers providing weak references. Because doing so requires allocating extra memory, and the whole point of intrusive pointers is to avoid that kind of overhead.

[Viacheslav Usov]

On Sun, Aug 7, 2016 at 8:54 PM, Nicol Bolas <jmck...@gmail.com> wrote:

I cannot say I have followed and really understood everything that you and the others discussed here, so I am trying to clarify this for myself.

> Indeed, the existence of `weak_ptr` is basically 99% of the problem with wanting complete control over the shared state.

If, hypothetically, we would somehow not have to deal with weak_ptr, are there other reasons why the hypothetical std::shared_ptr_state would still be impossible?

Cheers,

V.

[Viacheslav Usov]

On Sat, Aug 6, 2016 at 3:54 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Wait, I just realized something. You cannot embed the shared state in the object being managed. If you did, `weak_ptr` wouldn't be able to function at all.

You may be mistaken about the "function" of weak_ptr per the standard. You assume, as you said in a later message, that the shared state is supposed to outlive the object, with a further assumption that as soon as the state is only referenced by weak_ptr's, the object is deleted and storage deallocated. I do not think this is required by the standard and I know that in certain implementations make_shared just allocates everything in one big happy blob.

Perhaps it is the absence of clearly stipulated traits weak_ptr should have that makes them so complicated.

Cheers,

V.

[Edward Catmur]

On Mon, Aug 8, 2016 at 11:08 AM, Viacheslav Usov <via....@gmail.com> wrote:

On Sat, Aug 6, 2016 at 3:54 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Wait, I just realized something. You cannot embed the shared state in the object being managed. If you did, `weak_ptr` wouldn't be able to function at all.

You may be mistaken about the "function" of weak_ptr per the standard. You assume, as you said in a later message, that the shared state is supposed to outlive the object, with a further assumption that as soon as the state is only referenced by weak_ptr's, the object is deleted and storage deallocated. I do not think this is required by the standard and I know that in certain implementations make_shared just allocates everything in one big happy blob.

The storage does not have to be deallocated, but the object has to be destroyed (not deleted).

If the weak state is included in the shared state, and the shared state is included in the object (the intent of inheriting from shared_ptr_state) then the weak state will be destroyed when the object is destroyed, so any weak pointers will point to destroyed state; this would make it difficult to implement without relying on undefined or at least implementation-specific behavior to make referring to destroyed state OK.

If the weak state is external to the object, then that complicates the current case (where the shared state is external to the object and so can be composed with the weak state) and requires a separate allocation, at least in the shared_ptr{new T} and (new T)->shared_from_this() cases.

I think it might be possible to avoid a separate allocation for external weak state when using make_shared/allocate_shared, via the following process: the shared state constructor initializes its weak_state member pointer to nullptr; make_shared over-allocates and sets weak_state to the extra allocation after object construction but before any shared_ptr referring to the object is constructed; if a shared_ptr constructor is called with weak_state null, it allocates a weak state and sets weak_state to the separate allocation. However this incurs overhead on current users of shared_ptr and would likely break ABI compatibility.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 1:34 PM, 'Edward Catmur' via ISO C++ Standard - Discussion <std-dis...@isocpp.org> wrote:

> If the weak state is included in the shared state, and the shared state is included in the object (the intent of inheriting from shared_ptr_state) then the weak state will be destroyed when the object is destroyed, so any weak pointers will point to destroyed state; this would make it difficult to implement without relying on undefined or at least implementation-specific behavior to make referring to destroyed state OK.

Why does this need to invoke undefined or implementation-specific behaviour? If the storage for the object is not deleted, then the only effect is that of running the destructor. The memory location is not magically invalidated or made untouchable by that. If the destructor of the shared state is trivial, then, per [basic.life] it ends its lifetime when the storage is reused or released. And because the embedded shared state does not need much more than two reference counts, it is very hard to imagine why its destructor would be non-trivial.

Cheers,

V.

[Edward Catmur]

[basic.life] doesn't apply here; the shared state is a base subobject, so we are referred to [class.base.init], except I think [class.cdtor] is actually more relevant: "referring to any non-static member or base class of the object after the destructor finishes execution results in undefined behavior". I think this holds even if the reference is via a pointer (or reference) obtained while the most derived object was alive and the subobject is trivially destructible, though I could be wrong (the examples don't seem to cover this).

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 2:07 PM, 'Edward Catmur' via ISO C++ Standard - Discussion <std-dis...@isocpp.org> wrote:

> I think [class.cdtor] is actually more relevant: "referring to any non-static member or base class of the object after the destructor finishes execution results in undefined behavior".

In both n4296 and n4594, that follows "For an object with a non-trivial destructor", which is similar to what I said earlier.

Cheers,

V.

[Edward Catmur]

Right, but that means the most derived object, not the subobject. It'd be pretty silly if shared_ptr_state only worked for trivially destructible T.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 2:31 PM, 'Edward Catmur' via ISO C++ Standard - Discussion <std-dis...@isocpp.org> wrote:

> Right, but that means the most derived object, not the subobject.

Why? I can see that is the case for storage duration per [basic.stc.inherit], but for object lifetime? Can you quote the standard on that?

Cheers,

V.

[Edward Catmur]

[basic.life]/3: [ Note: [class.base.init] describes the lifetime of base and member subobjects.  — end note ]

Admittedly, [class.base.init] doesn't actually say when lifetime begins, and definitely doesn't describe when lifetime ends. That's probably a defect, which is why I looked at [class.cdtor] for guidance.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 3:31 PM, 'Edward Catmur' via ISO C++ Standard - Discussion <std-dis...@isocpp.org> wrote:

> Admittedly, [class.base.init] doesn't actually say when lifetime begins, and definitely doesn't describe when lifetime ends. That's probably a defect, which is why I looked at [class.cdtor] for guidance.

So if we interpret the standard as is, without assuming a defect, undefined behaviour is nowhere required to result when accessing members of a sub-object with a trivial destructor after the destructor finishes and before storage is reused or released. Right?

Cheers,

V.

[Edward Catmur]

Well, [class.cdtor] specifically says that base and member subobjects may only be accessed after the constructor begins and before the destructor finishes.

Actually, now I'm not sure. The subobject relation is not transitive, so [class.cdtor]/1 may not apply to subobjects of subobjects (and so forth) when the access to the subobject subobject does not touch the immediate subobject. We should probably take this query to std-discussion, if you're OK with that.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 4:02 PM, 'Edward Catmur' via ISO C++ Standard - Discussion <std-dis...@isocpp.org> wrote:

> Well, [class.cdtor] specifically says that base and member subobjects may only be accessed after the constructor begins and before the destructor finishes.

Yes, but then there is that thing about trivial dtors :)

> Actually, now I'm not sure. The subobject relation is not transitive, so [class.cdtor]/1 may not apply to subobjects of subobjects (and so forth) when the access to the subobject subobject does not touch the immediate subobject. We should probably take this query to std-discussion, if you're OK with that.

I am, and we are actually in std-discussion, but perhaps we should start a new thread, because this one is probably horrifying for other people at this stage.

Cheers,

V.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 13:58:11 PDT Viacheslav Usov wrote:
> Why does this need to invoke undefined or implementation-specific
> behaviour? If the storage for the object is not deleted, then the only
> effect is that of running the destructor. The memory location is not
> magically invalidated or made untouchable by that.

It might be. The compiler is free to add a memset(this, 0xcd, sizeof(*this))
at the end of the destructor, to facilitate detection of use-after-deletion.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 4:39 PM, Thiago Macieira <thi...@macieira.org> wrote:

> It might be. The compiler is free to add a memset(this, 0xcd, sizeof(*this)) at the end of the destructor, to facilitate detection of use-after-deletion.

Free according to what exactly? I think this re-iterates the argument we just had with Edward.

And, really, if it is use after DELETION, why is it free do so after DESTRUCTION? In my experience, that is done when memory is deallocated, not when the object is destructed.

Cheers,

V.

[Andrey Semashev]

On 08/08/16 14:58, Viacheslav Usov wrote:
> On Mon, Aug 8, 2016 at 1:34 PM, 'Edward Catmur' via ISO C++ Standard -
> Discussion <std-dis...@isocpp.org

> <mailto:std-dis...@isocpp.org>> wrote:
>
>> If the weak state is included in the shared state, and the shared
> state is included in the object (the intent of inheriting from
> shared_ptr_state) then the weak state will be destroyed when the object
> is destroyed, so any weak pointers will point to destroyed state; this
> would make it difficult to implement without relying on undefined or at
> least implementation-specific behavior to make referring to destroyed
> state OK.
>
> Why does this need to invoke undefined or implementation-specific
> behaviour? If the storage for the object is not deleted, then the only
> effect is that of running the destructor.

Running the body of the destructor may not be the only thing the
compiler has to do. For example, the compiler also has to ensure the
correct virtual functions are called during destruction (i.e. update the
pointer to the vtable). As John has mentioned, the compiler is also not
forbidden to clobber memory after destruction (e.g. fill with trapping
values).

I think [basic.life]/6 is pretty clear on this:

... The program has undefined behavior if:

...
— the glvalue is used to access a non-static data member or call a
non-static member function of the object, or
...

> The memory location is not magically invalidated or made untouchable by that.

It's not, but it no longer represents the object. It is raw storage with
unspecified contents, nothing more.

> If the destructor of
> the shared state is trivial, then, per [basic.life] it ends its lifetime
> when the storage is reused or released. And because the embedded shared
> state does not need much more than two reference counts, it is very hard
> to imagine why its destructor would be non-trivial.

First, the enclosing object's destructor can be non-trivial. Second, the
shared state may contain a deleter and an allocator that are supposed to
be used to destroy the object and deallocate the storage. Either of
these types may have non-trivial destructors.

[Andrey Semashev]

Also, if the allocator is destroyed while the object is destroyed,
arguably it will not be able to deallocate the storage.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 16:50:21 PDT Viacheslav Usov wrote:
> On Mon, Aug 8, 2016 at 4:39 PM, Thiago Macieira <thi...@macieira.org> wrote:
> > It might be. The compiler is free to add a memset(this, 0xcd,
>
> sizeof(*this)) at the end of the destructor, to facilitate detection of
> use-after-deletion.
>
> Free according to what exactly? I think this re-iterates the argument we
> just had with Edward.

Because it's undefined behaviour to use the object after destructor has
finished. You may reuse the storage, but you need to begin the liftetime of a
new object there.

Your discussion with Edward was whether it applied to a trivial object. Well,
this is not the case here:
- the shared state object is not guaranteed to be trivial
- the user's object cannot be constrained to be trivial

Conclusion: the whole object is must be assumed to be not trivial, which means
that the full storage may have been overwritten.

> And, really, if it is use after DELETION, why is it free do so after
> DESTRUCTION? In my experience, that is done when memory is deallocated, not
> when the object is destructed.

Your experience is not guarantee for all situations.

[Edward Catmur]

On Mon, Aug 8, 2016 at 4:21 PM, Andrey Semashev <andrey....@gmail.com> wrote:

On 08/08/16 18:18, Andrey Semashev wrote:

On 08/08/16 14:58, Viacheslav Usov wrote:

On Mon, Aug 8, 2016 at 1:34 PM, 'Edward Catmur' via ISO C++ Standard -
Discussion <std-dis...@isocpp.org

<mailto:std-discussion@isocpp.org>> wrote:

If the weak state is included in the shared state, and the shared

state is included in the object (the intent of inheriting from
shared_ptr_state) then the weak state will be destroyed when the object
is destroyed, so any weak pointers will point to destroyed state; this
would make it difficult to implement without relying on undefined or at
least implementation-specific behavior to make referring to destroyed
state OK.

Why does this need to invoke undefined or implementation-specific
behaviour? If the storage for the object is not deleted, then the only
effect is that of running the destructor.

Running the body of the destructor may not be the only thing the
compiler has to do. For example, the compiler also has to ensure the
correct virtual functions are called during destruction (i.e. update the
pointer to the vtable). As John has mentioned, the compiler is also not
forbidden to clobber memory after destruction (e.g. fill with trapping
values).

But it is forbidden to clobber memory of a trivially destructible complete object, right?

 

I think [basic.life]/6 is pretty clear on this:

... The program has undefined behavior if:

...
— the glvalue is used to access a non-static data member or call a
non-static member function of the object, or
...

We aren't accessing a non-static data member, we're accessing a non-static data member of a non-static data member. If [basic.life], [class.base.init] or [class.cdtor] said that the immediate subobject lifetime had ended, then [basic.life]/6 would apply, but all they say is that accessing the immediate subobject is illegal. We formed the pointer to weak state while the complete object was alive, so we aren't accessing the immediate subobject.

The memory location is not magically invalidated or made untouchable
by that.

It's not, but it no longer represents the object. It is raw storage with
unspecified contents, nothing more.

If the destructor of
the shared state is trivial, then, per [basic.life] it ends its lifetime
when the storage is reused or released. And because the embedded shared
state does not need much more than two reference counts, it is very hard
to imagine why its destructor would be non-trivial.

First, the enclosing object's destructor can be non-trivial. Second, the
shared state may contain a deleter and an allocator that are supposed to
be used to destroy the object and deallocate the storage. Either of
these types may have non-trivial destructors.

Also, if the allocator is destroyed while the object is destroyed, arguably it will not be able to deallocate the storage.

If we construct placement-construct allocator and/or deleter in raw storage contained within the complete object, we can defer destruction of the allocator and/or deleter until the weak state is disposed.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 5:18 PM, Andrey Semashev <andrey....@gmail.com> wrote:

> Running the body of the destructor may not be the only thing the compiler has to do. For example, the compiler also has to ensure the correct virtual functions are called during destruction (i.e. update the pointer to the vtable).

Again, this would require the destructor to be non-trivial.

> As John has mentioned, the compiler is also not forbidden to clobber memory after destruction (e.g. fill with trapping values).

I hope he will be able to substantiate that by quoting the standard.

> I think [basic.life]/6 is pretty clear on this:

That section applies only when the lifetime of the object has ended. While section 1.3 & 1.4 say that for an object with a trivial destructor the lifetime ends only when storage is reused or released, so section 6 is inapplicable in this case.

> First, the enclosing object's destructor can be non-trivial.

That does not make the sub-objects's destructor non-trivial. As I said earlier, this just re-iterates what Edward and I had discussed.

Cheers,

V.

[Nicol Bolas]

If you could ignore the existence of `weak_ptr`, then I don't see a reason why `shared_ptr_state` couldn't work.

Consider your entire thread of conversation below. It's all about how to access `shared_ptr_state` after the object containing it is destroyed. Well, the only reason it would be possible to access it later is because there's a `weak_ptr` out there. No `weak_ptr` means that's no longer an issue.

But you can't ignore `weak_ptr`. It is a fundamental part of the `shared_ptr` interface, and people who use `shared_ptr` have the right to expect `weak_ptr` to work. You can't even make getting a `weak_ptr` from a `shared_ptr` throw an exception if the managed object uses `shared_ptr_state`, because `weak_ptr`'s constructors are `noexcept`.

And having it return a NULL `weak_ptr` would be extremely rude, since again, they may have been relying on being able to weakly access that object.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 5:31 PM, Thiago Macieira <thi...@macieira.org> wrote:

> Because it's undefined behaviour to use the object after destructor has finished.

This does re-iterate what was discussed earlier. Sorry, I cannot afford cycling through that infinitely, so I'll be brief, and I do not mean it in a bad way.

> the shared state object is not guaranteed to be trivial

That's up to the implementation, which can certainly make it trivial.

> the user's object cannot be constrained to be trivial

That is irrelevant as discussed earlier.

> Conclusion: the whole object is must be assumed to be not trivial

Ditto.

> which means that the full storage may have been overwritten.

Non sequitur.

> Your experience is not guarantee for all situations.

You brought that up, you did not substantiate that by quoting the standard, my experience is contrary, where do you want to go from there?

Cheers,

V.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 5:42 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> If you could ignore the existence of `weak_ptr`, then I don't see a reason why `shared_ptr_state` couldn't work.

That's what I wanted to clarify. So, assuming we can deal with weak_ptr in some way, there is no other obstacle you could think of?

Cheers,

V.

[Andrey Semashev]

On 08/08/16 18:36, 'Edward Catmur' via ISO C++ Standard - Discussion wrote:
> On Mon, Aug 8, 2016 at 4:21 PM, Andrey Semashev
> <andrey....@gmail.com <mailto:andrey....@gmail.com>> wrote:
>
> On 08/08/16 18:18, Andrey Semashev wrote:
>
> Running the body of the destructor may not be the only thing the
> compiler has to do. For example, the compiler also has to ensure the
> correct virtual functions are called during destruction (i.e.
> update the
> pointer to the vtable). As John has mentioned, the compiler is
> also not
> forbidden to clobber memory after destruction (e.g. fill with
> trapping
> values).
>
> But it is forbidden to clobber memory of a trivially destructible
> complete object, right?

Yes, as that would make the destructor non-trivial. I hope we're not
considering limiting shared_ptr_state to trivially destructible classes
only, are we?

> I think [basic.life]/6 is pretty clear on this:
>
> ... The program has undefined behavior if:
>
> ...
> — the glvalue is used to access a non-static data member or call a
> non-static member function of the object, or
> ...
>
> We aren't accessing a non-static data member, we're accessing a
> non-static data member of a non-static data member.

I don't see the significance. If an object is destroyed then all its
non-static members are also destroyed, recursively so.

> If [basic.life],
> [class.base.init] or [class.cdtor] said that the immediate subobject
> lifetime had ended, then [basic.life]/6 would apply, but all they say is
> that accessing the immediate subobject is illegal. We formed the pointer
> to weak state while the complete object was alive, so we aren't
> accessing the immediate subobject.

I'm sorry, I don't follow.

I'm considering the case when all shared_ptrs to the object are
destroyed and weak_ptrs are left referencing the storage which is
supposed to hold the shared_ptr_state. These weak_ptrs will have to
access shared_ptr_state, and in order for that to be well-defined the
state has to be not destroyed. It cannot be so if the state is a member
of the object, no matter at what level of object nesting.

> If we construct placement-construct allocator and/or deleter in raw
> storage contained within the complete object, we can defer destruction
> of the allocator and/or deleter until the weak state is disposed.

I don't think this is a valid assumption. If the enclosing class'
destructor is not trivial, you cannot assume anything about the contents
of that raw storage after the destructor completes.

[Andrey Semashev]

On 08/08/16 18:38, Viacheslav Usov wrote:
> On Mon, Aug 8, 2016 at 5:18 PM, Andrey Semashev
> <andrey....@gmail.com <mailto:andrey....@gmail.com>> wrote:
>
>> Running the body of the destructor may not be the only thing the
> compiler has to do. For example, the compiler also has to ensure the
> correct virtual functions are called during destruction (i.e. update the
> pointer to the vtable).
>
> Again, this would require the destructor to be non-trivial.

Of course.

>> As John has mentioned, the compiler is also not forbidden to clobber
> memory after destruction (e.g. fill with trapping values).
>
> I hope he will be able to substantiate that by quoting the standard.

I don't think he has to. The standard doesn't specify the contents of
the storage after the non-trivial constructor completes, which means the
contents are, well, unspecified.

>> I think [basic.life]/6 is pretty clear on this:
>
> That section applies only when the lifetime of the object has ended.
> While section 1.3 & 1.4 say that for an object with a trivial destructor
> the lifetime ends only when storage is reused or released, so section 6
> is inapplicable in this case.

Right. But as I said in another post, I hope we're not limiting
shared_ptr_state to trivially destructible classes only.

>> First, the enclosing object's destructor can be non-trivial.
>
> That does not make the sub-objects's destructor non-trivial. As I said
> earlier, this just re-iterates what Edward and I had discussed.

The sub-object's destructor doesn't matter in this case because the
enclosing object's destructor is non-trivial and can leave the storage
in any state after completion.

[Nicol Bolas]

Your statement is rather like saying, "if the Grand Canyon weren't in the way, we could just walk there, right?" That's true... but it's not particularly useful.

[Thiago Macieira]

Sure.

Please make sure that the following function returns true:

struct ForwardDeclared;
bool f(shared_ptr<ForwardDeclared> ptr)
{
return weak_ptr(ptr).use_count() != 0;

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 6:16 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Your statement is rather like saying, "if the Grand Canyon weren't in the way, we could just walk there, right?" That's true... but it's not particularly useful.

That is useful, and if you think about it, you will certainly remember that as a programmer you must have done that sort of thing just about every day.

In this particular case, it is useful, because if it is only weak_ptr that is a problem, it is not a fundamental problem, without even going into all those complications we have been talking about recently.

Each weak_ptr can have a field that links it to another weak_ptr. The shared state can have a field that is the head of the list of all the weak_ptr's to the given shared state. The implementation can have a global mutex that synchronises access to all the weak_ptr lists. When a shared state goes down, it simply iterates over all the weak_ptrs that point to it, and makes their pointers to itself null.

I am not saying that is the best or even good way to deal with this. It is just a demonstration that the problem is tractable.

Cheers,

V.

[Edward Catmur]

On Mon, Aug 8, 2016 at 5:07 PM, Andrey Semashev <andrey....@gmail.com> wrote:

We aren't accessing a non-static data member, we're accessing a
non-static data member of a non-static data member.

I don't see the significance. If an object is destroyed then all its non-static members are also destroyed, recursively so.

That doesn't necessarily mean their lifetime has ended. Yes, non-trivial destructors are run, and those end the lifetime of the objects they are run on, but only up until a trivially destructible subobject is reached.

I'm considering the case when all shared_ptrs to the object are destroyed and weak_ptrs are left referencing the storage which is supposed to hold the shared_ptr_state. These weak_ptrs will have to access shared_ptr_state, and in order for that to be well-defined the state has to be not destroyed. It cannot be so if the state is a member of the object, no matter at what level of object nesting.

Why? The storage holding the shared state may be destroyed, but the lifetime of the storage may not have ended. Its lifetime has certainly not ended if the complete object is trivially destructible; at question is whether its lifetime has ended when the complete object is not trivially destructible.

 

If we construct placement-construct allocator and/or deleter in raw
storage contained within the complete object, we can defer destruction
of the allocator and/or deleter until the weak state is disposed.

I don't think this is a valid assumption. If the enclosing class' destructor is not trivial, you cannot assume anything about the contents of that raw storage after the destructor completes.

Again, why? User code cannot touch that raw storage; where is the license for the compiler to do so? I thought [class.cdtor]/1 was such license, but Viacheslav has persuaded me that may not be so clear.

As John has mentioned, the compiler is also not forbidden to clobber

memory after destruction (e.g. fill with trapping values).

I hope he will be able to substantiate that by quoting the standard.

I don't think he has to. The standard doesn't specify the contents of the storage after the non-trivial constructor completes, which means the contents are, well, unspecified.

Well, maybe it does specify the contents. If user code does not write to the storage during the non-trivial destructor sequence, and if the lifetime of the storage continues through the destructor call, and if it is legal to use a previously formed pointer or reference to access its contents, then its contents must be the same as before the destructor call, just as they would be were the complete object destructor trivial.

 

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 17:45:54 PDT Viacheslav Usov wrote:
> > the shared state object is not guaranteed to be trivial
>
> That's up to the implementation, which can certainly make it trivial.

Or, phrased it differently, that's up to the implementation, which means it
may be non-trivial. Note that they contain atomics, which are not copyable,
which means they aren't trivial.

> > the user's object cannot be constrained to be trivial
>
> That is irrelevant as discussed earlier.

Sorry, it's very much relevant because I don't think the earlier discussion's
conclusion was valid. The whole object is responsible for its storage and
could overwrite any of the sub-objects. The destructor is perfectly allowed to
do that

memset(this, 0xcd, sizeof(*this));

before returning.

> > Your experience is not guarantee for all situations.
>
> You brought that up, you did not substantiate that by quoting the standard,
> my experience is contrary, where do you want to go from there?

The point is that the experience of something working in a given situation is
not guarantee it will always work, in other situations.

You can walk with a basketball and you may have gotten away with it if the
referee didn't see it. That's not sufficient to guarantee that you'll always get
away with it.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 16:36:08 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> But it is forbidden to clobber memory of a trivially destructible complete
> object, right?

The trivial destructor can't do it, clearly, otherwise it wouldn't be trivial.

But a non-trivial destructor for an outer object that contains this sub-object
could.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 6:44 PM, Thiago Macieira <thi...@macieira.org> wrote:

> Or, phrased it differently, that's up to the implementation, which means it may be non-trivial. Note that they contain atomics, which are not copyable, which means they aren't trivial.

The issue here is the triviality of the destructor. That is defined in [class.dtor] and has nothing to do with copyability.

> Sorry, it's very much relevant because I don't think the earlier discussion's conclusion was valid. The whole object is responsible for its storage and could overwrite any of the sub-objects. The destructor is perfectly allowed to do that  memset(this, 0xcd, sizeof(*this)); before returning.

Which is instant UB for all but POD and similar types. Consider:

struct A

{

    std::unique_ptr<char> ptr;

    ~A() { memset(this, 0xcd, sizeof(*this)); }

};

What happens to ptr which is yet to be destructed after the memset?

> The point is that the experience of something working in a given situation is not guarantee it will always work, in other situations.

I did not invoke my experience only. There were other reasons, which you keep ignoring.

Cheers,

V.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 17:39:51 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> On Mon, Aug 8, 2016 at 5:07 PM, Andrey Semashev <andrey....@gmail.com>
> wrote:
> >> We aren't accessing a non-static data member, we're accessing a
> >> non-static data member of a non-static data member.
> >
> > I don't see the significance. If an object is destroyed then all its
> > non-static members are also destroyed, recursively so.
>
> That doesn't necessarily mean their lifetime has ended. Yes, non-trivial
> destructors are run, and those end the lifetime of the objects they are run
> on, but only up until a trivially destructible subobject is reached.

Sorry, that doesn't make sense. It has to mean that. When the lifetime of an
object ends, the lifetime of all of its sub-objects, recursively, must have
ended too.

If you meant that the memory continues to be comprised of bytes and those
bytes can be read, sure, they can. It doesn't mean that they continue to
contain what they used to. The contents may have changed because of the non-
trivial destructors that were run.

> > I don't think this is a valid assumption. If the enclosing class'
> > destructor is not trivial, you cannot assume anything about the contents
> > of
> > that raw storage after the destructor completes.
>
> Again, why? User code cannot touch that raw storage; where is the license
> for the compiler to do so? I thought [class.cdtor]/1 was such license, but
> Viacheslav has persuaded me that may not be so clear.

Why can't the user code touch that raw storage? If the sub-object is trivially
destructible, then it doesn't matter what bytes it contains when the that
destructor would have been run. So the user code *can* memset it and it's a
valid use of the sub-object.

Moreover, even if we say that the lifetime ends only when another object's
lifetime begins there, what's to prevent exactly that from happening? Why
couldn't the non-trivial outer destructor begin the lifetime of another sub-
object in the same storage bytes?

> > As John has mentioned, the compiler is also not forbidden to clobber
> >> memory after destruction (e.g. fill with trapping values).
> >>
> >> I hope he will be able to substantiate that by quoting the standard.
> >
> > I don't think he has to. The standard doesn't specify the contents of the
> > storage after the non-trivial constructor completes, which means the
> > contents are, well, unspecified.
>
> Well, maybe it does specify the contents. If user code does not write to
> the storage during the non-trivial destructor sequence, and if the lifetime
> of the storage continues through the destructor call, and if it is legal to
> use a previously formed pointer or reference to access its contents, then
> its contents must be the same as before the destructor call, just as they
> would be were the complete object destructor trivial.

I disagree. There's no guarantee that the byte contents of a non-trivial
object will remain the same after said destructor has run.

Proof by absurd: you're saying that all trivial sub-objects of any object must
retain their values as they had before the outer object's destructor was run.
But obviously everything is constructed from primitive types, which are
trivial. The only way to do that is if destructors are not allowed to write to
any of their sub-objects. Conclusion: all destructors are trivial.

Are all destructors trivial?

[Nicol Bolas]

On Monday, August 8, 2016 at 12:38:25 PM UTC-4, Viacheslav Usov wrote:

On Mon, Aug 8, 2016 at 6:16 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Your statement is rather like saying, "if the Grand Canyon weren't in the way, we could just walk there, right?" That's true... but it's not particularly useful.

That is useful, and if you think about it, you will certainly remember that as a programmer you must have done that sort of thing just about every day.

In this particular case, it is useful, because if it is only weak_ptr that is a problem, it is not a fundamental problem, without even going into all those complications we have been talking about recently.

Each weak_ptr can have a field that links it to another weak_ptr. The shared state can have a field that is the head of the list of all the weak_ptr's to the given shared state. The implementation can have a global mutex that synchronises access to all the weak_ptr lists. When a shared state goes down, it simply iterates over all the weak_ptrs that point to it, and makes their pointers to itself null.

There's a reason why I picked the "Grand Canyon" for that analogy. Because even if you could move the Grand Canyon, you would suddenly remember that the Grand Canyon was dug out by the Colorado River. Which feeds water to literally millions of people.

The point being that, even if you could overcome it, you will have caused massive harm to a bunch of other people. All just so that you can take a walk.

The benefits do not outweigh the disadvantages.

Also, I found a new problem for you.

Let's say you have some class `T` which inherits from `shared_ptr_state<T, D, A>`. What exactly happens if you derive a class `U` from `T`, then try to manage that in a shared_ptr?

If you were using regular shared_ptr logic, everything would be fine because its shared state would know about U. The shared state it builds would have `U` as its template parameter and therefore know how to deal with it. Indeed, it is UB to manage a base class via `shared_ptr`.

And that's exactly what you're tying to do. Because you derived from `shared_ptr_state<T, D, A>` explicitly, your class now cannot be derived from. Or at least, you cannot derive from it and then manage it with a `shared_ptr`. Even if you gave `T` a virtual destructor, the pointer that `shared_ptr_state` would store is still a pointer to `T`, not to `U`. So when it comes time to deallocate the memory for `T` with the allocator `A`, it will pass `A` the wrong pointer.

There might be a way to overcome that, but again, to what end?

[Andrey Semashev]

On 08/08/16 19:39, 'Edward Catmur' via ISO C++ Standard - Discussion wrote:
> On Mon, Aug 8, 2016 at 5:07 PM, Andrey Semashev
> <andrey....@gmail.com <mailto:andrey....@gmail.com>> wrote:
>
>
> We aren't accessing a non-static data member, we're accessing a
> non-static data member of a non-static data member.
>
> I don't see the significance. If an object is destroyed then all its
> non-static members are also destroyed, recursively so.
>
> That doesn't necessarily mean their lifetime has ended. Yes, non-trivial
> destructors are run, and those end the lifetime of the objects they are
> run on, but only up until a trivially destructible subobject is reached.
>
> I'm considering the case when all shared_ptrs to the object are
> destroyed and weak_ptrs are left referencing the storage which is
> supposed to hold the shared_ptr_state. These weak_ptrs will have to
> access shared_ptr_state, and in order for that to be well-defined
> the state has to be not destroyed. It cannot be so if the state is a
> member of the object, no matter at what level of object nesting.
>
> Why? The storage holding the shared state may be destroyed, but the
> lifetime of the storage may not have ended. Its lifetime has certainly
> not ended if the complete object is trivially destructible; at question
> is whether its lifetime has ended when the complete object is not
> trivially destructible.

The mentioned earlier [class.cdtor]/1 says that you can't access a
non-static data member - any member - after the non-trivial destructor
completed. It doesn't matter whether that member's destructor is trivial
or not.

I understand what you're saying, and I do think there is slight
inconsistency between this paragraph and lifetime rules for trivially
destructible objects. But those inconsistencies only contribute to my
opinion that accessing a trivially destructible member of a destroyed
non-trivially destructible object is UB. And frankly, IMO it should be UB.

I think lifetime rules could be updated so that trivial
constructors/destructors also act as points of starting ending the
object lifetime. A separate paragraph could be added stating that the
trivial constructors/destructors don't affect the underlying storage. It
would still be UB to access a trivial member of a destroyed non-trivial
object but at least the confusion wrt object lifetime would be resolved.

> If we construct placement-construct allocator and/or deleter in raw
> storage contained within the complete object, we can defer
> destruction
> of the allocator and/or deleter until the weak state is disposed.
>
> I don't think this is a valid assumption. If the enclosing class'
> destructor is not trivial, you cannot assume anything about the
> contents of that raw storage after the destructor completes.
>
> Again, why? User code cannot touch that raw storage; where is the
> license for the compiler to do so? I thought [class.cdtor]/1 was such
> license, but Viacheslav has persuaded me that may not be so clear.

I didn't say user can't touch the storage, he can. He just can't assume
any particular contents in that storage.

> The standard doesn't specify the contents
> of the storage after the non-trivial constructor completes, which
> means the contents are, well, unspecified.
>
> Well, maybe it does specify the contents. If user code does not write to
> the storage during the non-trivial destructor sequence, and if the
> lifetime of the storage continues through the destructor call, and if it
> is legal to use a previously formed pointer or reference to access its
> contents, then its contents must be the same as before the destructor
> call, just as they would be were the complete object destructor trivial.

Does the standard guarantee this anywhere? I didn't find any statements
that would amount to that guarantee - on the countrary, the standard
allows compilers a great deal of latitude in implementing constructors
and destructors involving any kind of magic.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 7:08 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> There's a reason why I picked the "Grand Canyon" for that analogy. Because even if you could move

The hypothesis was "it it weren't in the way" (a). Not "it were moved" (b). If I achieve something with (a), it is invalid to assume that should entail (b).

> Let's say you have some class `T` which inherits from `shared_ptr_state<T, D, A>`. What exactly happens if you derive a class `U` from `T`, then try to manage that in a shared_ptr?

I do not think I can answer this, because we have not really discussed how we could make them work together.

If you make some particular assumption, please give us more details.

Cheers,

V.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 7:17 PM, Andrey Semashev <andrey....@gmail.com> wrote:

> I understand what you're saying, and I do think there is slight inconsistency between this paragraph and lifetime rules for trivially destructible objects. But those inconsistencies only contribute to my opinion that accessing a trivially destructible member of a destroyed non-trivially destructible object is UB. And frankly, IMO it should be UB.

Let's punt the standard and let you explain why you think so. I mean, really, what makes you think non-trivial destructors of enclosing objects should have magic powers on the storage for members, which must stay valid till after the enclosing destructor finishes? No standard speak, just common sense at this point please?

Cheers,

V.

[Nicol Bolas]

The "assumption" is merely what you proposed: a `shared_ptr_state<T, D, A>` object which is a base class of `T`. You allocate an object of that type and manage it with a `shared_ptr`. And it will work just like a regular shared_ptr, except that it has no external shared state.

I'm simply explaining the logical conclusion of that line of thought. In the cleanup of the last `shared_ptr`, `A::deallocate(ptr)` must be executed, and `ptr` must be the pointer returned from the allocation function.

And the only `ptr` that `shared_ptr_state<T, D, A>` knows about is a pointer to `T`. It does not know about `U`. So how do you suggest that it learn about `U`? Can someone get a pointer to `U` down to the `shared_ptr_state`? If so, who and how?

[Nicol Bolas]

Once a non-trivial destructor has completed, the lifetime of all subobjects should be ended, trivial or not. Why? Because that leads to a reasonable, coherent object model.

A type with non-trivial destruction represents something more opaque than a raw block of bits. Even if it is composed of pieces that are raw blocks of bits, the whole is something more than the sum of its parts. Therefore, you should not be able to treat any part of it like a raw block of bits.

[Andrey Semashev]

On 08/08/16 20:36, Viacheslav Usov wrote:
> On Mon, Aug 8, 2016 at 7:17 PM, Andrey Semashev
> <andrey....@gmail.com <mailto:andrey....@gmail.com>> wrote:
>
>> I understand what you're saying, and I do think there is slight
> inconsistency between this paragraph and lifetime rules for trivially
> destructible objects. But those inconsistencies only contribute to my
> opinion that accessing a trivially destructible member of a destroyed
> non-trivially destructible object is UB. And frankly, IMO it should be UB.
>

> Let's punt the standard and let you explain /why/ you think so. I mean,
> really, /what/ makes you think non-trivial destructors of
> enclosing//objects should have magic powers on the storage for members,
> which /must/ stay valid till /after/ the enclosing destructor finishes?

> No standard speak, just common sense at this point please?

My common sense tells me that the object does not exist after it is
destroyed. No matter what kind of object this is. Accessing something
that does not exist should be prohibited by the standard.

You might say but there's the storage bytes and you would be true.
Except that the storage is not the object. The standard rather loosely
maps objects onto the storage, and I'd say this is good as it allows
more freedom to the implementers. You can say that the storage contains
serialized representation of the object, and the exact serialization
rules are not mandated by the standard. Consequently, unless you rely on
a particular implementation, there are only so many things you can do
with raw storage.

[Viacheslav Usov]

On Mon, Aug 8, 2016 at 7:46 PM, Nicol Bolas <jmck...@gmail.com> wrote:

> Once a non-trivial destructor has completed, the lifetime of all subobjects should be ended, trivial or not. Why? Because that leads to a reasonable, coherent object model.

That answers a different question, something like "why is it a good thing to end the lifetime of all subobjects...". I understand that, and what you say is reasonable, but that's just not the question I asked.

Cheers,

V.

[Edward Catmur]

On Mon, Aug 8, 2016 at 5:47 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 16:36:08 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> But it is forbidden to clobber memory of a trivially destructible complete
> object, right?

The trivial destructor can't do it, clearly, otherwise it wouldn't be trivial.

But a non-trivial destructor for an outer object that contains this sub-object
could.

A complete object is not a sub-object. What did you mean here?

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 19:02:05 PDT Viacheslav Usov wrote:
> > Sorry, it's very much relevant because I don't think the earlier
> > discussion's conclusion was valid. The whole object is responsible for its
> > storage and could overwrite any of the sub-objects. The destructor is
> > perfectly allowed to do that memset(this, 0xcd, sizeof(*this)); before
> > returning.
>
> Which is instant UB for all but POD and similar types. Consider:
>
> struct A
> {
> std::unique_ptr<char> ptr;
> ~A() { memset(this, 0xcd, sizeof(*this)); }
> };
>
> What happens to ptr which is yet to be destructed after the memset?

Fair enough, even though that's not what I meant. I meant that the compiler
could insert that after the sub-objects were destroyed, which wouldn't be UB.
That would happen after the unique_ptr destructor in your example. Nothing in
the standard prevents the compiler from doing that, which means it's
permissible.

But even then, we would still be able to memset the trivial sub-objects. Like
so:

struct B
{
some_trivial_object obj;
~B() { memset(&obj, 0xcd, sizeof(obj); }
};

Now rename some_trivial_object as shared_ptr_state.

> > The point is that the experience of something working in a given
> > situation is not guarantee it will always work, in other situations.
>
> I did not invoke my experience only. There were other reasons, which you
> keep ignoring.

I'm addressing the reasons. This point above was exclusively as a reply to

"And, really, if it is use after DELETION, why is it free do so after
DESTRUCTION? In my experience, that is done when memory is deallocated, not
when the object is destructed."

Your experience is biased towards the implementations you've got experience
with. You cannot claim experience as a rule for what is impossible.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 19:01:54 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> > The trivial destructor can't do it, clearly, otherwise it wouldn't be
> > trivial.
> >
> > But a non-trivial destructor for an outer object that contains this
> > sub-object
> > could.
>
> A complete object is not a sub-object. What did you mean here?

Each object may be a sub-object of a larger object.

The complete object wrapping sub-objects is allowed to do anything to the sub-
objects it is composed of.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 20:17:47 PDT Andrey Semashev wrote:
> I think lifetime rules could be updated so that trivial
> constructors/destructors also act as points of starting ending the
> object lifetime.

That doesn't work either.

int i = 0;
i.~int();
// am I allowed the following:
assert(i == 0);
i = 1;

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 19:36:07 PDT Viacheslav Usov wrote:
> Let's punt the standard and let you explain *why* you think so. I mean,
> really, *what* makes you think non-trivial destructors of enclosing objects
> should have magic powers on the storage for members, which *must* stay
> valid till *after* the enclosing destructor finishes? No standard speak,

> just common sense at this point please?

Because it allows for easier detection of mistakes like use-after-free. If the
compiler can fill in the storage area with a predefined pattern, and you in the
debugger see the values in that pattern, it immediately stands out and you
conclude you accessed some object after its destructors were run.

The fact that some sub-objects were non-trivial notwithstanding.

[Edward Catmur]

On Mon, Aug 8, 2016 at 6:03 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 17:39:51 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> On Mon, Aug 8, 2016 at 5:07 PM, Andrey Semashev <andrey....@gmail.com>
> wrote:
> >> We aren't accessing a non-static data member, we're accessing a
> >> non-static data member of a non-static data member.
> >
> > I don't see the significance. If an object is destroyed then all its
> > non-static members are also destroyed, recursively so.
>
> That doesn't necessarily mean their lifetime has ended. Yes, non-trivial
> destructors are run, and those end the lifetime of the objects they are run
> on, but only up until a trivially destructible subobject is reached.

Sorry, that doesn't make sense. It has to mean that. When the lifetime of an
object ends, the lifetime of all of its sub-objects, recursively, must have
ended too.

Why? Where does the Standard say that nested subobject lifetimes are nested isomorphically?

If you meant that the memory continues to be comprised of bytes and those
bytes can be read, sure, they can. It doesn't mean that they continue to
contain what they used to. The contents may have changed because of the non-
trivial destructors that were run.

Not by user-defined destructors, only by the compiler.

> > I don't think this is a valid assumption. If the enclosing class'
> > destructor is not trivial, you cannot assume anything about the contents
> > of
> > that raw storage after the destructor completes.
>
> Again, why? User code cannot touch that raw storage; where is the license
> for the compiler to do so? I thought [class.cdtor]/1 was such license, but
> Viacheslav has persuaded me that may not be so clear.

Why can't the user code touch that raw storage? If the sub-object is trivially
destructible, then it doesn't matter what bytes it contains when the that
destructor would have been run. So the user code *can* memset it and it's a
valid use of the sub-object.

Moreover, even if we say that the lifetime ends only when another object's
lifetime begins there, what's to prevent exactly that from happening? Why
couldn't the non-trivial outer destructor begin the lifetime of another sub-
object in the same storage bytes?

That's trivial to guard against; contain the trivially destructible subobject in a non-trivially destructible subobject controlled by the library. The user cannot touch the storage of the interposed object because it is non-trivially destructible, and the interposed object non-trivial destructor chooses not to touch the storage, so the only agent that conceivably could touch the storage is the compiler.

> > As John has mentioned, the compiler is also not forbidden to clobber
> >> memory after destruction (e.g. fill with trapping values).
> >>
> >> I hope he will be able to substantiate that by quoting the standard.
> >
> > I don't think he has to. The standard doesn't specify the contents of the
> > storage after the non-trivial constructor completes, which means the
> > contents are, well, unspecified.
>
> Well, maybe it does specify the contents. If user code does not write to
> the storage during the non-trivial destructor sequence, and if the lifetime
> of the storage continues through the destructor call, and if it is legal to
> use a previously formed pointer or reference to access its contents, then
> its contents must be the same as before the destructor call, just as they
> would be were the complete object destructor trivial.

I disagree. There's no guarantee that the byte contents of a non-trivial
object will remain the same after said destructor has run.

Proof by absurd: you're saying that all trivial sub-objects of any object must
retain their values as they had before the outer object's destructor was run.
But obviously everything is constructed from primitive types, which are
trivial. The only way to do that is if destructors are not allowed to write to
any of their sub-objects. Conclusion: all destructors are trivial.

Are all destructors trivial?

I'm sorry, I can't follow your reasoning. I'm not saying that destructors aren't allowed to write to subobjects, I'm saying that if they choose not to write to those subobjects, then those subobjects are not otherwise altered. I explicitly mentioned that as my first condition: "If user code does not write to the storage during the non-trivial destructor sequence".

[Andrey Semashev]

On 08/08/16 21:22, Thiago Macieira wrote:
> On segunda-feira, 8 de agosto de 2016 20:17:47 PDT Andrey Semashev wrote:
>> I think lifetime rules could be updated so that trivial
>> constructors/destructors also act as points of starting ending the
>> object lifetime.
>
> That doesn't work either.
>
> int i = 0;
> i.~int();
> // am I allowed the following:
> assert(i == 0);
> i = 1;

I don't think you should be allowed to do that. Mostly for sake of
consistency with other (non-trivial) types. Otherwise lifetime rules
make no real sense for trivial types.

[Edward Catmur]

On Mon, Aug 8, 2016 at 6:17 PM, Andrey Semashev <andrey....@gmail.com> wrote:

On 08/08/16 19:39, 'Edward Catmur' via ISO C++ Standard - Discussion wrote:

On Mon, Aug 8, 2016 at 5:07 PM, Andrey Semashev
<andrey....@gmail.com <mailto:andrey.semashev@gmail.com>> wrote:


        We aren't accessing a non-static data member, we're accessing a
        non-static data member of a non-static data member.

    I don't see the significance. If an object is destroyed then all its
    non-static members are also destroyed, recursively so.

That doesn't necessarily mean their lifetime has ended. Yes, non-trivial
destructors are run, and those end the lifetime of the objects they are
run on, but only up until a trivially destructible subobject is reached.

    I'm considering the case when all shared_ptrs to the object are
    destroyed and weak_ptrs are left referencing the storage which is
    supposed to hold the shared_ptr_state. These weak_ptrs will have to
    access shared_ptr_state, and in order for that to be well-defined
    the state has to be not destroyed. It cannot be so if the state is a
    member of the object, no matter at what level of object nesting.

Why? The storage holding the shared state may be destroyed, but the
lifetime of the storage may not have ended. Its lifetime has certainly
not ended if the complete object is trivially destructible; at question
is whether its lifetime has ended when the complete object is not
trivially destructible.

The mentioned earlier [class.cdtor]/1 says that you can't access a non-static data member - any member - after the non-trivial destructor completed. It doesn't matter whether that member's destructor is trivial or not.

Membership is not transitive. If we indirect a pointer to a non-direct nested member (a subobject of a subobject), have we accessed the intermediate member (the immediate subobject)?

 

I understand what you're saying, and I do think there is slight inconsistency between this paragraph and lifetime rules for trivially destructible objects. But those inconsistencies only contribute to my opinion that accessing a trivially destructible member of a destroyed non-trivially destructible object is UB. And frankly, IMO it should be UB.

I think lifetime rules could be updated so that trivial constructors/destructors also act as points of starting ending the object lifetime. A separate paragraph could be added stating that the trivial constructors/destructors don't affect the underlying storage. It would still be UB to access a trivial member of a destroyed non-trivial object but at least the confusion wrt object lifetime would be resolved.

That would be difficult, unless you mean it to only apply to subobject trivial constructors/destructors. Currently trivial complete objects are treated specially, and I don't see that changing any time soon.

 

        If we construct placement-construct allocator and/or deleter in raw
        storage contained within the complete object, we can defer
        destruction
        of the allocator and/or deleter until the weak state is disposed.

    I don't think this is a valid assumption. If the enclosing class'
    destructor is not trivial, you cannot assume anything about the
    contents of that raw storage after the destructor completes.

Again, why? User code cannot touch that raw storage; where is the
license for the compiler to do so? I thought [class.cdtor]/1 was such
license, but Viacheslav has persuaded me that may not be so clear.

I didn't say user can't touch the storage, he can. He just can't assume any particular contents in that storage.

Why not, if they ensure that the storage has particular contents at the time the destructor completes?

 

    The standard doesn't specify the contents
    of the storage after the non-trivial constructor completes, which
    means the contents are, well, unspecified.

Well, maybe it does specify the contents. If user code does not write to
the storage during the non-trivial destructor sequence, and if the
lifetime of the storage continues through the destructor call, and if it
is legal to use a previously formed pointer or reference to access its
contents, then its contents must be the same as before the destructor
call, just as they would be were the complete object destructor trivial.

Does the standard guarantee this anywhere? I didn't find any statements that would amount to that guarantee - on the countrary, the standard allows compilers a great deal of latitude in implementing constructors and destructors involving any kind of magic.

Yes, but that latitude is expressed by marking user code as having undefined behavior. If no undefined behavior obtains here, then the behavior is as specified by the object model.

[Edward Catmur]

On Mon, Aug 8, 2016 at 7:20 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 19:01:54 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> > The trivial destructor can't do it, clearly, otherwise it wouldn't be
> > trivial.
> >
> > But a non-trivial destructor for an outer object that contains this
> > sub-object
> > could.
>
> A complete object is not a sub-object. What did you mean here?

Each object may be a sub-object of a larger object.

The complete object wrapping sub-objects is allowed to do anything to the sub-
objects it is composed of.

Yes, but if it's a subobject then it's not a complete object. User code in a destructor can only modify recursive subobjects it has permission to access; it can't modify a subobject private subobject. Whether the compiler is allowed to do so is another matter, but it is at least constrained from doing so for trivially destructible complete objects.

[Edward Catmur]

On Mon, Aug 8, 2016 at 7:18 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 19:02:05 PDT Viacheslav Usov wrote:
> > Sorry, it's very much relevant because I don't think the earlier
> > discussion's conclusion was valid. The whole object is responsible for its
> > storage and could overwrite any of the sub-objects. The destructor is
> > perfectly allowed to do that  memset(this, 0xcd, sizeof(*this)); before
> > returning.
>
> Which is instant UB for all but POD and similar types.  Consider:
>
> struct A
> {
>     std::unique_ptr<char> ptr;
>     ~A() { memset(this, 0xcd, sizeof(*this)); }
> };
>
> What happens to ptr which is yet to be destructed after the memset?

Fair enough, even though that's not what I meant. I meant that the compiler
could insert that after the sub-objects were destroyed, which wouldn't be UB.
That would happen after the unique_ptr destructor in your example. Nothing in
the standard prevents the compiler from doing that, which means it's
permissible.

An object retains its value over its lifetime, so if the nested subobject outlives its complete object and can be observed to have done so the compiler is precluded from clobbering it.

 

But even then, we would still be able to memset the trivial sub-objects. Like
so:

struct B
{
        some_trivial_object obj;
        ~B() { memset(&obj, 0xcd, sizeof(obj); }
};

Now rename some_trivial_object as shared_ptr_state.

shared_ptr_state does not need to be itself trivial, it just (possibly) needs to have a trivial (eventual) subobject.

> > The point is that the experience of something working in a given
> > situation is not guarantee it will always work, in other situations.
>
> I did not invoke my experience only. There were other reasons, which you
> keep ignoring.

I'm addressing the reasons. This point above was exclusively as a reply to

"And, really, if it is use after DELETION, why is it free do so after
DESTRUCTION? In my experience, that is done when memory is deallocated, not
when the object is destructed."

Your experience is biased towards the implementations you've got experience
with. You cannot claim experience as a rule for what is impossible.

Experience can help, though. Consider:

#include <memory>

struct X { int i = 42; };

struct Y : X { ~Y() {} };

int main() {

  alignas(Y) char buf[sizeof(Y)];

  Y* p = new (buf) Y;

  int* p1 = &p->i;

  p->~Y();

  return *p1;              // ???

}

gcc returns 0; clang, ICC and MSVC return 42. So at least one compiler agrees with you (the others may, too, but they aren't saying).

[Thiago Macieira]

But that's exactly why the standard talks about the lifetime of trivial
objects ending only when the bytes are reused or when they are freed, not when
the (trivial) destructor runs.

On the other hand, the problem we're discussing is what happens when a non-
trivial destructor runs on a larger object containing a trivial sub-object.
It's the case from another email of mine:

struct A
{
~A();
int i;
};

A::~A() is allowed to change the values stored in A::i.

[Andrey Semashev]

On 08/08/16 21:48, 'Edward Catmur' via ISO C++ Standard - Discussion wrote:
> On Mon, Aug 8, 2016 at 6:17 PM, Andrey Semashev
> <andrey....@gmail.com <mailto:andrey....@gmail.com>> wrote:
>
> The mentioned earlier [class.cdtor]/1 says that you can't access a
> non-static data member - any member - after the non-trivial
> destructor completed. It doesn't matter whether that member's
> destructor is trivial or not.
>
> Membership is not transitive. If we indirect a pointer to a non-direct
> nested member (a subobject of a subobject), have we accessed the
> intermediate member (the immediate subobject)?

How a sub-object can be destroyed while its sub-sub-object is not? Or
are you saying that dereferencing a pointer to the destroyed sub-object
is UB and at the same time dereferencing a pointer to the destroyed
sub-sub-object is fine? I'm sorry, but neither of these make sense to me.

> I didn't say user can't touch the storage, he can. He just can't
> assume any particular contents in that storage.
>
> Why not, if they ensure that the storage has particular contents at the
> time the destructor completes?

As I said earlier, the user is not in position to ensure that unless he
targets a particular implementation and that implementation allows that.

[Andrey Semashev]

On 08/08/16 22:52, Thiago Macieira wrote:
> On segunda-feira, 8 de agosto de 2016 21:28:19 PDT Andrey Semashev wrote:
>> On 08/08/16 21:22, Thiago Macieira wrote:
>>> On segunda-feira, 8 de agosto de 2016 20:17:47 PDT Andrey Semashev wrote:
>>>> I think lifetime rules could be updated so that trivial
>>>> constructors/destructors also act as points of starting ending the
>>>> object lifetime.
>>>
>>> That doesn't work either.
>>>
>>> int i = 0;
>>> i.~int();
>>> // am I allowed the following:
>>> assert(i == 0);
>>> i = 1;
>>
>> I don't think you should be allowed to do that. Mostly for sake of
>> consistency with other (non-trivial) types. Otherwise lifetime rules
>> make no real sense for trivial types.
>
> But that's exactly why the standard talks about the lifetime of trivial
> objects ending only when the bytes are reused or when they are freed, not when
> the (trivial) destructor runs.

I know, that's why I suggested the change.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 19:24:18 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> > Sorry, that doesn't make sense. It has to mean that. When the lifetime of
> > an
> > object ends, the lifetime of all of its sub-objects, recursively, must
> > have
> > ended too.
>
> Why? Where does the Standard say that nested subobject lifetimes are nested
> isomorphically?

It doesn't say that they aren't, which is why we're having this discussion in
the first place. So we have to look at this from the point of view of what was
intended and what makes most sense.

I submit that recusive ending of lifetime is the option that makes most sense.
The other option requires too many exceptions to be implementable. It would
only apply to:

a) non-public members
b) of trivially destructible classes
c) which are members of at most one non-trivial, outer object

The lifetime of all other members in such non-trivial classes would be
considered ended.

> > If you meant that the memory continues to be comprised of bytes and those
> > bytes can be read, sure, they can. It doesn't mean that they continue to
> > contain what they used to. The contents may have changed because of the

> > non-trivial destructors that were run.

>
> Not by user-defined destructors, only by the compiler.

Why not? You're arguing that you could have taken a pointer to that memory
location and you could access it after the point where the trivial destructor
would have run. Well, if you can do that, then I can also do it from my own
destructor.

struct B;
struct A
{
B b;
std::shared_ptr_state state;
A() : state{}, b(&state) {}
};
struct B
{
void *ptr;
B(void *storage) : ptr(storage) {}
~B();
};

Since A::b's destructor is run after A::state, then I can use the storage area
in B's destructor. Whether A::state contained any non-public members is
irrelevant.

Plus what I had said:

> > Why can't the user code touch that raw storage? If the sub-object is
> > trivially
> > destructible, then it doesn't matter what bytes it contains when the that
> > destructor would have been run. So the user code *can* memset it and it's
> > a
> > valid use of the sub-object.
> >
> > Moreover, even if we say that the lifetime ends only when another object's
> > lifetime begins there, what's to prevent exactly that from happening? Why
> > couldn't the non-trivial outer destructor begin the lifetime of another
> > sub-
> > object in the same storage bytes?
>
> That's trivial to guard against; contain the trivially destructible
> subobject in a non-trivially destructible subobject controlled by the
> library. The user cannot touch the storage of the interposed object because
> it is non-trivially destructible, and the interposed object non-trivial
> destructor chooses not to touch the storage, so the only agent that
> conceivably could touch the storage is the compiler.

That doesn't work. If you put the trivial state inside a non-trivial wrapper,
then the wrapper's non-trivial destructor is run and its lifetime is clearly
ended. That means the storage bytes are free to be reused again.

> > Proof by absurd: you're saying that all trivial sub-objects of any object
> > must
> > retain their values as they had before the outer object's destructor was
> > run.
> > But obviously everything is constructed from primitive types, which are
> > trivial. The only way to do that is if destructors are not allowed to
> > write to
> > any of their sub-objects. Conclusion: all destructors are trivial.
> >
> > Are all destructors trivial?
>
> I'm sorry, I can't follow your reasoning. I'm not saying that destructors
> aren't allowed to write to subobjects, I'm saying that if they choose not
> to write to those subobjects, then those subobjects are not otherwise
> altered. I explicitly mentioned that as my first condition: "If user code
> does not write to the storage during the non-trivial destructor sequence".

That's not what you're saying. Your previous claims are different than what
you've just written. You want to guarantee that the contents of a trivial sub-
object of any object remain constant after the destructor of that object has
finished. To do that, you need to guarantee all cases, not just if "they [the
destructors] choose not to write to those subobjects".

If you can't guarantee in all cases, then you must assume that they are
different. If they can be different, then the compiler is allowed to assume no
one is going to use that storage.

[Edward Catmur]

On Mon, Aug 8, 2016 at 8:59 PM, Andrey Semashev <andrey....@gmail.com> wrote:

On 08/08/16 21:48, 'Edward Catmur' via ISO C++ Standard - Discussion wrote:

On Mon, Aug 8, 2016 at 6:17 PM, Andrey Semashev
<andrey....@gmail.com <mailto:andrey.semashev@gmail.com>> wrote:

    The mentioned earlier [class.cdtor]/1 says that you can't access a
    non-static data member - any member - after the non-trivial
    destructor completed. It doesn't matter whether that member's
    destructor is trivial or not.

Membership is not transitive. If we indirect a pointer to a non-direct
nested member (a subobject of a subobject), have we accessed the
intermediate member (the immediate subobject)?

How a sub-object can be destroyed while its sub-sub-object is not? Or are you saying that dereferencing a pointer to the destroyed sub-object is UB and at the same time dereferencing a pointer to the destroyed sub-sub-object is fine? I'm sorry, but neither of these make sense to me.

The fact that an object has been destroyed is not the definitive answer to whether it can be accessed. The Standard says when user-supplied destructors run and when the lifetime of complete objects and non-trivially destructible subobjects ends. It says that subobjects of non-trivially destructible objects cannot be accessed after the containing destructor has completed, but it does not say explicitly that sub-subobjects cannot be accessed. It does not say that the lifetime of trivially destructible subobjects is contained within the lifetime of their parent objects.

There are two possibilities here: either the language is defective, and sub-subobjects cannot be accessed, or it is written as intended, and sub-subobjects can be accessed as long as the immediate subobject is not accessed, up until the storage of the complete object is released or reused. My intuition leans towards the former possibility, but only weakly.

Your intuition in this regard is questionable, since you want the lifetime of all objects (including trivially destructible complete objects) to terminate when they are destroyed.

    I didn't say user can't touch the storage, he can. He just can't
    assume any particular contents in that storage.

Why not, if they ensure that the storage has particular contents at the
time the destructor completes?

As I said earlier, the user is not in position to ensure that unless he targets a particular implementation and that implementation allows that.

They can ensure the contents at the time the closing brace of the destructor is reached. What happens after that depends on the implementation, which is constrained to behave in accordance with the Standard.

[Edward Catmur]

On Mon, Aug 8, 2016 at 9:32 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 19:24:18 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> Where does the Standard say that nested subobject lifetimes are nested
> isomorphically?

It doesn't say that they aren't, which is why we're having this discussion in
the first place. So we have to look at this from the point of view of what was
intended and what makes most sense.

I submit that recusive ending of lifetime is the option that makes most sense.
The other option requires too many exceptions to be implementable. It would
only apply to:

 a) non-public members
 b) of trivially destructible classes
 c) which are members of at most one non-trivial, outer object

The lifetime of all other members in such non-trivial classes would be
considered ended.

I agree that recursive lifetimes (or, equivalently in effect, barring access to recursive subobjects of a non-trivially destructed object) seems to make more sense, but I've been surprised before.

Of your exceptions, (a) is unnecessary; a containing object destructor can write to the trivially destructible subobject subobject and it would retain the value written by the destructor; (b) absolutely; and I'm not sure I understand (c) - are you talking about virtual inheritance and if so how does that affect things?

You're arguing that you could have taken a pointer to that memory
location and you could access it after the point where the trivial destructor
would have run. Well, if you can do that, then I can also do it from my own
destructor.

struct B;
struct A
{
        B b;
        std::shared_ptr_state state;
        A() : state{}, b(&state) {}
};
struct B
{
        void *ptr;
        B(void *storage) : ptr(storage) {}
        ~B();
};

Since A::b's destructor is run after A::state, then I can use the storage area
in B's destructor. Whether A::state contained any non-public members is
irrelevant.

Ah, I see, nice example (though you'd have to use multiple inheritance, or more straightforwardly pass the pointer to storage to outside the object). Yes, that does rather break the scheme - at least, such code would have to be explicitly barred.

> > Why can't the user code touch that raw storage? If the sub-object is
> > trivially
> > destructible, then it doesn't matter what bytes it contains when the that
> > destructor would have been run. So the user code *can* memset it and it's
> > a
> > valid use of the sub-object.
> >
> > Moreover, even if we say that the lifetime ends only when another object's
> > lifetime begins there, what's to prevent exactly that from happening? Why
> > couldn't the non-trivial outer destructor begin the lifetime of another
> > sub-
> > object in the same storage bytes?
>
> That's trivial to guard against; contain the trivially destructible
> subobject in a non-trivially destructible subobject controlled by the
> library. The user cannot touch the storage of the interposed object because
> it is non-trivially destructible, and the interposed object non-trivial
> destructor chooses not to touch the storage, so the only agent that
> conceivably could touch the storage is the compiler.

That doesn't work. If you put the trivial state inside a non-trivial wrapper,
then the wrapper's non-trivial destructor is run and its lifetime is clearly
ended. That means the storage bytes are free to be reused again.

If you mean by using the destructed wrapper's storage directly, from user code running after the wrapper's destructor completes, then yes, you're right. I hadn't considered that that might occur.

> > Proof by absurd: you're saying that all trivial sub-objects of any object
> > must
> > retain their values as they had before the outer object's destructor was
> > run.
> > But obviously everything is constructed from primitive types, which are
> > trivial. The only way to do that is if destructors are not allowed to
> > write to
> > any of their sub-objects. Conclusion: all destructors are trivial.
> >
> > Are all destructors trivial?
>
> I'm sorry, I can't follow your reasoning. I'm not saying that destructors
> aren't allowed to write to subobjects, I'm saying that if they choose not
> to write to those subobjects, then those subobjects are not otherwise
> altered. I explicitly mentioned that as my first condition: "If user code
> does not write to the storage during the non-trivial destructor sequence".

That's not what you're saying. Your previous claims are different than what
you've just written. You want to guarantee that the contents of a trivial sub-
object of any object remain constant after the destructor of that object has
finished. To do that, you need to guarantee all cases, not just if "they [the
destructors] choose not to write to those subobjects".

If you can't guarantee in all cases, then you must assume that they are
different. If they can be different, then the compiler is allowed to assume no
one is going to use that storage.

Libraries can place constraints on user code; in this case the constraint would be on writing to the shared_ptr_state storage after the shared_ptr_state subobject is destroyed. The compiler can't make assumptions based on what user code might have done, since it doesn't know what the documented constraints on user code are.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 21:34:35 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> The fact that an object has been destroyed is not the definitive answer to
> whether it can be accessed. The Standard says when user-supplied
> destructors run and when the lifetime of complete objects and non-trivially
> destructible subobjects ends. It says that subobjects of non-trivially
> destructible objects cannot be accessed after the containing destructor has
> completed, but it does not say explicitly that sub-subobjects cannot be
> accessed. It does not say that the lifetime of trivially destructible
> subobjects is contained within the lifetime of their parent objects.
>
> There are two possibilities here: either the language is defective, and
> sub-subobjects cannot be accessed, or it is written as intended, and
> sub-subobjects can be accessed as long as the immediate subobject is not
> accessed, up until the storage of the complete object is released or
> reused. My intuition leans towards the former possibility, but only weakly.

I think we'll have to define what it means for the lifetime to end.

Each of the primitive types that compose the complete object that used to
exist continue to exist. They are still there. Such as in:

struct A
{
int i;
~A();
}:

a.~A();
a.i = 0; // begins new lifetime for A::i

However, their contents are all unspecified. Code cannot assume that values are
retained past destruction. Is that the same as the lifetime ending?

More importantly: what can the compiler assume about contents?

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 20:29:01 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> > Fair enough, even though that's not what I meant. I meant that the
> > compiler
> > could insert that after the sub-objects were destroyed, which wouldn't be
> > UB.
> > That would happen after the unique_ptr destructor in your example. Nothing
> > in
> > the standard prevents the compiler from doing that, which means it's
> > permissible.
>
> An object retains its value over its lifetime, so if the nested subobject
> outlives its complete object and can be observed to have done so the
> compiler is precluded from clobbering it.

A sub-object cannot outlive its containing object because, even if it could,
the contents are unspecified. I submit that "contents become unspecified" means
that the previous object's lifetime ended.

> > But even then, we would still be able to memset the trivial sub-objects.
> > Like
> > so:
> >
> > struct B
> > {
> > some_trivial_object obj;
> > ~B() { memset(&obj, 0xcd, sizeof(obj); }
> > };
> >
> > Now rename some_trivial_object as shared_ptr_state.
>
> shared_ptr_state does not need to be itself trivial, it just (possibly)
> needs to have a trivial (eventual) subobject.

You're just adding another level of indirection. If I took struct B above and
placed it in:

struct C { B b; };

Then both C and B have non-trivial destructors and the innermost trivial sub-
object cannot be guaranteed to retain its byte values. There's the case I had
above, plus one more: now we *know* that C::b's lifetime ended since the non-
trivial destructor has finished, so the storage bytes can be reused. The
compiler is free to use them as a scratch area for something.

> > Your experience is biased towards the implementations you've got
> > experience
> > with. You cannot claim experience as a rule for what is impossible.
>
> Experience can help, though. Consider:
>
> #include <memory>
>
> struct X { int i = 42; };
> struct Y : X { ~Y() {} };
>
> int main() {
> alignas(Y) char buf[sizeof(Y)];
> Y* p = new (buf) Y;
> int* p1 = &p->i;
> p->~Y();
> return *p1; // ???
> }
>
>
> gcc returns 0; clang, ICC and MSVC return 42. So at least one compiler
> agrees with you (the others may, too, but they aren't saying).

Again, my point is that you can't prove that "no one does X" by saying "no one
I checked has done X". It only proves attributes observable in your sample.

In this case, the sample proved that at least one implementation has done it.

[Edward Catmur]

On Mon, Aug 8, 2016 at 10:18 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 21:34:35 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> The fact that an object has been destroyed is not the definitive answer to
> whether it can be accessed. The Standard says when user-supplied
> destructors run and when the lifetime of complete objects and non-trivially
> destructible subobjects ends. It says that subobjects of non-trivially
> destructible objects cannot be accessed after the containing destructor has
> completed, but it does not say explicitly that sub-subobjects cannot be
> accessed. It does not say that the lifetime of trivially destructible
> subobjects is contained within the lifetime of their parent objects.
>
> There are two possibilities here: either the language is defective, and
> sub-subobjects cannot be accessed, or it is written as intended, and
> sub-subobjects can be accessed as long as the immediate subobject is not
> accessed, up until the storage of the complete object is released or
> reused. My intuition leans towards the former possibility, but only weakly.

I think we'll have to define what it means for the lifetime to end.

Each of the primitive types that compose the complete object that used to
exist continue to exist. They are still there. Such as in:

        struct A
        {
                int i;
                ~A();
        }:

        a.~A();
        a.i = 0;        // begins new lifetime for A::i

No, you can't do that; [class.cdtor]/1 says that referring to a.i is UB. (The examples given are the formation of pointers, but the implication is that forming an lvalue (which occurs first) is where the UB lies.)

 

However, their contents are all unspecified. Code cannot assume that values are
retained past destruction. Is that the same as the lifetime ending?

You can't inspect the value of an object you can't refer to, so the question is moot.

More importantly: what can the compiler assume about contents?

The compiler doesn't have to assume anything about contents. It has to maintain the illusion of the object model, and that's about it.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 22:11:35 PDT 'Edward Catmur' via ISO C++

True.

So I guess this boils down to a simple question: is the compiler allowed to
modify the bytes backing the storage of class types after all the destructors
of the sub-objects have run?

Note I didn't say "trivial".

Note also this should also apply to using the storage as scratch area for
other work, like running destructors of sibling sub-objects.

Finally, note that there's a dual question regarding constructors: is the
compiler allowed to assume that the contents of the bytes backing the storage
of a class are unspecified before the constructor is run? Or, more specifically,
can it fill in the storage area with a pattern value?

void f(int);
struct A
{
int i;
A() { f(i); } // is the compiler allowed to assume i is unspecified?
}:

void *ptr = ::operator new(sizeof(A));
memset(ptr, 0, sizeof(A));
new (ptr) A; // it will be called from here

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 22:28:30 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> > exist continue to exist. They are still there. Such as in:
> > struct A
> > {
> >
> > int i;
> > ~A();
> >
> > }:
> >
> > a.~A();
> > a.i = 0; // begins new lifetime for A::i
>
> No, you can't do that; [class.cdtor]/1 says that referring to a.i is UB.
> (The examples given are the formation of pointers, but the implication is
> that forming an lvalue (which occurs first) is where the UB lies.)

Fair enough.

You would say that, provided that ptr in the example below points to valid
storage area, is suitably aligned and of sufficient size, then:

new (SomeClass) ptr;

begins the lifetime of a new object, correct? Then I should be able to do:

int *ptr = &a.i;
a.~A();
new (ptr) int();

which must begin the lifetime of a new object.

> > However, their contents are all unspecified. Code cannot assume that
> > values are
> > retained past destruction. Is that the same as the lifetime ending?
>
> You can't inspect the value of an object you can't refer to, so the
> question is moot.

Indeed. But please understand the question below referring to a pointer that
was taken when referring was still valid.

> > More importantly: what can the compiler assume about contents?
>
> The compiler doesn't have to assume anything about contents. It has to
> maintain the illusion of the object model, and that's about it.

[Edward Catmur]

On Mon, Aug 8, 2016 at 10:20 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 20:29:01 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> > Fair enough, even though that's not what I meant. I meant that the
> > compiler
> > could insert that after the sub-objects were destroyed, which wouldn't be
> > UB.
> > That would happen after the unique_ptr destructor in your example. Nothing
> > in
> > the standard prevents the compiler from doing that, which means it's
> > permissible.
>
> An object retains its value over its lifetime, so if the nested subobject
> outlives its complete object and can be observed to have done so the
> compiler is precluded from clobbering it.

A sub-object cannot outlive its containing object because, even if it could,
the contents are unspecified. I submit that "contents become unspecified" means
that the previous object's lifetime ended.

I'm not familiar with that language. Is it a paraphrase of a clause in the Standard?

 

> > But even then, we would still be able to memset the trivial sub-objects.
> > Like
> > so:
> >
> > struct B
> > {
> >         some_trivial_object obj;
> >         ~B() { memset(&obj, 0xcd, sizeof(obj); }
> > };
> >
> > Now rename some_trivial_object as shared_ptr_state.
>
> shared_ptr_state does not need to be itself trivial, it just (possibly)
> needs to have a trivial (eventual) subobject.

You're just adding another level of indirection. If I took struct B above and
placed it in:

struct C { B b; };

Then both C and B have non-trivial destructors and the innermost trivial sub-
object cannot be guaranteed to retain its byte values. There's the case I had
above, plus one more: now we *know* that C::b's lifetime ended since the non-
trivial destructor has finished, so the storage bytes can be reused. The
compiler is free to use them as a scratch area for something.

The storage can be reused by the user after shared_ptr_state destructor completes, but that doesn't mean that the compiler is allowed to do the same thing. If I write int i = 42; int j = i; then I'm allowed to insert i = 99; in between, but the compiler isn't allowed to do that.

> > Your experience is biased towards the implementations you've got
> > experience
> > with. You cannot claim experience as a rule for what is impossible.
>
> Experience can help, though. Consider:
>
> #include <memory>
>
> struct X { int i = 42; };
> struct Y : X { ~Y() {} };
>
> int main() {
>   alignas(Y) char buf[sizeof(Y)];
>   Y* p = new (buf) Y;
>   int* p1 = &p->i;
>   p->~Y();
>   return *p1;              // ???
> }
>
>
> gcc returns 0; clang, ICC and MSVC return 42. So at least one compiler
> agrees with you (the others may, too, but they aren't saying).

Again, my point is that you can't prove that "no one does X" by saying "no one
I checked has done X". It only proves attributes observable in your sample.

In this case, the sample proved that at least one implementation has done it.

Yes. Interestingly, my impression (from playing around with inputs, not from inspecting compiler internals) is that gcc is eliding the constructor of Y and all access to the placement-constructed object. gcc's optimizer is known to be occasionally over-aggressive, so that doesn't necessarily mean that this is actually intended.

[Thiago Macieira]

On segunda-feira, 8 de agosto de 2016 22:41:06 PDT 'Edward Catmur' via ISO C++

Standard - Discussion wrote:
> > A sub-object cannot outlive its containing object because, even if it
> > could,
> > the contents are unspecified. I submit that "contents become unspecified"
> > means
> > that the previous object's lifetime ended.
>
> I'm not familiar with that language. Is it a paraphrase of a clause in the
> Standard?

No, I'm proposing this now. It would be stronger if the lifetime of another
object had begun there.

> The storage can be reused by the user after shared_ptr_state destructor
> completes, but that doesn't mean that the compiler is allowed to do the
> same thing. If I write int i = 42; int j = i; then I'm allowed to insert i
> = 99; in between, but the compiler isn't allowed to do that.

It is if we say it is. That was my point: if we rule that the lifetime ended,
then the compiler is allowed to reuse the storage for its own purposes.

> > Again, my point is that you can't prove that "no one does X" by saying "no
> > one
> > I checked has done X". It only proves attributes observable in your
> > sample.
> >
> > In this case, the sample proved that at least one implementation has done
> > it.
>
> Yes. Interestingly, my impression (from playing around with inputs, not
> from inspecting compiler internals) is that gcc is eliding the constructor
> of Y and all access to the placement-constructed object. gcc's optimizer is
> known to be occasionally over-aggressive, so that doesn't necessarily mean
> that this is actually intended.

I think it boils down to the same. If the side-effect on the storage area is
allowed to persist after the end of the destructor, then the compiler couldn't
have optimised the constructor out of existence either. It only did so because
it could prove that the member was not accessed while the object was alive.

[Edward Catmur]

On Mon, Aug 8, 2016 at 10:30 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 22:11:35 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> Libraries can place constraints on user code; in this case the constraint
> would be on writing to the shared_ptr_state storage after
> the shared_ptr_state subobject is destroyed. The compiler can't make
> assumptions based on what user code might have done, since it doesn't know
> what the documented constraints on user code are.

True.

So I guess this boils down to a simple question: is the compiler allowed to
modify the bytes backing the storage of class types after all the destructors
of the sub-objects have run?

Note I didn't say "trivial".

Note also this should also apply to using the storage as scratch area for
other work, like running destructors of sibling sub-objects.

Yes, absolutely. Well, if you confine your UB to scribbling on memory, anyway.

Inspired by your examples I've though up a way to conformantly check whether the compiler is scribbling (or thinks it might want to):

struct A { int i = 42; ~A() {} };

int main() {

  alignas(A) char buf[sizeof(A)];

  auto* p = new (buf) A;

  void* p1 = &p->i;

  p->~A();

  int x;

  std::memcpy(&x, p1, sizeof(x));

  return x;

}

If it is legal to an implementation to return anything other than 42, why is that? What is the user doing here that is UB, unspecified or implementation-defined?

Finally, note that there's a dual question regarding constructors: is the
compiler allowed to assume that the contents of the bytes backing the storage
of a class are unspecified before the constructor is run? Or, more specifically,
can it fill in the storage area with a pattern value?

        void f(int);
        struct A
        {
                int i;
                A() { f(i); }           // is the compiler allowed to assume i is unspecified?
        }:

        void *ptr = ::operator new(sizeof(A));
        memset(ptr, 0, sizeof(A));
        new (ptr) A;            // it will be called from here

That's easy; the A::i is 0. That's because its lifetime begins as soon as storage is obtained; it has an indeterminate value at that time, but the memset gives it a value of 0 and the default-initialization performed (or rather, not performed) by A::A() does nothing to change that. It's the same as zero-initialization.

[Edward Catmur]

On Mon, Aug 8, 2016 at 10:39 PM, Thiago Macieira <thi...@macieira.org> wrote:

On segunda-feira, 8 de agosto de 2016 22:28:30 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> > exist continue to exist. They are still there. Such as in:
> >         struct A
> >         {
> >
> >                 int i;
> >                 ~A();
> >
> >         }:
> >
> >         a.~A();
> >         a.i = 0;        // begins new lifetime for A::i
>
> No, you can't do that; [class.cdtor]/1 says that referring to a.i is UB.
> (The examples given are the formation of pointers, but the implication is
> that forming an lvalue (which occurs first) is where the UB lies.)

Fair enough.

You would say that, provided that ptr in the example below points to valid
storage area, is suitably aligned and of sufficient size, then:

        new (SomeClass) ptr;

begins the lifetime of a new object, correct? Then I should be able to do:

        int *ptr = &a.i;
        a.~A();
        new (ptr) int();

which must begin the lifetime of a new object.

I would certainly hope so; it'd be horrendous if that were illegal.

However, if it's legal then surely so must be:

int& i = *(new (ptr) int);

which does not begin the lifetime of a new object (default-initialization does not create objects), but rather forms an lvalue to an object that has been there all along.

[inkwizyt...@gmail.com]

On Tuesday, August 9, 2016 at 12:05:45 AM UTC+2, Thiago Macieira wrote:

On segunda-feira, 8 de agosto de 2016 22:41:06 PDT 'Edward Catmur' via ISO C++
Standard - Discussion wrote:
> > A sub-object cannot outlive its containing object because, even if it
> > could,
> > the contents are unspecified. I submit that "contents become unspecified"
> > means
> > that the previous object's lifetime ended.
>
> I'm not familiar with that language. Is it a paraphrase of a clause in the
> Standard?

No, I'm proposing this now. It would be stronger if the lifetime of another
object had begun there.

 

What If we invert this problem and made object be sub-object of control block? Most of this functionality is done by `make_share` but we could create it without any shared pointer pointing to it.
I will have limitation but as long we do not need inheriting that type (with it you will always have some overhead in `shared_from_this`) it could be effective as intrusive pointer.