Observable unique_ptr: Have anyone implemneted this?

[mihailn...@gmail.com]

Hello,

In many cases there is the need to track an object and use it only if it is still alive. This can't be achieved with raw pointers.

The only way, right now, is by using a shared_ptr + weak_ptr.

But this has the downside of making your object sharable, where one does not intent to (granted with use_count of 1 most of the time).

I think there is a middle ground - b/w

A) Object has explicit single-owner lifetime (unique_ptr), BUT pointers are potentially dangling and object can be destroyed at any time.

B) Pointers which never dangle and object is kept alive while using it, BUT the life time is no longer explicit - anyone can keep the object alive OR in, the case of shared_ptr<unique_ptr>, must have special shared_ptr instance which is "delegated" (commented) to be the owner.

The middle ground is:

A') Object has explicit single-owned lifetime with pointers which never dangle, BUT object might be destroyed at any time.

This middle ground is implementable in current C++, but is not very pretty, manly because there are no extension pointes to unique_ptr - one need to use it as member and reimplement its interface.

My question is - has anyone implemented such a pointer ("production level") OR you guys just use the shared_ptr in these scenarios?

[mihailn...@gmail.com]

Oh, boy, two typos in the subject alone, that's just great.

[Domen Vrankar]

Currently I'm just using shared_ptr + weak_ptr.

I've never implemented something like that but have given it some thought a while ago - specially regarding the same functionality but for stack allocated objects since they can be moved into a different scope with move assignment (never got far enough to even consider the feasibility carefully enough). The thing is that such object would have to block at the top destructor until all owning pointers retrieved by weak_ptr instances to it are freed and that could cause deal locks (in single threaded or multi threaded use case) if somebody were to store such owning pointer somewhere - but this functionality would be welcome at least in debug mode non the less (or I should simply start using static analyzers... - that's the haven't given enough feasibility thought part) and disabled checking in release mode.

Anyway you could simply create your own class that would contain a std::unique_ptr member and have all the blocking and giving weak_ptr objects there. Later if it would prove feasible this could be added to std::unique_ptr or some other smart ptr that contains WARNING-CAN-DEAD-LOCK-IN-DESTRUCTOR note (and for stack allocated wrapper you can still move or construct stack object inside a wrapper class and use it with operator-> - for a nicer wrapper we'll have to wait for dot operator overloading).

Regards,

Domen

[Domen Vrankar]

Btw if you change WARNING-CAN-DEAD-LOCK-IN-DESTRUCTOR for not knowing when exactly the destructor will get called you essentially get a std::shared_ptr so at that point it's no use even bothering and you use a std::shared_ptr instead...

Regards,

Domen

[mihailn...@gmail.com]

The idea is the handlers to not keep the object alive. There should be no "owning pointers", just plain pointer which get nullified correctly when the resource is destroyed.

If the handle participates to the lifetime in any, your are indeed well better off just using shared_ptr. The idea is to have a non-dangling raw pointer and nothing else.

[Domen Vrankar]

2016-10-17 13:00 GMT+02:00 <mihailn...@gmail.com>:

The idea is the handlers to not keep the object alive. There should be no "owning pointers", just plain pointer which get nullified correctly when the resource is destroyed.

If the handle participates to the lifetime in any, your are indeed well better off just using shared_ptr. The idea is to have a non-dangling raw pointer and nothing else.

Well for multi threaded application that wouldn't work as the owning thread can destroy the object while a different thread is using it (that's why you have to get std::shared_ptr from std::weak_ptr). Also in that case you would have to prevent calling reset and release std::unique_ptr member functions (possibly some other functions) even for single threaded programs as you could call a function from your non owning pointer that would reset std::unique_ptr and invalidate the calling pointer while being used.

Regards,

Domen

[Thiago Macieira]

Em segunda-feira, 17 de outubro de 2016, às 01:14:52 PDT,
mihailn...@gmail.com escreveu:

> A') Object has explicit single-owned lifetime with pointers which never
> dangle, BUT object might be destroyed at any time.
>
> This middle ground is implementable in current C++, but is not very pretty,
> manly because there are no extension pointes to unique_ptr - one need to
> use it as member and reimplement its interface.
>
> My question is - has anyone implemented such a pointer ("production level")
> OR you guys just use the shared_ptr in these scenarios?

I think we've had this discussion before.

What you're asking for requires a separate control block to be kept and
indicates whether the pointer is still valid or not. That block needs to
outlive the pointer, so it needs to be refcounted.

In other words, we've described the internals of shared_ptr.

--
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
Software Architect - Intel Open Source Technology Center

[Marc Mutz]

On Monday 17 October 2016 10:14:52 mihailn...@gmail.com wrote:
> Hello,
>
> In many cases there is the need to track an object and use it only if it is
> still alive. This can't be achieved with raw pointers.
>
> The only way, right now, is by using a shared_ptr + weak_ptr.
> But this has the downside of making your object sharable, where one does
> not intent to (granted with use_count of 1 most of the time).

The fundamental problem with what you're intending to do is this: unique_ptr
is a *unique* ptr. There are no other owners. Suppose there was some
weak_unique_ptr for unique_ptr the way we have weak_ptr for shared_ptr.

What does weak_unique_ptr::lock() return? It cannot return unique_ptr, since
then you might end up with two unique_ptrs owning the same object. You could
say that the unique_ptr returned from lock() has an empty deleter, or that
weak_unique_ptr can be directly dereferenced. but then you lose atomicity
between checking weak_unique_ptr for having been nulled out and deref'ing it
to access the object. Now, that may or may not be a problem, but it's a
problem that would have to be addressed for a "production-ready"
implementation.

That said, you could do the following:

Construct the object with make_shared, create a custom deleter for unqiue_ptr
and move the shared_ptr holding the object into the deleter of a unique_ptr
pointing to the object. You can type-erase the deleter by holding a
shared_ptr<void>.

struct ObservableUniquePtrDeleter {
std::shared_ptr<void> ptr;
void operator(void *) { ptr.reset(); }
};

template <typename T>
using ObservableUniquePtr = std::unique_ptr<T, ObservableUniquePtrDeleter>;
template <typename T, typename ...Args>
auto make_observable_unique(Args &&...args) {
auto p = std::make_shared<T>(std::forward<Args>(args)...);
auto t = p.get();
return ObservableUnqiuePtr<T>{t, std::move(p)};
}

template <typename T>
auto get_weak_ref(const ObservableUniquePtr<T> &ptr) {
std::weak_ptr<T> weak;
if (ptr)
weak = static_pointer_cast<T>(ptr.get_deleter().ptr);
return weak;
}

(completely untested)

>
> I think there is a middle ground - b/w
>
> A) Object has explicit single-owner lifetime (unique_ptr), BUT pointers
> are potentially dangling and object can be destroyed at any time.
> B) Pointers which never dangle and object is kept alive while using it, BUT
> the life time is no longer explicit - anyone can keep the object alive OR
> in, the case of shared_ptr<unique_ptr>, must have special shared_ptr
> instance which is "delegated" (commented) to be the owner.
>
>
> The middle ground is:
> A') Object has explicit single-owned lifetime with pointers which never
> dangle, BUT object might be destroyed at any time.
>
>
> This middle ground is implementable in current C++, but is not very pretty,
> manly because there are no extension pointes to unique_ptr - one need to
> use it as member and reimplement its interface.
>
>
> My question is - has anyone implemented such a pointer ("production level")
> OR you guys just use the shared_ptr in these scenarios?

--
Marc Mutz <marc...@kdab.com> | Senior Software Engineer
KDAB (Deutschland) GmbH & Co.KG, a KDAB Group Company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts

[mihailn...@gmail.com]

On Monday, October 17, 2016 at 3:57:46 PM UTC+3, Thiago Macieira wrote:

Em segunda-feira, 17 de outubro de 2016, às 01:14:52 PDT,
mihailn...@gmail.com escreveu:

...

I think we've had this discussion before.

What you're asking for requires a separate control block to be kept and
indicates whether the pointer is still valid or not. That block needs to
outlive the pointer, so it needs to be refcounted.

In other words, we've described the internals of shared_ptr.

...

The difference is, shared_ptr is deigned to give shared ownership to the resource itself.

The middle ground I am talking about is, the ownership of the resource does not change - it is still unique.

The shared lifetime is only of the control block, not the resource.

The resource does not participate on the shared lifetime of the control block and vice versa - they have different lifetimes.

[Thiago Macieira]

Em segunda-feira, 17 de outubro de 2016, às 06:09:36 PDT,
mihailn...@gmail.com escreveu:

> On Monday, October 17, 2016 at 3:57:46 PM UTC+3, Thiago Macieira wrote:
> > What you're asking for requires a separate control block to be kept and
> > indicates whether the pointer is still valid or not. That block needs to
> > outlive the pointer, so it needs to be refcounted.
> >
> > In other words, we've described the internals of shared_ptr.
> > ...
>
> The difference is, shared_ptr is deigned to give shared ownership to the
> resource itself.
> The middle ground I am talking about is, the ownership of the resource does
> not change - it is still unique.
> The shared lifetime is only of the control block, not the resource.

> *The resource does not participate on the shared lifetime of the control
> block and vice versa - they have different lifetimes.*

I understand the use is different.

My point is that it's the same control block, so you may as well use the same
class.

[mihailn...@gmail.com]

On Monday, October 17, 2016 at 4:06:43 PM UTC+3, Marc Mutz wrote:

On Monday 17 October 2016 10:14:52 mihailn...@gmail.com wrote:
> Hello,
>
> In many cases there is the need to track an object and use it only if it is
> still alive. This can't be achieved with raw pointers.
>
> The only way, right now, is by using a shared_ptr + weak_ptr.
> But this has the downside of making your object sharable, where one does
> not intent to (granted with use_count of 1 most of the time).

The fundamental problem with what you're intending to do is this: unique_ptr
is a *unique* ptr. There are no other owners. Suppose there was some
weak_unique_ptr for unique_ptr the way we have weak_ptr for shared_ptr.

What does weak_unique_ptr::lock() return? It cannot return unique_ptr, since
then you might end up with two unique_ptrs owning the same object. You could
say that the unique_ptr returned from lock() has an empty deleter, or that
weak_unique_ptr can be directly dereferenced. but then you lose atomicity
between checking weak_unique_ptr for having been nulled out and deref'ing it
to access the object. Now, that may or may not be a problem, but it's a
problem that would have to be addressed for a  "production-ready"
implementation.

..

 

Do you mean nullified by a different thread?

I don't assume that pointer is used from multiple threads - much like unique_ptr it is used from a single thread.

For multiple threads we shared_ptr, that's for sure.

All I care is the handles ("weak_ptr"s) to not dangle. That's it. Not unlike QPointer.

As I said, I have done this, but it is not very straightforward and I was curious if some has done it.

 

[Nicol Bolas]

On Monday, October 17, 2016 at 9:34:47 AM UTC-4, mihailn...@gmail.com wrote:

On Monday, October 17, 2016 at 4:06:43 PM UTC+3, Marc Mutz wrote:

On Monday 17 October 2016 10:14:52 mihailn...@gmail.com wrote:
> Hello,
>
> In many cases there is the need to track an object and use it only if it is
> still alive. This can't be achieved with raw pointers.
>
> The only way, right now, is by using a shared_ptr + weak_ptr.
> But this has the downside of making your object sharable, where one does
> not intent to (granted with use_count of 1 most of the time).

The fundamental problem with what you're intending to do is this: unique_ptr
is a *unique* ptr. There are no other owners. Suppose there was some
weak_unique_ptr for unique_ptr the way we have weak_ptr for shared_ptr.

What does weak_unique_ptr::lock() return? It cannot return unique_ptr, since
then you might end up with two unique_ptrs owning the same object. You could
say that the unique_ptr returned from lock() has an empty deleter, or that
weak_unique_ptr can be directly dereferenced. but then you lose atomicity
between checking weak_unique_ptr for having been nulled out and deref'ing it
to access the object. Now, that may or may not be a problem, but it's a
problem that would have to be addressed for a  "production-ready"
implementation.
..

 

Do you mean nullified by a different thread?

I don't assume that pointer is used from multiple threads

Then you're talking about a class that has a huge caveat to its usage. This is the 21st century; classes that only work when you're not using threads is a poor and dangerous assumption to be making.

I'm not saying that such a type should never exist. But if you have the need for such a thing, it's not something that would be broadly usable.

In any case, you have several routes of implementation:

1: Linked list. From an `observable_unique_ptr<T>`, you build a linked-list of pointers to existing `observable_ptr<T>` objects that share access to the unique pointer. Upon the destruction of the `observable_unique_ptr<T>`, you NULL-out all of those `observable_ptr<T>` objects.

The linked list can be done intrusively (each `T` provides an interface which the `observable_unique_ptr` exposes) or extrusively (`observable_unique_ptr` stores the linked list internally.).

2: Control block. Along with the `T`, you create a control block which is referenced by every `observable_ptr<T>`. To NULL them all out, you simply NULL out the control block. The control block can only be destroyed when all observable_ptrs of any kind that reference it are destroyed.

The thing about all of these cases is this: they create overhead. One of the biggest advantages of `unique_ptr` is having zero overhead. If you create this type, then you're saying that you're fine with overhead. At which point... why would you use this instead of `shared_ptr`, when `shared_ptr` actually is thread-safe?

Is that tiny bit of performance from not using atomic reference counts really that crucial to your application?

[mihailn...@gmail.com]

On Monday, October 17, 2016 at 5:12:47 PM UTC+3, Nicol Bolas wrote:

...

The thing about all of these cases is this: they create overhead. One of the biggest advantages of `unique_ptr` is having zero overhead. If you create this type, then you're saying that you're fine with overhead. At which point... why would you use this instead of `shared_ptr`, when `shared_ptr` actually is thread-safe?

Is that tiny bit of performance from not using atomic reference counts really that crucial to your application?

...

 

The semantics are the problem. The user is forced to use shared ownership where non-shared is needed.

Tracking lifetime does not imply lifetime participation. 

Considering we are told over and over again (for all good reasons) to prefer unique ownership, and considering the fact, unique_ptr (and its raw "handles") is already one-thread-at-a-time,

it certainly does not feel good when, in order to use tracking, you have to move to shared ownership as well - a feature you don't need.

[Nicol Bolas]

On Monday, October 17, 2016 at 11:17:10 AM UTC-4, mihailn...@gmail.com wrote:

On Monday, October 17, 2016 at 5:12:47 PM UTC+3, Nicol Bolas wrote:

...
The thing about all of these cases is this: they create overhead. One of the biggest advantages of `unique_ptr` is having zero overhead. If you create this type, then you're saying that you're fine with overhead. At which point... why would you use this instead of `shared_ptr`, when `shared_ptr` actually is thread-safe?

Is that tiny bit of performance from not using atomic reference counts really that crucial to your application?

...

 

The semantics are the problem. The user is forced to use shared ownership where non-shared is needed. Tracking lifetime does not imply lifetime participation.

But there is shared ownership. There is no way to do this where ownership is not conceptually being shared. Your way de-facto causes ownership of the object to be shared between two places: the code with the `observable_unique_ptr` and the code that accesses the `observing_ptr`.

You "solve" this problem by declaring that the code doing the accessing must synchronize its access with the code that owns the `observable_unique_ptr`, and that the code doing the accessing must never do anything that would cause the code that owns the `observable_unique_ptr` to relinquish ownership.

This has the effect of making the use of this feature fragile. And the whole point of having ownership handled by objects is to keep code from being fragile like this in the first place. Observe:

observable_unique_ptr<T> pt;

void do_stuff()
{
  pt = ...; //relinquishes object.
}

void observe(observing_ptr<T> hT)
{
  auto my_pt = hT.lock();

  if(my_pt)
  {
    do_stuff(); //Oops.

    my_pt->someFunc(); //Object has been destroyed.
  }
}

int main()
{
  pt = ...;

  observe(pt);
  return 0;
}

Oh sure, it's easy to tell in this case that this code will produce UB. But what about when a complex chain of events in `do_stuff` causes the changing of `pt`? If it takes several function calls winding through a half-dozen files that ultimately results in the release of the currently owned object, it would certainly not be easy to tell.

You might say that this would never happen to you, that you're a good enough coder to avoid that. But people who don't use smart pointers say the exact same thing: that they'll remember to release the memory at the right time. That they'll clean up all their pointers when they delete an object. That they won't use dynamic memory allocations incorrectly.

We invented smart pointers because we recognize that we always think we are cleverer than we actually are.

The function `observe` does not merely "observe" the pointer. It requires that if a non-NULL pointer was returned from `lock()`, that the pointer remain non-NULL until the end of the function. `shared/weak_ptr` can represent this requirement. Your observing unique pointer cannot.

There is no unique ownership here. There is a tacit requirement that the caller of `observe` will use their own tools to make sure that the requirement is upheld.

To be fair, we do something similar every time a function takes a `T&` or a `T*`. Every function effectively assumes that there is nothing it can do which will cause the destruction of the object being pointed-to/referenced. And it is the responsibility of the caller to make sure that this requirement is upheld.

But my overall point is that smart pointers are all about specifying ownership semantics. And the ownership semantics of `observe` is that the object must either exist or not exist. If it exists, it must continue to do so until `observe` is finished observing it.

And the smart pointer you desire does not match this ownership usage; `shared/weak_ptr` does.

Considering we are told over and over again (for all good reasons) to prefer unique ownership, and considering the fact, unique_ptr (and its raw "handles") is already one-thread-at-a-time,

That's true of 90% of the standard library types; you cannot access the same object from multiple threads without problems. Indeed, it's true of `shared_ptr` too.

But you can access different instances of a `shared_ptr` that manage the same object. You can't do that with `unique_ptr` because, by definition, for each object being managed, there is exactly and only one instance doing that management.

[mihailn...@gmail.com]

On Monday, October 17, 2016 at 7:00:24 PM UTC+3, Nicol Bolas wrote:

On Monday, October 17, 2016 at 11:17:10 AM UTC-4, mihailn...@gmail.com wrote:

...

But you can access different instances of a `shared_ptr` that manage the same object. You can't do that with `unique_ptr` because, by definition, for each object being managed, there is exactly and only one instance doing that management.

 

There is only one object which manages the object lifetime, but infinite number of pointers and references to that data.

No one is forbidding these any time soon, as long as one obeys the rules:

 - no cross-tread use of these at the same time;

 - no funky code which can jeopardize the original object;

 - you cannot store them unless you are sure, the original has not changed.

This is the current way to use unique_ptr.

The point is, the last rule can be relaxed to: you can store as much refs as you want, just test them for null before use.

The first two rules still apply as before, and must be obeyed.

In practice, relaxing the third rule considerably increases the use case of unique_ptr.

Every use of unique_ptr in place of shared_ptr is always a good thing!

[David Krauss]

On 2016–10–18, at 12:28 AM, mihailn...@gmail.com wrote:

In practice, relaxing the third rule considerably increases the use case of unique_ptr.

Every use of unique_ptr in place of shared_ptr is always a good thing!

You don’t seem to be proposing to expand the usage of unique_ptr though. It doesn’t have the control block needed by the observer. You’re proposing a new class which is under the hood almost exactly like shared_ptr, but with a less safe interface.

If you want a shared_ptr whose reference count is never more than 1, then (privately) subclass it and delete the copy constructor. To create the single-threaded observer, subclass weak_ptr and replace the lock() interface with something like,

    T * get() const { return this->lock().get(); }

This doesn’t yield the minor performance improvements, but it’s exactly the interface you describe. It can fairly be called something like observable_unique_ptr. If profiling indicates that observer::get() or the destructor is a bottleneck, then reimplement the library without a shared_ptr basis.

But it remains a different animal from unique_ptr due to the control block, which adds overhead and subtly changes the deleter semantics. And its added value is only performance at the expense of thread safety, which as others have already said, makes it a poor fit for the standard library.

[mihailn...@gmail.com]

On Tuesday, October 18, 2016 at 3:12:04 PM UTC+3, David Krauss wrote:

On 2016–10–18, at 12:28 AM, mihailn...@gmail.com wrote:

In practice, relaxing the third rule considerably increases the use case of unique_ptr.

Every use of unique_ptr in place of shared_ptr is always a good thing!

You don’t seem to be proposing to expand the usage of unique_ptr though. It doesn’t have the control block needed by the observer. You’re proposing a new class which is under the hood almost exactly like shared_ptr, but with a less safe interface.

...

I meant unique ownership as a concept, not unique_ptr as such. Pointers to unique ownership objects are always less safe.

No matter if implemented as an upgrade to unique_ptr or a downgrade from shared_ptr, there are uses for such a QPointer-like class.

I was not even proposing it, just asking if you all just use shared_ptr, though not entirely correct, or use some other tool.