Iterating over/transforming std::optional

[Mikhail Maltsev]

Hi, all!

Rust and Scala have types similar to std::optional:
https://doc.rust-lang.org/std/option/index.html
http://www.scala-lang.org/api/2.11.8/#scala.Option

Both of these languages support iterating over Option, (as if it was a container
with either 0 or 1 elements). Another feature which C++'s optional lacks
(compared to Rust and Scala) is transforming the value contained in std::optional.

I.e.:

std::string to_string(int i);
std::optional<std::string> to_opt_string(int i);

int main()
{
std::optional<int> test { 1 };
auto res = test.transform(to_string);
auto res2 = test.flat_transform(to_opt_string);

// res and res2 are of type std::optional<std::string>
}

What do you think about adding these features?

--
Mikhail Maltsev

[S.B.]

https://groups.google.com/a/isocpp.org/forum/#!topic/std-proposals/in38mPp2NiU

[Nicol Bolas]

On Sunday, September 25, 2016 at 9:34:49 AM UTC-4, Mikhail Maltsev wrote:

Hi, all!

Rust and Scala have types similar to std::optional:
https://doc.rust-lang.org/std/option/index.html
http://www.scala-lang.org/api/2.11.8/#scala.Option

Both of these languages support iterating over Option, (as if it was a container
with either 0 or 1 elements).

It seems to me that you can write an `as_range` function that takes `optional` easily enough. But really, iterating over an optional isn't something people want to do often enough to stick it in the standard library.

Another feature which C++'s optional lacks
(compared to Rust and Scala) is transforming the value contained in std::optional.

It seems to me that this is something you can write as a simple helper function, one which takes an optional and a function, then uses `std::invoke` on the optional if it is engaged.

Like range optional iteration, it is something that doesn't come up frequently enough to need a standard library function.

[Sean Middleditch]

On Sunday, September 25, 2016 at 1:59:21 PM UTC-7, Nicol Bolas wrote:

Like range optional iteration, it is something that doesn't come up frequently enough to need a standard library function.

Checking if an optional is engaged and then invoking code over it is the whole point of an optional. Doing it _safely_ without being exceedingly easy to misuse is also a very key functionality, something which today requires a good static analyzer to use.

This is asking for essential limited pattern matching in pure-library form for optional. Pattern matching being something just about everyone wants; see https://github.com/solodon4/Mach7 for instance.

The interface in C++17 is unsafe. It goes against the many lessons learned in the languages that pioneered the very concept of optionals/variants/algebraic-types. With C++ optional, you can just *opt and get a thrown exception; now we have Java's NullPointerException ported into C++, essentially.

The video isn't available yet, but you should really see Ben's CppCon '16 talk "Using Types Effectively" that explains the better way to use types like optional that we have learned from functional-programming. (The talk's slide content is available on his GitHub at https://github.com/elbeno/using-types-effectively/blob/master/presentation.org .) I might also suggest reading articles on using optional types that come straight from the FP world, e.g. articles like https://8thlight.com/blog/uku-taht/2015/04/29/using-the-option-type-effectively.html .

Better use of an optional _mandates_ the use of conditional unpacking. Traditionally, this was done with a full-weight pattern matching system:

// painfully verbose... but always safe, guaranteed (invented syntax, of course)

match (opt) {

  case some(&val): use(val);

  case none: ignore_or_something();

}

 More modern improvements allow this to be done in a simpler conditional statement:

// short and simple... not possible in C++ as a library

if (auto& val = some(opt))

  use(val); 

C++ doesn't have support for that in the language directly; the above would check the truth-iness of the unpacked value, not whether the value can be unpacked in the first place. The only ways to emulate it would be as unsafe as optional is today (in fact would just _be_ what optional is today). What we do have in C++ that can be used to emulate this, however, is the for-range statement:

// semantic abuse of for-range, but it works in C++ today with no language changes,

// and doesn't suffer the problems of lambdas ... but it can't support an else clause

for (auto& val : opt)

  use(val);

// negate the check - none returns a one-element range of std::ignore or something if the optional is disengaged

for (auto : none(opt))

  ignore_or_something();

We could also use lambdas/functions as you suggested, but that breaks in some fun ways; namely, it disables the use of return/break from the body. The best you can do would be a signature like the below. It's also just a pain to use reliably in loops, meaning you have to resort to _really_ ugly uses of find_if instead.

template <class A, class U, class V>

common_type_t<U, V> match(optional<A> value, function<U(A)> engaged, function<V()> disengaged);

// usable in the tail position of a function only

return match(val,

  [](T& val){ return use(val); },

  []{ return ignore_or_something(); });

// this won't actually work...

for (auto& opt : optionals_container)

  match(opt,

    [](T& val){

      if (condition(val))

        return val; // oops, this isn't useful, it "continues" in the loop and doesn't return/break

    }, []{ something_else(); });

// find_if suffers from the same general problem: you are able to use the return value incorrectly/unsafely

auto it = std::find_if(optionals_container.begin(), optionals_container.end(), [](optional<T>& opt){

  if (opt && conditional(*opt))

    return true;

  else {

    something_else();

    return false;

  }

});

auto& val = **opt; // what if it == end()? are you _sure_ the returned value is an engaged optional? the compiler can't help check your assumption

return val;

Now, all that said, I don't in my heart _like_ the idea of abusing the for-range statement for this purpose. But, as a library-only implementation, it's really the best option *ahem* that we have right now. I'd prefer real pattern matching, but the specifics aside, the claim that we don't need this or that it's an uncommon need is mistaken.

On a related note, the lack of a ::map method or its like on optional is also a huge shortcoming. Such a method would actually help remove a ton of the uses of the for-range usage discussed here:

// val becomes optional<B>, and is engaged iff opt was engaged

auto val = opt.map([](auto& v){ return B(v); });

// could be even simpler if we had a std::construct which I'll include below

auto val = opt.map(construct<B>); 

std::construct<T>

template <class T>

struct constructor_impl {

  template <class... Args>

  constexpr T operator()(Args&&... args) const { return T(std::forward<Args>(args)...); }

};

template <class T>

constexpr auto construct = constructor_impl<T>{};

[Ville Voutilainen]

On 26 September 2016 at 09:11, Sean Middleditch

<sean.mid...@gmail.com> wrote:
> The interface in C++17 is unsafe. It goes against the many lessons learned
> in the languages that pioneered the very concept of
> optionals/variants/algebraic-types. With C++ optional, you can just *opt and
> get a thrown exception; now we have Java's NullPointerException ported into
> C++, essentially.

*opt doesn't throw.

[Domen Vrankar]

2016-09-26 8:11 GMT+02:00 Sean Middleditch <sean.mid...@gmail.com>:

Better use of an optional _mandates_ the use of conditional unpacking. Traditionally, this was done with a full-weight pattern matching system:

// painfully verbose... but always safe, guaranteed (invented syntax, of course)

match (opt) {

  case some(&val): use(val);

  case none: ignore_or_something();

}

 More modern improvements allow this to be done in a simpler conditional statement:

// short and simple... not possible in C++ as a library

if (auto& val = some(opt))

  use(val); 

C++ doesn't have support for that in the language directly; the above would check the truth-iness of the unpacked value, not whether the value can be unpacked in the first place. The only ways to emulate it would be as unsafe as optional is today (in fact would just _be_ what optional is today). What we do have in C++ that can be used to emulate this, however, is the for-range statement:

// semantic abuse of for-range, but it works in C++ today with no language changes,

// and doesn't suffer the problems of lambdas ... but it can't support an else clause

for (auto& val : opt)

  use(val);

// negate the check - none returns a one-element range of std::ignore or something if the optional is disengaged

for (auto : none(opt))

  ignore_or_something();

I know that it's an extra unpacking line and more lines than pattern matching version but this works with C++17:
  if(opt) // or alternative opt.has_value() for non experimental version of optional
  {
      auto& val = opt.value();
  }

  else
  {

      // handle no value

  }

It looks much better to me than for loop emulation hack and good enough to wait for pattern matching. You could also write an invoke_if (or invoke_if_else) functor that executes a lambda but only if optional contains a value which would probably be even more in the direction of functional programming and can emulate pattern matching.

Regards,

Domen

[Sean Middleditch]

On Sun, Sep 25, 2016 at 11:16 PM, Ville Voutilainen <ville.vo...@gmail.com> wrote:

*opt doesn't throw.

So it does not. Which is, arguably, worse. I mean, I staunchly dislike exceptions, but adding yet another unnecessary set of easy-to-accidentally-invoke UB is not great.

Sadly, the committee/community wanted optional sooner rather than later, even if the language isn't ready to do it right yet, so here we are. :)

Point stands. Easy-to-grep and safer-to-teach access to optional/variant/etc. is something that should be in the library. Since we'd be targetting C++20 at a minimum, now, though, it's perhaps better to think about how any language features landing in that timeframe could potentially be used to make the access even better (because no, for-range abuse isn't great).

--

Sean Middleditch
http://seanmiddleditch.com

[Matt Calabrese]

On Mon, Sep 26, 2016 at 9:07 AM, Sean Middleditch <se...@middleditch.us> wrote:

On Sun, Sep 25, 2016 at 11:16 PM, Ville Voutilainen <ville.vo...@gmail.com> wrote:

*opt doesn't throw.

So it does not. Which is, arguably, worse. I mean, I staunchly dislike exceptions, but adding yet another unnecessary set of easy-to-accidentally-invoke UB is not great.

*opt should not throw since it is a precondition violation. Either you use the throwing version when you don't know if it is engaged or not, in which case you are using the exception for regular control flow (which is never advocated), or you accidentally dereferenced the optional when it was nullopt, in which case your program's invariants have been violated so you shouldn't be trying to recover (you have a bug).

That said, I do I agree with you that what we have is not completely ideal and is error prone. A "visit" or "match" can sometimes be better (though it can be overkill, depending on who you talk to). I do have a solution that should be in an upcoming paper if I complete it in time -- a revision of p0376 (the std::call paper). The first version from the pre-Oulu mailing is at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0376r0.html . In the upcoming revision, the "active_field_of" provider has been renamed "active_alternative_of" to be consistent with how variant refers to its fields as "alternatives" and instead of dealing directly with just std::variant, it and the other variant-related providers now deal with a VariantLike concept that I introduce (not dependent on concepts TS, but I'll try to make a concepts TS versions of the concept as well). Things that can model VariantLike include std::variant, but also std::optional, boost::variant, and the proposed std::expected. The implication here is that you can do the equivalent of n-ary "visit" or "match", etc. in the same way for variant, optional, expected, and others, along with several other kinds of provisions that the revision introduces (I now have over 30 providers, though only a very small, core subset will be proposed).

[Ville Voutilainen]

On 26 September 2016 at 19:07, Sean Middleditch <se...@middleditch.us> wrote:
> On Sun, Sep 25, 2016 at 11:16 PM, Ville Voutilainen
> <ville.vo...@gmail.com> wrote:
>>
>>
>> *opt doesn't throw.
>
>
> So it does not. Which is, arguably, worse. I mean, I staunchly dislike
> exceptions, but adding yet another unnecessary set of
> easy-to-accidentally-invoke UB is not great.

Right, it's not great, it's fantastic. If I do

if (opt) {
opt->foo();
opt->bar();
f(*opt);
g(*opt);
}

I don't expect the language to check every one of those indirections,
because I already checked them.

> Sadly, the committee/community wanted optional sooner rather than later,
> even if the language isn't ready to do it right yet, so here we are. :)

I doubt that definition of "right" is universally shared by everyone.

> Point stands. Easy-to-grep and safer-to-teach access to
> optional/variant/etc. is something that should be in the library. Since we'd

optional::value and optional::value_or don't cut it?

[Domen Vrankar]

Sadly, the committee/community wanted optional sooner rather than later, even if the language isn't ready to do it right yet, so here we are. :)

IMHO having a somewhat more error prone standard version is much better than different people reinventing it on projects where using boost libraries is forbidden for one reason or another other (I was in that situation twice so it's not a theoretical reason).

 

Point stands. Easy-to-grep and safer-to-teach access to optional/variant/etc. is something that should be in the library. Since we'd be targetting C++20 at a minimum, now, though, it's perhaps better to think about how any language features landing in that timeframe could potentially be used to make the access even better (because no, for-range abuse isn't great).

My point regarding abusing for range was that if alternative is not that terrible compared to it (especially with if statements with initializers) because in both cases I have to think what I'm doing but while reading other people's code if won't confuse me while for will raise aha-one-or-more flag in my head which has a greater potential of confusing me and making the code harder to expand/modify.

That's why I think that standardizing iterators on optional would be a bad idea since it would probably confuse far more people than it would make interface less error prone. The interface isn't nearly as bad that we would need alternatives at any cost.

Regards,

Domen

[Nicol Bolas]

On Monday, September 26, 2016 at 1:42:03 PM UTC-4, Domen Vrankar wrote:

Sadly, the committee/community wanted optional sooner rather than later, even if the language isn't ready to do it right yet, so here we are. :)

IMHO having a somewhat more error prone standard version is much better than different people reinventing it on projects where using boost libraries is forbidden for one reason or another other (I was in that situation twice so it's not a theoretical reason).

Further, it really should be noted that `optional` as it stands is no more error-prone than a pointer. So it's not like this tool is beyond the normal stuff we have to deal with; it's just the same thing in a slightly different form.

[szollos...@gmail.com]

Hi,

How about...

std::optional<int> getOpt(int, int);

if (X& x : getOpt(137, 42))
    std::cout << x * 2;

.. akin to range for? I.e., we could have ':' as the bind operation (in monadic terms, not std::bind()), which binds getOpt()'s return value to the body of the 'if'. (For those who don't know, I'm working extending for-loops where we could use the same.)
The nice property of this syntax is that it works with pointers as well:

int a = 1;
int* pa = &a;

if (int& x : pa)
    std::cout << x * 2;

It way less error-prone than *pa or *opt_a, also it's got a simple syntax and zero overhead. In fact, the  compiler it might optimize out the optional itself much quicker than normally in this case (given it can inline getOpt()) and convert getOpt()'s return statemens to return to a given label in the caller, namely to 'if' and 'else' (given that type in each return of getOpt() can be statically deduced to X or nullopt_t). Thus it's both a syntax sugar and a hint for the optimizer.

What do you think?

Thanks,
-lorro

[Tony V E]

On Mon, Sep 26, 2016 at 4:19 PM, <szollos...@gmail.com> wrote:

Hi,

How about...

std::optional<int> getOpt(int, int);

if (X& x : getOpt(137, 42))
    std::cout << x * 2;

.. akin to range for? I.e., we could have ':' as the bind operation (in monadic terms, not std::bind()), which binds getOpt()'s return value to the body of the 'if'. (For those who don't know, I'm working extending for-loops where we could use the same.)
The nice property of this syntax is that it works with pointers as well:

int a = 1;
int* pa = &a;

if (int& x : pa)
    std::cout << x * 2;

It way less error-prone than *pa or *opt_a, also it's got a simple syntax and zero overhead. In fact, the  compiler it might optimize out the optional itself much quicker than normally in this case (given it can inline getOpt()) and convert getOpt()'s return statemens to return to a given label in the caller, namely to 'if' and 'else' (given that type in each return of getOpt() can be statically deduced to X or nullopt_t). Thus it's both a syntax sugar and a hint for the optimizer.

What do you think?

+1

I was thinking about this, and heading in exactly the same direction.

 

Thanks,
-lorro

--

Be seeing you,

Tony

[Ville Voutilainen]

Way ahead of you:
https://cplusplus.github.io/EWG/ewg-closed.html#140

[szollos...@gmail.com]

Hi,

'EWG found the motivation for the proposal insufficient.' - why oh why...
(sometimes I feel we needed random macro parameter separators like ':', ';' beside ',', just to be able to implement these dropped features).

-lorro

[Sean Middleditch]

On Mon, Sep 26, 2016 at 1:59 PM, <szollos...@gmail.com> wrote:

Hi,

'EWG found the motivation for the proposal insufficient.' - why oh why...

Read the paper. It's arguably a very weak presentation of the FP fundamentals necessary to understand the use cases.

At this point, a one-off syntax that does nothing more than checked-assignment is pretty uninteresting. Structured bindings at the very least needs to be taken into account and how that would extend to full-on pattern matching.

[Ville Voutilainen]

Which is indeed one of the reasons why EWG found that proposal
unconvincing. If we want
pattern matching, perhaps we should go for pattern matching rather
than minor tweaks to
tiny facilities of the language, like an if-statement.

[szollos...@gmail.com]

Hi,

You're right that it can probably be presented better; however, I don't think it's an FP/non-FP question. Quite the contrary: it's a perfect little tool when porting tons of legacy code where you 'just know' when you can dereference a nullptr or when an object can / can't be in invalid state. And I guess it's a very common problem.
(BTW, to me the 'perfect' optional would have a visitor with an optional else; but ymmw and I understand you can't force it on everyone).

Thanks,
-lorro