[Proposal] Strict matching comprehensions

[Eric Meadows-Jönsson]

## Background

`for` comprehensions are one of the most powerful features in Elixir. It supports both enumerable and bitstring generators, filters through boolean expressions and pattern matching, collectibles with `:into` and folding with `:reduce`.

One of the features are automatic filtering by patterns in generators:

```elixir

list = [{:ok, 1}, {:ok, 2}, {:error, :fail}, {:ok, 4}]

for {:ok, num} <- list, do: num

#=> [1, 2, 4]

```

Generator filtering is very powerful because it allows you to succinctly filter out data that is not relevant to the comprehension in the same expression that you are generating elements out of your enumerable/bitstrings. But the implicit filtering can be dangerous because changes in the shape of the data will silently be removed which can cause hard to catch bugs.

The following example can show how this can be an issue when testing `Posts.create/0`. If a change causes the function to start returning `{:ok, %Post{}}` instead of the expected `%Post{}` the test will pass even though we have a bug.

```elixir

test "create posts" do

  posts = Posts.create()

  for %Post{id: id} <- posts, do: assert is_integer(id)

end

```

The example uses a test to highlight the issue but it can just as well happen in production code, specially when refactoring in other parts of the code base than the comprehension.

Elixir is a dynamically typed language but dynamic typing errors are less of an issue compared to many other dynamic languages because we are usual strict in the data we accept by using pattern matching and guard functions. `for` is by design not strict on the shape of data it accepts and therefor loses the nice property of early failure on incorrect data.

## Proposal

I propose an alternative comprehension macro called `for!` that has the same functionality as `for` but instead of filtering on patterns in generators it will raise a `MatchError`.

```elixir

posts = [{:ok, %Post{}}]

for! %Post{id: id} <- posts, do: assert is_integer(id)

#=> ** (MatchError) no match of right hand side value: {:ok, %Post{}}

```

Pattern matching when not generating values with `=` remains unchanged.

`for!` gives the developer an option to be strict on the data it accepts instead of silently ignoring data that does not match.

## Other considerations

You can get strict matching with `for` today by first assigning to a variable. This way you can also mix filtering and strict matching patterns.

```elixir

posts = [{:ok, %Post{}}]

for post <- posts,

    %Post{id: id} = post,

    do: assert is_integer(id)

#=> ** (MatchError) no match of right hand side value: {:ok, %Post{}}

```

Another alternative is to introduce a new operator such as `<<-` (the actual token can be anything, `<<-` is only used as an example) for raising pattern matches instead of introducing a completely new macro.

```elixir

posts = [{:ok, %Post{}}]

for %Post{id: id} <<- posts, do: assert is_integer(id)

#=> ** (MatchError) no match of right hand side value: {:ok, %Post{}}

```

A downside of adding new functions or macros is that it doesn't compose as well compared to adding options (or operators) to existing functions. If we want to add another variant of comprehensions in the future we might be in the position that we need 4 macros, and then 8 and so on.

Another benefit of adding an operator is that you can mix both `<-` and `<<-` in a single comprehension.

The downside of an operator is that it adds more complexity for the language user. We would also need an operator that is visually close to `<-` but still distinctive enough that they are easy to separate since their behavior are very difference.

[Christopher Keele]

This feature would be very useful, I've experience this signature-change pain point before too (and kind of have been avoiding `for` ever since, TBH).

I'm reluctant to increase the surface area of the language itself, what do you think about adding a `:strict` option to `for` instead of a new special form/kernel macro/operator?

[Adam Lancaster]

I also love the proposal.

It's a shame we can't re-use the `with` semantics of `=` raising a match error in the for.

My two cents is `for!` makes the most sense, and follows the conventions of other functions.

Best

Adam

-- 
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/42adcfba-12d8-4469-a156-f412b0d290a9n%40googlegroups.com.

[Tallak Tveide]

I would like to add a solution within the existing language:

```elixir

> list = [{:ok, 1}, {:ok, 2}, {:error, :fail}, {:ok, 4}]

> for el <- list, do: ({:ok, num} = el; num)

** (MatchError) no match of right hand side value: {:error, :fail}

```

I think this is reasonable.

Acctually the built in filtering in `for` caught me off guard, I was expecting for to fail unless all elements matched. So for me the better solution would be to always make matching in `for` strict. But I guess this is too late now for backwards compatibility. Another alternative to `for!` would be:

```elixir

> list = [{:ok, 1}, {:ok, 2}, {:error, :fail}, {:ok, 4}]

> for {:ok, num} <- list, strict: true, do: num

** (MatchError) no match of right hand side value: {:error, :fail}

```

I don't like the use of the exclamation mark in `for!` because it has little meaning relative to the existing use of the exclamation mark in Elixir.

[Christopher Keele]

> for {:ok, num} <- list, strict: true, do: num

Agreed, this is more or less exactly what I was pitching.

[José Valim]

My concern with :strict is that it changes the behavior considerably of the generators but it may show up only quite later on, far from them, especially if you have multiple filters.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/f4d5c0be-567a-4a7d-9b39-68202226c788n%40googlegroups.com.

[Christopher Keele]

> My concern with :strict is that it changes the behavior considerably of the generators but it may show up only quite later on, far from them, especially if you have multiple filters.

Could you elaborate? I don't quite think I understand, particularly "[the behaviour] may show up only quite later on"

Does "quite later" here refer to code distance (the MatchError's stacktrace would point away from/bury the for location)? Or temporal distance?

[José Valim]

Sorry, I meant to someone reading the code. The strict option is modifying the behavior of the operator <-, which may be quite before it in the text.

I prefer for! in this case as it is upfront.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0ce03abc-61bb-4423-b6a8-704d1d62169fn%40googlegroups.com.

[Christopher Keele]

That's fair enough! Though from my perspective both for! and strict: true would be about equally far from the <- where matches fail. But I can see the keyword format getting lost in the filters and other keywords.

You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/LEUD2alHPiE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4K01hBRkjLaRPj5ktViNNjYqdFbKdysvFcDVG%3DgBp78dA%40mail.gmail.com.

[Paul Schoenfelder]

I’m generally in favor of the option to have stricter semantics, but to me the introduction of `for!` feels out of sync with other special forms, none of which are bang-form. Furthermore, especially in contrast to `with`, you end up with this weird dichotomy with the `<-` operator, where sometimes it means a filtering match, and other times where it means strict match. That kind of syntactical inconsistency in a language feels like a bad precedent to set, despite what feels like a reasonable compromise. It’s also notable to me that there are easy ways to program defensively to force match errors if you want them, within the current syntax, but obviously that comes at the cost of more verbosity.

I’m not sure what the right answer is, but this feels to me like rushing to solve a specific problem without spending enough time considering how it meshes with the rest of the language in terms of cognitive complexity, particularly for those new to the language.

Anyway, that’s my two cents. I’m a fan of the concept for sure, but would almost prefer to see the semantics changed in a major version bump, to match `with`, even if that meant manually updating a bunch of my code, because at least it keeps the language self consistent. I’ll admit I’m probably an outlier on that though.

Paul 

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAD9kT2QPn_prFiS%2BR9eemqA43DMvvOB8NrAweL2PgE_ZR2g6Cg%40mail.gmail.com.

[Adam Lancaster]

I'm definitely sympathetic to that idea.

I think part of what internal consistency buys us is predictability and therefore a quicker path to a good mental model about what the code is going to do. But the mental model is the more important thing. Which is just to say if we don't have internal consistency but we can get to a good mental model, then I think it might be okay.

I think given other functions that follow the same idea, seeing a `for!` would certainly communicate "right this is expected to raise under some condition" - at least to me. 

There's also not an obvious way to have `for` mimic `with` when I think about it because say you do this:

```

for [a, _] = [1, 2], do: ... 

```

there is no way to distinguish it from a filter - where `=` should not raise a match error.

I think you'd have to more clearly de-mark the difference between the generators and the filters, which feels like a big change.

Best

Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/8a6cf634-cda5-4445-8230-4b7b69ed5ca8%40www.fastmail.com.

[Bruce Tate]

for! +1 

-bt

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/5E4E868D-88E9-4B5E-9A3B-25F330530687%40a-corp.co.uk.

--

Regards,

Bruce Tate

CEO

Groxio, LLC.

512.799.9366

br...@grox.io

grox.io

[Paul Schoenfelder]

In my opinion, internal consistency is part of the mental model, so inconsistency reflects a flaw in the model. That said, I think that's probably talking past you on this a bit, and I get what your point is: ultimately if it is reasonably intuitive, consistency can be allowed to fall by the wayside a bit. I guess where I disagree is that I'm not sure this will intuitive for someone not already steeped in the language. The point I was getting at by comparing `for` and `with` is that they both make use of the same `<-` operator in a way that is consistent across both forms, but with `for!` that falls apart.

Now back to `for!`. Even though it looks just like `for`, the `<-` operator starts to behave like `=`. If you are skimming code and happen to miss the single character difference between the two (`for` vs `for!`), you will wind up with a very different idea about what the same code does. The human brain is terrible at distinguishing small differences like this, it's why you can typo things like `behavior` and `behaviour` and read right over it without noticing, sometimes even when you are _trying_ to notice those things.

I think it would be far better for us to use a new operator in place of `<-`, rather than a new special form that looks basically identical to an existing one, but works differently in subtle ways. Not to mention, the operator approach would allow one to mix both `<-` and the new operator together in the same `for`, should it be useful to do so. In any case, I don't really have a strong opinion on what that operator is specifically, but I am much more in favor of that direction, than I am `for!`.

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/65609056-4A25-45FA-B91F-84D4DF292129%40a-corp.co.uk.

[Adam Lancaster]

That makes sense!

I guess you could make a new operator available outside of the `for` too, like back in a `with`...  Maybe `<-!` 

Best

Adam

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4f04033a-d509-460e-8205-ad23e1251b1e%40www.fastmail.com.

[Zach Daniel]

I like the new operator concept as well. The semantics of it could be extended to `with`,

E.g

with {:ok, res} <- thing(),

        {:ok, handled} <<- handle_res(thing),

        …

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4f04033a-d509-460e-8205-ad23e1251b1e%40www.fastmail.com.

[Stefan Chrobot]

How about an "else" block for the items that don't match?

for {:ok, item} <- items do

  # ...

else

  # ...

end

This would be consistent with <- in "with". I'm assuming the intention would be to prefer "for!" over "for" for most of the cases so the issue with this is a need to figure out a terse syntax for raising a MatchError without having to type something like "else _ -> raise MatchError"; plus also include what was being attempted in the error message. Is "else raise" an option?

Best,

Stefan

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CE301440-828B-41A5-B388-75CD6FF94699%40a-corp.co.uk.

[Sabiwara Yukichi]

A strict alternative is definitely a great idea!

for! sounds good and I like it.

Just an alternative for the record if we want to go for a different operator instead: maybe we could use the `in` operator to distinguish with the `<-`?

It reads naturally, `in` is already semantically used for enumerables, it is different enough from `<-` to make the difference in behavior clear and it is common in several other languages (python, JS...).

for {key, value} in map do

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CE301440-828B-41A5-B388-75CD6FF94699%40a-corp.co.uk.

[José Valim]

I am against introducing a new operator because I think that will be just unclear. Regardless if we pick "<!-" or "<<-", I don't think the notation would be clear to everyone reading the code. Between for!, a strict option, and the operator, the operator is, in my opinion, the most unreadable.

Furthermore, the operator doesn't have a use in "with", because "with" can use = for strict matches.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7b6srxk_vE45h-qY--g61t-Lzqgvix46jX2GxQRE46FGA%40mail.gmail.com.

[Stefan Chrobot]

Ideally, <- and = would have the same semantics in "for" and "with", but I'm not sure how confusing "for {:ok, post} = posts, do: ..." would be.

Given all that, I find "for!" to be the best approach, except for the name. As stated before, a bang version feels like a precedent. I think "fors" ("for strict") is better (like def and defp), or maybe a totally different name for a loop (while?, loop?, rep?, iter?).

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KFQ_2BNCXmFdJuxJC2XWD-hSpbod5_wt%2Bka%3DSBpfFcpg%40mail.gmail.com.

[Christopher Keele]

For the purpose of understanding internal consistency, I reviewed the behaviour of all native testing/matching constructs in a gist here:

In the gist I have a stream-of-consciousness set of observations describing how I arrive at these conclusions, but the TL;DR is:

I'd propose either

• extending for to support else features and raise ForClauseError if an item doesn't match
• introducing a new construct every that raises an EveryClauseError if an item doesn't match

Additionally, for consistency's sake I'd recommend strongly against

• introducing a new "arrow operator", ex <<-
• introducing a new construct for!
• extending for to support a strict: keyword argument

[Zach Daniel]

Ultimately if we don’t prefer the operator, I’d rather just see:

for var <- list do

  %{destructured: data} = var

end

Over the `for!` operator. I don’t know why the operator seems so much better than `for!` to me though.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CACzMe7YNuuhqq9S%2BJu6W_829XVBbPZar9vj5pAjkcan99NO82w%40mail.gmail.com.

[Christopher Keele]

Another option would be extending for to accept matches and set bindings in its macro's list of clauses, like with (which also mixes <- with =)

list = [1]

for option <- list, option = {key, value} do

  {option, key, value}

end

# proposal would raise runtime MatchError

# today raises compiletime CompileError: undefined function key/0

I actually like this even more than the other proposals I've advocated for. It'd bring the only other <- operator-using-construct (for) closer to the featureset of with and raises the intuitive MatchError instead of the ForClauseError that would be more consistent with other implementations.

[Christopher Keele]

Re: Zachary

> I’d rather just see:

> 

> for var <- list do

>   %{destructured: data} = var

> end

My understanding is that part of the intent is to have a way to surface hard requirements about the data shape to the top level of the construct, so it doesn't get lost within the block. I agree I'd prefer this convention over the addition of new constructs or operators increasing the surface area language, but if it was added to the language I'd use the feature since I imagine that'd become the new convention.

[Allen Madsen]

Re: Christopher Keele

> Another option would be extending for to accept matches and set bindings in its macro's list of clauses, like with (which also mixes <- with =)

This is already supported. You can do:

list = [1]

for option <- list, {key, value} = option do

  {option, key, value}

end

Allen Madsen
http://www.allenmadsen.com

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/ea66088e-445f-40d9-a9d3-702a700fb623n%40googlegroups.com.

[Christopher Keele]

Re: Allen Madsen

It's curious that for supports the match clause {key, value} = option works but option = {key, value} does not.

This feels like a bug in the implementation. (Interestingly, both formulations work in the -> clause.) I will open a github issue for further discussion.

Re: in general

The thing we can't do today using that match clause feature in for today is "guarded clauses" (ie failing rather than filtering with a guard mechanism). There's no place to put the guard, so this necessitates a new construct.

Today a guard in the <- clause acts as a filter, and you can't attach a guard to a match clause. We can't use with's else mechanism because we don't expect data shapes in multi-clause fors to have universally addressable failure modes. (That rules out my for -> else proposal above.)

I'm beginning to arrive at the conclusion that we'd definitely need to introduce a new construct for this (if we decide to proceed). I prefer:

every {key, value} when is_integer(value) <- [a: 1, b: "2"], do: {key, value+1}
# EveryClauseError: no clause matching {:b, "2"}

[Allen Madsen]

Re: Christopher Keele

> It's curious that for supports the match clause {key, value} = option works but option = {key, value} does not.

> This feels like a bug in the implementation. (Interestingly, both formulations work in the -> clause.) I will open a github issue for further discussion.

To me, that makes sense, since the latter case, key and value haven't been defined yet and matching and binding happens on the left side of =. The way you wrote it could work if you defined key and value first and pinned option with ^.

Re: In General

I also prefer the every construct over the other things presented so far.

Allen Madsen
http://www.allenmadsen.com

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/dbc2b3fa-d9b9-49f4-ba59-34805a630660n%40googlegroups.com.