Proposal: Named Function Arguments

[Brandon Gillespie]

Ordering of arguments is an age-old programming challenge (creating many bugs). Type checking and pattern matching help, but are also limited. We currently have two indirect ways of having named arguments, but both have challenges:

1. Keyword lists — pattern matching must match the exact order of the listed arguments, which doesn't help the issue here which is ordering, let alone optional arguments, leading to extraneous in-function calls to Keyword.get.
2. Maps — while this supports optional arguments, it has the overhead of creating and destructing a map for a simple function call. Benchmarking this vs ordered arguments shows it's 1.79x slower than simply using ordered arguments. That's an overhead that builds up on every function (I didn't benchmark keyword lists, sorry).
3. Existing syntax to handle named arguments as keywords should still be handled, for backwards compatibility, making it harder to do this sort of thing.

To preface the proposal, I realize this may simply not be possible with the limitations of the parser/compiler that we have today.

Proposal:

Add a compile-time named/pinned argument syntax in the function declaration head, which allows the naming of arguments as if it were a keyword list, but instead, the keys are mapped to the variable names/pins and, if necessary, are rearranged to keep the ordering correct.

This will not work with conventional keyword arguments, where one expects a keyword list—if using the named/pinned syntax, keyword arguments may not be used. It's one or the other, not both.

If this named syntax exists on a function head, then a calling function may use the `name: value` syntax and it will align the values to the named argument at compile time (no keyword lists are used at runtime).

Example:

Using `&` as a reference for the naming (or possibly if not that, the asterisk):

  def do_a_thing(&new, &old)

Would accept any of these calls, and in all cases the variable values within the called function would align properly:

  do_a_thing("newer value", "older value")
  do_a_thing(new: "newer value", old: "older value")
  do_a_thing(old: "older value", new: "newer value")

Optional named arguments in the same manner as optional arguments today:

  def do_a_thing(&new, &old, &optional \\ "extra info")

Optional ideas:

1. If rearranging the arguments at compile time is not easily feasible, it could simply just raise a compiler error when out of order, and then strip the names when they are properly ordered.
2. If using named/pinned arguments, always require them to be named (thus, the first example above would be invalid). However, this comes with its own challenge, notably what about when using pipelines? Perhaps just allow that.
3. If it's too challenging to use the same syntax as keyword lists, another naming convention could be used. It's just ... uglier.  Perhaps if using `*` instead it would be on both sides (function head, and calling function), such as:

  def do_a_thing(*new, *old)
  ...
  do_a_thing(*new: "newer value", *old: "older value")

[Marc-André Lafortune]

If I understand your proposal, calling `do_a_thing(old: "older value", new: "newer value")` would need to know the signature of `do_a_thing` at *compile-time*. Among other things, two modules wouldn't be able to call each other's functions this way, as one couldn't be compiled without the other... Also there would be no way to know if `do_a_thing` uses named function arguments, so any function call using keywords would introduce a compile-time dependency.

That being said, I miss Ruby's named arguments, and wish the default syntax for Elixir was building maps instead of keywords, but that can't change. Having a nice syntax for map arguments with defaults could be interesting though.

[Felipe Stival]

This library implements this functionality, albeit on a peculiar way: https://hex.pm/packages/defnamed. It suffers from the issues explained by Marc-André.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/0928ddde-8ca2-4110-a223-d62b711529a7n%40googlegroups.com.

[José Valim]

The tricky part is that mixing multiple-clause pattern matching with defaults is very complicated due to all the ambiguity that arises. And Elixir has opted into pattern matching for this particular feature set.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAKC64%2Bz6yHpK%3DJpN1fd7fgz9Z-m2UmsA0a852obGrjCpXzN1-w%40mail.gmail.com.

[Brandon Gillespie]

The challenge I've seen, faced, and had team members face time and time again, is that pattern matching only goes so far. The limitations are:

1. Maps are slower (significantly enough to avoid IMHO), but do give the flexibility desired (1.79x slower, which could compound if in the wrong place)
2. Keyword lists must be matched in exact order, and flexibility with them is limited

It's the frequent edge cases that end up causing problems: they end up being a little clunky in practical use.

-Brandon

On 10/27/22 2:30 PM, José Valim wrote:

[...]

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4%2BsNgEUVdYQv_N6cu90O%2BzheNs1UPo5bZPw8-tT4UROhA%40mail.gmail.com.

[José Valim]

> 1. Maps are slower (significantly enough to avoid IMHO), but do give the flexibility desired (1.79x slower, which could compound if in the wrong place)

Do you have a source for the 1.79 slower?

I don't expect maps to be slower but, more importantly, the difference here should be negligible because options rarely exceed more than a dozen keys and in those scenarios lookups are in the order of single-digit microseconds (source) in the worst case. So it seems counter productive to say you wouldn't pick the more flexible option because an operation will take 1.75us instead of 1us (assuming it is indeed slower). :)

Plus maps have other benefits such as reading multiple keys at once which will certainly offset keyword lists.

> Keyword lists must be matched in exact order, and flexibility with them is limited

Yes, this is a pain point I see people running into. Especially given that you are not supposed to match on keyword lists.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/e77b7315-625a-4c8e-0e63-bbd03116fbbe%40cold.org.

[Brandon Gillespie]

On 10/27/22 3:42 PM, José Valim wrote:

> 1. Maps are slower (significantly enough to avoid IMHO), but do give the flexibility desired (1.79x slower, which could compound if in the wrong place)

Do you have a source for the 1.79 slower?

I don't expect maps to be slower but, more importantly, the difference here should be negligible because options rarely exceed more than a dozen keys and in those scenarios lookups are in the order of single-digit microseconds (source) in the worst case. So it seems counter productive to say you wouldn't pick the more flexible option because an operation will take 1.75us instead of 1us (assuming it is indeed slower). :)

Plus maps have other benefits such as reading multiple keys at once which will certainly offset keyword lists.

1.79 times, as I read it, not 1.79us. And of course benchmarks being highly subjective, now that I retooled it it's at 2.12x slower (see notes at the very bottom for likely reasons why).

What drove me to look into this is a very specific situation we face in my project where we process a high frequency of events on a databus per second, and it was bogging down. As I isolated easy wins (external database calls) I started looking into other inefficiencies.

Specifically this is a call that looks at the before and after state of a struct and from that decides if changes should be handled or not.

I expect many will likely find fault with the implementation of this benchmark, but it is for a specific use case... for the room in general: let's not digress on if there are better ways to do a thing or not :)

The gist includes three scenarios:

• ordered — arguments are given in order and pipelined (this is what we are using now)
  
• map_get — arguments are culled ala Map.take()
• map_pattern — arguments use a pattern match w/a dictionary

The #'s:

Name                  ips        average  deviation         median         99th %
ordered            5.06 M      197.65 ns ±21980.53%         143 ns         230 ns
map_get            3.86 M      259.27 ns ±28310.93%         183 ns         494 ns
map_pattern        2.39 M      418.72 ns ±11140.31%         426 ns         551 ns

Comparison:
ordered            5.06 M
map_get            3.86 M - 1.31x slower +61.61 ns
map_pattern        2.39 M - 2.12x slower +221.07 ns

The gist / benchmark that created the above: https://gist.github.com/srevenant/ed28a0cf2a167379a21eb482291313ce

Why the change?

I did change it from when I ran it previously to come up with the 1.79x number, because it was referencing our code base, and in the previous benchmark I had less arguments it was matching against. I added a few more arguments to amplify the situation, as you can see.

I do realize this is NOT the heart of my performance problem, so I hope others can avoid digressing into "you should do it this other way" type discussions :D

This is just something that came out of exploring some performance issues.

And the suggestion for efficient named arguments came because we'd switched to ordered arguments, and then the code was flipped in one place (a, b) vs (b, a) (face-palm)

-Brandon

[Zach Daniel]

What if a syntax for matching on keyword lists that allowed for items in any position was added to Elixir? Something like (just shooting from the hip) `[…foo: bar]` ? Then you could have your cake and eat it too, right?

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/f3307a40-b4fe-c66f-0576-a162202dfee2%40cold.org.

[José Valim]

 

1.79 times, as I read it, not 1.79us. And of course benchmarks being highly subjective, now that I retooled it it's at 2.12x slower (see notes at the very bottom for likely reasons why).

Correct. What I did is to take a reference value of 1us and multiplied it by 1.79, to say that at this scale those numbers likely won't matter.

The gist includes three scenarios:

Thanks for sharing. I won't go deep into this, as requested, but I want to point out that the conclusion "maps are slower (significantly enough to avoid)" is still incorrect for the benchmarks above.

All of those benchmarks are using map patterns because both map.field and Map.get are also pattern matching on maps.

map.field is equivalent to:

case map do

  %{field: value} -> value

  %{} -> :erlang.error(:badkey)

  _ -> :erlang.error(:badmap)

end

Map.get/2 is equivalent to:

case map do

  %{field: value} -> value

  %{} -> nil

end

 

To further drive this point home, you could rewrite the map_get one as:

  def map_get(engine) do
    map_get_take(engine.persist, engine, @take_keys, [])
  end

  defp map_get_take(engine, persist, [a | rest], out) do
    case {engine, persist} do
      {%{^a => value}, %{^a => value}} ->
        map_get_take(engine, persist, rest, [{a, value} | out])

      _ ->
        map_get_take(engine, persist, rest, out)
    end
  end

  defp map_get_take(_, _, [], out), do: out

And the numbers likely won't matter or be roughly the same. The point is: you are effectively benchmarking different algorithms and not the difference between map_get or map_pattern.

I am only calling this out because I want to be sure no one will have "maps are slower (significantly enough to avoid)" as a take away from this discussion.

> What if a syntax for matching on keyword lists that allowed for items in any position was added to Elixir? Something like (just shooting from the hip) `[…foo: bar]` ? Then you could have your cake and eat it too, right?

Valid patterns and guards are dictated by the VM. We can't compile keyword lists lookups to any valid pattern matching and I would be skeptical about proposing such because we should avoid adding linear lookups to patterns.

It is worth taking a step back. It is not only about asking "can we have this feature?". But also asking (at least) if the feature plays well with the other constructs in the language and if we can efficiently implement it (and I believe the answer is no to both).

[Brandon Gillespie]

Fair enough :)

If I understand what you are saying: they are all maps because the source data comes from a map, and it's the method of extracting data from the map that differs (the algorithm), not the inherent nature of a map itself.

I agree, and apologize for the mistaken assertion.

However, what I didn't benchmark as i think about it, is what I often will see, which is the creation of a map simply to pass arguments — and this is more relevant to the request/need. The example was based on existing structs/maps and not creating them at each function call time.

Instead, for example:

       def do_a_thing(%{key2: value2, key1: value1}) do ...

I think it's becoming a common pattern to then construct the maps as part of the call, ala:

       do_a_thing(%{key1: 10, key2: 20})

Is this an expensive pattern because it generates a map only for the next function to extract the keys and ignore the map?

-Brandon

--
You received this message because you are subscribed to a topic in the Google Groups "elixir-lang-core" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elixir-lang-core/Dbl6CL5TU5A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4L37yu8KVbhuM0gNkVYOzCeoXaKzTBk4aY4OLLRdgRRLg%40mail.gmail.com.

[José Valim]

> Is this an expensive pattern because it generates a map only for the next function to extract the keys and ignore the map?

It depends on what you are comparing it with. Compared to simply passing arguments, it is likely slower. Compared to keyword lists, likely faster.

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9f60ba0c-8403-e93f-d5fb-b3f55df88d14%40cold.org.

[Jake Wood]

So the original proposal here is for introducing a named parameter syntax. The reason I like named parameters is b/c the order of parameters doesn't matter – when they do matter it's easy for refactoring to introduce hard to catch bugs. Pattern matching has been proposed as the idiomatic way to achieve argument order not mattering. If I understand correctly, the recommendation is to stuff arguments into a map just before a function call that itself immediately destructures them. While this approach does address my primary concern (ie parameter order), it has to be slower, right? I can imagine this having a non-trivial effect in a pipeline on a hot-path.

So the question for me, really, is how much quicker is passing ordered arguments vs creating then destructuring a map? If it's negligible then it's negligble, but if it's not then it would be nice to have an alternative.

- Jake

[Paul Schoenfelder]

The problem is that in order to implement named function arguments, both the caller and callee must agree on the calling convention (i.e. how arguments are passed, in what order, etc.). In general, it isn't possible to know this in all circumstances, due to things like a callee being implemented in another language, dynamic apply via apply/3, function captures, and hot code loading. I think the issue here isn't whether or not the feature would be nice (I think we all agree that it would be), or how it performs, but that implementing it has a number of unresolvable issues.

Paul

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/01432858-e854-4747-921a-230e6bbd7489n%40googlegroups.com.

[Wiebe-Marten Wijnja]

Thank you for starting this interesting discussion!

While I don't think the suggested solution (introducing special pattern matching syntax) is viable, for the reasons already mentioned by others,
I do think the problem itself warrants further consideration.

Currently, handling keyword arguments is done in an ad-hoc fashion.
Approaches between different codebases and even between different parts of the same codebase vary significantly.
Especially w.r.t. error handling.
Even in Elixir's own codebase this is apparent. Some (non-exhaustive) examples:
- Passing wrong options to `if` raises an ArgumentError with the text "invalid or duplicate keys for if, only "do" and an optional "else" are permitted"
- Passing wrong options to `defimpl` raises an ArgumentError with the text "unknown options given to defimpl, got: [foo: 10, bar: 20]"
- Passing wrong options to `for` raises a CompileError with the text "unsupported option :foo given to for"
- Passing wrong options to `inspect` ignores the option(s) silently.
- Passing wrong options to `GenServer.start_link` ignores the option(s) silently.

Other differences are between whether only keyword lists are accepted, or maps with atom keys also, or possibly anything implementing the `Access` protocol.
And in some places the options are used to create a special struct representing the parsed options, which is allowed to be passed as well directly.

This makes me think that we might want to look into standardizing:
- How to indicate which options are mandatory and which options have defaults.
- What kind of exception is raised when incorrect values are passed (and with what message).
- By default raise whenever unrecognized options are passed; the alternative of ignoring unrecognized options as an explicit opt-in choice.

I think we could introduce a macro that embeds the code to do these things and turn the result into a map inside the function where it is called.
For the reason mentioned by José before (supporting multiple function clauses with different pattern matching and defaults)
it makes more sense to call this macro in the function body rather than embellish the function head with some special form.
What I haven't been able to figure out yet is how to call this macro (`parse_options`?), or in which module in Elixir core it should live. (`Keyword`? Or in a new `Option` module?)

I haven't written a proof-of-concept yet but I am pretty sure that it is possible to write an implementation that needs to traverse the list --or map--
that is passed in to the function only once. (Stopping earlier when the number of keys inside do not match.)
This should be performant enough for general usage.
If there is a problem, I think that raising an ArgumentError (but with a different error message detailing what options are missing or unrecognized)
might be the clearest way to indicate to the caller that they are using the function incorrectly.

The diligent reader might notice that there certainly is some overlap between this new macro and initializing a struct with enforced keys.

~Marten / Qqwy

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/01432858-e854-4747-921a-230e6bbd7489n%40googlegroups.com.

[Zach Daniel]

Couldn’t anyone who wants to do something like this just use a tool like nimble_options (at least in the near to mid term)? That’s what I do. https://github.com/dashbitco/nimble_options

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/50a4057e-1d53-77fe-6cf5-1d7804f32b8b%40resilia.nl.

[José Valim]

Keyword.validate! is meant to solve part of the option validation, except typing.

NimbleOptions can solve typing but ideally I would try to embed the basic type validation in the type system.

So in the long term, I would say type system + Keyword.validate!.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/l9squzpm.82ffe77f-1d5b-4c9f-9adf-83a8ed0cf0e8%40we.are.superhuman.com.

[Wiebe-Marten Wijnja]

Thank you; I had missed the introduction of `Keyword.validate!` 😊

What is your stance on enhancing it to turn the result into a map or struct?
Both for performance (performance characteristics of small maps are similar to those of tuples).
And to make more illegal states unrepresentable: `Keyword.validate!` does not allow duplicate keys, but the output is another keyword list, which does not enforce this invariant. (i.e. "parse, don't validate")

~Qqwy / Marten

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAGnRm4KA%2BzATOV%2BW7AQOXeKKkxxg%2BcCfOv-nztsPS5JRu6ezdg%40mail.gmail.com.