Operator precedence of `in` vs `!`

[Luke Imhoff]

(Sorry, for the long post, but lots of quoted AST)

As part of intellij-elixir, I have fairly thorough tests now to ensure my Grammar Kit grammar (an LL grammar with parsing expression grammar style positive and negative look-ahead) parses the same quoted form as Elixir's yecc grammar (an LALR grammar like you'd write for yacc) parsing the same way since one is left-most derivation and the original is right-most derivation.  These tests have shown an inconsistency in precedence.  The precedence according to elixir_parser.yrl is as follows:

Left       5 do.

Right     10 stab_op_eol.     %% ->

Left      20 ','.

Nonassoc  30 capture_op_eol.  %% &

Left      40 in_match_op_eol. %% <-, \\ (allowed in matches along =)

Right     50 when_op_eol.     %% when

Right     60 type_op_eol.     %% ::

Right     70 pipe_op_eol.     %% |

Right     80 assoc_op_eol.    %% =>

Right     90 match_op_eol.    %% =

Left     130 or_op_eol.       %% ||, |||, or

Left     140 and_op_eol.      %% &&, &&&, and

Left     150 comp_op_eol.     %% ==, !=, =~, ===, !==

Left     160 rel_op_eol.      %% <, >, <=, >=

Left     170 arrow_op_eol.    %% < (op), (op) > (|>, <<<, >>>, ~>>, <<~, ~>, <~, <~>, <|>)

Left     180 in_op_eol.       %% in

Right    200 two_op_eol.      %% ++, --, .., <>

Left     210 add_op_eol.      %% + (op), - (op)

Left     220 mult_op_eol.     %% * (op), / (op)

Left     250 hat_op_eol.      %% ^ (op) (^^^)

Nonassoc 300 unary_op_eol.    %% +, -, !, ^, not, ~~~

Left     310 dot_call_op.

Left     310 dot_op.          %% .

Nonassoc 320 at_op_eol.       %% @

Nonassoc 330 dot_identifier.

So, my reading of this is that unary_op_eol should be encapsulated by any operator with a lower number (so early in the above list).  This appears to hold true when quoting test expressions with Code.string_to_quoted/1:

iex(intellij_elixir@tartarus)27> Code.string_to_quoted("+one when +two")

{:ok,

 {:when, [line: 1],

  [{:+, [line: 1], [{:one, [line: 1], nil}]},

   {:+, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)28> Code.string_to_quoted("+one :: +two")  

{:ok,

 {:::, [line: 1],

  [{:+, [line: 1], [{:one, [line: 1], nil}]},

   {:+, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)29> Code.string_to_quoted("!one :: !two")

{:ok,

 {:::, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)30> Code.string_to_quoted("!one | !two") 

{:ok,

 {:|, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)31> Code.string_to_quoted("!one = !two")

{:ok,

 {:=, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)32> Code.string_to_quoted("!one || !two") 

{:ok,

 {:||, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)33> Code.string_to_quoted("!one && !two")

{:ok,

 {:&&, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)34> Code.string_to_quoted("!one == !two")

{:ok,

 {:==, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)35> Code.string_to_quoted("!one < !two") 

{:ok,

 {:<, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)36> Code.string_to_quoted("!one |> !two")

{:ok,

 {:|>, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

For everything from when through arrow (like `|>`), this can be summarized as the infix binary operator being the outer call, but for `in`, it switches:

iex(intellij_elixir@tartarus)37> Code.string_to_quoted("!one in !two")

{:ok,

 {:!, [line: 1],

  [{:in, [line: 1],

    [{:one, [line: 1], nil}, {:!, [line: 1], [{:two, [line: 1], nil}]}]}]}}

Ok, so I could assume that for some reason that at precedence 180, something weird happens and everything past `in` should be inside `!`, which is unary and so higher precedence (number 300), but everything after `in` also has the infix operator on the outside:

iex(intellij_elixir@tartarus)38> Code.string_to_quoted("!one ++ !two")

{:ok,

 {:++, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)39> Code.string_to_quoted("!one + !two") 

{:ok,

 {:+, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)40> Code.string_to_quoted("!one * !two")

{:ok,

 {:*, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

iex(intellij_elixir@tartarus)41> Code.string_to_quoted("!one ^^^ !two")

{:ok,

 {:^^^, [line: 1],

  [{:!, [line: 1], [{:one, [line: 1], nil}]},

   {:!, [line: 1], [{:two, [line: 1], nil}]}]}}

So, what's happening in the grammar to make `in` special?  If I'm misunderstanding how to read the operator precedence it could lead to systematic errors in intellij-elixir's translation from LALR to LL.  (And I had plenty of problems converting LALR to LL, so I'm fine with it being a problem with me reading the grammar, I just would like help understanding how to read it properly.)

[José Valim]

There is nothing in the grammar actually. We manipulate the AST in the Erlang code part of the parser. I am on my phone but I can send more info later if desired.

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/194f16ba-f1bc-4f08-b6dd-501c5bcc6f0c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Luke Imhoff]

Links to the relevant files and line ranges in those files on Github would be useful.  I've been working on this branch for 3 months already, so waiting a few more days to land it isn't a big deal.  Thanks for your help.

[José Valim]

Here we go:

https://github.com/elixir-lang/elixir/blob/master/lib/elixir/src/elixir_parser.yrl#L596-L597

https://github.com/elixir-lang/elixir/blob/master/lib/elixir/src/elixir_parser.yrl#L624

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

--

You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/CAFmZThoiuXg8rrm%2BWdmfn5FpyWvr0vYuUEk-k6%2BoXuQ1SBgL1A%40mail.gmail.com.

[Luke Imhoff]

https://github.com/elixir-lang/elixir/blob/master/lib/elixir/src/elixir_parser.yrl#L596-L597

Integrated as https://github.com/KronicDeth/intellij-elixir/commit/49ffef5a33ca1102213821efb4e2140f97622ca1 into intellij-elixir.  Such simple pattern matching and rearrangement is so much more code in Java using JInterface.  Thanks for the links.