Syntax extension for Matlab complex number notation

[Jaco]

I have implemented a syntax extension that provides a Matlab complex number notation, for example "5i" denotes the imaginary value 5. The implementation defines "i" as a suffix operator. It looks as follows:

-{ block:

mlp.lexer:add "i"

mlp.expr.suffix:add{ "i", prec = 90, builder = |a, _| +{complex(-{a}) } }

 }

table.print( +{5i} )

table.print( +{xi} )

It works as expected in the examples above. "xi" is parsed as an identifier and "5i" as a complex number. However, the following results in a "Unexpected expr token `Keyword "i"" error:

table.print( +{i} )

I need to be able to capture the identifier "i" and replace it with "complex(1)". How can I do this? 

[Fabien Fleutot]

On Wed, Oct 19, 2011 at 12:14 PM, Jaco <jvdm...@emss.co.za> wrote: 

table.print( +{i} )

I need to be able to capture the identifier "i" and replace it with "complex(1)". How can I do this? 

 

You don't; you want "i" to also be a plain expression keyword, in addition to being a suffix. Simply add:

mlp.expr:add{ "i", builder = || +{complex(1)} }

-- Fabien.

[Jaco]

Ok, that makes sense - thanks!

By the way, I've made the builder more strict so that only numbers are allowed to be suffixed with "i". The builder now gives an error if the operand is not a number as shown below. Is this the correct way to handle such errors, or is there a better/nicer way to fail?

-{ block:

-{ extension "match" } 

 

function complex_builder(t)

match t with

| `Number{ n } -> return +{complex(-{`Number{ n }})}

| _ -> error("Invalid use of operator i")

end

end

 

mlp.lexer:add "i"

mlp.expr.suffix:add{ "i", prec = 90, complex_builder }

mlp.expr:add{ "i", builder = || +{complex(1)} }

 

 }

table.print( +{5i} )

table.print( +{xi} )

table.print( +{i} )

[Fabien Fleutot]

On Wed, Oct 19, 2011 at 2:12 PM, Jaco <jvdm...@emss.co.za> wrote:

Ok, that makes sense - thanks!

By the way, I've made the builder more strict so that only numbers are allowed to be suffixed with "i". The builder now gives an error if the operand is not a number as shown below. Is this the correct way to handle such errors, or is there a better/nicer way to fail?

-{ block: [snip] }

Seems fine if you want to forbid it, but I don't get why you want to reject stuff like "a i + b" or "cos(phi) + sin(phi) i". 

[Jaco]

The preferred way to write your examples would be:

a*i + b

cos(phi) + sin(phi)*i

exp(i*t)

This is the way Matlab does it too, so I'd rather keep the stricter checking for now.

[Jaco]

Ok, I've run into a bit of a problem here. One of my requirements is that the extended Lua syntax must still be compatible with standard Lua. By reserving "i" like this you use the ability to have a variable "i" and that is fairly common in many Lua programs. I have thus taken the approach used by Scilab (which is a Matlab "clone") where all constants are prefixed with %, e.g. "%i" or "%pi". My implementation is now the following:

-{ block:

mlp.lexer:add "i"

mlp.expr.suffix:add{ "i", prec = 90, |a,_| +{complex(-{a})} }

 

-- Needed to add this so the variable "i" can be used in expressions 

mlp.expr:add{ "i", builder = || +{i} } 

mlp.lexer:add "%"

mlp.expr.prefix:add { "%", prec = 100, builder = |_, a| `Index{ +{const}, `String{ a[1] } } }

 }

const = {

i  = complex(1),

pi = 3.14159265

}

table.print( +{5i} )

table.print( +{xi} )

table.print( +{%i} )

table.print( +{i} )

The above approach allows me to use the variable "i" in expressions. The problem I've run into is when I try to assign to a variable "i" as follows:

i = 1          // gives error "Invalid use of operator i"

local i = 1    // gives error "Identifier expected"

t = { i = 1 }  // this works

t = {}

t.i = 1        // gives error "Identifier expected"

Can you provide any suggestions on how to make assignments to "i" work in this context?

[Fabien Fleutot]

My overall feeling is that you're trying to violate Lua's spirit for something that isn't worth it. Lua grammar is simple, relies on few rules and little context, that's part of what makes Metalua workable.

Remember that the only reason why you're making "i" so special, is that you want to be able to write "2+3i" instead of "2+3*i". Is it worth messing up the whole grammar for that? If you try hard enough, you can get this to work, but your grammatical changes will have affected the whole language, and will interfere with most other extensions. You'll have lost modularity.

This being said, if you think it's really important to support "2+3i", you can do it in less disruptive ways:

• you can change the number lexer to accept suffix i;
• you can change the default expression suffix parser.

I'll outline how to tackle the latter (the lexer hasn't been written with the intent of been modified on the fly). The expression suffixes are handled by a multisequence, i.e. there can be no more than one suffix parser which doesn't start with a keyword. Notice btw that this limitation wouldn't be hard to overcome; but it discourages you from writing ambiguous grammars, so it's not a limitation, it's a feature :) Anyway, in our case, the non-keyword expression suffix parser is already used to handle string arguments without parentheses, such as <<some_function  "string_arg">>. Look in mlp_expr.lua:

   suffix = { name="expr suffix op",

      [...]

      default = { name="opt_string_arg", parse = mlp.opt_string, builder = function(f, arg) 

         return {tag="Call", f, arg } end } } }

You want to replace it with a suffix parser which accepts both strings, and the identifier "i". This way we do NOT declare "i" as a keyword, and it will still be accepted as variable or field name. I think something like this should work (untested code):

local d = mlp.expr.suffix.default

function d.parse(lx)

    match lx:peek() with

    | `String{ ... } | +{i} -> return lx:next()

    | _ -> return nil

    end

end

function d.builder(expr, suffix)

    match expr, suffix with

    | `Number{...}, +{i} -> return +{complex(-{expr})}

    | _, +{i} -> error "only use 'i' suffix on numbers"

    | _, `String{...} -> return `Call{ expr, suffix }

    end

end

One  important closing remark: you say you want to remain backward compatible with plain Lua, but you've already lost that with your use of single quotes as suffixes, rather than string delimiters. No Lua program using single quote delimited strings will be accepted by your dialect. And there's no reasonable way around this: f' * g' is a valid expression both in plain Lua and in your dialect, yet means something different.

[Jaco]

Thanks Fabien for your detailed response. I agree with you - such a change is not worth it. It would have been nice syntactic sugar, but I never realised how many side effects would be involved. This is an exploration for me and trying ideas out. Some will be discarded along the way. For now I've decided to discard this approach and to rather just use the "%" prefix operator for constants. Complex values can then be written as "2 + %i*3", or "a + %i*b". That is a completely acceptable and valid solution and is also very similar to what Scilab provides.

[Jaco]

I tried the approach you suggested above, but ran into problems because any statement like "i = ..." causes problems because the i is seen as suffix for the previous line/statement. I then tried the option of changing the number lexer. It works quite well. Here is my implementation:

--------------------------------------------------------------------------------

-- Number extractor modified to include complex number notation

--------------------------------------------------------------------------------

function lexer.lexer:extract_number()

   -- Number

   local imag

   local j = self.src:match(self.patterns.number_hex, self.i)

   if not j then

      j = self.src:match (self.patterns.number_mantissa[1], self.i) or

          self.src:match (self.patterns.number_mantissa[2], self.i)

      if j then

         j = self.src:match (self.patterns.number_exponant, j) or j;

         imag = self.src:match ("^[ij]()", j);

      end

   end

   if not j then return end

   -- Number found, interpret with tonumber() and return it

   local n = tonumber (self.src:sub (self.i, j-1))

   if not imag then

      self.i = j

      return "Number", n

   else

      self.i = imag

      return "Imag", n

   end

end

--------------------------------------------------------------------------------

-- Default parser for primary expressions modified to support imaginary numbers

--------------------------------------------------------------------------------

function id_or_literal (lx)

   local a = lx:next()

   if not (a.tag=="Id" or a.tag=="String" or a.tag=="Number" or a.tag=="Imag") then

      local msg

      if a.tag=='Eof' then

         msg = "End of file reached when an expression was expected"

      elseif a.tag=='Keyword' then

         msg = "An expression was expected, and `"..a[1]..

               "' can't start an expression"

      else

         msg = "Unexpected expr token " .. _G.table.tostring (a, 'nohash')

      end

      gg.parse_error (lx, msg)

   end

   if a.tag=="Imag" then

      return {tag="Call", {tag="Id", "complex"}, {tag="Number", 0}, {tag="Number", a[1]}}

   else

      return a

   end

end

-- Replace the default id_or_literal parser with the new one

mlp.expr.primary.default = id_or_literal

 

[Fabien Fleutot]

On Mon, Oct 31, 2011 at 2:41 PM, Jaco <jvdm...@emss.co.za> wrote:

I tried the approach you suggested above, but ran into problems because any statement like "i = ..." causes problems because the i is seen as suffix for the previous line/statement.

First, some generic comments:

• You're hit by a pretty common problem: Lua is designed so that it's almost always possible to guess where statements start and finish, without having to look for ";" statement separators, thus making them optional. Unfortunately, this property is maintained, in plain Lua syntax, through a lot of small, unassuming design details, which are easy to unwillingly break. If I were to design Metalua independently from plain Lua, I'd certainly make ";" separators mandatory.
  
  It would also make sense to make a syntax extension whose only purpose is to make semicolons mandatory: other extensions which break the guessable-end-of-statements property would  load it, and users would know that if they want one of these super-fancy extensions, the price to pay is to put semicolons everywhere in the files using them.
  
  Notice that Lua itself sometimes has problems: Lua 5.1 will refuse "f(x)\n(g)(x)", because it's not sure whether it should be understood as "f(x); g(y)" or as "f(x)(g)(y)" (f being a third degree function). To do that, Lua check for line returns: it thus mixes lexing concerns with parsing ones, which is pretty ugly.
  
  
• Your experiments are interesting, but from a pragmatic PoV, I think that sticking to a global definition "i = imag(1)" and forcing to write "2+3*i" instead of "2+3i" would be more user-friendly in the long term: you're less likely to create weird shenanigans, it only adds 1 character in complex constants definitions, and it keeps your grammar simpler.
  
  
• Metalua's lexer is not intended to be modified, although it happens to be possible. A big challenge in extensible languages is extensions composition: several, independently defined extensions should work well when combined together, at least when the grammar changes they define aren't formally incompatible. Hacks in the lexer are highly likely to stump on each others' toes, and as such should be avoided.

Now for your specific problem: the reason why you go and hack the parser is that you want no space between the number and the "i". Actually, I think what you don't want is a "\n", but "2 + 3 i" should be accepted. This can be done at the parsing stage, because tokens and tree nodes carry on lineinfo elements: tree elements have a lineinfo field, with two sub-fields first and last, which each contain a list of 4 elements:

• the line number
• the column number within the line
• the character offset, from the beginning of the source file/string
• the name of the source file

This gives you the locations of the first and last characters of any given node in an AST. this means that you can, in the suffix builder, check that the "i" token is on the same line as the number it suffixes. Unfortunately, we hit a limitation in Metalua here,  mostly because we mix lexing and parsing in a pretty filthy way: 

• when parsing the suffix, we don't have access to the suffixed expression yet, so we have to decide whether we parse the "i" depending on whether it's on the same line as the number
• when  building the resulting expression, if "i" is not on the same line as the number, it's too late to "unparse" the "i", at least without committing unspeakable hacks.

So if we find something such as "x=3\ni=4", we'll have an error such as "i suffix must be on the same line as the number; if i is meant to be a variable, had a semicolon". You'd force the user to write"x=3;\ni=4" in this case. It seems acceptable to me, and that's what Lua 5.1 does when facing "f(x)\ng(y)"ambiguities: force the user to remove ambiguities rather than silently picking one interpretation for him.

function d.builder(expr, suffix)

    match expr, suffix with

    | `Number{last={n1,...}, ...}, `Id{ first={n2,...} "i" },  ->

        if n1==n2 then return +{complex(-{expr})}

        else error "put the i suffix on the same line as the number" end

    | _, +{i} -> error "only use 'i' suffix on numbers"

    | _, `String{...} -> return `Call{ expr, suffix }

    end

end

However, let me repeat myself: in my opinion, all this shows that the "i suffix" syntax extension isn't worth it. My rule of thumb to evaluate a potential extension's worth is: "will it modify the way I think of my code when I'll use it?". Avoiding a "*" in complex constants is not worth any mess.

[Jaco]

Thanks again for your insightful and detailed comments. I view what I am currently doing purely as experiments - that's the best way to learn and to see where the problem areas are. What the final version of the numerical Lua (or Lualab as I call it) will look like depends very much on the outcomes of these experiments.