Syntax extension for Matlab 'end' keyword when used in matrix indexing

[Jaco]

I'm trying to see if I can add something similar to the Matlab 'end' keyword. The following shows how it works in Matlab:

// Create a 3 x 4 matrix

M = [1 2 3 4; 5 6 7 8; 9 10 11 12]

// Index last row of 2nd column

M(end,2)

// Index last column of 1st row

M(1,end)

// Zero last 3 columns of 2nd row

M(2, end-2:end) = 0

// Add a row to the bottom

M(end+1, :) = [13 14 15 16]

What happens internally is that Matlab calls the end(obj, k, n) method on the matrix, where obj is the matrix, k is the index position where end is used and n is the number of indices in the indexing expression. For example, M(end,2) results in M(end(M,1,2), 2) and M(1,end) results in M(1, end(M,2,2)).

The use of the 'end' keyword is problematic in Lua. I'd like to use dollar ($) as the keyword which is similar to what Scilab does for denoting the last index for a particular dimension. The dollar sign does not interfere with existing Lua syntax. For now I've decided to use the function call as the indexing mechanism so that it reads exactly like Matlab. The extended Lua syntax should then read as follows. Note that $ can be used in expressions too.

// Index last row of 2nd column

v = M($,2)

// Index last column of 1st row

v = M(1,$)

// Index 2nd last column of 1st row

v = M(1,$-1)

For the Lua implementation the use of $ inside an indexing expression should be transformed into a end_index(k,n) method invocation on the matrix object, where k is the index position and n is the number of indices in the indexing expression. The above examples will then be translated into the following:

// Index last row of 2nd column

v = M( M:end_index(1,2), 2 )

// Index last column of 1st row

v = M( 1, M:end_index(2,2) )

// Index 2nd last column of 1st row

v = M(1, M:end_index(2,2)-1 )

From my existing knowledge about Metalua I can see two possible ways of approaching this:

1. Replace the standard function call builder in the expression parser with a new one that traverses down all its arguments and does the necessary substitution of $ when found. This traversal should stop on operands which are function calls. This implies that $ only works within the nearest function call scope and not further up. I view this as an acceptable limitation.
2. Do a post-lexing transform before the AST is used to generate bytecode. Maybe one can query the resulting AST with the new query language for all occurrences of the $ keyword. For each one found, the AST structure can be traversed upwards until the first function call node is found. When found  all the necessary information is available to replace the $ keyword with its correct end_index() method invocation. 

I would appreciate any feedback and/or comments on my implementation ideas above.

[Fabien]

The idiomatic Lua way to do that is to dynamically support negative indexes: -1 is the last row/column, -2 the one just before etc. Wouldn't that solve your problem nicely, without need for any syntax wizardry?

[Jaco]

Thanks, great suggestion. It's much simpler and more intuitive than the additional syntax elements.

[Jaco]

Darn, after looking at the different use cases the use of negative indexes is not a solution for some cases. The two problematic cases are when the last index value is used in ranges or refer to beyond the current range. Below are examples. I use the double colon :: as the range operator, where 1::3 results in {1, 2, 3}. The use of :: on its own implies all rows or columns.

// Extract 2nd column of matrix excluding first two rows

A = B(3::$, 2)

// Add a row at the bottom of matrix

M($+1, ::) = {13, 14, 15, 16}

The adding of a row or a column to a matrix is a very typical use case. The above two examples cannot be solved with negative indexes. Lets look at each case individually.

Ranges

The range 1::-1 is an invalid range. A possible solution is to use notation like k:: to indicate from k to the end in increments of 1, and k::i:: to indicate from k to the end in increments of i. To make it symmetrical one can also use ::n to indicate from the start to n, but the start will always be 1, so its not really necessary. In my current implementation the :: operator returns a range object and not the expanded list. In this case it can return a range object with a special value for the end value which indicates "to the end". The above example will then read as follows

// Extract 2nd column of matrix excluding first two rows

A = B(3::, 2)

Adding rows/columns

One cannot refer to 1 beyond the end (or k beyond the end in general) using negative indexes. A possible solution is to introduce a special prefix operator like #+ that means "the end plus ...". The # reminds of the table size operator of Lua. This operator will return a special index object which can be interpreted at run-time by the matrix object. The will look as follows:

// Add a row at the bottom of matrix

M(#+1, ::) = {13, 14, 15, 16}

Referring to the end row/column or before

For this case the negative indexes will work well, for example:

// Replace the second-last row

M(-2, ::) = {13, 14, 15, 16}

Any comments or suggestions?

[Fabien]

On Thu, Oct 27, 2011 at 11:47 AM, Jaco <jvdm...@emss.co.za> wrote:

Any comments or suggestions?

When you can avoid macros, avoid them. It seems to me that what you want could be achieved with a special value end_of_range, which supports __add and __sub metamethods. I'd suggest that you implement your solution that way, without syntax extension, so that

1. you figure out all the details hard to anticipate without actually implementing
2. you can assess how much syntax extension is actually required to get a usable language.

Premature focus on superficial syntax is a sure recipe to pointless nitpicking, and missing the elephants in the room; start with a robust back-end.