beginner's AST problem

[peter lavalle]

I'm trying to use SableCC 3.3 to generate a parser with AST construction

minidsl.grammar is the grammar file, which should allow a list of variable declarations
MiniDSLTest.java tries to parse test01.minidsl but fails at character 1 of line 5 (the start of the token "matrix") where it expects "matrix"

Obviously I suspect the elves are changing my source code when I hit "run," but on the off chance that there's a more rational solution; I would like to ask if anyone has experience that will shed some light on how to move forward with this and make the JUnit test pass

[Phuc Luoi]

You can make JUnit test like

...
String validInput = "<whaterver you want parse>";
try{
PushbackReader reader = new PushbackReader(new
StringReader(validInput));
Lexer l = new Lexer(reader);
Parser p = new Parser(l);
Tree root = p.parse();
}catch(LexerException ex)
{
fail(validInpt + " is a valid input");
}catch(ParserException ex)
{
fail(validInput + " is a valid input");
}

[Phuc Luoi]

To your question: I think you must put the token k_matrix before the
token name in your grammar. Also:

Tokens

// copied from http://sablecc.sourceforge.net/grammars/ISOC.grammar.html
comment = '//' not_eol* eol;


blank = (cr | lf| tab| ' '|eol)+;

k_matrix = 'matrix';
name = (letter_lower|letter_upper) (letter_lower|letter_upper|numerial)*;

I thought it was problem with Lexer, so
I modified your test class to get out which token is the last token.




import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.PushbackReader;
import java.io.Reader;

import junit.framework.TestCase;
import compiler.minidsl.lexer.Lexer;
import compiler.minidsl.lexer.LexerException;
import compiler.minidsl.node.Start;
import compiler.minidsl.parser.Parser;
import compiler.minidsl.parser.ParserException;


public class MiniDSLTest extends TestCase {

public void test01Time() throws Exception {
try{
final Start parsedResource = parseResource("test01.minidsl");
}catch(ParserException ex)
{
String mes = "last token:" + ex.getToken().getClass().getName() + "
" + ex.toString();
throw new Exception(mes);
}
}

public Start parseResource(String string) throws ParserException,
LexerException, IOException {

return parseInputStream(getClass().getResourceAsStream(string));
}

public Start parseInputStream(InputStream inputStream) throws
ParserException, LexerException, IOException {
final Reader reader = new InputStreamReader(inputStream);
final PushbackReader pushbackReader = new PushbackReader(reader);
final Lexer lexer = new Lexer(pushbackReader);
final Parser parser = new Parser(lexer);

return parser.parse();
}
}

[Etienne Gagnon]

Hi Peter,

The declaration order of tokens is important with SableCC 3.3. The rules for matching tokens are:

1. The lexer finds the longest string that matches one token definition (or more).
2. If more than one token definition is involved, the first declaration is selected.

These are the rules used by traditional lexer generators (Lex/Flex).

An easy way to check that tokens are matched as you expect is to use a debug lexer. See this archive message:

http://lists.sablecc.org/pipermail/sablecc-user/msg00004.html

SableCC 4 is much better at handling this: it actually selects the right token definition (usually there's only one logical possibility) or, otherwise, issues a semantic error.

Have fun!

Etienne

Etienne Gagnon, Ph.D.
http://sablecc.org

On 2012-07-20 05:08, Phuc Luoi wrote:

To your question: I think you must put the token k_matrix before the
token name in your grammar. Also:
[...]

[peter lavalle]

Thanks, that sounds quite logical actually. I'll give that a shot when I get home.

[peter lavalle]

Yeah - that was it. Token ordering, now I know how to use it

Thank you very much