the line number of token is error (antlr-4.5.3)

[Lu Taoxxx]

I have a file, there are "0D 0D 0A" (\r\r\n) at the end of each line. Editor tool(Nodepad++) regards it as two lines, but from 'ANTLR' point view, this is one line.

How to change ANTLR to follow editor tool rule. (the version of ANTLR is 4.5.3)

[Jim Idle]

Your file is in an invalid format. Notepad++ is basically just fooled and not knowing quite what to do, it displays as two lines. 

Such things happen sometimes when people check files out from VCS on say Unix, then copy them to windows without a good conversion.

The solution is to fix your file as there is nothing wrong with ANTLR. If for some reason you cannot fix this file, then you will have to track the line numbers yourself with something like:

   EOL: '\r' '\n'?  { incrementLine(); } ;

Then override the error reporting.

But as 0D was only ever valid for EOL on old MacOS systems, and 0D 0A is Windows, then your file is not a valid format for any system ever ;). Fix your file, and save yourself a lot of problems.

If you use gvim you can do it with one :s command. If you have a lot of files, then a short program should do it.

Jim

On Thu, May 19, 2016 at 11:06 AM, Lu Taoxxx <luya...@gmail.com> wrote:

I have a file, there are "0D 0D 0A" (\r\r\n) at the end of each line. Editor tool(Nodepad++) regards it as two lines, but from 'ANTLR' point view, this is one line.

How to change ANTLR to follow editor tool rule. (the version of ANTLR is 4.5.3)

--
You received this message because you are subscribed to the Google Groups "antlr-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to antlr-discussi...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lu Taoxxx]

I can't change the file. and "incrementLine()" isn't ANTLR builtin API. 

So the workaround solution I used is "read all lines using JDK api, and join them using '\n'".

BufferedReader bufferedReader = new BufferedReader(reader);
StringBuilder sb = new StringBuilder();
String line;
while ((line = bufferedReader.readLine()) != null) {
    sb.append(line);
    sb.append('\n');
}
in = new ANTLRInputStream(sb.toString());

在 2016年5月19日星期四 UTC+8下午3:40:34，Jim Idle写道：

[Jim Idle]

It’s much easier than that I think. Though I still think you shoudl find out the root cause of the file format error and fix that, all you need do is write your own inputstream that extends the supplied one. In the LA() method, just throw away the 0x0D characters, which means your parser will only see LF for EOL and everything will work as intended.

Something like this

package com.armorize.vicara.compiler.toolchain;

import java.io.IOException;

import org.antlr.v4.runtime.ANTLRFileStream;

public class MyInputStream extends ANTLRFileStream {

    public MyInputStream(String fileName) throws IOException {
        super(fileName);
    }

    @Override
    public int LA(int i) {

        int c;

        while ((c = super.LA(i)) == 0x0D) {
        }

        return c;
    }
}

This is also how I generally implement case in sensitivity (by returning all c in lowercase), instead of A: 'A' |'a';

Jim

​

[Lu Taoxxx]

Thanks, I think your solution is reasonable.

在 2016年5月20日星期五 UTC+8下午1:53:15，Jim Idle写道：