Retrieving the underlying source text

[Niels Basjes]

Hi,

I wrote a parser ( https://github.com/nielsbasjes/yauaa ) and when I simply skip the spaces in my lexer the getText() method no longer includes these skipped tokens.

Effectively the input string "foo  bar" is output as "foobar" by getText.

So for my project (where in some cases I need the original including the spaces) I wrote a getSourceText function that retrieves the substring of the underlying source instead of concatenating the parsed tokens (like getText() does).

As a test I implemented this first in my project 

See https://github.com/nielsbasjes/yauaa/blob/SPACE_Skip/src/main/java/nl/basjes/parse/useragent/utils/AntlrUtils.java#L32 

And to see if it would be a usable fit I implemented it as a new method in antlr4 itself.

See: https://github.com/nielsbasjes/antlr4/commit/b1ce20cd00dd59d6a00b35526a70bd9958fd7947

Now before I put effort into writing tests and ensuring the code is clean: 

Is this an enhancement that is desirable in ANTLR?

Or is this a call that should not go into ANTLR a be a part of the specific application?

Niels Basjes