Negative numbers in text/scanner - is this a bug or by design?

[dvi...@gmail.com]

Hi.

I've run into this little problem using the text/scanner package, and wanted to ask if it's a bug or not. 

Basically, since this package parses ints and floats, I figured negative ints and floats are fine. Turns out the minus sign is read as a separate token before the number. 

Example:

func main() {
        var s scanner.Scanner
        s.Init(strings.NewReader("Foo 1 2.5 -3 -0.54"))
        s.Mode = scanner.GoTokens
        for tok := s.Scan(); tok != scanner.EOF; tok = s.Scan() {
                fmt.Println("Tok: ", s.TokenText())
        }
}

Outputs:

Tok:  Foo
Tok:  1
Tok:  2.5
Tok:  -
Tok:  3
Tok:  -
Tok:  0.54

If it's not a bug it should be documented IMHO. 

Thanks!

[Rob Pike]

That package is intended as a scanner for Go tokens ("literals as defined by the Go language"), and in Go the minus sign is a separate token.

-rob

[dvi...@gmail.com]

Thanks Rob, I saw that the default was the "go tokens", but figured it was just a default in a generic library. 

So maybe it's worth making this behavior configurable, among the other flags of the scanner?  

I found it very very useful and easy for parsing a tiny DSL to query some data structures (yay Greenspun's 10th rule! ;-) ), so I assure you it's a nice utility beyond parsing go text.

BTW It resides under the text/ package, and there is a go/scanner package, this makes it a bit confusing.

[Jan Mercl]

On Sat, Aug 3, 2013 at 3:58 PM, <dvi...@gmail.com> wrote:
> I found it very very useful and easy for parsing a tiny DSL to query some
> data structures (yay Greenspun's 10th rule! ;-) ), so I assure you it's a
> nice utility beyond parsing go text.

Design the tokenizer of your DSL to expect ]unary] '+'/'-' as a
separate token as well, which is pretty standard anyway[0]; and you're
good to go.

[0] Eg.: http://www.lysator.liu.se/c/ANSI-C-grammar-l.html

-j

[Jim Robinson]

I think Rob's point was that the implementation really was specific

to parsing Go tokens, that it was not intended to be generic, so I

am guessing that it's unlikely he's going to want to move it.

[Dvir Volk]

On Saturday, August 3, 2013 5:42:28 PM UTC+3, Jan Mercl wrote:

On Sat, Aug 3, 2013 at 3:58 PM,  <dvi...@gmail.com> wrote:
Design the tokenizer of your DSL to expect ]unary] '+'/'-' as a
separate token as well, which is pretty standard anyway[0]; and you're
good to go.

that is of course what I did, I just found the behavior of the package not to be what I expected intuitively, and thought it might be a bug.

[Rob Pike]

It's a bit of an moribund package, not used much and showing its age.

-rob

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.