Problem with tokenisation?

[Dominique de Waleffe]

Given the tokens:

#token COMMA ","
#token DOT "."
#token FLOAT "[0-9]{[.,][0-9]+}"
#token ICODE "[0-9][0-9][0-9][0-9][0-9][0-9]"
#token NUMBER "[0-9]+"

I would expect the following:

123456, 35678.

to yield the tokens ICODE COMMA NUMBER DOT
(ignoring the spaces).

However, I get errors saying "123456," is an invalid token and the same
for "35678.".

I seems strange to me, but then, I'm new to PCCTS.
Can someone explain what I'm doing wrong or am expecting wrongly?

Thanks a lot,
Dominique

[Scott Stanchfield]

For starters, you need to write a "space" rule, such as

#token "[\ \t]"

-- Scott

--
Scott Stanchfield McCabe & Associates -- Columbia, Maryland
(These are not necessarily the opinions of McCabe & Associates)
Visit McCabe & Associates at http://www.mccabe.com

[Dominique de Waleffe]

Scott Stanchfield wrote:
>
> For starters, you need to write a "space" rule, such as
>
> #token "[\ \t]"

> I wrote

> > However, I get errors saying "123456," is an invalid token and the same
> > for "35678.".

> --
> Scott Stanchfield McCabe & Associates -- Columbia, Maryland

The problem happens even with a space token #token SPACE "\ \t" so I
don't think the problem lies there. (I also call zzkip() for spaces).

Those token defs are part of a larger grammar
Any other idea?

Dominique

[John D. Mitchell]

In article <3118A5...@miscrit.be>, Dominique de Waleffe wrote:
>Scott Stanchfield wrote:
>> For starters, you need to write a "space" rule, such as
>> #token "[\ \t]"

[...]

>The problem happens even with a space token
>#token SPACE "\ \t" so I
>don't think the problem lies there. (I also call zzkip() for spaces).

Your definition of SPACE if very different from Scott's. Try again.

Take care,
John