unicode regepxs

[Werner LEMBERG]

Tim,


it seems to me that pyggy is not ready yet to handle UTF-8 encoded
regexps. Any plan to implement this? I would need it urgently :-)


Werner

[Tim Newsham]

> it seems to me that pyggy is not ready yet to handle UTF-8 encoded
> regexps. Any plan to implement this? I would need it urgently :-)

I really dont know too much about UTF-8 or unicode. I supposed
the front end could take in characters from a file and use
pythons utf8-to-string conversion functions to convert it to
a unicode string, and then run that through the lexer (or have
the lexer do this automatically). However as it stands the
lexer tables are defined as arrays of 256 possibilities, for
8-bit characters. Updating the lexer generator to handle
unicode wouldnt be a trivial undertaking.

I will put it on my ToDo list to look at but it definitely wont
be done "soon".

> Werner

Tim Newsham
http://www.lava.net/~newsham/

[Tim Newsham]

> it seems to me that pyggy is not ready yet to handle UTF-8 encoded
> regexps. Any plan to implement this? I would need it urgently :-)

It also occurs to me that you could still write a lexer for
UTF-8 characters without having support in the lexer, since
UTF-8 is just a sequence of 8-bit characters. This would probably
be more work than writing it in a lexer generator that supported
UTF-8, but might not be that hard.

[Werner LEMBERG]

> It also occurs to me that you could still write a lexer for UTF-8
> characters without having support in the lexer, since UTF-8 is just
> a sequence of 8-bit characters. This would probably be more work
> than writing it in a lexer generator that supported UTF-8, but might
> not be that hard.

I think so too, but it becomes *very* ugly. For example, instead of
using

[äöü]

I had to write

\xc3(\xa4|\xb6|\xbc)

or something similar. Maybe you can provide a function which does
this conversion automatically.


Werner