Spanish tokenizer
David T.
I'm implementing a es_ES tokenizer for pyenchant and need some help.
I have copyed the en_EN.py to es_ES.py and I have added more
valid_chars like "á", "é", "í", "ó", "ú" but when I'm going to use it
I obtain a beautiful Traceback like that:
File "/usr/lib/pymodules/python2.6/enchant/tokenize/es.py", line 107,
in _consume_alpha_utf8
u = text[offset:offset+incr].decode("utf8")
AttributeError: 'array.array' object has no attribute 'decode'
It's happend when evaluating words containing some accentuated char.
Please help me and show me what more I have to do for to build a right
tokenizer for my language.
Thank's in advance.
Best Regards,
David T.
Ryan Kelly
> Hello all,
>
> I'm implementing a es_ES tokenizer for pyenchant
Great. Would you consider contributing it for inclusion in the main
distribution?
> and need some help.
>
> I have copyed the en_EN.py to es_ES.py and I have added more
> valid_chars like "á", "é", "í", "ó", "ú" but when I'm going to use it
> I obtain a beautiful Traceback like that:
>
>
> File "/usr/lib/pymodules/python2.6/enchant/tokenize/es.py", line 107,
> in _consume_alpha_utf8
> u = text[offset:offset+incr].decode("utf8")
> AttributeError: 'array.array' object has no attribute 'decode'
>
> It's happend when evaluating words containing some accentuated char.
>
> Please help me and show me what more I have to do for to build a right
> tokenizer for my language.
Looks like a bug in my _comsume_alpha_utf8 implementation - it assumes
the input is a str instance but it might be e.g. a mutable character
array. I just added a test case to reproduce it in the english
tokeniser, and the corresponding bugfix.
You can grab the updated file from github here:
http://github.com/rfk/pyenchant/blob/master/enchant/tokenize/en.py
Or if you want just the diff, view it here:
http://github.com/rfk/pyenchant/commit/7fa60b5c45d191a7f518ae4c9ef31c98e54f1dfe
Cheers,
Ryan
--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
ry...@rfk.id.au | http://www.rfk.id.au/ramblings/gpg/ for details
David T
On Thu, 2010-09-02 at 05:58 -0700, David T. wrote:Great. Would you consider contributing it for inclusion in the main
> Hello all,
>
> I'm implementing a es_ES tokenizer for pyenchant
distribution?
Looks like a bug in my _comsume_alpha_utf8 implementation - it assumes
> and need some help.
>
> I have copyed the en_EN.py to es_ES.py and I have added more
> valid_chars like "á", "é", "í", "ó", "ú" but when I'm going to use it
> I obtain a beautiful Traceback like that:
>
>
> File "/usr/lib/pymodules/python2.6/enchant/tokenize/es.py", line 107,
> in _consume_alpha_utf8
> u = text[offset:offset+incr].decode("utf8")
> AttributeError: 'array.array' object has no attribute 'decode'
>
> It's happend when evaluating words containing some accentuated char.
>
> Please help me and show me what more I have to do for to build a right
> tokenizer for my language.
the input is a str instance but it might be e.g. a mutable character
array. I just added a test case to reproduce it in the english
tokeniser, and the corresponding bugfix.
You can grab the updated file from github here:
http://github.com/rfk/pyenchant/blob/master/enchant/tokenize/en.py
Or if you want just the diff, view it here:
http://github.com/rfk/pyenchant/commit/7fa60b5c45d191a7f518ae4c9ef31c98e54f1dfe
Ryan Kelly
Just to clarify, does it work when you pass a unicode string to
SpellChecker rather than a bytestring?
> Other problem is tokenizeing, when is checking some at the
> end accentuated word the tokenizer discard the last char how in this
> example:
>
>
> word: sentenció
> token: sentenci (missing ó)
This is expected if you're simply adding characters to valid_chars -
it's designed for things like apostrophes and hyphens that can appear
within a word but at the start or end of a word.
Instead you'll probably need to modify the various _consume_alpha
methods to make them recognise the additional characters.
> and then I get this traceback:
>
>
> Traceback (most recent call last):
> File "/usr/lib/pymodules/python2.6/enchant/checker/__init__.py",
> line 246, in next
> if self.dict.check(word):
> File "/usr/lib/pymodules/python2.6/enchant/__init__.py", line 550,
> in check
> self._raise_error()
> File "/usr/lib/pymodules/python2.6/enchant/__init__.py", line 524,
> in _raise_error
> raise eclass(default)
> enchant.Error: Unspecified Error
>
>
> I'm lost, please help.
Please send through your modified tokenizer, and any testcases that
you've been using. I'll try to take a look tomorrow.
Ryan
> --
> You received this message because you are subscribed to the Google
> Groups "pyenchant users" group.
> To post to this group, send email to pyencha...@googlegroups.com.
> To unsubscribe from this group, send email to pyenchant-users
> +unsub...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/pyenchant-users?hl=en.
David T
This is expected if you're simply adding characters to valid_chars -
> Other problem is tokenizeing, when is checking some at the
> end accentuated word the tokenizer discard the last char how in this
> example:
>
>
> word: sentenció
> token: sentenci (missing ó)
it's designed for things like apostrophes and hyphens that can appear
within a word but at the start or end of a word.
Instead you'll probably need to modify the various _consume_alpha
methods to make them recognise the additional characters.
Please send through your modified tokenizer, and any testcases that
> and then I get this traceback:
>
>
> Traceback (most recent call last):
> File "/usr/lib/pymodules/python2.6/enchant/checker/__init__.py",
> line 246, in next
> if self.dict.check(word):
> File "/usr/lib/pymodules/python2.6/enchant/__init__.py", line 550,
> in check
> self._raise_error()
> File "/usr/lib/pymodules/python2.6/enchant/__init__.py", line 524,
> in _raise_error
> raise eclass(default)
> enchant.Error: Unspecified Error
>
>
> I'm lost, please help.
you've been using. I'll try to take a look tomorrow.
Ryan Kelly
>
>
> Just to clarify, does it work when you pass a unicode string
> to
> SpellChecker rather than a bytestring?
>
>
> Yes, it is. But I repeat you I'm lost with this code/decode staffs,
> maybe I'm doing something wrong.
I'm not even sure I totally understand unicode handling myself. This
article "all about python and unicode" might help:
http://boodebr.org/main/python/all-about-python-and-unicode
As a general rule, if your text has characters outside the ASCII range
you're much better off keeping it as a unicode string.
If you need to encode it to a bytestring, then pyenchant requires it to
be encoded in UTF8.
> But the en_EN tokenizer works well, only fail when finish with last
> accentuated words.
>
> Could you give me an example of how to do this and what controls the
> end of a word in the EN tokenizer please?.
The guts of this are the "next" method on the tokenizer class. Here is
the important bit of the loop, which locates the start and end of a
word:
# Find start of next word (must be alpha)
while offset < len(text):
# If this returns non-zero then the next
# character is alphabetic.
incr = self._consume_alpha(text,offset)
if incr:
break
offset += 1
curPos = offset
# Find end of word using, allowing valid_chars
while offset < len(text):
# If this returns zero, the next character
# is non-alphabetic.
incr = self._consume_alpha(text,offset)
if not incr:
if text[offset] in self._valid_chars:
incr = 1
else:
break
offset += incr
This basically skips over characters until it finds one that is "alpha"
- i.e. it belongs to a word. It then skips over characters looking for
one that's not alpha, and the result is a word.
So if a character is being ignored, it's probably because the
_consume_alpha method is not recognising it.
> > and then I get this traceback:
> >
> >
> > Traceback (most recent call last):
> > File
> "/usr/lib/pymodules/python2.6/enchant/checker/__init__.py",
> > line 246, in next
> > if self.dict.check(word):
> > File "/usr/lib/pymodules/python2.6/enchant/__init__.py",
> line 550,
> > in check
> > self._raise_error()
> > File "/usr/lib/pymodules/python2.6/enchant/__init__.py",
> line 524,
> > in _raise_error
> > raise eclass(default)
> > enchant.Error: Unspecified Error
> >
> >
> > I'm lost, please help.
Hmmm, this looks like you're passing invalid data into enchant. My
guess is that it's corrupted utf8 data.
Are you sure you're encoding your unicode strings properly into utf8?
> Please send through your modified tokenizer, and any testcases
> that you've been using. I'll try to take a look tomorrow.
>
>
> Sorry but I'm using pyenchant from an application and testing from
> this.
> I do that in this way:
>
>
> self._checker = SpellChecker(language)
> self._checker.next()
> self._checker.word
>
>
> An example text could be:
> """En esta- acts al arrezndadur, entrega al arrndadur, Ia cantidad de
> 160% an cnnceptc dc fianza la cual sera davuelta al arra·n~datari¤ a
> la tinalizacién da esta ccntratc, qua se antandaré pmrrcgadu pur
> anuaiidades sicmpra que subsista la necesidad de su ccupaciém pm parte
> dal arrandatariu."""
>
>
> All works ok less with the word "antandaré" that mark only "antandar"
> and obviate the "é". After check this word, when I correct or ignore
> this word and do next again, then fails.
I have tried this text with the default english tokenizer and it appears
to work OK. Can you please confirm whether using the english tokenizer
produces the same results for you? i.e. try something like:
>>> from enchant.tokenize.en import tokenize
>>> print list(tokenize("hello antandaré world"))
[('hello', 0), ('antandar\xc3\xa9', 6), ('world', 17)]
>>>
> I hope you can help me/guide me or something. I think only need to set
> this letters for my language: á é í ó ú but don't know where I have to
> put in the tokenizer code, is difficult for me, sorry.
Sorry, I know the tokenizer code is much more complicated than it needs
to be. If I were to start again from scratch I think I'd do it
differently and try to simplify things a little, but it works for now so
I'm hesitant to touch it :-)
Cheers,
Ryan
projetmbc
I'm interesting by the Spanish tokenizer so as to have just little
changes to produce a French tokenizer.
Is-it possible to have a copy of the code to see if I could find time
to do it rapidly ?
David T
Ryan Kelly
> Thank you very much for all your responses. I have tested better and
> there are no any problem now. No ignore last accentuated words and no
> any problems :-) Sorry for confusing you I was testing with a old
> version.
No worries, glad to hear it's working OK.
>
> I would like to build a es_ES tokenizer but the english works well for
> that and it's no neccesary. What do you think?
If there's no need for a language-specific tokenizer, I think it's
better to keep just the default english one. Less code to maintain is
always a plus :-)
projetmbc
work.
It could be usefull to have a call like get_tokenizer("standard") so
as to indicate that the tokenize can be used with several "latin"
languages.