HELP with unicode!!!!!

[dizzydoc]

Hi ,

i have lost a lot of time looking for a solution.

My problem is, while i a reading csv in python using

csv_file = open(path_file, "rb")
reader = csv.reader(csv_file)

whenever i perform any operation on the string returned in each cell i
get this error

*** UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0'
in position 34: ordinal not in range(128)

i tried decoding the value as

val.decode('utf-8')

...this helped to reduce a few encodings but gets stuck for
the encoding "\xa0"
Please HELP!

Thanks

[Alexandr Aibulatov]

Write at first line on your py file:
# -*- coding: utf-8 -*-

2012/5/1 dizzydoc <htc.kara...@gmail.com>:

> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to django-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-users?hl=en.
>

[Ian Clelland]

On Tuesday, May 1, 2012, dizzydoc wrote:

Hi ,

i have lost a lot of time looking for a solution.

My problem is, while i a reading csv in python using

csv_file  = open(path_file, "rb")
reader = csv.reader(csv_file)

whenever i perform any operation on the string returned in each cell i
get this error

*** UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0'
in position 34: ordinal not in range(128)

It's a long-standing limitation with the python csv module that it does not handle Unicode well. In fact, it's one of the first things mentioned in the docs:

 http://docs.python.org/library/csv.html

Further down that page, though, is a snippet of code that you should be able to use (search the page for "unicode_csv_reader")

If that doesn't work, come back here with some code that you've tried and hopefully we can find a solution

Ian

i tried decoding the value as

val.decode('utf-8')

...this helped to reduce a few encodings but gets stuck for
the encoding "\xa0"
Please HELP!

--
Regards,
Ian Clelland
<clel...@gmail.com>

[JHeasly]

Also, have a look at this slidedeck: http://farmdev.com/talks/unicode/

It's from a PyCon 2008 talk, "Unicode In Python, Completely Demystified" A video of the talk is at http://www.youtube.com/watch?v=cecJ9o5GGmw, but the audio's miserable, and the slides are pretty good, so it may be all you need to get your head around what's going on.

Also, in your code example, the error says UnicodeEncodeError, not Decode, so try val.encode('utf-8')(not val.decode('utf-8')).

— John

[Kejun He]

Hi,

  it may because of the csv file sometimes, try to save the file as utf-8 or utf-8 no-bom.