WTF? Printing unicode strings
Ron Garret
u'\xbd'
>>> print _
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
>>>
John Salerno
Not sure if this really helps you, but:
½
>>>
Fredrik Lundh
so stdout on your machine is ascii, and you don't understand why you
cannot print a non-ascii unicode character to it? wtf?
</F>
Ron Garret
Fredrik Lundh <fre...@pythonware.com> wrote:
I forgot to mention:
>>> sys.getdefaultencoding()
'utf-8'
>>> print u'\xbd'
Robert Kern
> I forgot to mention:
>
>>>>sys.getdefaultencoding()
>
> 'utf-8'
A) You shouldn't be able to do that.
B) Don't do that.
C) It's not relevant to the encoding of stdout which determines how unicode
strings get converted to bytes when printing them:
>>> import sys
>>> sys.stdout.encoding
'UTF-8'
>>> sys.getdefaultencoding()
'ascii'
>>> print u'\xbd'
½
--
Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco
Ron Garret
Robert Kern <rober...@gmail.com> wrote:
> Ron Garret wrote:
>
> > I forgot to mention:
> >
> >>>>sys.getdefaultencoding()
> >
> > 'utf-8'
>
> A) You shouldn't be able to do that.
What can I say? I can.
> B) Don't do that.
OK. What should I do instead?
> C) It's not relevant to the encoding of stdout which determines how unicode
> strings get converted to bytes when printing them:
>
> >>> import sys
> >>> sys.stdout.encoding
> 'UTF-8'
> >>> sys.getdefaultencoding()
> 'ascii'
> >>> print u'\xbd'
> 1⁄2
OK, so how am I supposed to change the encoding of sys.stdout? It comes
up as US-ASCII on my system. Simply setting it doesn't work:
>>> import sys
>>> sys.stdout.encoding='utf-8'
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: readonly attribute
>>>
rg
Robert Kern
> In article <mailman.5906.1147989...@python.org>,
> Robert Kern <rober...@gmail.com> wrote:
>
>>Ron Garret wrote:
>>
>>>I forgot to mention:
>>>
>>>
>>>>>>sys.getdefaultencoding()
>>>
>>>'utf-8'
>>
>>A) You shouldn't be able to do that.
>
> What can I say? I can.
See B).
>>B) Don't do that.
>
> OK. What should I do instead?
See below.
>>C) It's not relevant to the encoding of stdout which determines how unicode
>>strings get converted to bytes when printing them:
>>
>>>>>import sys
>>>>>sys.stdout.encoding
>>
>>'UTF-8'
>>
>>>>>sys.getdefaultencoding()
>>
>>'ascii'
>>
>>>>>print u'\xbd'
>>
>>1⁄2
>
> OK, so how am I supposed to change the encoding of sys.stdout? It comes
> up as US-ASCII on my system. Simply setting it doesn't work:
You will have to use a terminal that accepts UTF-8.
Serge Orlov
> In article <mailman.5906.1147989...@python.org>,
> Robert Kern <rober...@gmail.com> wrote:
>
> > Ron Garret wrote:
> >
> > > I forgot to mention:
> > >
> > >>>>sys.getdefaultencoding()
> > >
> > > 'utf-8'
> >
> > A) You shouldn't be able to do that.
>
> What can I say? I can.
>
> > B) Don't do that.
>
> OK. What should I do instead?
Exact answer depends on what OS and terminal you are using and what
your program is supposed to do, are you going to distribute the program
or it's just for internal use.
Ron Garret
"Serge Orlov" <Serge...@gmail.com> wrote:
I'm using an OS X terminal to ssh to a Linux machine.
But what about this:
>>> f2=open('foo','w')
>>> f2.write(u'\xFF')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
position 0: ordinal not in range(128)
>>>
That should have nothing to do with my terminal, right?
I just found http://www.amk.ca/python/howto/unicode, which seems to be
enlightening. The answer seems to be something like:
import codecs
f = codecs.open('foo','w','utf-8')
but that seems pretty awkward.
rg
Robert Kern
> I'm using an OS X terminal to ssh to a Linux machine.
Click on the "Terminal" menu, then "Window Settings...". Choose "Display" from
the combobox. At the bottom you will see a combobox title "Character Set
Encoding". Choose "Unicode (UTF-8)".
> But what about this:
>
>>>>f2=open('foo','w')
>>>>f2.write(u'\xFF')
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
> position 0: ordinal not in range(128)
>
> That should have nothing to do with my terminal, right?
Correct, that is a different problem. f.write() expects a string of bytes, not a
unicode string. In order to convert unicode strings to byte strings without an
explicit .encode() method call, Python uses the default encoding which is
'ascii'. It's not easily changeable for a good reason. Your modules won't work
on anyone else's machine if you hack that setting.
> I just found http://www.amk.ca/python/howto/unicode, which seems to be
> enlightening. The answer seems to be something like:
>
> import codecs
> f = codecs.open('foo','w','utf-8')
>
> but that seems pretty awkward.
<shrug> About as clean as it gets when dealing with text encodings.
Paul Boddie
>
> But what about this:
>
> >>> f2=open('foo','w')
> >>> f2.write(u'\xFF')
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
> position 0: ordinal not in range(128)
> >>>
>
> That should have nothing to do with my terminal, right?
Correct. But first try to answer this: given that you want to write the
Unicode character value 255 to a file, how is that character to be
represented in the file?
For example, one might think that one could just get a byte whose value
is 255 and write that to a file, but what happens if one chooses a
Unicode character whose value is greater than 255? One could use two
bytes or three bytes or as many as one needs, but what if the lowest 8
bits of that value are all set? How would one know, if one reads a file
back and gets a byte whose value is 255 whether it represents a
character all by itself or is part of another character's
representation? It gets complicated!
The solution is that you choose an encoding which allows you to store
the characters in the file, thus answering indirectly the question
above: encodings determine how the characters are represented in the
file and allow you to read the file and get back the characters you put
into it. One of the most common encodings suitable for the storage of
Unicode character values is UTF-8, which has been designed with the
above complications in mind, but as long as you remember to choose an
encoding, you don't have to think about it: Python takes care of the
difficult stuff on your behalf. In the above code you haven't made that
choice.
So, to answer the above question, you can either...
* Use the encode method on Unicode objects to turn them into plain
strings, then write them to a file - at that point, you are
writing specific byte values.
* Use the codecs.open function and other codecs module features to
write Unicode objects directly to files and streams - here, the
module's infrastructure deals with byte-level issues.
* If you're using something like an XML library, you can often pass a
normal file or stream object to some function or method whilst
stating the output encoding.
There is no universally correct answer to which encoding should be used
when writing Unicode character values to files, contrary to some
beliefs and opinions which, for example, lead to people pretending that
everything is in UTF-8 in order to appease legacy applications with the
minimum of tweaks necessary to stop them from breaking completely.
Thus, Python doesn't make a decision for you here.
Paul
Ron Garret
Robert Kern <rober...@gmail.com> wrote:
> Ron Garret wrote:
>
> > I'm using an OS X terminal to ssh to a Linux machine.
>
> Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
> from
> the combobox. At the bottom you will see a combobox title "Character Set
> Encoding". Choose "Unicode (UTF-8)".
It was already set to UTF-8.
> > But what about this:
> >
> >>>>f2=open('foo','w')
> >>>>f2.write(u'\xFF')
> >
> > Traceback (most recent call last):
> > File "<stdin>", line 1, in ?
> > UnicodeEncodeError: 'ascii' codec can't encode character u'\xff' in
> > position 0: ordinal not in range(128)
> >
> > That should have nothing to do with my terminal, right?
>
> Correct, that is a different problem. f.write() expects a string of bytes,
> not a
> unicode string. In order to convert unicode strings to byte strings without
> an
> explicit .encode() method call, Python uses the default encoding which is
> 'ascii'. It's not easily changeable for a good reason. Your modules won't
> work
> on anyone else's machine if you hack that setting.
OK.
> > I just found http://www.amk.ca/python/howto/unicode, which seems to be
> > enlightening. The answer seems to be something like:
> >
> > import codecs
> > f = codecs.open('foo','w','utf-8')
> >
> > but that seems pretty awkward.
>
> <shrug> About as clean as it gets when dealing with text encodings.
OK. Thanks.
rg
Serge Orlov
> In article <1147992722.9...@j73g2000cwa.googlegroups.com>,
> "Serge Orlov" <Serge...@gmail.com> wrote:
>
> > Ron Garret wrote:
> > > In article <mailman.5906.1147989...@python.org>,
> > > Robert Kern <rober...@gmail.com> wrote:
> > >
> > > > Ron Garret wrote:
> > > >
> > > > > I forgot to mention:
> > > > >
> > > > >>>>sys.getdefaultencoding()
> > > > >
> > > > > 'utf-8'
> > > >
> > > > A) You shouldn't be able to do that.
> > >
> > > What can I say? I can.
> > >
> > > > B) Don't do that.
> > >
> > > OK. What should I do instead?
> >
> > Exact answer depends on what OS and terminal you are using and what
> > your program is supposed to do, are you going to distribute the program
> > or it's just for internal use.
>
> I'm using an OS X terminal to ssh to a Linux machine.
In theory it should work out of the box. OS X terminal should set
enviromental variable LANG=en_US.utf-8, then ssh should transfer this
variable to Linux and python will know that your terminal is utf-8.
Unfortunately AFAIK OS X terminal doesn't set that variable and most
(all?) ssh clients don't transfer it between machines. As a workaround
you can set that variable on linux yourself . This should work in the
command line right away:
LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Or put the following line in ~/.bashrc and logout/login
export LANG=en_US.utf-8
Robert Kern
> In article <mailman.5917.1147995...@python.org>,
> Robert Kern <rober...@gmail.com> wrote:
>
>>Ron Garret wrote:
>>
>>>I'm using an OS X terminal to ssh to a Linux machine.
>>
>>Click on the "Terminal" menu, then "Window Settings...". Choose "Display"
>>from
>>the combobox. At the bottom you will see a combobox title "Character Set
>>Encoding". Choose "Unicode (UTF-8)".
>
> It was already set to UTF-8.
Then take a look at your LANG environment variable on your Linux machine. For
example, I have LANG=en_US.UTF-8 on my Linux machine, and I can ssh into it from
a UTF-8-configured Terminal.app and print unicode strings just fine.
Ron Garret
"Serge Orlov" <Serge...@gmail.com> wrote:
No joy.
ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
ron@www01:~$
rg
Serge Orlov
> > > I'm using an OS X terminal to ssh to a Linux machine.
> >
> > In theory it should work out of the box. OS X terminal should set
> > enviromental variable LANG=en_US.utf-8, then ssh should transfer this
> > variable to Linux and python will know that your terminal is utf-8.
> > Unfortunately AFAIK OS X terminal doesn't set that variable and most
> > (all?) ssh clients don't transfer it between machines. As a workaround
> > you can set that variable on linux yourself . This should work in the
> > command line right away:
> >
> > LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> >
> > Or put the following line in ~/.bashrc and logout/login
> >
> > export LANG=en_US.utf-8
>
> No joy.
>
> ron@www01:~$ LANG=en_US.utf-8 python -c "print unichr(0xbd)"
> Traceback (most recent call last):
> File "<string>", line 1, in ?
> UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
> position 0: ordinal not in range(128)
> ron@www01:~$
What version of python and what shell do you run? What the following
commands print:
python -V
echo $SHELL
$SHELL --version
Ron Garret
"Serge Orlov" <Serge...@gmail.com> wrote:
ron@www01:~$ python -V
Python 2.3.4
ron@www01:~$ echo $SHELL
/bin/bash
ron@www01:~$ $SHELL --version
GNU bash, version 2.05b.0(1)-release (i386-pc-linux-gnu)
Copyright (C) 2002 Free Software Foundation, Inc.
ron@www01:~$
Serge Orlov
That's recent enough. I guess the distribution you're using set LC_*
variables for no good reason. Either unset all enviromental variables
starting with LC_ and set LANG variable or overide LC_CTYPE variable:
LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"
Should be working now :)
Serge Orlov
I've pulled myself together and installed linux in vwware player.
Apparently there is another way linux distributors can screw up. I
chose debian 3.1 minimal network install and after answering all
installation questions I found that only ascii and latin-1 english
locales were installed:
$ locale -a
C
en_US
en_US.iso88591
POSIX
In 2006, I would expect utf-8 english locale to be present even in
minimal install. I had to edit /etc/locale.gen and run locale-gen as
root. After that python started to print unicode characters.
Laurent Pointal
This is default encoding for evaluation of expressions in u"..."
strings, this has nothing to do with printing.
For the output encoding, see sys.stdout.encoding.
>>> import sys
>>> sys.stdout.encoding
'cp850'
>>>
A+
Laurent.
John Salerno
AFAIK, I'm all ASCII (at least, I never made explicit changes to the
default Python install), so how am I able to print out the character?
Robert Kern
> AFAIK, I'm all ASCII (at least, I never made explicit changes to the
> default Python install), so how am I able to print out the character?
Because sys.stdout.encoding isn't determined by your Python configuration, but
your terminal's.
sk...@pobox.com
Robert> Because sys.stdout.encoding isn't determined by your Python
Robert> configuration, but your terminal's.
Learn something every day. I take it "646" is an alias for "ascii" (or vice
versa)?
% python
Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
[GCC 3.4.1] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'646'
>>> import codecs
>>> codecs.lookup("646")
(<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii.StreamReader at 0x819aa4c>, <class encodings.ascii.StreamWriter at 0x819aa1c>)
Skip
John Salerno
> Robert> Because sys.stdout.encoding isn't determined by your Python
> Robert> configuration, but your terminal's.
>
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?
Hmm, not that this helps me any :)
>>> import sys
>>> sys.stdout.encoding
'cp1252'
>>> import codecs
>>> codecs.lookup('cp1252')
(<bound method Codec.encode of <encodings.cp1252.Codec instance at
0x009D6670>>, <bound method Codec.decode of <encodings.cp1252.Codec
instance at 0x009D6698>>, <class encodings.cp1252.StreamReader at
0x009CF360>, <class encodings.cp1252.StreamWriter at 0x009CF330>)
>>>
sk...@pobox.com
John> Hmm, not that this helps me any :)
>>>> import sys
>>>> sys.stdout.encoding
John> 'cp1252'
Sure it does. You can print Unicode objects which map to cp1252. I assume
that means you're on Windows or that for some perverse reason you have your
Mac's Terminal window set to cp1252. (Does it go there? I'm at work right
now so I can't check).
Skip
John Salerno
You're right, I'm on XP. I just couldn't make sense of the lookup call,
although some of the names looked like .NET classes.
Robert Kern
> Robert> Because sys.stdout.encoding isn't determined by your Python
> Robert> configuration, but your terminal's.
>
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?
>
> Python 2.4.2 (#1, Feb 23 2006, 12:48:31)
> [GCC 3.4.1] on sunos5
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.stdout.encoding
> '646'
> >>> import codecs
> >>> codecs.lookup("646")
> (<built-in function ascii_encode>, <built-in function ascii_decode>, <class encodings.ascii.StreamReader at 0x819aa4c>, <class encodings.ascii.StreamWriter at 0x819aa1c>)
Yes. In encodings/aliases.py in the standard library:
"""
aliases = {
# Please keep this list sorted alphabetically by value !
# ascii codec
'646' : 'ascii',
Ron Garret
"Serge Orlov" <Serge...@gmail.com> wrote:
Nope:
ron@www01:~$ export | grep LC
ron@www01:~$
> Either unset all enviromental variables
> starting with LC_ and set LANG variable or overide LC_CTYPE variable:
>
> LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"
>
> Should be working now :)
Nope:
ron@www01:~$ LC_CTYPE=en_US.utf-8 python -c "print unichr(0xbd)"
Traceback (most recent call last):
File "<string>", line 1, in ?
UnicodeEncodeError: 'ascii' codec can't encode character u'\xbd' in
position 0: ordinal not in range(128)
rg
Ron Garret
"Serge Orlov" <Serge...@gmail.com> wrote:
That's it. Thanks!
rg
"Martin v. Löwis"
> Learn something every day. I take it "646" is an alias for "ascii" (or vice
> versa)?
Usage of "646" as an alias for ASCII is primarily a Sun invention. When
ASCII became an international standard, its standard number became
ISO/IEC 646:1968. It's not *quite* the same as ASCII, as it leaves a
certain number of code points unassigned that ASCII defines (most
notably, the dollar sign, and the square and curly braces). What Sun
means is probably the "International Reference Version" of ISO 646,
which is (now) identical to ASCII.
Regards,
Martin