Different default encodings with Anaconda between Spyder and command prompt

[Evan Carter]

I get different values for sys.getdefaultencoding in Spyder than when I run the same Python in other ways. It seems that similar behavior was recently removed from PyDev.

I'm running Spyder on a Mac. I installed it via Anaconda.

Within Spyder, from a standard Python interpreter accessed via the or from an iPython console, my default encoding is UTF-8:

Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15) 

[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

Imported NumPy 1.7.1, SciPy 0.13.0, Matplotlib 1.3.1

Type "scientific" for more details.

>>> import sys

>>> sys.getdefaultencoding()

'UTF-8'

>>> 

but when I run what seems to be the same Python from the command prompt, I get:

Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)

[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import sys

>>> sys.getdefaultencoding()

'ascii'

This caused an error that was fairly difficult to debug. Can I make Spyder respect the system's default encoding, or otherwise force the two to be the same?

Here's some related discussion that didn't quite answer the question for me.

Here is someone describing how difficult it is to change Python's default encoding.

[Adrian Klaver]

On 06/14/2014 07:30 PM, Evan Carter wrote:
> I get different values for sys.getdefaultencoding in Spyder than when I
> run the same Python in other ways. It seems that similar behavior was
> recently removed from PyDev

> <https://sw-brainwy.rhcloud.com/tracker/PyDev/315>.

>
> I'm running Spyder on a Mac. I installed it via Anaconda.
>
> Within Spyder, from a standard Python interpreter accessed via the or
> from an iPython console, my default encoding is UTF-8:
>
> Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)
> [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>
> Imported NumPy 1.7.1, SciPy 0.13.0, Matplotlib 1.3.1
> Type "scientific" for more details.
> >>> import sys
> >>> sys.getdefaultencoding()
> 'UTF-8'
> >>>
>
>
> but when I run what seems to be the same Python from the command prompt,
> I get:
>
> Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)
> [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.getdefaultencoding()
> 'ascii'
>
>
> This caused an error that was fairly difficult to debug. Can I make
> Spyder respect the system's default encoding, or otherwise force the two
> to be the same?
>

My guess is it has something to do with this:

http://hints.macworld.com/article.php?story=20100713130450549

So check the environ in each case above to see:

import os
os.environ


--
Adrian Klaver
adrian...@aklaver.com

[Evan Carter]

Thank you for the reply, Adrian. I do not find PYTHONIOENCODING in either environment. Could it be the SPYDER_ENCODING environment variable that's causing the behavior?

Running [(k,v) for k, v in os.environ.items() if 'utf' in v.lower() or 'ascii' in v.lower()] in Spyder gives me

[('SPYDER_ENCODING', 'UTF-8')]

while when I run the same command from a Python shell created from the terminal I get

[('LANG', 'en_US.UTF-8')]

[Adrian Klaver]

On 06/15/2014 03:12 PM, Evan Carter wrote:
> Thank you for the reply, Adrian. I do not find PYTHONIOENCODING in
> either environment. Could it be the SPYDER_ENCODING environment variable
> that's causing the behavior?
>
> Running [(k,v) for k, v in os.environ.items() if 'utf' in v.lower() or
> 'ascii' in v.lower()] in Spyder gives me
>
> [('SPYDER_ENCODING', 'UTF-8')]
>
> while when I run the same command from a Python shell created from the
> terminal I get
>
> [('LANG', 'en_US.UTF-8')]
>

I know from the Postgres mailing lists that OS X has some non-standard
ways of working with locales/encoding. I do not use OS X enough to
really understand what is going on. I can point to others that have more
practical experience in this area, so:

http://blog.remibergsma.com/2012/07/10/setting-locales-correctly-on-mac-osx-terminal-application/

or the suggestions from the first link I sent:
"
So typing export PYTHONIOENCODING=utf-8 prior to invoking the Python
interpreter does the trick, or you could just add this setting to your
environment file: ~/.MacOSX/environment.plist.

As the other commenters have noted, in Python 2 you can print things of
class str directly to the terminal (which are bytestrings) but things of
class unicode need to be converted to bytestrings (str) before they can
be printed. The easiest way to do this is
myunicodestring.encode("utf-8") but if you want to be able to say “print
myunicodestring” without encoding first in the interactive shell, you
can try putting export LC_CTYPE=en_US.utf-8 in your .bash_profile, just
so that Python knows that Terminal.app wants its input (Python’s output)
to be in UTF-8. I think newer versions of Terminal do this automatically
(10.6), but older ones did not.
"

Whatever method you choose it would seem you need to give OS X an assist
in making the correct choice. From the looks of it Anaconda does this by
setting the SPYDER_ENCODING, my guess in its version of the Python
startup file.

--
Adrian Klaver
adrian...@aklaver.com

[Evan Carter]

You are right that Spyder is messing with the encoding in its sitecustomize.py:

encoding = None
try:
    import locale
except ImportError:
    pass
else:
    loc = locale.getdefaultlocale()
    if loc[1]:
        encoding = loc[1]

if encoding is None:
    encoding = "UTF-8"

try:
    sys.setdefaultencoding(encoding)
    os.environ['SPYDER_ENCODING'] = encoding
except AttributeError:
    # Python 3
    pass

If I am so bold as to add encoding = "ascii" then sys.getdefaultencoding() does indeed give 'ascii' when run from the console, but I don't know what the consequences will be of messing with the encoding for all of Spyder. I'd like to change the encoding only for the Python interpreter in the console, not the Python environment in which Spyder is running. Unfortunately, adding os.environ['SPYDER_ENCODING'] = 'ascii' doesn't work: the environment variable stays set but sys.getdefaultencoding() still gives "UTF-8".

Unfortunately setting the PYTHONIOENCODING environment variable before I run a Python program from the command line doesn't work either:

$ export PYTHONIOENCODING=UTF-8
$ export | grep PYTHONIOENCODING
declare -x PYTHONIOENCODING="UTF-8"
$ python
Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)
[GCC 4.0.1 (Apple Inc. build 5493)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'

[Joseph Martinot-Lagarde]

Le 15/06/2014 04:30, Evan Carter a écrit :
> I get different values for sys.getdefaultencoding in Spyder than when I
> run the same Python in other ways. It seems that similar behavior was
> recently removed from PyDev

> <https://sw-brainwy.rhcloud.com/tracker/PyDev/315>.

>
> I'm running Spyder on a Mac. I installed it via Anaconda.
>
> Within Spyder, from a standard Python interpreter accessed via the or
> from an iPython console, my default encoding is UTF-8:
>
> Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)
> [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
>
> Imported NumPy 1.7.1, SciPy 0.13.0, Matplotlib 1.3.1
> Type "scientific" for more details.
> >>> import sys
> >>> sys.getdefaultencoding()
> 'UTF-8'
> >>>
>
>
> but when I run what seems to be the same Python from the command prompt,
> I get:
>
> Python 2.7.6 |Anaconda 1.8.0 (x86_64)| (default, Jan 10 2014, 11:23:15)
> [GCC 4.0.1 (Apple Inc. build 5493)] on darwin
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.getdefaultencoding()
> 'ascii'
>
>
> This caused an error that was fairly difficult to debug. Can I make
> Spyder respect the system's default encoding, or otherwise force the two
> to be the same?
>
> Here

> <https://groups.google.com/forum/#!searchin/spyderlib/getdefaultencoding/spyderlib/v5ELu04_DXU/8DThvxoYMZkJ>'s

> some related discussion that didn't quite answer the question for me.
>

> Here <http://www.ianbicking.org/illusive-setdefaultencoding.html> is

> someone describing how difficult it is to change Python's default encoding.
>

> --
> You received this message because you are subscribed to the Google
> Groups "spyder" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to
> spyderlib+...@googlegroups.com
> <mailto:spyderlib+...@googlegroups.com>.
> To post to this group, send email to
> spyd...@googlegroups.com
> <mailto:spyd...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/spyderlib.
> For more options, visit https://groups.google.com/d/optout.

I already opened a bug in the tracker:
https://code.google.com/p/spyderlib/issues/detail?id=1830. It is more a
reminder that a real bug report, so feel free to add more information
(or a pull request on https://bitbucket.org/spyder-ide/spyderlib !).

---
Ce courrier électronique ne contient aucun virus ou logiciel malveillant parce que la protection avast! Antivirus est active.
http://www.avast.com

[Adrian Klaver]

On 06/16/2014 09:30 PM, Evan Carter wrote:
> You are right that Spyder is messing with the encoding in its
> sitecustomize.py

> <https://code.google.com/p/spyderlib/source/browse/spyderlib/widgets/externalshell/sitecustomize.py>:

Some Googling seems to indicate this SO question has the answer:

http://stackoverflow.com/questions/7165108/in-osx-lion-lang-is-not-set-to-utf8-how-fix/8161863#8161863

Try adding or editing the ~/.profile file for it to correctly export
your locale settings upon initiating a new session.

export LC_ALL=en_US.UTF-8
export LANG=en_US.UTF-8

These two lines added to the file should suffice to set the locale
[replace en_US for your desired locale, and check beforehand that it is
indeed installed on your system (locale -a)].

After that, you can start a new session and check using locale:


If that does not work, I would say this is question for some OS X
experts, of which I am not.



--
Adrian Klaver
adrian...@aklaver.com

[Evan Carter]

Unfortunately not even editing .profile works for me; I can see that the variables are set but Python's behavior is the same. While working on it I did notice that sys.getfilesystemencoding() returns "utf-8", with or without the new lines in .profile, so the problem may be Python-specific.

I found another Stack Overflow question that says

The default of Python 2.x is to use ascii. You can change it(can't remember how) but it is not recommended, because it will break libraries that use ascii strings. It is all changed in 3.x. where UTF is the standard.  

If this is correct then what I really want is to change the Spyder console to use ascii rather than changing Python run in other contexts to use utf-8.

I've just checked and seen that when I run sys.getdefaultencoding() in Spyder's internal console I get 'ascii'. So I guess I can just remove the code to set the default encoding to utf-8 from sitecustomize.py without breaking the environment that Spyder itself runs in. Is that correct?

I wonder why Spyder defaults to utf-8 in the first place, if it's not what the program uses internally.

[Carlos Córdoba]

Hi Evan,

By changing the encoding in sitecustomize.py, you would do it only for our Python and IPython consoles and not for our internals, so it’s perfectly fine if you want to follow this path.

Cheers,
Carlos

El 18/06/14 22:54, Evan Carter escribió:

--
You received this message because you are subscribed to the Google Groups "spyder" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spyderlib+...@googlegroups.com.
To post to this group, send email to spyd...@googlegroups.com.

​