UnicodeEncodeError pyinstaller-1.5-rc1 on Windows

[Stefan Fruehwirth]

Hi,

I get an UnicodeEncodeError when running the generated executable out
of this line of code in a script:

---
print u'confusing cat: \u0283i-wa\u0127\u0259d'
---

That's all. Full Traceback:

---
Traceback (most recent call last):
File "<string>", line 1, in <module>
File ".\pyInstallerUnicodeIssue\build
\pyi.win32\pyInstallerUnicodeIssue\outPYZ1.pyz/encodings.cp850", line
12, in encode
UnicodeEncodeError: 'charmap' codec can't encode character u'\u0283'
in position 15: character maps to <undefined>
---

Executable was generated with the standard Makespec/Build process - no
modifications. Spec and other files can be posted as needed. Is this
some sort of configuration error or should I consider it a bug and
post it to trac?

Thanks,
Stefan

[Giovanni Bajo]

It is well possible that the console/stdout is configured with a
different encoding within PyInstaller rather than the interpreter
version. For instance, PyInstaller does not use the computer's site.py
module, so if the encoding for stdout is set there, there can be a
different behaviour.

I don't know off the top of my head how Python decides which encoding
must be used for stdout on Windows (or any other platform), nor how (and
if) it can be programatically changed from within an application. If you
want to pursue this bug, you will have to find out how this whole thing
works, and then we can decided whether or how to fix this issue.
--
Giovanni Bajo :: ra...@develer.com
Develer S.r.l. :: http://www.develer.com

My Blog: http://giovanni.bajo.it
Last post: Compile-time Function Execution in D

[Stefan Fruehwirth]

On 11 Jan., 11:37, Giovanni Bajo <ra...@develer.com> wrote:

> It is well possible that the console/stdout is configured with a
> different encoding within PyInstaller rather than the interpreter
> version. For instance, PyInstaller does not use the computer's site.py
> module, so if the encoding for stdout is set there, there can be a
> different behaviour.

Yes, good hint, thanks. For stdout there is a simple (not very clean)
workaround (http://stackoverflow.com/questions/492483/setting-the-
correct-encoding-when-piping-stdout-in-python):

---
import sys
import codecs
sys.stdout = codecs.getwriter('utf8')(sys.stdout)
---

Anyway, the application in which I first discovered this error still
threw the exception because somewhere in library code - I wasn't able
to find the place - there must be some operation which does not
properly encode its input. Obviously sys.stdout is just one of the
writers that uses the wrong encoding when compiled with pyInstaller.

> I don't know off the top of my head how Python decides which encoding
> must be used for stdout on Windows (or any other platform), nor how (and
> if) it can be programatically changed from within an application. If you
> want to pursue this bug, you will have to find out how this whole thing
> works, and then we can decided whether or how to fix this issue.

I finally got my app working after setting the env variable
PYTHONIOENCODING to "utf_8" at both build time and execution time
(http://daveagp.wordpress.com/2010/10/26/what-a-character/). The
strange thing is that I failed to re-produce the error in an
environment without PYTHONIOENCODING - even with a completely empty
build and dist dir :-o

I suggest to somehow manage PYTHONIOENCODING in the Makespec/Build
process.

Stefan

[Giovanni Bajo]

OK, I would appreciate if you could open a trac ticket now with all
these references inside, so that I will take care once the makespec-ng
branch is merged (which makes easier to let the user finetune additional
options in the spec file).

[Stefan Fruehwirth]

On 11 Jan., 16:51, Giovanni Bajo <ra...@develer.com> wrote:

> OK, I would appreciate if you could open a trac ticket now with all
> these references inside, so that I will take care once the makespec-ng
> branch is merged (which makes easier to let the user finetune additional
> options in the spec file).

OK. Ticket: http://www.pyinstaller.org/ticket/310

Stefan