sys.getfilesystemencoding() always returns None

Daniel Hyams

unread,

Jan 22, 2011, 3:32:22 PM1/22/11

to PyInstaller

On linux; seems to work fine in windows and OSX.

Attached is a small script and test .spec that demonstrate. If I run
fse.py as a script, I get 'UTF-8' as output. If I run it frozen, it
returns None.

--
Daniel Hyams
dhy...@gmail.com

fse.py

fse.spec

Hartmut Goebel

unread,

Jan 22, 2011, 4:53:28 PM1/22/11

to pyins...@googlegroups.com

Am 22.01.2011 21:32, schrieb Daniel Hyams:
> On linux; seems to work fine in windows and OSX.
>
> Attached is a small script and test .spec that demonstrate. If I run
> fse.py as a script, I get 'UTF-8' as output. If I run it frozen, it
> returns None.

Which version of PyInstaller are you using?

I can not reproduce this. Please check your environment is set up
correctly by adding these lines to your script:

import pprint, os
pprint.pprint(dict(os.environ))

--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult mit Mitglied bei http://www.7-it.de

dhyams

unread,

Jan 23, 2011, 10:48:02 PM1/23/11

to pyins...@googlegroups.com

I tried with a 1.4 series SVN, the current 1.5rc1, and the current svn as of tonight; all return None for sys.getfilesystemencoding() when frozen. Here is the environmental output from the two suggested lines above:

dhyams@linux-573c> ./fse.exe

{'COLORFGBG': '0;15',

'COLORTERM': '1',

'CPU': 'i686',

'CSHEDIT': 'emacs',

'DBUS_SESSION_BUS_ADDRESS': 'unix:abstract=/tmp/dbus-lRBT1Um4Ix,guid=fdd04d3f13d7b8b7639e5c0f4d2e15e9',

'DESKTOP_SESSION': 'default',

'DESKTOP_STARTUP_ID': '',

'DISPLAY': ':0',

'DM_CONTROL': '/var/run/xdmctl',

'ENV': '/etc/bash.bashrc',

'FROM_HEADER': '',

'GS_LIB': '/home/dhyams/.fonts',

'GTK2_RC_FILES': '/etc/gtk-2.0/gtkrc:/usr/share/themes//QtCurve/gtk-2.0/gtkrc:/home/dhyams/.gtkrc-2.0-qtengine:/home/dhyams/.gtkrc-2.0',

'GTK_IM_MODULE': 'cedilla',

'G_BROKEN_FILENAMES': '1',

'G_FILENAME_ENCODING': '@locale,UTF-8,ISO-8859-15,CP1252',

'HISTSIZE': '1000',

'HOME': '/home/dhyams',

'HOST': 'linux-573c',

'HOSTNAME': 'linux-573c',

'HOSTTYPE': 'i386',

'INFODIR': '/usr/local/info:/usr/share/info:/usr/info',

'INFOPATH': '/usr/local/info:/usr/share/info:/usr/info',

'INPUTRC': '/home/dhyams/.inputrc',

'JAVA_BINDIR': '/usr/lib/jvm/jre/bin',

'JAVA_HOME': '/usr/lib/jvm/jre',

'JAVA_ROOT': '/usr/lib/jvm/jre',

'JRE_HOME': '/usr/lib/jvm/jre',

'KDE_FULL_SESSION': 'true',

'KDE_SESSION_UID': '1000',

'KDE_SESSION_VERSION': '4',

'KONSOLE_DBUS_SERVICE': ':1.26',

'KONSOLE_DBUS_SESSION': '/Sessions/5',

'LANG': 'en_US.UTF-8',

'LANGUAGE': 'en_US',

'LD_LIBRARY_PATH': './:/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:',

'LESS': '-M -I',

'LESSCLOSE': 'lessclose.sh %s %s',

'LESSKEY': '/etc/lesskey.bin',

'LESSOPEN': 'lessopen.sh %s',

'LESS_ADVANCED_PREPROCESSOR': 'no',

'LOGNAME': 'dhyams',

'LS_COLORS': 'no=00:fi=00:di=01;34:ln=00;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=41;33;01:ex=00;32:*.cmd=00;32:*.exe=01;32:*.com=01;32:*.bat=01;32:*.btm=01;32:*.dll=01;32:*.tar=00;31:*.tbz=00;31:*.tgz=00;31:*.rpm=00;31:*.deb=00;31:*.arj=00;31:*.taz=00;31:*.lzh=00;31:*.lzma=00;31:*.zip=00;31:*.zoo=00;31:*.z=00;31:*.Z=00;31:*.gz=00;31:*.bz2=00;31:*.tb2=00;31:*.tz2=00;31:*.tbz2=00;31:*.avi=01;35:*.bmp=01;35:*.fli=01;35:*.gif=01;35:*.jpg=01;35:*.jpeg=01;35:*.mng=01;35:*.mov=01;35:*.mpg=01;35:*.pcx=01;35:*.pbm=01;35:*.pgm=01;35:*.png=01;35:*.ppm=01;35:*.tga=01;35:*.tif=01;35:*.xbm=01;35:*.xpm=01;35:*.dl=01;35:*.gl=01;35:*.wmv=01;35:*.aiff=00;32:*.au=00;32:*.mid=00;32:*.mp3=00;32:*.ogg=00;32:*.voc=00;32:*.wav=00;32:',

'LS_OPTIONS': '-N --color=tty -T 0',

'MACHTYPE': 'i686-suse-linux',

'MAIL': '/var/spool/mail/dhyams',

'MANPATH': '/usr/local/man:/usr/share/man',

'MINICOM': '-c on',

'MORE': '-sl',

'NNTPSERVER': 'news',

'OLDPWD': '/dhyams/My Dropbox/CurveExpertPro/dist/pyinstaller-svn-1-23-2011/bugreports/dist',

'OSTYPE': 'linux',

'PAGER': 'less',

'PATH': '/sbin:/usr/local/python/bin:/sbin:/usr/local/python/bin:/home/dhyams/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/X11R6/bin:/usr/games:/usr/lib/jvm/jre/bin:/usr/lib/mit/bin:/usr/lib/mit/sbin:/opt/kde3/bin',

'PROFILEHOME': '',

'PROFILEREAD': 'true',

'PWD': '/dhyams/My Dropbox/CurveExpertPro/dist/pyinstaller-svn-1-23-2011/bugreports/dist/fse',

'PYTHONPATH': '.',

'PYTHONSTARTUP': '/etc/pythonstart',

'QT_IM_MODULE': 'xim',

'QT_IM_SWITCHER': 'imsw-multi',

'QT_PLUGIN_PATH': '/home/dhyams/.kde4/lib/kde4/plugins/:/usr/lib/kde4/plugins/',

'QT_SYSTEM_DIR': '/usr/share/desktop-data',

'SESSION_MANAGER': 'local/linux-573c:@/tmp/.ICE-unix/4604,unix/linux-573c:/tmp/.ICE-unix/4604',

'SHELL': '/bin/bash',

'SHLVL': '2',

'SSH_AGENT_PID': '4275',

'SSH_ASKPASS': '/usr/lib/ssh/x11-ssh-askpass',

'SSH_AUTH_SOCK': '/tmp/ssh-CsOgc4038/agent.4038',

'TERM': 'xterm',

'TMPDIR': '/tmp',

'USER': 'dhyams',

'USE_FAM': '',

'WINDOWID': '48234497',

'WINDOWMANAGER': '/usr/bin/startkde',

'WINDOWPATH': '7',

'XAUTHLOCALHOSTNAME': 'linux-573c',

'XCURSOR_THEME': 'Oxygen_White',

'XDG_CONFIG_DIRS': '/etc/xdg',

'XDG_DATA_DIRS': '/usr/share:/etc/opt/kde3/share:/opt/kde3/share',

'XDG_SESSION_COOKIE': '6569d80d17bd3a5ee8c684c9493eedd8-1294865894.311789-1311049069',

'XDM_MANAGED': 'method=classic,auto',

'XKEYSYMDB': '/usr/share/X11/XKeysymDB',

'XMODIFIERS': '@im=local',

'XNLSPATH': '/usr/share/X11/nls',

'XSESSION_IS_UP': 'yes',

'_': './fse.exe',

'_MEIPASS2': './'}

sys.getfilesystemencoding = None

Hartmut Goebel

unread,

Jan 24, 2011, 4:25:22 AM1/24/11

to pyins...@googlegroups.com

Am 24.01.2011 04:48, schrieb dhyams:
> I tried with a 1.4 series SVN, the current 1.5rc1, and the current svn
> as of tonight; all return None for sys.getfilesystemencoding() when
> frozen. Here is the environmental output from the two suggested lines
> above:

Thanks. I missed the more important part of the question: How does this
differ from your "normal" environment?

> 'LANG': 'en_US.UTF-8',
>
>
> 'LANGUAGE': 'en_US',
>
>

Not being a locacle export, I'm missing "LC_CTYPE" here, which is
related to encodings (see
<http://www.debian.org/doc/manuals/intro-i18n/ch-locale.en.html>.

Daniel Hyams

unread,

Jan 24, 2011, 9:55:39 PM1/24/11

to pyins...@googlegroups.com

Unfortunately, I don't really see anything noteworthy about the
difference between the environment in the frozen mode or not:

dhyams@linux-573c:> diff SCRIPT_ENV FROZEN_ENV
36c36
< 'LD_LIBRARY_PATH':
'/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:',
---
> 'LD_LIBRARY_PATH': 'dist/fse/:/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:/usr/local/wx/2.9.1.1/lib:/usr/local/wx/2.8-ucs4/lib:',
57a58
> 'PYTHONPATH': 'dist/fse',
86,87c87,89
< '_': '/usr/local/python/bin/python'}
---
> '_': 'dist/fse/fse.exe',
> '_MEIPASS2': 'dist/fse/'}

--
Daniel Hyams
dhy...@gmail.com

Hartmut Goebel

unread,

Jan 25, 2011, 5:05:20 PM1/25/11

to pyins...@googlegroups.com

Am 25.01.2011 03:55, schrieb Daniel Hyams:
> Unfortunately, I don't really see anything noteworthy about the
> difference between the environment in the frozen mode or not:

Ouch, this will become a hard one.

You may try to run

LC_ALL=$LANG ./fse

any maybe other LC_* variables. As I said: I'm not a locale guru.

If this fails, too, it may be something very tricky. Please try to debug
both python fse.py and ./fse via strace.

--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen

http://www.goebel-consult.de/blog

Daniel Hyams

unread,

Jan 25, 2011, 6:49:31 PM1/25/11

to pyins...@googlegroups.com

Hmm. I can confirm that this behavior does *not* happen on an Ubuntu
10.10 virtual box that I put together, but does on my SuSe 11.1
testbed, which is what I develop on. I'm not sure what the difference
is, but it is likely that there is something fishy with the SuSe box.
When I get a chance, I will create a clean SuSe 11.1 virtualbox and
try it.

--
Daniel Hyams
dhy...@gmail.com

Jeremy Sanders

unread,

Nov 4, 2011, 4:42:21 PM11/4/11

to pyins...@googlegroups.com

On Tuesday, January 25, 2011 10:05:20 PM UTC, Hartmut Goebel wrote:

You may try to run
LC_ALL=$LANG ./fse
any maybe other LC_* variables. As I said: I'm not a locale guru.
If this fails, too, it may be something very tricky. Please try to debug
both python fse.py and ./fse via strace.

Hi - I'm hitting the sys.getfilesystemencoding() problem with pyinstaller too, which is breaking saving unicode filenames in my PyQt program. Is this fixed yet?

I've compared strace between the systems. The working system has this, which the non-working system does not.

20188 open("/usr/local/lib/python2.7/encodings/__init__.pyc", O_RDONLY|O_LARGEFILE) = 6
20188 fstat64(6, {st_mode=S_IFREG|0644, st_size=4394, ...}) = 0
20188 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f4a000
20188 read(6, "\3\363\r\n@\250\203Mc\0\0\0\0\0\0\0\0\3\0\0\0@\0\0\0s\216\0\0\0d\0"..., 4096) = 4096
20188 fstat64(6, {st_mode=S_IFREG|0644, st_size=4394, ...}) = 0
20188 read(6, "\1\4\3\t\1\22\1\"\1\3\1\3\1\26\1\24\1\24\1$\1$\0016\0016\1\3\1\3\1\26"..., 4096) = 298
20188 read(6, "", 4096) = 0
20188 close(6) = 0

So it looks like the program (after run through pyinstaller), is loading a module from the Python directory, rather from its own dist directory.Should pyinstaller be including the encodings module and it is not?

Jeremy

Jeremy Sanders

unread,

Nov 6, 2011, 1:04:13 PM11/6/11

to PyInstaller

On Nov 4, 8:42 pm, Jeremy Sanders <jeremysand...@gmail.com> wrote:

> So it looks like the program (after run through pyinstaller), is loading a
> module from the Python directory, rather from its own dist directory.Should
> pyinstaller be including the encodings module and it is not?

If I explicitally include "import encodings" in my test script, it
still seems to fail when trying to load the encodings module. I'm
wondering whether the problems are caused by encodings/__init__.py is
doing some messing around with __import__ to load encoding modules:
# Import is absolute to prevent the possibly malicious
import of a
# module with side-effects that is not in the 'encodings'
package.
mod = __import__('encodings.' + modname,
fromlist=_import_tail,
level=0)

Hartmut Goebel

unread,

Nov 7, 2011, 5:17:09 AM11/7/11

to pyins...@googlegroups.com

Am 04.11.2011 21:42, schrieb Jeremy Sanders:
> So it looks like the program (after run through pyinstaller), is
> loading a module from the Python directory, rather from its own dist
> directory.Should pyinstaller be including the encodings module and it
> is not?

This looks as if you package does not include the encodings module.

Does your .spec file include CONFIGDIR/support/useUnicode.py? If not:
does your config.dat set unicode=1?

If useUnicode.py is included please follow the instructions at
<http://www.pyinstaller.org/wiki/HowtoReportBugs#Beforesubmittingareport:Makesureeverythingispackaged>
to track this down. Thanks.

--
Schönen Gruß - Regards
Hartmut Goebel
Dipl.-Informatiker (univ.), CISSP, CSSLP

Goebel Consult
Spezialist für IT-Sicherheit in komplexen Umgebungen
http://www.goebel-consult.de

Monatliche Kolumne: http://www.cissp-gefluester.de/
Goebel Consult ist Mitglied bei http://www.7-it.de

Reply all

Reply to author

Forward