Trouble getting pyinstaller to import Queue in an imported module.

[Coyot Linden (Glenn Glazer)]

Hi, all.

There is a top level script, called SL_Launcher.  This script may decide to launch another script, download_update as a subprocess:

download_process = subprocess.Popen([path_to_downloader, ...args to download_update] ...args to subprocess)

where path_to_downloader is the pyinstaller created executable in the same directory as the pyinstaller created executable of SL_Launcher.

Inside download_update, it imports a module called InstallerUserMessage.py, which isolates all of the Tkinter handling for the various other modules. For test purposes, InstallerUserMessage has a main() that exercises the various methods. For the implementation of a progressbar, it imports Queue.

When I run the main() method from InstallerUserMessage, it runs without error.

However, when I run SL_Launcher.exe or just download_update.exe, I get:

LOADER: Running SL_Launcher.py
Traceback (most recent call last):
 File "SL_Launcher.py", line 30, in <module>
   import InstallerUserMessage
 File "c:\cygwin64\tmp\pip-build-xzzmth\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
 File "InstallerUserMessage.py", line 33, in <module>
   import Queue
ImportError: No module named Queue
Failed to execute script SL_Launcher
LOADER: OK.
LOADER: Cleaning up Python interpreter.
LOADER: Back to parent (RC: -1)
LOADER: Doing cleanup
LOADER: Freeing archive status for c:\Program Files (x86)\SecondLifeTestCoyotVMP\SL_Launcher.exe

or

LOADER: Running download_update.py
Traceback (most recent call last):
 File "download_update.py", line 33, in <module>
   import os
 File "c:\cygwin64\tmp\pip-build-xzzmth\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
 File "InstallerUserMessage.py", line 33, in <module>
   import Queue
ImportError: No module named Queue
Failed to execute script download_update
LOADER: OK.
LOADER: Cleaning up Python interpreter.
LOADER: Back to parent (RC: -1)
LOADER: Doing cleanup
LOADER: Freeing archive status for c:\Program Files (x86)\SecondLifeTestCoyotVMP\download_update.exe

The only other error output message is this:

LOADER: Error activating the context: ActivateActCtx:
An attempt to set the process default activation context failed because the process default activation context was already set.

If I add import Queue to SL_Launcher (it is already in download_update), there is no problem with those import statements.  If I reorder the import statements in InstallerUserMessage to put Queue at the end (I typically alphabetize them), it has no problem finding os, sys, threading, time, Tkinter, ttk, and webbrowser, and dies on Queue.

By way of environment notes, Windows 7, Python 2.7.11 and pyinstaller 3.2. I am not using any custom spec files. This code works on Mac as Python scripts. 

Is there something I need to do to get pyinstaller to find Queue in an imported module?  I find it curious and strange that it found all of other imports but not Queue.

Best,

coyot

-- 
https://wiki.lindenlab.com/wiki/User:Coyot/Tao_of_Linden
https://wiki.lindenlab.com/wiki/Shrek_Ears

[Coyot Linden (Glenn Glazer)]

Anyone have any thoughts on this?  It's now also showing up in download_update:

LOADER: Running download_update.py
Traceback (most recent call last):
 File "download_update.py", line 33, in <module>
   import os
 File "c:\cygwin64\tmp\pip-build-xzzmth\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module
 File "InstallerUserMessage.py", line 33, in <module>
   import Queue
ImportError: No module named Queue
Failed to execute script download_update

I have no idea why Queue would be different than any other Python module.

Best,

coyot

[Coyot Linden (Glenn Glazer)]

On 9/29/16 13:20, Coyot Linden (Glenn Glazer) wrote:
> I have no idea why Queue would be different than any other Python module.

My colleague, Nat, asks:

"Is the name Queue used internally to PyInstaller somehow or other?"

Best,

coyot

[Steve Barnes]

> --
> You received this message because you are subscribed to the Google
> Groups "PyInstaller" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pyinstaller...@googlegroups.com
> <mailto:pyinstaller...@googlegroups.com>.
> To post to this group, send email to pyins...@googlegroups.com
> <mailto:pyins...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/pyinstaller.
> For more options, visit https://groups.google.com/d/optout.

An interesting little experiment that might throw some light on the
situation:
In [24]: import Queue
In [25]: Queue.__file__
Out[25]: 'c:\\python27\\lib\\Queue.pyc'
In [26]: Queue?
Type: module
String form: <module 'Queue' from 'c:\python27\lib\Queue.pyc'>
File: c:\python27\lib\queue.py
Docstring: A multi-producer, multi-consumer queue.

In [27]: import queue
In [28]: queue.__file__
Out[28]: 'c:\\python27\\lib\\site-packages\\queue\\__init__.pyc'
In [29]: queue?
Type: module
String form: <module 'queue' from
'c:\python27\lib\site-packages\queue\__init__.pyc'>
File: c:\python27\lib\site-packages\queue\__init__.py
Docstring: <no docstring>

https://docs.python.org/2.7/library/queue.html?highlight=queue is worth
a look as opposed to
https://docs.python.org/3.5/library/queue.html?highlight=queue

--
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect
those of my employer.

[Hartmut Goebel]

Hi,

Am 30.09.2016 um 07:12 schrieb Steve Barnes:

File:        c:\python27\lib\queue.py
[…]
File:        c:\python27\lib\site-packages\queue\__init__.py

Thanks Steve, for tracking this down.

@Coyote: This sounds like you mixed your Python 2.7 and your Python 3.4 environment. Since Windows file-system ignores case when searching for files, you'll end up with this kind of errors. I strongly suggest to use a virtual environment when freezing your software with PyInstaller.

--
Schönen Gruß
Hartmut Goebel
Dipl.-Informatiker (univ), CISSP, CSSLP, ISO 27001 Lead Implementer
Information Security Management, Security Governance, Secure Software Development

Goebel Consult, Landshut
http://www.goebel-consult.de

Blog: http://www.goebel-consult.de/blog/bestanden-iso-27001-lead-implementer
Kolumne: http://www.cissp-gefluester.de/2010-06-adobe-und-der-maiszunsler

[Steve Barnes]

On 30/09/2016 09:45, Hartmut Goebel wrote:
> Hi,
>
> Am 30.09.2016 um 07:12 schrieb Steve Barnes:
>
>> File: c:\python27\lib\queue.py
>> […]
>> File: c:\python27\lib\site-packages\queue\__init__.py
>
> Thanks Steve, for tracking this down.
>
> @Coyote: This sounds like you mixed your Python 2.7 and your Python 3.4
> environment. Since Windows file-system ignores case when searching for
> files, you'll end up with this kind of errors. I strongly suggest to use
> a virtual environment when freezing your software with PyInstaller.
>
>

I think that one of six, __future__, backports or some other package
might be responsible for this. However, setting up a virtualenv for
builds is ALWAYS a good idea so that you know what your dependencies are
and so that upgrading a package in your main environment doesn't break
something that build before.

[Coyot Linden (Glenn Glazer)]

On 9/30/16 03:03, Steve Barnes wrote:

On 30/09/2016 09:45, Hartmut Goebel wrote:

> Hi,
> 
> Am 30.09.2016 um 07:12 schrieb Steve Barnes:
> 

>> File:        c:\python27\lib\queue.py
>> […]
>> File:        c:\python27\lib\site-packages\queue\__init__.py

> 
> Thanks Steve, for tracking this down.
> 
> @Coyote: This sounds like you mixed your Python 2.7 and your Python 3.4
> environment. Since Windows file-system ignores case when searching for
> files, you'll end up with this kind of errors. I strongly suggest to use
> a virtual environment when freezing your software with PyInstaller.
> 
> 

I think that one of six, __future__, backports or some other package
might be responsible for this.  However, setting up a virtualenv for
builds is ALWAYS a good idea so that you know what your dependencies are
and so that upgrading a package in your main environment doesn't break
something that build before.

For the record, there is no Python 3 on our systems.  Here's another interesting experiment.  If I launch a python shell on Windows and do:

>>> import Queue
>>> import queue

I do not get an error.  But if I do this on Mac, I get:

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
ImportError: No module named queue

because the UNIX in MacOS recognizes case. 

Best,

coyot

[Coyot Linden (Glenn Glazer)]

On 9/29/16 22:12, Steve Barnes wrote:
> An interesting little experiment that might throw some light on the
> situation:
> In [24]: import Queue
> In [25]: Queue.__file__
> Out[25]: 'c:\\python27\\lib\\Queue.pyc'
> In [26]: Queue?
> Type: module
> String form: <module 'Queue' from 'c:\python27\lib\Queue.pyc'>
> File: c:\python27\lib\queue.py
> Docstring: A multi-producer, multi-consumer queue.
>
> In [27]: import queue
> In [28]: queue.__file__
> Out[28]: 'c:\\python27\\lib\\site-packages\\queue\\__init__.pyc'
> In [29]: queue?
> Type: module
> String form: <module 'queue' from
> 'c:\python27\lib\site-packages\queue\__init__.pyc'>
> File: c:\python27\lib\site-packages\queue\__init__.py
> Docstring: <no docstring>
>
> https://docs.python.org/2.7/library/queue.html?highlight=queue is worth
> a look as opposed to
> https://docs.python.org/3.5/library/queue.html?highlight=queue

Steven's experiment (which repros exactly for me) got me curious and I
did some searching on python, {Q,q}ueue, lib and site-packages.

Here's an interesting thread where the Docker folks were having the same
problem:

https://github.com/docker/compose/issues/2108

And here are two Pyinstaller bug threads describing the problem:

https://github.com/pyinstaller/pyinstaller/issues/1939
https://github.com/pyinstaller/pyinstaller/issues/1935

In the latter one, Codewarrior0 says that the problem is due to not
being able to freeze two modules with the same (up to case) name and
also claims he fixed it.

Did this fix get into Pyinstaller 3.2? If not, do we know when it will
become available?

I do not think that a conflict with non-existent, other versions of
python on the machine are the problem, but rather the bug as described
in #1935.

Best,

coyot

[Steve Barnes]

> --

A further experiment:

virtualenv TestVM
New python executable in F:\toolbuild\TestVM\Scripts\python.exe
Installing setuptools, pip, wheel...done.

F:\toolbuild>testvm\Scripts\activate.bat

(TestVM) F:\toolbuild>python -c"import queue;"

Traceback (most recent call last):

File "<string>", line 1, in <module>

ImportError: No module named queue

(TestVM) F:\toolbuild>pip install future
Collecting future
Installing collected packages: future
Successfully installed future-0.15.2

(TestVM) F:\toolbuild>python -c"import queue;"

(TestVM) F:\toolbuild>
<NO ERROR>

So the ministry queue package, (queue), arrives with "future" - I would
guess that it is necessary for some reason - I think that it allows
import queue by shimming the existing Queue module but obviously it
causes pyinstaller issues.

It would be interesting to try the same on the Mac to see if this
introduces the problem but I do not have one to hand.

[Coyot Linden (Glenn Glazer)]

Tried to repro on Mac, but for some reason virtualenv won't install future even though I can outside of it:

ggl...@Coyots-MBP.attlocal.net:~$ pip install future
Collecting future
  Using cached future-0.15.2.tar.gz
Installing collected packages: future
  Running setup.py install for future ... done
Successfully installed future-0.15.2

ggl...@Coyots-MBP.attlocal.net:~$ virtualenv pytest
New python executable in /Users/gglazer/pytest/bin/python
Installing setuptools, pip, wheel...done.

ggl...@Coyots-MBP.attlocal.net:~$ source pytest/bin/activate
(pytest) ggl...@Coyots-MBP.attlocal.net:~$ python -c"import queue;"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named queue

(pytest) ggl...@Coyots-MBP.attlocal.net:~$ pip install future
Collecting future
  Using cached future-0.15.2.tar.gz
Could not import setuptools which is required to install from a source distribution.

Traceback (most recent call last):

  File "/Users/gglazer/pytest/lib/python2.7/site-packages/pip/req/req_install.py", line 372, in setup_py
    import setuptools  # noqa
  File "/Users/gglazer/pytest/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module>
    from setuptools.extern.six.moves import filter, filterfalse, map
  File "/Users/gglazer/pytest/lib/python2.7/site-packages/setuptools/extern/__init__.py", line 1, in <module>
    from pkg_resources.extern import VendorImporter
ImportError: No module named extern

Best,

coyot

[Steve Barnes]

A further test in Windows 10 bash shell and I get exactly the same
behaviour.

Since I now cannot pip uninstall future it also looks like it might be a
slightly misbehaving package.

[Coyot Linden (Glenn Glazer)]

Thanks, Steve.  I very much appreciate your time looking at this.  I think the solution lies in Codewarrior0's fix and whether that is available yet.

Best,

coyot

[Coyot Linden (Glenn Glazer)]

I may have an additional clue.

If I do evil things and monkey with the sys.path like this:

 if os.name == 'nt':
   packages = os.path.join(os.path.splitdrive(sys.path[0])[0],'\\Python27\\lib\\site-packages')
   sys.path.insert(0, packages)

Then it works in the downloader.  The problem is that the point of using Queue is send messages from the downloader (progress) to the Tkinter class (InstallerUserMessage), so both classes need to have an import of Queue.  Adding the code above to both doesn't work when the downloader imports InstallerUserMessage in order to instantiate a class to send messages to.

The sys.path, as seen by IUM is:

['C:\\Python27\\lib\\site-packages', 'C:\\Python27\\lib\\site-packages', 'C:\\Users\\coyot\\AppData\\Local\\Temp\\_MEI12~1']

It replicates the site-packages dir because it inherits sys.path from the downloader which did this once for its own sake.

And the traceback is:

Traceback (most recent call last):

 File "download_update.py", line 38, in <module>

 File "c:\cygwin64\tmp\pip-build-xzzmth\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module

 File "InstallerUserMessage.py", line 39, in <module>
   #for hyperlinks

ImportError: No module named Queue
Failed to execute script download_update

Note that the complaint is from the imported IUM class trying to import Queue, not from downloader itself.

Best,

coyot

[Coyot Linden (Glenn Glazer)]

In a continuing attempt to get pyinstaller to create an executable that doesn't throw an exception, we tried a hack.  There is nothing special about a canonical Python module, it is just another script.  So, we copied Queue.py to our source tree, renamed it and then did something like:

if os.name = 'nt':
   import SLQueue as Queue
#posix env
else:
   import Queue

in the downloader and message modules.

This worked great, with no error on import from either bespoke module.  But on running the downloader, we got this:

 File "c:\cygwin64\tmp\pip-build-xzzmth\pyinstaller\PyInstaller\loader\pyimod03_importers.py", line 389, in load_module

 File "site-packages\urllib3\connectionpool.py", line 28, in <module>
 File "site-packages\urllib3\packages\six.py", line 203, in load_module
 File "site-packages\urllib3\packages\six.py", line 115, in _resolve
 File "site-packages\urllib3\packages\six.py", line 82, in _import_module

ImportError: No module named Queue

which is to say that the PyPi version of urllib3 also relies on Queue via the six package also fails.  The idea of having to bring in local versions of every module that depends on Queue is hard to stomach. 

Again, on these systems, importing any of the these modules via the command interpreter works without error, as does executing the original script via python <cmd> <args>.

Best,

coyot