Pyinstaller

[Luke M]

Luke,

 

I use pyinstaller to create executables because of its compatibility with the ctypes package and DLLs. However, I can't seem to get it to handle the imports for pyqtgraph. I know that pyinstaller has a process to handle hidden imports, but with they way the imports are done here, I haven't figured out a way around it.

 

For me, pyinstaller will package the executable fine, but it breaks when you run it and it gets to import pyqtgraph. In pyqtgraph.frozenSupport, it throws a WindowsError because it is looking for c:\path\to\exe?175104\pyqtgraph\grapicsItems/*.* which obviously doesn't exist.

 

Have you tested pyqtgraph with pyinstaller? Do you have any thoughts?

 

Thanks,

Luke M

[Luke Campagnola]

I have not tested with pyinstaller yet, only py2exe. Christian Gavin wrote some excellent documentation on how he got py2exe running. It is available on the webpage, although some things have changed since he wrote it. 

I would love to be able to support pyinstaller as well, though. Would you be willing to write up a short set of instructions for how you are packaging with pyinstaller? I could then follow what you are doing and try to debug the pyqtgraph issues.

Luke

 

[Luke M]

Sure I could definitely write up a simple example with some instructions at least showing how I'm using it and where it fails. I'll try to post something early next week. Thanks!

[Luke M]

A very simple example of a python script to package using pyinstaller would be: 

 

#test.py
import pyqtgraph as pg
print 'Hello World'

For this simple example, using pyinstaller2.0 you could just run pyinstaller.py for test. However, for more complicated examples I typically have to moify the spec file so here are the steps I would typically follow:

1. Put the above .py file in the pyinstaller2.0 directory
2. From that directory, run
python utils/Makespec.py --console --onedir test.py
3. If necessary (not for this simple case) I would edit the .spec file that this created for any hooks that I need (typically scipy and h5py, see documentation)
4. Probably not relevant to pyqtgraph, but if I'm using any DLLs, I'd add the necessary tuple to a.binaries in the spec file (see this pyinstaller ticket)
5. Finally, run:
   pyinstaller.py test\test.spec

If you do this without the pyqtgraph import statement, the executable will run without issue. If you leave the import in, the executable will be created, but when you run it, you'll see the WindowsError that I mentioned above.

 

[Luke M]

Hi Luke,

 

I was curious if you had a chance to take a look at this. Did I provide enough information? 

 

 

[Luke Campagnola]

Yes, sorry this is taking so long. It looks like there is no way to do dynamic imports the way I have them set up, so I am working on caching the names that need to be imported. Stay tuned.. 

Luke

 

[Luke M]

No worries. I was afraid that would be the case. Thanks for the update! 

 

[Mathew Schwartz]

I was wondering if you have worked on pyinstaller compatibility at all?  Actually I dont care if it is pyinstaller or some other method for packaging python files but I would like to be able to distribute a contained file so other people could use the program.  Has anyone successfully done this?  My attempts at pyinstaller have been futile.  

[Luke Campagnola]

On Sun, Jun 9, 2013 at 1:21 AM, Mathew Schwartz <umc...@gmail.com> wrote:

I was wondering if you have worked on pyinstaller compatibility at all?  Actually I dont care if it is pyinstaller or some other method for packaging python files but I would like to be able to distribute a contained file so other people could use the program.  Has anyone successfully done this?  My attempts at pyinstaller have been futile.  

I've used py2exe in the past. There's actually a document linked on the website about packaging with py2exe, but it's a little out of date (http://pyqtgraph.org/Bundling%20applications%20with%20PyQtGraph_R16.pdf). There's been some discussion about the parts of the document that are no longer relevant, specifically issues with os.listdir and packaging the .png files:  https://groups.google.com/d/msg/pyqtgraph/TzZKRg7SP_0/zYEWeDoMFmkJ

 

Luke

[Mathew Schwartz]

great thanks for the document.   I notice it doesnt mention the opengl part of pyqtgraph, before i get too far into it do you know if that will cause problems or not?

--
-- [ You are subscribed to pyqt...@googlegroups.com. To unsubscribe, send email to pyqtgraph+...@googlegroups.com ]
---
You received this message because you are subscribed to the Google Groups "pyqtgraph" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyqtgraph+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Luke Campagnola]

On Sun, Jun 9, 2013 at 8:05 AM, Mathew Schwartz <umc...@gmail.com> wrote:

great thanks for the document.   I notice it doesnt mention the opengl part of pyqtgraph, before i get too far into it do you know if that will cause problems or not?

I can't think of any reason this should cause trouble (pyqtgraph.opengl uses the same import mechanisms as .graphicsItems and .widgets), but I I have not tried it. 

[Luke Lee]

Hi all,

 

I'm new to the group and figured this discussion needed another person named Luke. I'm actually in the process of trying to get pyqtgraph into my own pyinstaller setup and thought I'd share what I did to get this to work in a hackish way.

 

It's true that pyinstaller cannot handle dynamic imports. I haven't verified this, but I'm assuming this is because pyinstaller uses some type of custom import hook to introspect what your application attempts to import and adds it to the package. So, the importAll() function in pyqtgraph\__init__.py is a problem. Pyinstaller will not detect any of these 'imports' done in importAll() because they aren't really going through the true import mechanism, no additional modules end up in sys.modules, etc.

 

I was able to run pyinstaller 'successfully' on the demo program, arrow.py, without any modifications. However, when running the build exe from pyinstaller I would get errors from frozenSupport.listdir(). These errors are because of the following chain of events:

 

- arrow.py imports pyqtgraph (running in sys.frozen = 1 state)

- pyqtgraph/__init__.py executes importAll() function to dynamically import all submodules and packages

- frozenSupport.listdir() detects we are in frozen mode and tries to import the .py files from the current directory

- frozenSupport.listdir() fails to find the specified .py files because pyinstaller did not include any of the pyqtgraph submodules and subpackages since their imports went undected during introspection

 

So, long story short I 'fixed' my pyinstaller setup by removing the dynamic imports in pyqtgraph/__init__.py and pyqtgraph/GraphicsScene/exportDialog.py. Below is my hackish attempt at finding all the dynamic imports and replacing them with 'real' imports:

 

1. I modified the importAll() function in pyqtgraph/__init__.py to print out the actual import statements being 'generated' on the fly.

2. I just imported pyqtgraph once and piped all the prints to a text file. 

3. I pasted all these import lines into pyqtgraph/__init__.py and removed the calls to importAll.

I don't know how well diffs show up in a Google group, so I put up a gist, https://gist.github.com/durden/6165525, with my modified importAll() function and a raw listing of all the dynamic imports that happen.  So, anyone facing this problem can use this in the future, if nothing is changed in the actual package source.

Finally, is this dynamic importing just for convenience?  If so, it might be a bit tedious, but I think there are a few possible solutions:

1. List all these needed attributes, classes, etc. in the proper module __all__ to explicitly declare the 'public' interface.

2. Move the import lines I generated in the above gist to the proper <module>/__init__.py so that each module/package explicitly imports it's 'public' interface.

Solution #1 would still allow the main pyqtgraph/__init__.py to use 'import *' to easily make everything available from a single namespace.  However, it's a little difficult for new developers to immediately see where a module/class/attribute originates from.

I think #2 is probably a more clear and explicit way to 'solve' this, if this is even deemed a 'problem.'  #2 will be tedious in the event that a new class/attribute, etc. is added in future releases because the corresponding __init__.py would require updating.  However, this solution has the added benefits of being explicit and allow developers to clearly see where each module/class/attribute originates from.

[Luke Lee]

It's also worth taking a careful look at all the imports the importAll() function generates.  For example, the importing tries to make every attribute in a module available in the pyqtgraph namespace, including things like, QtCore, QtGui, OrderedDict, etc.

Is the availability of these attributes at the pyqtgraph namespace done for a reason or just a side-effect of importAll all making everything available except 'hidden' attributes that start with '_'?

[Mathew Schwartz]

Hi Luke Lee,

Thanks for this info.  Any chance you have a setup file you could share with me? pyinstaller has been a very confusing experience for me and it would be nice if you had already built the hooks and everything that i could copy off of.

Mat

--

[Luke Lee]

Hi Mat,

I don't have a setup file.  However, I do have this gist,  https://gist.github.com/durden/6165525, that contains all the modifications I needed to the pyqtgraph source.  Below are the commands/steps I've used to successfully use pyinstaller 2.x with pyqtgraph:

1.  Create a new file for setting PyQt API version if you use version 2 of the PyQt API with Python 2.x:

       - You just need to follow the instructions here: http://www.pyinstaller.org/wiki/Recipe/PyQtChangeApiVersion

2. Remove dynamic imports from pyqtgraph/__init__.py:

       - Open pyqtgraph/__init__.py and comment out the two calls to 'importAll' and replace with listing of real imports.

       - These changes are shown in the dynamic_imports.py file from this gist: https://gist.github.com/durden/6165525#file-dynamic_imports-py

 3. Remove dynamic import from pyqtgraph/exporters/__init__.py:

       - Open up pyqtgraph/exporters/__init__.py

       - Comment out dynamic imports and replace with real imports similar to this gist: https://gist.github.com/durden/6165525#file-modified_exporters_init-py

4. Test you pyqtgraph setup/environment by building one of the example scripts:

        - Try building the Arrow.py example specifically because this is the only one I've tested so far.

        - Run pyinstaller.py  --runtime-hook <rthook_file_from_above.py> Arrow.py

        - This should produce two directories: dist/ and build/.

        - Go into the dist/arrow/ directory and try running arrow.exe (if on Windows)

Let me know if you have any other questions.  These instructions will hopefully get you close.  A few things to keep in mind:

1. I only tried Pyinstaller 2.x.  Pyinstaller 1.x has a few more steps involved to create the executable.

2. I've only tried Pyinstaller 2.x on Windows.  My linux/os x setup just installs the packages locally and runs, no need for installer.

3. I've only tested the one Arrow.py example.  I haven't written my own pyqtgraph scripts yet.  I wanted to make sure Pyinstaller would work before investing too much time developing with it.

Luke Lee

[Jonathan Berry]

Thanks for this post, Luke. I was able to use your steps 2 and 3 to build my pyqtgraph application using pyinstaller. However, I found that using step 3 breaks exporting a graph using the right click->Export option. I was able to tweak it a little bit to use the code after the "importModules" line to create the Exporters list the way pyqtgraph expects. See here: https://gist.github.com/plasmoidia/6909107

Jonathan

[Luke Lee]

Jonathan,

Thanks for the update.  That makes sense, I never tried to run any code that used the exporting directly after I applied my diffs.  I was just trying to get the entire package to load for testing.

Thanks again for fix.

Luke Lee

[Luke M]

All,

It has been a while since I worked on this, but I recently came back to it and noticed that the changelog mentions that 0.9.9 will abandon the dynamic import system. Figuring this might help, I cloned the latest development code from github and installed it. Unfortunately, once installed, it didn't work (when I imported pyqtgraph, I got a never ending string of errors). I'm not really sure what's going on there, and since it was a development version anyway I figured I'd drop it and went back and installed 0.9.8 (using python setup.py install). Out of curiosity, I tried pyinstaller anyway (because v2.1 had been released since I last tried it) and to my surprise, it worked fine!  Now if I completely wipe out pyqtgraph and install 0.9.8 from scratch, it doesn't work (throws errors in frozenSupport.listdir).

It is awesome that I was able to get it to work and I've already distributed my application to some co-workers who don't have python installed. However, my question is this: What is it about installing 0.9.8 over the development code that allowed this to work? I just want to make sure I'll be able to re-create it in the future...

Thanks,

Luke M

[Luke Campagnola]

On Mon, Mar 24, 2014 at 7:23 PM, Luke M <lcj...@gmail.com> wrote:

It has been a while since I worked on this, but I recently came back to it and noticed that the changelog mentions that 0.9.9 will abandon the dynamic import system. Figuring this might help, I cloned the latest development code from github and installed it. Unfortunately, once installed, it didn't work (when I imported pyqtgraph, I got a never ending string of errors). I'm not really sure what's going on there, and since it was a development version anyway I figured I'd drop it and went back and installed 0.9.8 (using python setup.py install). Out of curiosity, I tried pyinstaller anyway (because v2.1 had been released since I last tried it) and to my surprise, it worked fine!  Now if I completely wipe out pyqtgraph and install 0.9.8 from scratch, it doesn't work (throws errors in frozenSupport.listdir).

It is awesome that I was able to get it to work and I've already distributed my application to some co-workers who don't have python installed. However, my question is this: What is it about installing 0.9.8 over the development code that allowed this to work? I just want to make sure I'll be able to re-create it in the future...

Hah, I have no idea  :)

Generally installing multiple versions on top of each other is bad news. Let's focus on what was wrong with the develop branch--this has been quite stable for me and it would be great to have proper pyinstaller support for 0.9.9.

[Luke M]

On Monday, March 24, 2014 7:40:04 PM UTC-4, Luke Campagnola wrote:

Hah, I have no idea  :)

Generally installing multiple versions on top of each other is bad news. Let's focus on what was wrong with the develop branch--this has been quite stable for me and it would be great to have proper pyinstaller support for 0.9.9.

Not being an expert with such things, I assumed that running setup.py install would overwrite existing installs... obviously wrong. That being said, troubleshooting what is wrong with my install of the develop branch is difficult because the python console literally cycles endlessly and I can't really get a grasp of what it's doing. I was able to catch a screenshot, attached below. This is using python 2.7.2 (and after realizing how far behind I was, 2.7.6). Do you have any thoughts?

[Luke Campagnola]

On Mon, Mar 24, 2014 at 8:21 PM, Luke M <lcj...@gmail.com> wrote:

Not being an expert with such things, I assumed that running setup.py install would overwrite existing installs... obviously wrong.

I'm pretty sure I can make the installer raise an error if pg is already installed.. I'll give that a try.

 

That being said, troubleshooting what is wrong with my install of the develop branch is difficult because the python console literally cycles endlessly and I can't really get a grasp of what it's doing. I was able to catch a screenshot, attached below. This is using python 2.7.2 (and after realizing how far behind I was, 2.7.6). Do you have any thoughts?

First: you can use the task manager to kill the python process; that'll allow you to copy-paste the error in the future.

The exceptions I see there aren't even inside pyqtgraph; they are coming from pyreadline. Of course it's possible I did something wrong to make pyreadline unhappy.. 

Try this:

    git checkout 9cfc3a9f851e

    python examples/

    git checkout develop

 

If it doesn't work there, then we might need to try git bisect to determine which commit causes the problem.. Thanks for your help!

[Luke M]

Good call on the task manager.

I'm not sure I fully understood your other instructions. I checked out the version you suggested. Running python examples/ worked. Then I checked out develop, python examples/ still worked. I tried deleting pyqtgraph.....egg from site-packages and re-installing from both checkouts (python setup.py install from the repo directory) and both still produced the same error on import.

Not sure if it's related, but at the very beginning of the installation process I get the following:

This appears to be a git checkout, but an error occurred while attempting to determine a version string for the current commit.

Traceback (most recent call last):

  File "tools\setupHelpers.py", line 103, in getVersionStrings

    gitVersion = getGitVersion(tagPrefix='pyqtgraph-')

  File "tools\setupHelpers.py", line 50, in getGitVersion

    tagNames = check_output(['git', 'tag'], universal_newlines=True).strip().split('\n')

  File "C:\Python27\lib\subprocess.py", line 566, in check_output

    process = Popen(stdout=PIPE, *popenargs, **kwargs)

  File "C:\Python27\lib\subprocess.py", line 709, in __init__

    errread, errwrite)

  File "C:\Python27\lib\subprocess.py", line 957, in _execute_child

    startupinfo)

WindowsError: [Error 2] The system cannot find the file specified
running install
.
.
.

[Luke Campagnola]

On Mon, Mar 24, 2014 at 10:35 PM, Luke M <lcj...@gmail.com> wrote:

I'm not sure I fully understood your other instructions. I checked out the version you suggested. Running python examples/ worked. Then I checked out develop, python examples/ still worked. I tried deleting pyqtgraph.....egg from site-packages and re-installing from both checkouts (python setup.py install from the repo directory) and both still produced the same error on import.

Interesting..  so you only get the error for installed versions of the package??

The error comes from this: https://code.google.com/p/colorama/issues/detail?id=47

The commit I had you check out was the one immediately before colorama was added to the library, so that should have fixed it; perhaps there were still files lingering from the previous install?

Try the following, if you don't mind:

1. Make a clean install from the latest develop branch

2. Download this file: https://colorama.googlecode.com/issues/attachment?aid=470001000&name=win32.py&token=jRG3YVsrkvIQA34_qx-oOLBBXjM%3A1395715708721

3. Move it to your installed pyqtgraph/debug/colorama/ folder (not the source folder, but the one under C:\python27\lib\site-packages or similar) 

4. Let me know it it helps :)

[Luke Lee]

At the risk of derailing this thread, what was the reasoning behind abandoning the dynamic imports?  This was probably a relatively large change.  So, I assume there was a big motivator behind it.

I haven't ever used dynamic imports in a large project.  So, I'd love to hear the pros and cons just for my own education.

[Luke Campagnola]

On Tue, Mar 25, 2014 at 9:57 AM, Luke Lee <durd...@gmail.com> wrote:

At the risk of derailing this thread, what was the reasoning behind abandoning the dynamic imports?  This was probably a relatively large change.  So, I assume there was a big motivator behind it.

I haven't ever used dynamic imports in a large project.  So, I'd love to hear the pros and cons just for my own education.

Dynamic import pros:

* No need to keep track of every module in a large collection; new files are automatically imported.

Cons:

* Makes py2exe more difficult

* Makes pyinstaller impossible

* Causes a variety of similar problems in other environments where the assumptions about where to locate module files are not valid.

So basically, after the fifth or so complaint about dynamic import issues, I decided it was not work the hassle. I doubt I will ever get a complaint about it being difficult to add new GraphicsItems to the package :)

[Luke Lee]

That all makes sense.  Thanks for the reply!

--
You received this message because you are subscribed to a topic in the Google Groups "pyqtgraph" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pyqtgraph/kgOhk3C91xQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pyqtgraph+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pyqtgraph/CACZXET9m6uUh2qdetqtEf5LRG%2Bdqg9D%2ByE7wue%2Bq6kAf-mERNA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Luke M]

On Monday, March 24, 2014 10:56:39 PM UTC-4, Luke Campagnola wrote:

Interesting..  so you only get the error for installed versions of the package??

The error comes from this: https://code.google.com/p/colorama/issues/detail?id=47

The commit I had you check out was the one immediately before colorama was added to the library, so that should have fixed it; perhaps there were still files lingering from the previous install?

Try the following, if you don't mind:

1. Make a clean install from the latest develop branch

2. Download this file: https://colorama.googlecode.com/issues/attachment?aid=470001000&name=win32.py&token=jRG3YVsrkvIQA34_qx-oOLBBXjM%3A1395715708721

3. Move it to your installed pyqtgraph/debug/colorama/ folder (not the source folder, but the one under C:\python27\lib\site-packages or similar) 

4. Let me know it it helps :)

Yes I only got the error for the installed version.

I followed your instructions, though I couldn't find pyqtgraph/debug. Is it possible you meant pyqtgraph/util/colorama? I put your file in "C:\Python27\Lib\site-packages\pyqtgraph-0.9.8-py2.7.egg\pyqtgraph\util\colorama" and it looks like it worked. I can import pyqtgraph without errors and I'm still able to bundle a simple program using pyinstaller.

[Luke Campagnola]

Yep, that's what I meant :)

Thanks for testing this! I'll include it shortly..

 

[Luke M]

On Tuesday, March 25, 2014 7:15:58 PM UTC-4, Luke Campagnola wrote:

Yep, that's what I meant :)

Thanks for testing this! I'll include it shortly..

 

No problem, thanks for all your help!