Py2exe and Export dialog

[JHallas]

Hi,

I'm not sure if you provide support for bundling to exe, but I figured this is relevant to many people using pyqtgraph.  

I have an application that uses PlotWidget to graph some data.  All the functionality works great when running directly.  When I create an exe with py2exe, I've worked all the errors so that the application runs and graphs, but when I right click on the graph and click export, the following error is generated in the log:

Traceback (most recent call last):

  File "pyqtgraph\GraphicsScene\GraphicsScene.pyc", line 546, in showExportDialog

  File "pyqtgraph\GraphicsScene\exportDialog.pyc", line 36, in show

AttributeError: 'module' object has no attribute 'ViewBox'

If I remove reference to ViewBox in exportDialog then when I run my script directly I can export only the full window, which is fine for me.  However, when I bundle to an exe and try to export I get the following error:

Traceback (most recent call last):

  File "pyqtgraph\GraphicsScene\GraphicsScene.pyc", line 546, in showExportDialog

  File "pyqtgraph\GraphicsScene\exportDialog.pyc", line 36, in show

AttributeError: 'module' object has no attribute 'PlotItem'

Any guesses on where I should look?

[Christian Gavin]

I spent myself some time to bundle to exe. I had to override os.listdir( ) so that we can also list files inside a zip archive, and a few other changes. I don't know if Luke merged these changes or not into the latest pyqtgraph, I did this a few month ago.

I think the export dialog dynamically imports other modules, and thus py2exe did not see them at the time it tried to resolve all references.

Adding in your code an explicit import for anything that is missing (ViewBox, PlotItem, and maybe others) should likely fix it.

Regards,
Christian

-- [ You are subscribed to pyqt...@googlegroups.com. To unsubscribe, send email to pyqtgraph+...@googlegroups.com ]

[JHallas]

Hi,

I followed the tutorial on his website which allowed the zip archive thing to work and fix a couple of other bugs.  I also had to include LayoutWidget and a couple of other modules manually.  The pyc files for ViewBox and PlotItem are included in the library.zip file as I eventually included the entire pyqtgraph folder.  

Adding an explicit import statement to my main script has no effect on this bug.  If I add an explicit import of those modules in the exportDialog script I get errors when I run my script directly.  Looking through the log file on that it seems that it fails when it gets into some loop importing graphicsScene.

Any ideas for other things I can try?

[Christian Gavin]

Hi,

I wrote the tutorial, but I tested it with revision 174 of pyqtgraph, this was a while ago.

I definitely need to run the whole procedure again with the latest version and update the guide accordingly.

I'll need a few days to do that but if you find anything else in the meantime, please let me know so the guide can be updated. Usually the problems I encountered were with __import__ which was trying to look for contents inside a zip file.

It's possible the export dialog attempts to dynamically import modules by using something else than os.listdir, in which case the replacement for os.listdir will not be used.

Regards,
Christian

[Christian Gavin]

Ok, I got it...

The export dialog refers to pg.ViewBox, pg.PlotItem and others, and for some reason, these names are not imported into the pg namespace when running as a bundled application.

I modified pyqtgraph/__init__.py as follows.

Change:

importAll('graphicsItems')
importAll('widgets', excludes=['MatplotlibWidget', 'RemoteGraphicsView'])

to (added lines in bold)

importAll('graphicsItems')
importAll('graphicsItems/ViewBox', ('axisCtrlTemplate_pyqt', 'axisCtrlTemplate_pyside'))
importAll('graphicsItems/PlotItem', ('plotConfigTemplate_pyqt', 'plotConfigTemplate_pyside'))
importAll('widgets', excludes=['MatplotlibWidget', 'RemoteGraphicsView'])

The modification above forces the modules under the ViewBox and PlotItem to be imported directly into the pg namespace, so that the exportDialog can now use them. I put some modules into the exclusion list so that they do not get imported, since they could cause errors trying to load pyqt or pyside (I am guessing you are using PyQt4 most likely).

With that change, you should be able to show the exports dialog without any errors (at least, it worked for me).

Luke, could this change be included into the next release of pyqtgraph? I didn't see any side effects running the application from the interpreter or bundled.

Regards,
Christian

[Christian Gavin]

Ok, almost there...

I forgot one more change, also in pyqtgraph/__init__.py.

The importAll function should now read (see changes in bold):

def importAll(path, excludes=()):
    d = os.path.join(os.path.split(__file__)[0], path)
    files = []
    for f in os.listdir(d):
        if os.path.isdir(os.path.join(d, f)) and f != '__pycache__':
            files.append(f)
        elif f[-3:] == '.py' and f != '__init__.py':
            files.append(f[:-3])
        
    # we have to use a "package.package.module" notation...
    path = path.replace("/", ".")
   
    for modName in files:
        if modName in excludes:
            continue
        mod = __import__(path+"."+modName, globals(), locals(), fromlist=['*'])
        if hasattr(mod, '__all__'):
            names = mod.__all__
        else:
            names = [n for n in dir(mod) if n[0] != '_']
        for k in names:
            if hasattr(mod, k):
                globals()[k] = getattr(mod, k)

Regards,
Christian

On 10/9/2012 10:44 AM, JHallas wrote:

[Christian Gavin]

Urg, the export dialog now shows up, but all export file types are missing :-(

Needs more investigating. Sorry for declaring victory too early...

Regards,
Christian

On 10/9/2012 10:44 AM, JHallas wrote:

[Luke Campagnola]

Thanks!

I'm going to work on this later today; hopefully I'll have some changes pushed up by tomorrow.

Luke

[Christian Gavin]

I also have some minor changes to the guide for bundling apps.

I'll email them when ready.

Here is what I am getting when running the export dialog inside a bundled app (with all the changes I made to __init__.py).

Traceback (most recent call last):

  File "pyqtgraph\GraphicsScene\exportDialog.pyc", line 122, in exportClicked
AttributeError: 'NoneType' object has no attribute 'export'

Maybe some more things missing that we need to import?

Best,
Christian

[JHallas]

I thought that maybe the changes had already been pushed and updated to version 204.  My program still runs directly, but now when I bundle the application and try to run it I get pages and pages errors reference scipy.weave:

Traceback (most recent call last):

  File "pyqtgraph\graphicsItems\PlotCurveItem.pyc", line 324, in boundingRect

  File "pyqtgraph\graphicsItems\GraphicsItem.pyc", line 188, in pixelVectors

  File "pyqtgraph\functions.pyc", line 1380, in invertQTransform

Exception: This function depends on scipy.weave library, but it does not appear to be usable.

I deleted the pyqtgraph folder and reinstalled, and the error is still present.  Any ideas?

[Christian Gavin]

In the main module of your application, after you import the bundle utils with:

from pyqtgraphBundleUtils import *

Add this before importing pyqtgraph:

# problem with scipy...
from email.mime import message, image, text, multipart, audio

I am not sure exactly why scipy depends on mime, but the line above did it for me.

The changes I proposed in the last few emails are not part of 204.

Regards,
Christian

[JHallas]

Hi,

I imported the modules you said and no change.  Have you tried bundling with version 204 of pyqtgraph?

[Christian Gavin]

Yes, bundling worked for me with 204 (I got the same exact issue you encountered with weave), with the exception of the export dialog which now shows up but has an empty file type list.

There might be other dependencies...

Try this. In pyqtgraph/functions.py,

change

try:
    import scipy.weave
    USE_WEAVE = True
except ImportError:
    USE_WEAVE = False

to

    import scipy.weave
    USE_WEAVE = True

and see what error you get in the log after running the bundled app.

Christian

[JHallas]

Now I get a lot of these:

Traceback (most recent call last):

  File "pyqtgraph\graphicsItems\PlotCurveItem.pyc", line 324, in boundingRect

  File "pyqtgraph\graphicsItems\GraphicsItem.pyc", line 188, in pixelVectors

  File "pyqtgraph\functions.pyc", line 1381, in invertQTransform

NameError: global name 'USE_WEAVE' is not defined

[JHallas]

Ok.  After a little bit more troubleshooting I figured it out that for some reason the Py2exe was no longer automatically including the scipy package.  I then included the entire scipy package, and got errors relating to email.mime.  After adding back in the email.mime imports it all works and now I'm back to square one, error for export dialog.  Thanks for the help.

[Christian Gavin]

So, your export dialog shows up but the file types are empty?

Thanks,
Christian

[JHallas]

Negative.  I haven't added the changes you mentioned yet.  I figured I'd wait until you and Luke figure it out.  Although something that I noticed earlier may help:  by default py2exe didn't include anything from pyqtgraph.exporters when I ran it.  Maybe that's the problem?  

I've opted to use GUI2Exe frontend for Py2exe along with your bundleUtils script.  To make sure I don't have any problems with modules missing, I just set GUI2exe to import all of scipy and pyqtgraph.

[Christian Gavin]

Ah, I haven't tried GUI2exe. I need to give it a shot.

It's possible that the exporters are missing from the bundle, yes. I think Luke or myself should be able to figure it out soon.

Christian

[Luke Campagnola]

On Wed, Oct 10, 2012 at 12:54 PM, JHallas <jmha...@gmail.com> wrote:

I thought that maybe the changes had already been pushed and updated to version 204.  My program still runs directly, but now when I bundle the application and try to run it I get pages and pages errors reference scipy.weave:

Traceback (most recent call last):

  File "pyqtgraph\graphicsItems\PlotCurveItem.pyc", line 324, in boundingRect

  File "pyqtgraph\graphicsItems\GraphicsItem.pyc", line 188, in pixelVectors

  File "pyqtgraph\functions.pyc", line 1380, in invertQTransform

Exception: This function depends on scipy.weave library, but it does not appear to be usable.

Sorry, that's a new bug; fixed in r205.

Luke

[Christian Gavin]

Sorry, that's a new bug; fixed in r205.

Great. I love real time bug fixing :-)

Speaking of releases, the current versioning model is to just use the revision number in source control.

I would suggest at some point use a major.minor numbering scheme, and each new version under that scheme should undergo a bit more validation, eg. ensure it can be bundled with Py2exe and Py2app, and all the features work reasonably well in either mode (bundled or not), on Mac, Linux, and Windows.

Currently, the latest version is a development release, always evolving. It would be nice to have a 1.0 at some time, which would constitute a good reference point, and a way to differentiate with the always up-to-date development releases. Also a 1.0 represents something people know is solid and they can safely use for production.

Of course this is not a rant :-) pyqtgraph is excellent as it is. I just like the major / minor numbering model. Many open source projects lack this numbering model, or they stay stuck at 0.x for many years and this doesn't give confidence to users to try them.

Best,
Christian

[Luke Campagnola]

On Wed, Oct 10, 2012 at 4:06 PM, Christian Gavin <cga...@logitech.com> wrote:

Great. I love real time bug fixing :-)

Speaking of releases, the current versioning model is to just use the revision number in source control.

I would suggest at some point use a major.minor numbering scheme, and each new version under that scheme should undergo a bit more validation, eg. ensure it can be bundled with Py2exe and Py2app, and all the features work reasonably well in either mode (bundled or not), on Mac, Linux, and Windows.

You are absolutely correct and I've been thinking about these things a bit lately (also been thinking about my dissertation, which doesn't leave much time for anything else :)

Something pretty high on my todo-list for pyqtgraph is to start implementing unit tests in a wide variety of environments. Right now, my testing procedure is just to run through all of the examples and make sure everything looks ok. Obviously this leaves much to be desired, since the majority of bug reports are now from people not running exactly my combination of linux + pyqt + numpy/scipy. 

 

Currently, the latest version is a development release, always evolving. It would be nice to have a 1.0 at some time, which would constitute a good reference point, and a way to differentiate with the always up-to-date development releases. Also a 1.0 represents something people know is solid and they can safely use for production.

The current plan is:

   1) Implement unit tests on Win/Linux/OSX with some free parameters: scipy, scipy.weave, python version (2.6, 2.7, 3), PyQt version (4.8, 4.9) or Pyside, py2exe, py2app

   2) Have a brief feature-freeze period, perhaps a month or two

   3) Roll the dev branch over to stable pyqtgraph-1.0

At that point, I may start using new version numbers for the dev branch as well.

Of course this is not a rant :-)

Please rant; I appreciate the feedback :)

Luke 

[JHallas]

I updated to 205 and inserted the changes you suggested to __init__.py and it works great.  I think the reason yours isn't working is that exporters isn't automatically put into library.zip.  Thanks for all of your help.

[Christian Gavin]

Excellent. I just need to figure out how to include the exporters with py2exe.

Best,
Christian

[JHallas]

I think if you specifically import them into your main program py2exe will pick it up.  

from pyqtgraph.exporters import *

or you can add it to the includes options on py2exe

[JHallas]

Or, you can do what I did for a long time when a certain file was missing, and just open library.zip and add in the missing files.  With winrar I just copy the file from the folder and paste it into the zip where I want it.

[Christian Gavin]

Yup, I added the 2 following packages to the py2exe config file:

"email.mime", "pyqtgraph.exporters"

and it works without having to import these in the code.

Christian

[Luke Campagnola]

Update: I have integrated Christian's os.listdir replacement and added a few other changes to support frozen environments. I decided not to install the new listdir globally--it makes me really nervous to mess with low-level services like that  (see what happened when I messed up numpy.concatenate).

I have pushed the changes to the bzr repository, but I'm going to hold off on building packages until I do a little more testing. 

I'll probably add better support for pixmaps tomorrow (so there's no more need to copy auto.png manually).

Luke

[Luke Campagnola]

Update 2:  converted all PNGs to pickled python pixmaps  (sorry, couldn't resist), so it is no longer necessary to copy PNG files when building with py2exe.

I have also included some of the bundling tutorial scripts (with modifications) under pyqtgraph/tools. I figure this is easier than trying to copy the text out of the PDF  :)

Luke

[Christian Gavin]

Great.

Once we have a release with this, the bundling guide can be greatly
simplified.

Christian

[Luke Campagnola]

On Thu, Oct 11, 2012 at 4:55 PM, Christian Gavin <cga...@logitech.com> wrote:

Great.

Once we have a release with this, the bundling guide can be greatly simplified.

Oh right, release (I forgot that part). 

r207 is up :)

[Christian Gavin]

I was wondering...

Many issues when in frozen environment are due to not importing modules the standard way. For example some modules are imported directly into the root pyqtgraph namespace, even though these modules are located in packages 1 level or more below.

Why does pyqtgraph have to dynamically import modules in some directories? Can't all the modules be imported into the code with regular import statements, with the proper package path (eg. ".graphicsItems.TextItem")?

Are the graphic items and others changing so often that it would not be practical to list them one by one for importing? It doesn't seem to be the case to me.

Thanks,
Christian

[Luke Campagnola]

On Thu, Oct 11, 2012 at 5:47 PM, Christian Gavin <cga...@logitech.com> wrote:

I was wondering...

Many issues when in frozen environment are due to not importing modules the standard way. For example some modules are imported directly into the root pyqtgraph namespace, even though these modules are located in packages 1 level or more below.

Why does pyqtgraph have to dynamically import modules in some directories? Can't all the modules be imported into the code with regular import statements, with the proper package path (eg. ".graphicsItems.TextItem")?

That's a good question. This was done originally to avoid the hassle of keeping these import lists up to date and to make it easier for people to just drop in new graphicsItems, widgets, expoerters, etc. Obviously it has had some unforseen consequences, but those appear to be largely resolved, is that correct? (In my tests with py2exe, it was not necessary to explicitly specify any extra modules, including the exporters).  

Luke

[Christian Gavin]

Hmm. I had to explicitly specify a few packages (email.mime and pyqtgraph.exporters) in the py2exe configuration file for it to work for me (otherwise, scipy would barf that "message" module cannot be found, and the exporters list would be empty in the export dialog). On OS X, I didn't have to specify either one, but I did have to add an import statement for the modules in exporters, even though exporters was included in site-packages.zip.

We could get away with the whole os.listdir and special importing function if we kept an import list though. Or, we could have a small python script that people run once and that keeps the import list up-to-date... :-) I am not sure how often people drop in new exporters or widgets and if they do, it could be documented to add an import line.

Christian

Luke