pumpThread, PyQt thread not closing

[thirstydevil]

I've noticed a few issues using PyQt and pumpThread. It seems that
randomly when users closes Maya it will do 1 of these:

1) Close successfully.
2) Not closes. Instead require 2 close requests by the user.
3) Closes but leaves 1 Maya thread running ( the pumpThread ) and
still consuming Maya's memory footprint

In an attempt to understand the problem I've reconfigured pumpThread
based on Jon's and another user's version posted here (can't remember
who and cant be bothered to hunt for it again, sorry to who ever you
are). I'll blat out my thoughts and you guys can hopefully correct me
or confirm my understanding.

-----------------------------------------------------------------------------------------------------------------------------------------------------------------

I've not done any multi threading stuff before, but this is what I
understand is happening with pumpThread and the way in which it's
operating.

pumpThread is spawning 1 sub thread from Maya's main thread. The UI
is executing in that new thread.
threading.Thread( target = pumpQt, args = () )

Whenever a maya cmd is called in the PyQt UI code it is called with
utils.executeDeferred( processor ) in pumpQt block.

This intern is executed in maya's main thread when it can. I simply
added the following line to the pumpQt whille loop print "pump..."
makes sure a maya cmd is called every pumpThread cycle.

As this loop is executing quicker than maya's main loop and stack of
evalDeffered print statements build up in the main thread. In my
pumpThread module I've created a global for the while loop (gPump).
Using killPumpThread I can instantly stop and close the thread. But
Maya is still printing "pump..." untill the stack of evalDeffered cmds
have finished.

Here's my test code with my new pumpThread module

import pumpThread
pumpThread.initializePumpThread()
# Wait 20 secs
pumpThread.killPumpThread()
# only a small number of prints executed after thread is closed. Maybe
2-5 cycle depending

# Load my PyQt UI and leave open and pumping for 20secs
pumpThread.killPumpThread()
# window closes quickly and tread die's in Task Manager. But 926
lines of pump... printed before maya is finished

So how does that relate to the 3 errors above?

1) Close successfully.
Thread has closed in time and Maya has no evaldeffers in the stack

2) Not closes. Instead require 2 close requests by the user.
Thread has closed in time but maya is still pumping

3) Closes but leaves 1 Maya thread running ( the pumpThread ) and
still consuming Maya's memory footprint
Thread didn't close, UI could have been left open or maya not closing
sun threads properly?

It seems I can't do anything about error 2 as this is down to how
much and often the threads are communicating. But I can help on error
3 by calling pumpThread.killPumpThread() with a script job on Maya
exit.

"RANT ON"
I've spent nearly 6 months researching PyQt in Maya and so far
(especially if you are having to support 64bit) I have to say it's
allot of pain to maintain. Maya doesn't play nicely and will crash
with no warning if you try to do something wrong or something it's not
happy with. When you have 12 UI modules and 500 - 3000 lines of code
in each it is a real headache to comment out blocks of code to find
the line that is crashing Maya. Sometimes it's not you. I had one
example where by I was creating a QIcon as a module global, it was
being used allot in the UI. This line had been in for months, but all
of a sudden was crashing Maya. I had to put the QIcon inside a class
to solve the Maya crash. Developing a complex UI as I have with PyQT
with lots of dependencies on Maya tools, like Pymel and clr means that
you loose the ability to run the UI in Wings with debugging.
Something you really need. Instead. I think using C++ Qt would give
you the hooks to run in debug mode with Maya. You wouldn't have to
compile 64bit PyQt as well. But then you loose the benefits of
developing in a scripting environment.
"RANT OFF"

I'm sure that other studios trying to implement PyQT into there
pipeline have faced these issues. I was wondering if anyone else
could shed some light of the pit falls and hoops you are jumping
through.

Anyway here is my edited pumpThread module

-Dave
----------------------------------------------------------------------------------------------------------------------------------------------------------

from PyQt4 import QtCore, QtGui
import maya.utils as utils
import sys
import time
import threading
import gc
import maya.cmds as cmds

pumpedThread = None
app = None
gPump = True

def get_app():
global app
return app

def set_app(i_app):
global app
testAppInstance = QtCore.QCoreApplication.instance()
if testAppInstance:
app = testAppInstance
else:
app = i_app

def get_pt():
global pumpedThread
return pumpedThread

def set_pt(i_pt):
global pumpedThread
pumpedThread = i_pt

def pumpQt():
global app
global gPump
def processor():
app.processEvents()
while gPump:
time.sleep(0.01)
utils.executeDeferred( processor )
#print "pump..."

def killProcess():
global gPump
gPump = False

def killPumpThread():
if get_app():
get_app().closeAllWindows()
if get_pt():
killProcess()
set_pt(None)
set_app(None)
gc.collect()


class QThreader(QtCore.QThread):
'''An attempt to use QThread instead of Threading. Counld get
this to pump in the same fasion'''
def __init__(self, funcPointer, executeDeferred=True):
super(QThreader, self).__init__()
self.runThread = True
self.funcPointer = funcPointer
self.executeDeferred = executeDeferred

def run(self):
time = long(1)
while self.runThread:
self.usleep(time)
if self.executeDeferred:
utils.executeDeferred(self.funcPointer())
else:
self.funcPointer()

def killThread(self):
self.runThread=False

def startThread(self):
self.runThread=True


def initializePumpThread():
global gPump
gPump = True
if get_pt() == None:
set_app(QtGui.QApplication(sys.argv))
set_pt(threading.Thread( target = pumpQt, args = () ))
get_pt().start()

[Chris G]

Have you tried setting Thread.deamon / Thread.setDaemon() ?
If the thread is not daemonic, it may block python finalization.

-Chris

[David Moulder]

Hmm.  1st I've heard of daemonic.  I'll give it a go tomorrow.

Cheers

-Dave

[da...@thirstydevil.co.uk]

Hi guys,

Tried get_pt().setDaemon(True)...
This seemed to make no difference what so ever. Maya is still
randomly leaving 1 thread open.

So with no other option. I've added a scriptJob to pumpThread. This
has helped reduce the amount of times the thread is left open.
Happens about half the time now. But it's still a problem. So, what
else can I do? It's seems that I'll have to write a service to check
for a Maya application with just 1 thread. Then manually kill the
thread.

Alternatively I could do this as a python script when maya starts/or
closes. Anybody know of a way to "kill a process tree" from default
python?

-Dave

On Sep 16, 10:37 pm, David Moulder <da...@thirstydevil.co.uk> wrote:
> Hmm.  1st I've heard of daemonic.  I'll give it a go tomorrow.
>
> Cheers
>
> -Dave
>

[Chris G]

When you say "when users closes Maya" does that mean the user is going
through something that calls killPumpThread ? Or are they bypassing
that? Do you see problems in both cases?

[David Moulder]

pumpThread is running in the background when any PyQt window is opened because that UI has called initializePumpThread().
So the users are :-

Opening a PyQt window.
Then close that PyQt Gui.
Then close Maya as normal.

7/10 times 1 thread will be left open.
So I've setup a script job that calls killPumpThread on maya exit.  This has reduced the number of time's 1 thread is still left open.  This happens on both 2009, 2010, 2009x64 and 2010x64, XP and Vista.  The UI is nothing but a empty QDialog for testing. But infact, if you remove a PyQt UI from the problem and just put a print in the pumpQt() call.  Open maya and manualy start a thread by initializePumpThread() and repeat the above steps (without my script job calling killPumpThread ) then maya might close, and might not.  And by that I mean you might have to press the close button 10x before the Maya will close.

# The print call to test pumpThread

def pumpQt():
        global app
        global gPump
        def processor():
                app.processEvents()

                print "Pump"

        while gPump:
                time.sleep(0.01)
                utils.executeDeferred( processor )

I've solved the problem today in a hacky but reliable way.  In the initializePumpThread() I'm using DotNet via clr to determin if a maya process is running with one thread.  If so I kill it, cleaning up any rouge bad pumpThreads.  Also in the scriptJob I'm using DotNet to get the current process and kill it.  The whole thing is evalDeffered but with some heavy testing, it's saving all the prefs for maya correctly, before it kill's Maya for good.

When I'm at work if anyone's interested I'll post the new pumpThread hacks here.  I'm wondering if not alot of people have come across this problem?  It's very consistant problem here.  Either people have'nt noticed it, are not using PyQt in production.  If you are using PyQt and could post that you are :) then I'd like to hear from you.

-Dave

[Chris G]

Thats an imaginative workaround! =)

I think the issue is that eventually there is a huge backlog of idle
events that starts slowing everything down.
If we can ensure that we never add another idle event until the
previous one is processed we should be good. So maybe try using an
Event:

def pumpQt():
global app
global gPump

processorEv = threading.Event()
def processor():
app.processEvents()
processorEv.set()
while gPump:
utils.executeDeferred(processor)
# block until processor has run
processorEv.wait()
# aha, processor has completed, so now we sleep and repeat
processorEv.clear()
time.sleep(0.01)
print "pump..."

-Chris

[David Moulder]

Hmm,  I see what your getting at, I'll read up on the subject and then try out the theory below.  Threading is something I've never had to deal with before, I have a lot to read up on today... :)

-Dave