COM server threads/unloading

[Uncle Enzo]

Hello,

I have noticed that my multithreaded COM servers are not unloading
when their clients have unloaded.

To reproduce this in a simple case, I am using the py2exe COM server
recipe. Running the server example with the client code below results
in the testserver.exe loading after the dispatch call, and unloading
after the server reference is set to None.

Then I added the googlelogger import statement and the two lines
initializing and calling the log method. The logger that has two
threads. Using the same client code results in the testserver.exe
loading after the dispatch call, but it does not unload after the
server reference is set to None, nor does it unload after the client
terminates.

What is the proper way to get a multithreaded python COM server to
unload when there are no external references to it?

Thanks in advance,

Enzo

ps, comments about my logger design are welcome, but I'm mostly
interested in the threading issues :)

-----------------------------

Python 2.2.1 (#34, Apr 9 2002, 19:34:33) [MSC 32 bit (Intel)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win32com.client
>>> server = win32com.client.Dispatch("Python.TestServer")
>>> print server.Hello('enzo')
Hello enzo
>>> server = None
>>> ^Z

----------------------------
file: testcomserver.py

import googlelogger
import sys
import pythoncom

if hasattr(sys, 'importers'):
# we are running as py2exe-packed executable
pythoncom.frozen = 1

class HelloWorld:
_reg_clsctx_ = pythoncom.CLSCTX_LOCAL_SERVER
if hasattr(sys, 'importers'):
# In the py2exe-packed version, specify the module.class
# to use. In the python script version, python is able
# to figure it out itself.
_reg_class_spec_ = "__main__.HelloWorld"
_reg_clsid_ = "{592C15CE-894D-11D6-9A35-0050DA05A582}"
_reg_desc_ = "Python Test COM Server"
_reg_progid_ = "Python.TestServer"
_public_methods_ = ['Hello']
_public_attrs_ = ['softspace', 'noCalls']
_readonly_attrs_ = ['noCalls']
def __init__(self):
self.softspace = 1
self.noCalls = 0
self.mylogger = googlelogger.getLogger()
# __init__()

def Hello(self, who):
self.noCalls = self.noCalls + 1
# insert "softspace" number of spaces
self.mylogger.log('caller', str(who))
return "Hello" + " " * self.softspace + str(who)

if __name__ == '__main__':
if hasattr(sys, 'importers'):
# running as packed executable.
if '--register' in sys.argv[1:] or '--unregister' in
sys.argv[1:]:
# --register and --unregister work as usual
import win32com.server.register
win32com.server.register.UseCommandLine(HelloWorld)
else:
# start the server.
from win32com.server import localserver
localserver.main()
else:
import win32com.server.register
win32com.server.register.UseCommandLine(HelloWorld)

----------------------
file: setup.py

from distutils.core import setup
import py2exe

setup(name='testCOMserver',
scripts=['testCOMserver.py'])

------------------------
file: googlelogger.py

import threading
import Queue
import sys
import pavlnchcfg
import time
import win32com.client

# This thread waits for ("name", "value") items to appear in the
queue.
# It then checks for a log named "name", creates it if it doesn't
exist,
# and appends the message to that log, with a timestamp.
class Writer(threading.Thread):
def setLogger(self, logger):
#print "setting self._logger to ", logger
self._logger = logger

def run(self):
print "starting Writer.run thread"
while 1 :
# blocks until there are items in the queue
event = self._logger._logeventqueue.get(1)
#get returns a tuple, (name, message)
if event[0] == 'QUIT': #magic quit event
print "logger.Writer.run: got magic event: returning"
return

logname = event[0]
message = event[1]
try:
filename = self._logger._dir + '\\' + logname + ".txt"
logfile = open(filename, 'a')
logfile.write(time.asctime() + " " + message + '\n')
logfile.flush()
logfile.close()
print "logger.Writer.run: logged ", message, " to ",
logname
except:
print "logger.Writer.run: with to log: ", logname, "
failed with message: ", message

class Logger:
# class variables
_logeventqueue = Queue.Queue() #one thread safe multiple user
queue
_dir = ""

def __init__(self):
try:
Logger._dir="C:\\" # for our example
print "logger.Logger.init: got log directory",
Logger._dir, "from config"
except:
Logger._dir = "C:\\"
print "logger.Logger.init: No log directory specified in
config, _dir set to C:\\"

writer = Writer()
writer.setName("writerthread")
writer.setLogger(self)
writer.start()

def log(self, logname, message):
event = logname, message #making the event tuple
# event tuple is tuple[0] = name, tuple[1] = event
Logger._logeventqueue.put(event)

# factory pattern as module instance
_loggerInstance = None

def getLogger():
import logger
if logger._loggerInstance == None:
logger._loggerInstance = Logger()
return logger._loggerInstance
else:
return logger._loggerInstance

if __name__ == '__main__': #test main
import logger
print "starting main"
l = getLogger()
print l
l.log("DEBUG", "l, debug message 1")
l.log("USAGE", "l, usagestat 1")
l.log("DEBUG", "l, debug message 2")
lg = getLogger()
print lg
lg.log("DEBUG", "lg, debug message 1")
lg.log("USAGE", "lg, usagestat 1")
lg.log("DEBUG", "lg, debug message 2")
l.log("QUIT", "QUIT")
print "done"

[Mark Hammond]

There are a couple of things you should:
* Ensure CoUninitialize() is called for each thread (you would have
needed to call CoInitialize[Ex] manually for these threads.
* Call CoUninitialize() on the main thread. You did not need to call
* Ensure pythoncom._GetInterfaceCount() and pythoncom._GetGatewayCount()
both return zero. If not, your Python program still has a COM
reference, and this should be nuked.

As far as I know, if all these conditions are met everytihing will
unload correctly.

Mark.

[Uncle Enzo]

Thanks for your reply, Mark. I have a couple of follow-up questions...

Mark Hammond <mham...@skippinet.com.au> wrote in message

> * Ensure CoUninitialize() is called for each thread (you would have
> needed to call CoInitialize[Ex] manually for these threads.
> * Call CoUninitialize() on the main thread.

I noticed in the documentation that CoUninitialize() is not called for
the main Python thread automatically. But, how will my server know
when to call it?Should I ask my clients make a Release-like call?

Also, my testcomserver is simple, but it imports the multithreaded
logger. Does proper use of pythoncom require me to add pythoncom
Co(Un)Initialize() calls to the logger?

Thanks in advance,

Enzo

[Kris J. Zaragoza]

In article <10d944cb.02062...@posting.google.com>, Uncle Enzo wrote:
> Then I added the googlelogger import statement and the two lines
> initializing and calling the log method. The logger that has two
> threads. Using the same client code results in the testserver.exe
> loading after the dispatch call, but it does not unload after the
> server reference is set to None, nor does it unload after the client
> terminates.
>
> What is the proper way to get a multithreaded python COM server to
> unload when there are no external references to it?

Mark makes some good suggestions in his reply to you. However, your
problem appears to have nothing to do with COM and everything to do
with threading. In your googlelogger module, you create a thread that
runs in an infinite loop. You built a mechanism to shut down the
thread by sending it a magic QUIT command. However, your COM
component never sends it that command. Your server doesn't shut down
because you have threads still running waiting on a queue that will
never contain more messages.

As I see it, you have two options. First, you can build a mechanism
to explicitly send the logger the message to shut down. Second, you
can set the logging threads to be daemon threads with setDaemon().
This will flag the thread such that it will not keep the process from
exiting when all other threads have terminated. It's a quick and easy
way to terminate the thread without having to explicitly tell it to
shut down.

Then again, as Tim Peters writes, "Explicit is better than implicit."

-Kris

--
Kris J. Zaragoza | On the face of it, Microsoft complaining about
kzar...@attbi.com | the source license used by Linux is like the
| event horizon calling the kettle black.
| -- Adam Barr, article on kuro5hin.org

[Uncle Enzo]

Thanks Kris, for your reply. I have a follow-up question...

kzar...@attbi.com (Kris J. Zaragoza) wrote:
>(snip) Your server doesn't shut down

> because you have threads still running waiting on a queue that will
> never contain more messages.
>
> As I see it, you have two options. First, you can build a mechanism
> to explicitly send the logger the message to shut down.

How would you do this? Start another thread that periodically checks
mainthread.GetInterfaceCount(), and if it returns zero put in the quit
event and then return? I'd rather not have a shutdown or release
method exposed to users.

Thanks,

Enzo

[Kris J. Zaragoza]

In article <10d944cb.02062...@posting.google.com>, Uncle Enzo wrote:

> Thanks Kris, for your reply. I have a follow-up question...

No problem. Glad I could help. :)

>> As I see it, you have two options. First, you can build a mechanism
>> to explicitly send the logger the message to shut down.
>
> How would you do this? Start another thread that periodically checks
> mainthread.GetInterfaceCount(), and if it returns zero put in the quit
> event and then return? I'd rather not have a shutdown or release
> method exposed to users.
>

If you don't want to use an explicit shutdown call, look into defining
a __del__ method on your COM object. These get called when the
reference count on Python objects reaches zero, and I think they get
called when the reference count on the COM object goes to zero and the
object is released. (Mark the man could say more on this.) You could
send the magic quit command to the background thread in there.

I'd dig into this more for you, but I'm actually on vacation and
borrowing a pathetically slow dialup line from my cousin. The docs
for the Python COM extensions should be able to tell you more.

Good luck!

[Uncle Enzo]

Thanks for your suggestions Kris, SetDaemon() did the trick!

Enzo