For example, taking your example and changing it not to be dependent
on web.py I get:
import multiprocessing
import os
def x(y):
print os.getpid(), 'x', y
return y
def application(environ, start_response):
status = '200 OK'
output = 'Hello World!'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
print 'create pool'
pool = multiprocessing.Pool(processes=1)
print 'map call'
result = pool.map(x, [1])
print os.getpid(), 'doit', result
return [output]
If I fire off a request to this it appears to work correctly,
returning me hello world string and log the appropriate messages.
[Tue May 03 09:40:36 2011] [info] [client 127.0.0.1] mod_wsgi
(pid=32752, process='hello-1',
application='hello-1.example.com|/mptest.wsgi'): Loading WSGI script
'/Library/WebServer/Sites/hello-1/htdocs/mptest.wsgi'.
[Tue May 03 09:40:36 2011] [error] create pool
[Tue May 03 09:40:36 2011] [error] map call
[Tue May 03 09:40:36 2011] [error] 32753 x 1
[Tue May 03 09:40:36 2011] [error] 32752 doit [1]
However, the process then appears to receive a signal from somewhere
causing it to shutdown:
[Tue May 03 09:40:36 2011] [info] mod_wsgi (pid=32752): Shutdown
requested 'hello-1'.
[Tue May 03 09:40:41 2011] [info] mod_wsgi (pid=32752): Aborting
process 'hello-1'.
The multiprocessing module does issue signals, so it may be the source of this.
One thought was that this may be occurring when the pool is destroyed
at the end of the function call, so I moved the creation of pool to
module scope.
import multiprocessing
import os
print 'create pool'
pool = multiprocessing.Pool(processes=1)
def x(y):
print os.getpid(), 'x', y
return y
def application(environ, start_response):
status = '200 OK'
output = 'Hello World!'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
print 'map call'
result = pool.map(x, [1])
print os.getpid(), 'doit', result
return [output]
This though will not even run:
[Tue May 03 09:47:31 2011] [info] [client 127.0.0.1] mod_wsgi
(pid=32893, process='hello-1',
application='hello-1.example.com|/mptest.wsgi'): Loading WSGI script
'/Library/WebServer/Sites/hello-1/htdocs/mptest.wsgi'.
[Tue May 03 09:47:31 2011] [error] create pool
[Tue May 03 09:47:31 2011] [error] map call
[Tue May 03 09:47:31 2011] [error] Process PoolWorker-1:
[Tue May 03 09:47:31 2011] [error] Traceback (most recent call last):
[Tue May 03 09:47:31 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/process.py",
line 231, in _bootstrap
[Tue May 03 09:47:31 2011] [error] self.run()
[Tue May 03 09:47:31 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/process.py",
line 88, in run
[Tue May 03 09:47:31 2011] [error] self._target(*self._args, **self._kwargs)
[Tue May 03 09:47:31 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py",
line 57, in worker
[Tue May 03 09:47:31 2011] [error] task = get()
[Tue May 03 09:47:31 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/queues.py",
line 339, in get
[Tue May 03 09:47:31 2011] [error] return recv()
[Tue May 03 09:47:31 2011] [error] AttributeError: 'module' object has
no attribute 'x'
The browser also then hangs at that point.
Part of the issue here may be that WSGI script files are not really
standard Python modules in that the basename of the WSGI script file
doesn't match a module in sys.modules. If the multiprocessing module
tries to do magic stuff with imports to find original code to execute
in sub process it isn't going to work.
Specifically, may be related to:
http://code.google.com/p/modwsgi/wiki/IssuesWithPickleModule
If I attempt to move x() into being a nested function as:
import multiprocessing
import os
print 'create pool'
pool = multiprocessing.Pool(processes=1)
def application(environ, start_response):
status = '200 OK'
output = 'Hello World!'
response_headers = [('Content-type', 'text/plain'),
('Content-Length', str(len(output)))]
start_response(status, response_headers)
def x(y):
print os.getpid(), 'x', y
return y
print 'map call'
result = pool.map(x, [1])
print os.getpid(), 'doit', result
return [output]
Then one does get pickle errors, albeit for a different reason:
[Tue May 03 09:52:59 2011] [info] [client 127.0.0.1] mod_wsgi
(pid=33010, process='hello-1',
application='hello-1.example.com|/mptest.wsgi'): Loading WSGI script
'/Library/WebServer/Sites/hello-1/htdocs/mptest.wsgi'.
[Tue May 03 09:52:59 2011] [error] create pool
[Tue May 03 09:52:59 2011] [error] map call
[Tue May 03 09:52:59 2011] [error] Exception in thread Thread-1:
[Tue May 03 09:52:59 2011] [error] Traceback (most recent call last):
[Tue May 03 09:52:59 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
line 522, in __bootstrap_inner
[Tue May 03 09:52:59 2011] [error] self.run()
[Tue May 03 09:52:59 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py",
line 477, in run
[Tue May 03 09:52:59 2011] [error] self.__target(*self.__args,
**self.__kwargs)
[Tue May 03 09:52:59 2011] [error] File
"/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/multiprocessing/pool.py",
line 225, in _handle_tasks
[Tue May 03 09:52:59 2011] [error] put(task)
[Tue May 03 09:52:59 2011] [error] PicklingError: Can't pickle <type
'function'>: attribute lookup __builtin__.function failed
So, it is doing pickling in some form, which isn't going to work for
stuff in WSGI script file.
If you really want to pursue this, then suggest you move this code
outside of the WSGI script file and put it in a standard module on the
Python module search path you have set up for application.
Overall though, I would recommend against using multiprocessing module
from inside of mod_wsgi.
Graham
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7be84885-54d4-4417-adb3-42f1a0122a54n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/79fda9dd-a8d7-4c5a-a5ec-794074e4cf14n%40googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d9d9077c-53bf-45e8-a9e4-948a355ac2e1n%40googlegroups.com.