WSGIApplicationGroup %{GLOBAL}
WSGIDaemonProcess wsgi_app_ssl processes=10 threads=1 python-path="/path_to_venv..."
WSGIProcessGroup wsgi_app_ssl
WSGIScriptAlias /service/review_v2 /path/doServiceRequest_review.wsgi
WSGIScriptAlias /service/status_update_tasks_v2 /path/doServiceRequest_ctl_v2.wsgi
.....
Application handling is a standard
from webob import Request, Response
def __call__(self, environment, responseApplication):
myRequest = Request(environment)
.....
After a single request is processed with a file that is being uploaded in a FieldStorage
The issue is that after the request, the file descriptor is still open, but deleted (using lsof).
This file appears to be open in every process.of httpd. (same filename).
a) Is the apache configuration correct in this case?
b) Am I missing something here - i.e. is WebOB at fault here?
WebOB uses cgi - which has a cleanup __del__ which is supposed to close the file - but.I have not debugged down that far....
On 3 Mar 2021, at 12:39 pm, Ezra Peisach <ezra.p...@rcsb.org> wrote:Ok, this will be complicated.Recently moving to Python3,Running mod_wsgi from apache,We needed to add:WSGIApplicationGroup %{GLOBAL}
due to third party code (numpy, lxml). This means that requests are served by primary python process
WSGIDaemonProcess wsgi_app_ssl processes=10 threads=1 python-path="/path_to_venv..."
WSGIProcessGroup wsgi_app_ssl
We then have some scripts:WSGIScriptAlias /service/review_v2 /path/doServiceRequest_review.wsgi
WSGIScriptAlias /service/status_update_tasks_v2 /path/doServiceRequest_ctl_v2.wsgi
.....
Application handling is a standard
from webob import Request, Response
def __call__(self, environment, responseApplication):
myRequest = Request(environment)
.....
After a single request is processed with a file that is being uploaded in a FieldStorage
The issue is that after the request, the file descriptor is still open, but deleted (using lsof).
This file appears to be open in every process.of httpd. (same filename).
a) Is the apache configuration correct in this case?
b) Am I missing something here - i.e. is WebOB at fault here?
WebOB uses cgi - which has a cleanup __del__ which is supposed to close the file - but.I have not debugged down that far....
Thank you for your response.
If both applications use numpy and lxml, is it safe to use the same global WSGIApplicationGroup, but use separate process groups for each applcation? The applications are related, but do not interact with each other, except through database and filesystem.
I will try this.
Independently, for webob, I believe if a file upload request is
larger than 10Kb, it buffers to a temporary file, but never closes
at end, relying on pythonic cleanup when class scope is exited.
That I can report independently.
--
You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/rvOgQsj-kN0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/9D3B14DB-861B-4B3E-AD3F-91E855A9F8D6%40gmail.com.
On 3 Mar 2021, at 10:11 pm, Ezra Peisach <ezra.p...@rcsb.org> wrote:Thank you for your response.
If both applications use numpy and lxml, is it safe to use the same global WSGIApplicationGroup, but use separate process groups for each applcation? The applications are related, but do not interact with each other, except through database and filesystem.
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/f2847256-7248-33ce-b9f3-792d570117ba%40rcsb.org.
Thank you for the followup.
With your suggestion - two process groups, application-group on WSGIScriptAlias - I am seeing file descriptor open on what appears to be the parent and process, and two subprocesses.
I have a sneaky suspicion that webob is causing a resource leak.
Without application-group specified - a similar pattern.
Reducing file upload to under 10k - reduces to a single leak per
process.
I will take an independent server - and reduce everything down to as minimal a test case as possible - and see if webob or my code is doing something odd with the Request. My reading of the code is that it should fall out of scope and cleanup. Similar python class arrangement suggests that cleanups should be happening - but I need more testing.
from lsof:
httpd 38004 xdev 11u REG
253,0 695699 151322661 /tmp/#151322661 (deleted)
httpd 38004 xdev 12u REG 253,0
694951 151322663 /tmp/#151322663 (deleted)
httpd 38004 38254 xdev 11u REG 253,0
695699 151322661 /tmp/#151322661 (deleted)
httpd 38004 38254 xdev 12u REG 253,0
694951 151322663 /tmp/#151322663 (deleted)
httpd 38004 38255 xdev 11u REG 253,0
695699 151322661 /tmp/#151322661 (deleted)
httpd 38004 38255 xdev 12u REG 253,0
694951 151322663 /tmp/#151322663 (deleted)
httpd 38004 38256 xdev 11u REG 253,0
695699 151322661 /tmp/#151322661 (deleted)
httpd 38004 38256 xdev 12u REG 253,0
694951 151322663 /tmp/#151322663 (deleted)
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/718C9466-2575-4660-A8D5-3F3E48D921DB%40gmail.com.
I have tracked down the issue in webob.
A reference assignment copy of a cgi FieldStorage within webob results in resources not cleaning up.
A minimal test case is:
def __call__(self, environment, responseApplication):
""" Request callable entry
point
"""
myRequest = Request(environment)
myResponse = Response()
myResponse.status = '200 OK'
myResponse.content_type = 'text/html'
p =
myRequest.params
return myResponse(environment,responseApplication)
results in the resource leak - when running in daemon mode. If you get rid of the "p = ...." - then no resource leak.
I will report to webob developers. I am just going to see if I
can use mod_wsgi-express to reproduce the scenario.
Thank you Graham for explaining a better way to configure what I
am trying to do.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/718C9466-2575-4660-A8D5-3F3E48D921DB%40gmail.com.
On 3 Mar 2021, at 10:46 pm, Ezra Peisach <ezra.p...@rcsb.org> wrote:Thank you for the followup.
With your suggestion - two process groups, application-group on WSGIScriptAlias - I am seeing file descriptor open on what appears to be the parent and process, and two subprocesses.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/803283b2-85d4-35ec-335f-907bbed53c2a%40rcsb.org.