Unable to get bucket brigade for request
Alex Ezell
We recently tried to deploy mod_wsgi (v2.3) with Apache 2.x in our
Django stack. On our beta machines, with little to no load, it worked
fantastically.
However, as soon as we moved it to production and it was under a
heavier load, we began getting these errors:
[Thu Oct 16 11:26:30 2008] [error] [client 10.3.10.154] (70007)The
timeout specified has expired: mod_wsgi (pid=17615): Unable to get
bucket brigade for request., referer: https://www.example.com/app2/import/new/
There were some other configuration problems outside of mod_wsgi that
forced us to rollback the mod_wsgi implementation as well. However, we
have still not been able to discover the reason for or replicate this
error in any of our testing.
That URL that is the referrer typically does a file upload, so I
suspect those uploads were getting lost and some anecdotal evidence
from our users while mod_wsgi was enabled indicates lost or bad file
uploads.
My question then is:
1) Is there information on this error that I could peruse?
2) Does anyone know whether this is just a sorry configuration for
mod_wsgi or is it something deeper and more nefarious?
3) Perhaps this has nothing to do with mod_wsgi, in which case, I
apologize for wasting your time, but would love a pointer to somewhere
better I might ask this question.
Here are the only mod_wsgi directives we had in our Apache conf
(filepaths are illustrative):
WSGIScriptAlias /app2 /home/python/wsgi.py
WSGIDaemonProcess python_app display-name=wsgi-app
WSGIProcessGroup python_app
Thanks for your time.
/alex
Graham Dumpleton
>
> Hi all,
> We recently tried to deploy mod_wsgi (v2.3) with Apache 2.x in our
> Django stack. On our beta machines, with little to no load, it worked
> fantastically.
What operating system/variant/version are you using?
Which Apache MPM are you using, prefork or worker?
> However, as soon as we moved it to production and it was under a
> heavier load, we began getting these errors:
>
> [Thu Oct 16 11:26:30 2008] [error] [client 10.3.10.154] (70007)The
> timeout specified has expired: mod_wsgi (pid=17615): Unable to get
> bucket brigade for request., referer: https://www.example.com/app2/import/new/
>
> There were some other configuration problems outside of mod_wsgi that
> forced us to rollback the mod_wsgi implementation as well. However, we
> have still not been able to discover the reason for or replicate this
> error in any of our testing.
>
> That URL that is the referrer typically does a file upload, so I
> suspect those uploads were getting lost and some anecdotal evidence
> from our users while mod_wsgi was enabled indicates lost or bad file
> uploads.
>
> My question then is:
>
> 1) Is there information on this error that I could peruse?
Only really what I make up as I go along. ;-)
> 2) Does anyone know whether this is just a sorry configuration for
> mod_wsgi or is it something deeper and more nefarious?
Have you overridden the default Apache configuration setting of:
TimeOut 300
If so, what is it set to.
Would it be possible in Apache configuration to set:
LogLevel debug
This will generate a lot more output in Apache error log, but it may
cause Apache input filters to dump out information about what bucket
brigade access is failing.
> 3) Perhaps this has nothing to do with mod_wsgi, in which case, I
> apologize for wasting your time, but would love a pointer to somewhere
> better I might ask this question.
A work around for this may be to not delegate the specific URL which
is performing the upload to the daemon process group but allow it to
be run in the context of embedded mode. This will mean though that
Apache worker process will incur a hit on memory usage as will now run
part of your Django application. If this is acceptable, configuration
would be:
<Location /some/url/to/upload/handler>
WSGIProcessGroup %{GLOBAL}
</Location>
I am not entirely sure this will help, but when using daemon mode
there is a slight difference in the way that request content is dealt
with. In daemon mode the Apache input bucket brigade is accessed
directly where as in embedded mode an older Apache API for dealing
with request content is used which deals with buckets rather than
mod_wsgi doing it directly.
FWIW, I had someone contact me recently about a different Apache
module they were writing where they were seeing issues with that same
time of bucket code. They were wandering if I saw similar issues with
mod_wsgi. What they were seeing wasn't the same as what you have, they
were actually getting an end sentinel when there would still have been
data rather than it failing with an error. They were seeing issue on
CentOS 5.
When I get a chance I'll try and track through code to see if I can
see under what circumstances bucket brigade access may fail.
Graham
Alex Ezell
<graham.d...@gmail.com> wrote:
>
> 2008/11/21 Alex Ezell <aez...@gmail.com>:
>>
>> We recently tried to deploy mod_wsgi (v2.3) with Apache 2.x in our
>> Django stack. On our beta machines, with little to no load, it worked
>> fantastically.
>
> What operating system/variant/version are you using?
Ubuntu 7.10
> Which Apache MPM are you using, prefork or worker?
It's the prefork MPM.
>> 1) Is there information on this error that I could peruse?
>
> Only really what I make up as I go along. ;-)
That works for me. :)
>> 2) Does anyone know whether this is just a sorry configuration for
>> mod_wsgi or is it something deeper and more nefarious?
>
> Have you overridden the default Apache configuration setting of:
>
> TimeOut 300
>
> If so, what is it set to.
We have not overridden this directive in our conf files, so it'd be
using that default.
> Would it be possible in Apache configuration to set:
>
> LogLevel debug
>
> This will generate a lot more output in Apache error log, but it may
> cause Apache input filters to dump out information about what bucket
> brigade access is failing.
I can try this, but one challenge is that we only ever see the issue
under production load. I can try to hook up 'ab' or something and
hammer a test machine. I might be able to force the error with this
logging turned on. If so, I'll post anything that looks curious.
>> 3) Perhaps this has nothing to do with mod_wsgi, in which case, I
>> apologize for wasting your time, but would love a pointer to somewhere
>> better I might ask this question.
>
> A work around for this may be to not delegate the specific URL which
> is performing the upload to the daemon process group but allow it to
> be run in the context of embedded mode. This will mean though that
> Apache worker process will incur a hit on memory usage as will now run
> part of your Django application. If this is acceptable, configuration
> would be:
>
> <Location /some/url/to/upload/handler>
> WSGIProcessGroup %{GLOBAL}
> </Location>
>
> I am not entirely sure this will help, but when using daemon mode
> there is a slight difference in the way that request content is dealt
> with. In daemon mode the Apache input bucket brigade is accessed
> directly where as in embedded mode an older Apache API for dealing
> with request content is used which deals with buckets rather than
> mod_wsgi doing it directly.
Thanks for this workaround. I think we'll poke through the logging
stuff above before we do this. I'm a little reticent to have Apache
handling the Django requests though this alternate API sounds
promising. I might hold this in reserve if other efforts prove
fruitless.
> When I get a chance I'll try and track through code to see if I can
> see under what circumstances bucket brigade access may fail.
Thanks so much for your help and detailed response. I really
appreciate it. We know that mod_wsgi is the way to go in the long run,
so we're excited to get past this little bump in the road.
/alex