intermittent 404 with TG2, mod_wsgi & toscawidgets
Diez B. Roggisch
I'm baffled by a problem that leaves me out of any idea how to solve so far.
I've got a TG2-based application that amongst other things makes use of
tw.jquery to fetch jquery JS-files.
It works flawless when directly served by paster.
Today I wanted to test the application deployment under mod_wsgi.
So I created a bootstrapping-wsgi-script (my whole installation resides in a
virtualenv and is running from there standalone without a hitch)
The page itself is rendered & loaded properly.
However, various (and changing) parts of the subsequently fetched static
content, especially the JS-files, throws a 404.
But the fun part is: if I use firebug to fetch the url in question into
another tab, *IT GETS DOWNLOADED*... most of the time, that is. If I
press "reload" very often, I get intermittent 404s also.
I'm running apache2, mod_wsgi 2.0, python2.5.
THis is the startup message:
http://paste.turbogears.org/paste/9664
The vhost-conf:
http://paste.turbogears.org/paste/9666
The boostrap-wsgi-script:
http://paste.turbogears.org/paste/9667
If there is *anything* enlightening to say about this, I'm more than glad to
hear it.
I had the high hopes that such kind of behavior that I've seen PHP exhibit
would be something from a better forgotten dark past...
diez
Graham Dumpleton
> Hi,
>
> I'm baffled by a problem that leaves me out of any idea how to solve so far.
>
> I've got a TG2-based application that amongst other things makes use of
> tw.jquery to fetch jquery JS-files.
>
> It works flawless when directly served by paster.
>
> Today I wanted to test the application deployment undermod_wsgi.
>
> So I created a bootstrapping-wsgi-script (my whole installation resides in a
> virtualenv and is running from there standalone without a hitch)
>
> The page itself is rendered & loaded properly.
>
> However, various (and changing) parts of the subsequently fetched static
> content, especially the JS-files, throws a 404.
>
> But the fun part is: if I use firebug to fetch the url in question into
> another tab, *IT GETS DOWNLOADED*... most of the time, that is. If I
> press "reload" very often, I get intermittent 404s also.
>
> I'm running apache2,mod_wsgi2.0, python2.5.
Are you still using mod_python? If not, disable it out of Apache as it
can cause problems for mod_wsgi.
Did you completely 'stop' and then 'start' Apache after making any
recent changes, or at least touch the WSGI script file so daemon
processes would be reloaded.
> The vhost-conf:
>
> http://paste.turbogears.org/paste/9666
Why don't you have a ServerName directive inside VirtualHost? If only
serving one site and using ServerName from main configuration context,
you probably don't need the VirtualHost container then.
> The boostrap-wsgi-script:
>
> http://paste.turbogears.org/paste/9667
>
> If there is *anything* enlightening to say about this, I'm more than glad to
> hear it.
>
> I had the high hopes that such kind of behavior that I've seen PHP exhibit
> would be something from a better forgotten dark past...
TurboGears?
Post the lines from the Apache access log which show the 404 errors
for the request. Are the URLs there what you expect to see?
For more in depth debugging of request and response at WSGI level, use
second recipe in:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_and_Response
to capture the complete request and response. Look at them to see if
they are what you expect.
Graham
Diez B. Roggisch
> > I'm running apache2,mod_wsgi2.0, python2.5.
>
> Why aren't you using latest mod_wsgi 2.3?
Because I've set up the machine a few months ago and wasn't aware that there
is a newer version. I now upgraded.
> > THis is the startup message:
> >
> > http://paste.turbogears.org/paste/9664
>
> Are you still using mod_python? If not, disable it out of Apache as it
> can cause problems for mod_wsgi.
I now disabled it.
> Did you completely 'stop' and then 'start' Apache after making any
> recent changes, or at least touch the WSGI script file so daemon
> processes would be reloaded.
Yes, complete restarts.
> > The vhost-conf:
> >
> > http://paste.turbogears.org/paste/9666
>
> Why don't you have a ServerName directive inside VirtualHost? If only
> serving one site and using ServerName from main configuration context,
> you probably don't need the VirtualHost container then.
Because I just took what the ubuntu system was coming with, and don't know
much about apache configuration. Do you think it might affect the problem?
> > The boostrap-wsgi-script:
> >
> > http://paste.turbogears.org/paste/9667
> >
> > If there is *anything* enlightening to say about this, I'm more than glad
> > to hear it.
> >
> > I had the high hopes that such kind of behavior that I've seen PHP
> > exhibit would be something from a better forgotten dark past...
>
> Can you make it clear whether the 404 is generated by Apache or by
> TurboGears?
After putting a logging-middleware in front of my application, it appears that
the python/TG-code is responsible, not apache. See below for more remarks.
> Post the lines from the Apache access log which show the 404 errors
> for the request. Are the URLs there what you expect to see?
>
> For more in depth debugging of request and response at WSGI level, use
> second recipe in:
>
> http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Tracking_Request_
>and_Response
>
> to capture the complete request and response. Look at them to see if
> they are what you expect.
Thanks, I did that. After doing the update, things changed *slightly*: the
problem remains that loading a page itself works, yet *some* of it's referred
resources (mostly javascript, sometimes images) produce a 404.
However, the behavior that putting the url in question (e.g.
http://localhost/collaborate/toscawidgets/resources/ableton.collaborate.frontend/public/javascript/jquery/cluetip/jquery.cluetip.js
) into a tab, and pressing F5 frequently would show the intermittent 404, or
sometimes the resource, seems to be gone.
So far, the resource seems to be returned all the time.
I created a diff that I attached between two requests for the same resource,
one failing, one successful.
As one can see, there is only very little difference, and most of the
differences are to be expected.
I assume there is no chance to drop the apache into the PDB... so I will need
to have to investigate this via turning on logging inside the TG-app.
Thanks for your help so far,
Diez
Lukasz Szybalski
Try changing the settings in a wsgi to a deamon mode with 10 threads
and 3 processes(or what ever number you need it to be) and then test
your 404 again. This should fix your problem.
Lucas
Diez B. Roggisch
> However, the behavior that putting the url in question (e.g.
> http://localhost/collaborate/toscawidgets/resources/ableton.collaborate.fro
>ntend/public/javascript/jquery/cluetip/jquery.cluetip.js ) into a tab, and
> pressing F5 frequently would show the intermittent 404, or sometimes the
> resource, seems to be gone.
This was unfortunately not true, it showed the old behavior - a
non-deterministinc sequence of 200s and 404s.
> So far, the resource seems to be returned all the time.
>
> I created a diff that I attached between two requests for the same
> resource, one failing, one successful.
>
> As one can see, there is only very little difference, and most of the
> differences are to be expected.
>
> I assume there is no chance to drop the apache into the PDB... so I will
> need to have to investigate this via turning on logging inside the TG-app.
By now I found out that the culprit is tw.core.resources.ResourcesApp.
Every now and then, it fails to enumerate the registered resources.
I could confirm that this depends on the PID. If the requests hit the same PID
that the app was started with, things are ok.
If it hits another one, they fail.
So it looks as if there are instances of the application lying around that
aren't properly initialized... Currently I find this very intimidating :(
Diez
Diez B. Roggisch
> >>
> >> to capture the complete request and response. Look at them to see if
> >> they are what you expect.
> >
> > Thanks, I did that. After doing the update, things changed *slightly*:
> > the problem remains that loading a page itself works, yet *some* of it's
> > referred resources (mostly javascript, sometimes images) produce a 404.
> >
> > However, the behavior that putting the url in question (e.g.
> > http://localhost/collaborate/toscawidgets/resources/ableton.collaborate.f
> > and pressing F5 frequently would show the intermittent 404, or sometimes
> > the resource, seems to be gone.
>
> Try changing the settings in a wsgi to a deamon mode with 10 threads
> and 3 processes(or what ever number you need it to be) and then test
> your 404 again. This should fix your problem.
That didn't help.
Setting the process to 1 *did* help, not to surprising. Any other suggestions.
Diez
Graham Dumpleton
On Oct 15, 11:09 pm, "Diez B. Roggisch" <de...@web.de> wrote:
> I assume there is no chance to drop the apache into the PDB... so I will need
> to have to investigate this via turning on logging inside the TG-app.
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Python_Interactive_Debugger
Graham
Diez B. Roggisch
Thanks, I should have read the docs more careful.
I did solve the problem - sort of. It has nothing to do with mod_wsig.
Instead, some laziness on TG2/Pylons/Whomever's behalf is the culprit.
The thing that happens is pretty simple: to serve static resources outside TW,
they need to be registered.
This happens inside the controller-modules (widgets are supposed to be
globally declared, not instantiated on a per-request-base)
Now if a request hits a process that hasn't served any controller-based url,
the whole controller-hierarchy isn't loaded. Thus no registration, and thus
the 404.
Naturally the problem would go away once you put enough load on the system so
that all processes are hit by an actual controller-served request. Obviously
that can't be a solution, as it would mean that the first few dozen users or
so get errors.
I will investigate what can be done.
Diez
Lukasz Szybalski
I wonder if that is the case for tg1 as well. When I was serving my
mod_wsgi tg1 app using embedded mode that is what exactly was
happening, except it got fixed when I went into a deamon mode maybe
because the load was higher per process?!
Lucas
Lukasz Szybalski
Could you send me or post it to paste your final wsgi conf file and
apache conf file?
Thanks,
Lucas