[web2py] IOError, scalability concerned

[Thadeus Burgess]

What could possibly be causing this?

python 2.6
web2py trunk
apache/mod_wsgi 2.6

Any idea's on how I can narrow this down, or stop this? The pages
consist of static html (cached in RAM), and a page with a giant
SQLFORM on it. It kind of concerns me about the scalability of web2py,
as the errors rapidly increase as web traffic increases.

Traceback (most recent call last):
File "gluon/main.py", line 396, in wsgibase
request.body = copystream_progress(request) ### stores request body
File "gluon/main.py", line 143, in copystream_progress
copystream(source, dest, size, chunk_size)
File "gluon/fileutils.py", line 302, in copystream
data = src.read(size)
IOError: request data read error

--
Thadeus

[Thadeus Burgess]

Just some more information.

Static files are served by apache (as to the book)
Migrate = True

copystream seems to allure to the fact a file is being uploaded (which
is impossible because the application does not store any files)

Might this be a robot / scanner attempting to upload things? If so,
how to stop this?

--
Thadeus

[Thadeus Burgess]

It is not a robot... I performed an IP lookup on some of the IP
addresses that caused the error, and they come from ISP and typically
in residential areas, and spread out. So this is a big issue since
robots do not cause it.

[Graham Dumpleton]

On May 10, 12:28 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> What could possibly be causing this?

A user not waiting for a request to complete before clicking on
another link or pressing reload. In other words, client dropped
original connection.

Graham

[mdipierro]

Does this result in a ticket or console error? Do you get a lot of
requests/sec from the same IP?

On May 9, 9:49 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[mdipierro]

Not sure about the problem but I had a few instances of people
clicking reload a lot (and I mean a lot). So I use this:

# drop incoming connections if IP make more than 10 connection
attempts to port 80 within 100
seconds
$IPT -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent
--set
$IPT -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent
--update --seconds 60 --hitcount 10 -j DROP

[Thadeus Burgess]

Ticket. I usually see them the next day when I check admin.

No, it is usually just one IP but it happens to a lot of people at the
same time, scaled by the amount of traffic being put on the server.
Roughly about 10% of the requests generate this error.

> A user not waiting for a request to complete before clicking on
> another link or pressing reload. In other words, client dropped
> original connection.

Obviously this is a web2py issue then, because I don't have any
problems when I go click-happy on other web sites.

--
Thadeus

[mdipierro]

How many requests/day do you have?
I have never seen tickets like this.
Clearly this is an issue. I run 2.5. Could this be a problem with 2.6
and web2py?

Massimo

On May 9, 10:51 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Ticket. I usually see them the next day when I check admin.
>
> No, it is usually just one IP but it happens to a lot of people at the
> same time, scaled by the amount of traffic being put on the server.
> Roughly about 10% of the requests generate this error.
>
> > A user not waiting for a request to complete before clicking on
> > another link or pressing reload. In other words, client dropped
> > original connection.
>
> Obviously this is a web2py issue then, because I don't have any
> problems when I go click-happy on other web sites.
>
> --
> Thadeus
>

[Graham Dumpleton]

On May 10, 1:51 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Ticket. I usually see them the next day when I check admin.
>
> No, it is usually just one IP but it happens to a lot of people at the
> same time, scaled by the amount of traffic being put on the server.

Then it is likely that the ISP or network all the requests were routed
by dropped all the connections for whatever reason.

> Roughly about 10% of the requests generate this error.
>
> > A user not waiting for a request to complete before clicking on
> > another link or pressing reload. In other words, client dropped
> > original connection.
>
> Obviously this is a web2py issue then, because I don't have any
> problems when I go click-happy on other web sites.

Part of the problem is that there is no standard for what type of
Python exception is generated by a dropped connection. The mod_python
and mod_wsgi package so happens to use IOError, but different
descriptions. Other WSGI servers are within their rights to use a
completely different Python exception or yet another description
against an IOError. Thus, it becomes really hard for a generic
framework that can be hosted in various ways to make a judgement as to
whether a failure on read was due to a particular type of error. Thus
it becomes hard to ignore errors for loss of connection. You also by
ignoring them, limit an applications ability to take some special
action when connections are dropped.

It therefore isn't obvious what to do and most Python frameworks will
as a result just pass the exception up the stack and cause a 500
response. If you have a mailout option for errors back to system
administrators then you obviously may get an lot of emails. Best you
might do is for that mailout middleware to allow a user to supply
their own rules, ie., exception types and desription regex, for things
that should be ignored as far as mailout message to admin.

Graham

[mdipierro]

gave this some more thought (still not enought)
The problem is IOError which happens probably because users with slow
connection click reload.

Right now this is percaived by web2py an internal error, caught by
main.py:441 and logged.

Perhaps right before line 441 in main.py we should have something like

except IOError:
session._unlock(response)
return []

the problem is that IOError may also come from a filesystem IO and in
that case you definitively want a ticket issued.
There should be a way to deal with socket IO errors and filesystem IO
errors differently.

[mdipierro]

Graham is right. The best we can do is catch

try:
data = src.read(size)
except:
raise MySocketException

and catch it later as I suggested above. What do you think?

Massimo

On May 9, 11:52 pm, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:

[Thadeus Burgess]

So you are saying that a user with a slow connection (which consists
of a large portion of the sites user base) clicks reload, so their
browser stops communicating with that request, so web2py catches it as
an IOError... so then is this safe to ignore?

Below is a log that I found in apache2.error.log

[Sun Jan 31 13:42:50 2010] [error] [client ******] (70007)The timeout
specified has expired: mod_wsgi (pid=3961): Unable to get bucket
brigade for request., referer: http://mysite.com/page1
[Sun Jan 31 13:42:51 2010] [error] [client ******] mod_wsgi
(pid=3926): Exception occurred processing WSGI script
'/web2py/wsgihandler.py'.
[Sun Jan 31 13:42:51 2010] [error] [client ******] IOError: failed to write data

--
Thadeus

[mdipierro]

How about replace?

parse_get_post_vars(request, environ)

with

try:
parse_get_post_vars(request, environ)
except IOError:
return []

[mdipierro]

Yes. it is safe to ignore, but it still causes a minor slow down
because web2py issues a ticket and that may be unnecessary.

On May 10, 12:01 am, Thadeus Burgess <thade...@thadeusb.com> wrote:
> So you are saying that a user with a slow connection (which consists
> of a large portion of the sites user base) clicks reload, so their
> browser stops communicating with that request, so web2py catches it as
> an IOError... so then is this safe to ignore?
>
> Below is a log that I found in apache2.error.log
>
> [Sun Jan 31 13:42:50 2010] [error] [client ******] (70007)The timeout
> specified has expired: mod_wsgi (pid=3961): Unable to get bucket
> brigade for request., referer:http://mysite.com/page1
> [Sun Jan 31 13:42:51 2010] [error] [client ******] mod_wsgi
> (pid=3926): Exception occurred processing WSGI script
> '/web2py/wsgihandler.py'.
> [Sun Jan 31 13:42:51 2010] [error] [client ******] IOError: failed to write data
>
> --
> Thadeus
>
> On Sun, May 9, 2010 at 11:52 PM, Graham Dumpleton
>

[Thadeus Burgess]

What if there was an IO error with cgi.FieldStorage ?



--
Thadeus

[Thadeus Burgess]

Looking at the sites analytics, there are roughly about 1000 requests
a day, of which I will receive 100 of these tickets, give or take 20.

[mdipierro]

True. What about:

def parse_get_post_vars(request, environ):
....
try:

request.body = copystream_progress(request) ### stores request
body

except:
return False
...
return True

#in wsgibase
...
if not parse_get_post_vars(request, environ): return []

On May 10, 12:10 am, Thadeus Burgess <thade...@thadeusb.com> wrote:
> What if there was an IO error with cgi.FieldStorage ?
>
> --
> Thadeus
>

[mdipierro]

strange. I get 50,000 requests/day (~10,000 non-static files) and not
one error like this. I use apache_mod_wsgi.

On May 10, 12:11 am, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Looking at the sites analytics, there are roughly about 1000 requests
> a day, of which I will receive 100 of these tickets, give or take 20.
>
> --
> Thadeus
>

> On Mon, May 10, 2010 at 12:10 AM, Thadeus Burgess <thade...@thadeusb.com> wrote:
> > What if there was an IO error with cgi.FieldStorage ?
>
> > --
> > Thadeus
>

[Graham Dumpleton]

On May 10, 3:11 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> Looking at the sites analytics, there are roughly about 1000 requests
> a day, of which I will receive 100 of these tickets, give or take 20.

If you want a quick solution, stick nginx in front as a proxy and you
likely would eliminate the errors. This is because nginx helps to
isolate you from slow clients and some of the problems they can cause.
You can also offload static file serving to nginx as well, speeding
that up and allowing you to use less resources in Apache to serve same
number of requests.

I'll comment later when have time on Massimo's comments.

Graham

> On Mon, May 10, 2010 at 12:10 AM, Thadeus Burgess <thade...@thadeusb.com> wrote:
> > What if there was an IO error with cgi.FieldStorage ?
>
> > --
> > Thadeus
>

[Thadeus Burgess]

My original concerns are eased, as I thought 120 people out of the
1000 that visited the site were getting errors [1], which seems to not
be the case.

If I were to go to all of the trouble to install nginx, I might as
well stop using apache and install cherokee. I have been running my
blog on cherokee with no problems.

Massimo, looks reasonable... we can always change it later.

[1] the requests are unique visits, no robots included

[Thadeus]

As soon as you post the fix I will update the web2py version and keep
an ear out if anyone has any connectivity issues.

Thanks.

On May 10, 9:35 am, Thadeus Burgess <thade...@thadeusb.com> wrote:
> My original concerns are eased, as I thought 120 people out of the
> 1000 that visited the site were getting errors [1], which seems to not
> be the case.
>
> If I were to go to all of the trouble to install nginx, I might as
> well stop using apache and install cherokee. I have been running my
> blog on cherokee with no problems.
>
> Massimo, looks reasonable... we can always change it later.
>
> [1] the requests are unique visits, no robots included
>
> --
> Thadeus
>
> On Mon, May 10, 2010 at 1:53 AM, Graham Dumpleton
>

[mdipierro]

In trunk, give it a try.

[Graham Dumpleton]

On May 10, 3:01 pm, mdipierro <mdipie...@cs.depaul.edu> wrote:
> Graham is right. The best we can do is catch
>
> try:
>    data = src.read(size)
> except:
>    raise MySocketException
>
> and catch it later as I suggested above. What do you think?

You can't technically do either of the things you have suggested, ie.,
catch all or catch just IOError. In WSGI you might not be reading
direct from the original wsgi.input but from a WSGI middleware. It may
introduce its own sort of IOError for totally different reasons and
may raise exceptions for other reasons. If you blindly catch all
exceptions or even just of type IOError, you may mask totally
unrelated errors and make it extremely difficult for people to debug
things.

As I said in my original post, the best you can do is to propagate up
the original exception, but have any system which catches uncaught
exceptions and mails out details be able to be configured by a user
(not by default), such that the user can specify exception types and
exception descriptions which can be ignored for the particular hosting
system or combination of WSGI middleware they use.

Graham

[mdipierro]

Good point. I guess I would revert the latest change to wsgibase()
let me me think about this some more.

On May 10, 5:48 pm, Graham Dumpleton <graham.dumple...@gmail.com>

[Thadeus Burgess]

There are alot of these tickets being generated, roughly 10%, this is
annoying from a prospective of attempting to determine the tickets
that are real! Let alone the amount of files it is placing in the
errors folder quickly exceeds the recommended max file count for a
folder.

I wish it would at least not generate a ticket. However, it might come
down to needing a cron script that will clean up any tickets that
contain IOError.

--
Thadeus

[mdipierro]

A smarter approach could be not generate a ticket if the socket is
closed but I think still Graham objection would still be valid.

On May 10, 6:51 pm, Thadeus Burgess <thade...@thadeusb.com> wrote:
> There are alot of these tickets being generated, roughly 10%, this is
> annoying from a prospective of attempting to determine the tickets
> that are real! Let alone the amount of files it is placing in the
> errors folder quickly exceeds the recommended max file count for a
> folder.
>
> I wish it would at least not generate a ticket. However, it might come
> down to needing a cron script that will clean up any tickets that
> contain IOError.
>
> --
> Thadeus
>