avoid out of range error from googlebot

27 views
Skip to first unread message

BlueShadow

unread,
Nov 20, 2013, 10:23:54 AM11/20/13
to web...@googlegroups.com
Hi,
to increase my speed I have a fast downloadfunction for my images:
def fast_download():
   session
.forget(response)
   cache
.action(time_expire=604800)(lambda: 0)()
   
# very basic security (only allow fast_download on your_table.upload_field):
   
if not request.args(0).startswith("Images."):
       
return download()
   
   
#---------check which picture we are allowed to display--------
   
#----get the db entry and determine whether it is a small or big pic----
   rows
=db(db.Images.thumb == request.args(0)).select()
   
try:
       rows
[0]
   
except:
       small
=0
       rows
=db(db.Images.Image == request.args(0)).select()
   
else:
       small
=1

   
#print rows[0],small
   row
=rows[0]
   
if ...#some stuff to determine which image is shown (not relevant)


   from PIL import Image
   img=row.thumb
   filename = os.path.join(request.folder,'uploads',img)

       
return response.stream(open(filename,'rb'))

which works quite good.
My Problem is that the google image bot tries quite frequently to access the function directly. which gives an out of range error. (268 in the last 3 weeks)
part of one error ticket:
File "/home/www-data/web2py/applications/Movie_Alarm/controllers/default.py", line 327, in fast_download
row=rows[0]

HTTP_FROM:
googlebot(at)googlebot.com


HTTP_USER_AGENT:
Googlebot-Image/1.0
REQUEST_URI:
/fast_download/Images.thumb.a791255ac1014c88.7466d626e61696c2e6a7067.jpg


I tried to put fast_download in the disallowed part of the robots.txt
but it seems to have no effect.
and I hate having so many errors in my error database :)
have you guys any ideas how I can prevent this from happening again?

Niphlod

unread,
Nov 20, 2013, 11:21:39 AM11/20/13
to web...@googlegroups.com
shouldn't this question be raised to google than web2py ?
if this is unsolvable, wrap the entire thing in a try except raising a HTTP(404) to avoid logging the error.

Kevin Bethke

unread,
Nov 20, 2013, 11:41:02 AM11/20/13
to web...@googlegroups.com
that was the answer I was looking for. how do I direct the request to a 404 page or some other function?


--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/fCwBUaMzfsg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Niphlod

unread,
Nov 20, 2013, 4:51:09 PM11/20/13
to web...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages