Error related to googlebot?

72 views
Skip to first unread message

Tor Nordam

unread,
Jul 12, 2010, 6:10:49 PM7/12/10
to Django users
I'm currently writing a blog framework in django, and I have it up an
running (on www.presskanne.com). During the last 20 hours or so, I
have gotten about 60 emails with error 500 messages. They seem to
originate from different pages all over the site, but they all have
two things in common:

The tracebacks all show that the problem is in importing one
particular model

The clients are all either googlebot or yahoo slurp

Also, I can't reproduce any of the errors myself. When I visit the
offending urls, everything either seems fine, or in the case of a
nonexistent page, I get a nice 404 message.

So my question is if there is anything in particular which might cause
the page to crash when a searchbot tries to load it, but not do the
same for a regular user.

Tim Chase

unread,
Jul 12, 2010, 6:21:26 PM7/12/10
to django...@googlegroups.com, Tor Nordam
On 07/12/2010 01:10 PM, Tor Nordam wrote:
> So my question is if there is anything in particular which might cause
> the page to crash when a searchbot tries to load it, but not do the
> same for a regular user.

Well, additional information about the HTTP transaction might
help. Bots may or may not include any of a subset of the following:

Refe(r)rer
Cookies
JavaScript

So you might try tweaking your browser (FF has some nice plugins
for controlling these various bits) to refrain from processing
any of these and see if it makes a difference.

-tkc

Tor Nordam

unread,
Jul 12, 2010, 9:09:57 PM7/12/10
to Django users
Thank you for your reply. I have now tried disabling cookies,
javascript and referer, and I was still unable to reproduce any of the
errors.

Here is a sample errormessage, with everything included:

Traceback (most recent call last):

File "/usr/local/lib/python2.6/site-packages/django/core/handlers/
base.py", line 80, in get_response
response = middleware_method(request)

File "/usr/local/lib/python2.6/site-packages/django/middleware/
common.py", line 57, in process_request
if (not _is_valid_path(request.path_info, urlconf) and

File "/usr/local/lib/python2.6/site-packages/django/middleware/
common.py", line 143, in _is_valid_path
urlresolvers.resolve(path, urlconf)

File "/usr/local/lib/python2.6/site-packages/django/core/
urlresolvers.py", line 301, in resolve
return get_resolver(urlconf).resolve(path)

File "/usr/local/lib/python2.6/site-packages/django/core/
urlresolvers.py", line 214, in resolve
for pattern in self.url_patterns:

File "/usr/local/lib/python2.6/site-packages/django/core/
urlresolvers.py", line 243, in _get_url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns",
self.urlconf_module)

File "/usr/local/lib/python2.6/site-packages/django/core/
urlresolvers.py", line 238, in _get_urlconf_module
self._urlconf_module = import_module(self.urlconf_name)

File "/usr/local/lib/python2.6/site-packages/django/utils/
importlib.py", line 35, in import_module
__import__(name)

File "/home/users/presskanne/dcalcuttagutta/urls.py", line 7, in
<module>
admin.autodiscover()

File "/usr/local/lib/python2.6/site-packages/django/contrib/admin/
__init__.py", line 24, in autodiscover
import_module('%s.admin' % app)

File "/usr/local/lib/python2.6/site-packages/django/utils/
importlib.py", line 35, in import_module
__import__(name)

File "/home/users/presskanne/dcalcuttagutta/articles/admin.py", line
10, in <module>
from dcalcuttagutta.articles.models import Article, ArticleVersion,
Comment, CommentVersion, Category, Tag

File "/home/users/presskanne/dcalcuttagutta/articles/models.py", line
19, in <module>
from dcalcuttagutta.members.models import Member

File "/home/users/presskanne/dcalcuttagutta/members/models.py", line
82, in <module>
class MemberCreationForm(UserCreationForm):

File "/home/users/presskanne/dcalcuttagutta/members/models.py", line
105, in MemberCreationForm
language_interface = forms.ModelChoiceField(queryset =
Language.objects.filter(interface_translation = True)) #Language of
calcuttagutta interface

File "/usr/local/lib/python2.6/site-packages/django/db/models/
manager.py", line 141, in filter
return self.get_query_set().filter(*args, **kwargs)

File "/usr/local/lib/python2.6/site-packages/django/db/models/
query.py", line 550, in filter
return self._filter_or_exclude(False, *args, **kwargs)

File "/usr/local/lib/python2.6/site-packages/django/db/models/
query.py", line 568, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))

File "/usr/local/lib/python2.6/site-packages/django/db/models/sql/
query.py", line 1131, in add_q
can_reuse=used_aliases)

File "/usr/local/lib/python2.6/site-packages/django/db/models/sql/
query.py", line 1026, in add_filter
negate=negate, process_extras=process_extras)

File "/usr/local/lib/python2.6/site-packages/django/db/models/sql/
query.py", line 1182, in setup_joins
field, model, direct, m2m = opts.get_field_by_name(name)

File "/usr/local/lib/python2.6/site-packages/django/db/models/
options.py", line 291, in get_field_by_name
cache = self.init_name_map()

File "/usr/local/lib/python2.6/site-packages/django/db/models/
options.py", line 321, in init_name_map
for f, model in self.get_all_related_m2m_objects_with_model():

File "/usr/local/lib/python2.6/site-packages/django/db/models/
options.py", line 396, in get_all_related_m2m_objects_with_model
cache = self._fill_related_many_to_many_cache()

File "/usr/local/lib/python2.6/site-packages/django/db/models/
options.py", line 410, in _fill_related_many_to_many_cache
for klass in get_models():

File "/usr/local/lib/python2.6/site-packages/django/db/models/
loading.py", line 167, in get_models
self._populate()

File "/usr/local/lib/python2.6/site-packages/django/db/models/
loading.py", line 64, in _populate
self.load_app(app_name)

File "/usr/local/lib/python2.6/site-packages/django/db/models/
loading.py", line 78, in load_app
models = import_module('.models', app_name)

File "/usr/local/lib/python2.6/site-packages/django/utils/
importlib.py", line 35, in import_module
__import__(name)

File "/home/users/presskanne/dcalcuttagutta/media_files/models.py",
line 17, in <module>
from dcalcuttagutta.articles.models import Tag

ImportError: cannot import name Tag


<WSGIRequest
GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'DOCUMENT_ROOT': '/home/users/presskanne/www',
'GATEWAY_INTERFACE': 'CGI/1.1',
'HTTP_ACCEPT': '*/*',
'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
'HTTP_CONNECTION': 'close',
'HTTP_FROM': 'googlebot(at)googlebot.com',
'HTTP_HOST': 'www.presskanne.com',
'HTTP_USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://
www.google.com/bot.html)',
'PATH': '/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/
local/bin:/home/josdal/bin',
'PATH_INFO': u'/articles/104',
'PATH_TRANSLATED': 'redirect:/dcalcuttagutta.fcgi/articles/104/104',
'QUERY_STRING': '',
'REDIRECT_STATUS': '200',
'REDIRECT_UNIQUE_ID': 'TDsoSFUTR6UAATjeM1sAAACg',
'REDIRECT_URL': '/articles/104',
'REMOTE_ADDR': '66.249.66.22',
'REMOTE_PORT': '39148',
'REQUEST_METHOD': 'GET',
'REQUEST_URI': '/articles/104',
'SCRIPT_FILENAME': '/home/users/presskanne/www/dcalcuttagutta.fcgi',
'SCRIPT_NAME': u'',
'SERVER_ADDR': '85.19.71.165',
'SERVER_ADMIN': 'webm...@subsys.no',
'SERVER_NAME': 'www.presskanne.com',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.1',
'SERVER_SIGNATURE': '<address>Apache Server at www.presskanne.com Port
80</address>\n',
'SERVER_SOFTWARE': 'Apache',
'UNIQUE_ID': 'TDsoSFUTR6UAATjeM1sAAACg',
'wsgi.errors': <flup.server.fcgi_base.OutputStream object at
0x802e88050>,
'wsgi.input': <flup.server.fcgi_base.InputStream object at
0x802e7df90>,
'wsgi.multiprocess': False,
'wsgi.multithread': True,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)}>

Doug

unread,
Jan 24, 2014, 4:44:27 PM1/24/14
to django...@googlegroups.com
I am having this same exact problem. Did you ever find a resolution? 

Thanks.
Doug

James Turley

unread,
Jan 24, 2014, 4:48:22 PM1/24/14
to django...@googlegroups.com
Are you doing any user agent sniffing anywhere? If you make a request having spoofed your UA to match one of the offending Google/Yahoo bots, do you get the same 500 errors? Anything in robots.txt?


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/08289acf-2fef-4953-b77c-153001ec8ef9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Doug

unread,
Jan 25, 2014, 6:31:03 PM1/25/14
to django...@googlegroups.com
Thanks for the response.

I'm not doing any user agent sniffing, and I don't have a robots.txt. 


On Monday, July 12, 2010 2:10:49 PM UTC-4, Tor Nordam wrote:

Doug

unread,
Jan 25, 2014, 6:35:08 PM1/25/14
to django...@googlegroups.com
Here's an example stack trace:

Traceback (most recent call last):

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/core/handlers/base.py", line 103, in get_response
    resolver_match = resolver.resolve(request.path_info)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/core/urlresolvers.py", line 319, in resolve
    for pattern in self.url_patterns:

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/core/urlresolvers.py", line 347, in url_patterns

    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/core/urlresolvers.py", line 342, in urlconf_module
    self._urlconf_module = import_module(self.urlconf_name)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)

  File "/home/possiblecity/possiblecity/production/possiblecity/possiblecity/urls.py", line 11, in <module>
    autocomplete_light.autodiscover()

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/autocomplete_light/registry.py", line 237, in autodiscover
    _autodiscover(registry)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/autocomplete_light/registry.py", line 190, in _autodiscover
    import_module('%s.autocomplete_light_registry' % app)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)

  File "/home/possiblecity/possiblecity/production/possiblecity/possiblecity/apps/lotxlot/autocomplete_light_registry.py", line 4, in <module>
    from .models import Lot

  File "/home/possiblecity/possiblecity/production/possiblecity/possiblecity/apps/lotxlot/models.py", line 17, in <module>
    from apps.comments.models import Comment

  File "/home/possiblecity/possiblecity/production/possiblecity/possiblecity/apps/comments/models.py", line 10, in <module>
    from actstream.models import followers

  File "/home/possiblecity/lib/python2.7/actstream/models.py", line 22, in <module>
    User = get_user_model()

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 127, in get_user_model
    user_model = get_model(app_label, model_name)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/db/models/loading.py", line 230, in get_model
    self._populate()

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/db/models/loading.py", line 75, in _populate
    self.load_app(app_name)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/db/models/loading.py", line 96, in load_app

    models = import_module('.models', app_name)

  File "/home/possiblecity/.virtualenvs/prod/lib/python2.7/site-packages/django/utils/importlib.py", line 35, in import_module
    __import__(name)

  File "/home/possiblecity/possiblecity/production/possiblecity/possiblecity/apps/core/models.py", line 4, in <module>
    from actstream.models import Follow

ImportError: cannot import name Follow


<WSGIRequest
path:/lots/4778/,

GET:<QueryDict: {}>,
POST:<QueryDict: {}>,
COOKIES:{},
META:{'HTTP_ACCEPT': '*/*',

 'HTTP_ACCEPT_ENCODING': 'gzip,deflate',
 'HTTP_CONNECTION': 'close',
 'HTTP_FORWARDED_REQUEST_URI': '/lots/4778/',
 'HTTP_FROM': 'googlebot(at)googlebot.com',
 'HTTP_HOST': 'possiblecity.co',
 'HTTP_HTTPS': 'off',
 'HTTP_HTTP_X_FORWARDED_PROTO': 'http',

 'HTTP_USER_AGENT': 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)',
 'HTTP_X_FORWARDED_FOR': '66.249.73.20',
 'HTTP_X_FORWARDED_HOST': 'possiblecity.co',
 'HTTP_X_FORWARDED_PROTO': 'http',
 'HTTP_X_FORWARDED_SERVER': 'possiblecity.co',
 'HTTP_X_FORWARDED_SSL': 'off',
 'PATH_INFO': u'/lots/4778/',
 'QUERY_STRING': '',
 'RAW_URI': '/lots/4778/',
 'REMOTE_ADDR': '66.249.73.20',
 'REMOTE_PORT': '80',
 'REQUEST_METHOD': 'GET',
 'SCRIPT_NAME': u'',
 'SERVER_NAME': 'possiblecity.co',
 'SERVER_PORT': '80',
 'SERVER_PROTOCOL': 'HTTP/1.0',
 'SERVER_SOFTWARE': 'gunicorn/17.5',
 'gunicorn.socket': <socket._socketobject object at 0x2a1fd70>,
 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f0d45ef51e0>,
 'wsgi.file_wrapper': <class gunicorn.http.wsgi.FileWrapper at 0x2552b48>,
 'wsgi.input': <gunicorn.http.body.Body object at 0x3d7e9d0>,
 'wsgi.multiprocess': False,
 'wsgi.multithread': False,

 'wsgi.run_once': False,
 'wsgi.url_scheme': 'http',
 'wsgi.version': (1, 0)}>

On Monday, July 12, 2010 2:10:49 PM UTC-4, Tor Nordam wrote:

Doug

unread,
Jan 25, 2014, 6:40:47 PM1/25/14
to django...@googlegroups.com
Also, if I spoof the user agent (using a chrome plugin), I don't get the error.


On Monday, July 12, 2010 2:10:49 PM UTC-4, Tor Nordam wrote:

Stefano Probst

unread,
Jan 26, 2014, 8:50:24 PM1/26/14
to django...@googlegroups.com
Hi,
have you try it with "open a Page like Google" ( https://support.google.com/webmasters/answer/158587 )?
Reply all
Reply to author
Forward
0 new messages