Re: Intermittent mysql connection errors

[Guy]

After more testing it appears that this is an O/S layer issue and not specifically a django issue.

On Thursday, October 25, 2012 3:07:16 PM UTC-7, Guy wrote:

I'm experiencing an annoying problem with a couple of django apps. Both apps are games built on Django that make regular calls to the db using python-db. We have error emails turned on, and every so often (the interval doesn't seem to follow a pattern but it's usually no more than once in a half-hour) we get an email with the following info (I've obscured/changed some identifying info, for the record):

OperationalError at /analytics/

(2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")

Request Method:	POST
Request URL:	http://hostname.domain/analytics
Django Version:	1.3
Exception Type:	OperationalError
Exception Value:	(2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")
Exception Location:	/usr/lib/pymodules/python2.6/MySQLdb/connections.py in __init__, line 170
Python Executable:	/usr/sbin/uwsgi-1.2.3
Python Version:	2.6.5
Python Path:	['app_path', '.', '', '/usr/lib/python2.6', '/usr/lib/python2.6/plat-linux2', '/usr/lib/python2.6/lib-tk', '/usr/lib/python2.6/lib-old', '/usr/lib/python2.6/lib-dynload', '/usr/lib/python2.6/dist-packages', '/usr/lib/python2.6/dist-packages/PIL', '/usr/lib/pymodules/python2.6', '/usr/local/lib/python2.6/dist-packages', '/usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103', 'app_path/src', 'app_path/src/expedition', 'app_path/src/expedition/settings']
Server time:	Thu, 25 Oct 2012 14:14:08 -0700

Traceback

• /usr/local/lib/python2.6/dist-packages/django/core/handlers/base.py in get_response

  111.                         response = callback(request, *callback_args, **callback_kwargs)

  Local Vars

  Variable	Value
  exceptions	<module 'django.core.exceptions' from '/usr/local/lib/python2.6/dist-packages/django/core/exceptions.pyc'>
  e	OperationalError(2003, "Can't connect to MySQL server on 'x.x.x.x' (111)")
  callback_args	()
  receivers	[(<function _rollback_on_exception at 0x1f9a578>, None)]
  middleware_method	<newrelic.api.object_wrapper.ObjectWrapper object at 0x1fe0350>
  self	<django.core.handlers.wsgi.WSGIHandler object at 0x1d6c810>
  settings	<django.conf.LazySettings object at 0x1abded0>
  request	<WSGIRequest GET:<QueryDict: {}>, POST:<could not parse>, COOKIES:{}, META:{'CONTENT_LENGTH': '275', 'CONTENT_TYPE': 'application/x-www-form-urlencoded', 'DOCUMENT_ROOT': '/opt/nginx-1.2.1/html', 'HTTP_ACCEPT': '*/*', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us', 'HTTP_CONTENT_LENGTH': '275', 'HTTP_CONTENT_TYPE': 'application/x-www-form-urlencoded', 'HTTP_HOST': 'hostname.domain', 'HTTP_USER_AGENT': 'HTTPagent/1.2 CFNetwork/609 Darwin/13.0.0', 'HTTP_VIA': 'HTTP/1.1 akrmspsrvz4ts212.wnsnet.attws.com', 'HTTP_X_FORWARDED_FOR': 'x.x.x.x, x.x.x.x', 'HTTP_X_REAL_IP': 'x.x.x.x', 'PATH_INFO': u'/analytics/', 'QUERY_STRING': '', 'REMOTE_ADDR': 'x.x.x.x', 'REMOTE_PORT': '49328', 'REQUEST_METHOD': 'POST', 'REQUEST_URI': '/analytics/', 'SCRIPT_NAME': u'', 'SERVER_NAME': 'hostname.domain', 'SERVER_PORT': '80', 'SERVER_PROTOCOL': 'HTTP/1.0', 'uwsgi.core': 15, 'uwsgi.node': 'hostname', 'uwsgi.version': '1.2.3', 'wsgi.errors': <open file 'wsgi_errors', mode 'w' at 0x1b9e420>, 'wsgi.file_wrapper': <built-in function uwsgi_sendfile>, 'wsgi.input': <newrelic.api.web_transaction.WSGIInputWrapper object at 0x309b5d0>, 'wsgi.multiprocess': True, 'wsgi.multithread': True, 'wsgi.run_once': False, 'wsgi.url_scheme': 'http', 'wsgi.version': (1, 0)}>
  callback	<newrelic.api.object_wrapper.ObjectWrapper object at 0x7fc0c825ed10>
  resolver	<RegexURLResolver expedition.urls (None:None) ^/>
  urlresolvers	<module 'django.core.urlresolvers' from '/usr/local/lib/python2.6/dist-packages/django/core/urlresolvers.pyc'>
  callback_kwargs	{}
  response	None
  urlconf	'expedition.urls'

• /usr/local/lib/python2.6/dist-packages/newrelic-1.5.0.103/newrelic/api/object_wrapper.py in __call__

  198.                 self._nr_instance, args, kwargs)

The db connection works fine most of the time, it just seems like intermittently it can't connect. I've done a couple of things, like enabled tcp_tw_reuse (it seems I usually have about 150-200 connections to mysql in TIME_WAIT on any one web server), checked the mysql server and web servers message logs for too many connections errors, and looked at the mysql log to see if there's any info there. I also wrote a script that connects to the db and sends an email to me if it doesn't get expected info, initially I ran it with a cron job and didn't get any emails from it, but obviously that was only every minute. I'm now running it every second using the watch command but haven't so far had any emails.

It's very strange, and I'm not sure why it's happening. As I said, we have another django app (on completely separate infrastructure) that has the same issue. I don't believe there's a network issue but I wouldn't rule it out, but I'm curious if anyone else on here has seen this behaviour before, and if so, I'd like to know how you fixed it. Any help would be appreciated.

Thanks,

Guy

[Nazeem S]

Hi,

 

Could you please share what was the solution, I'm experiencing the same.