Dreamhost, Django and MySQL server connection

[Edgars Jekabsons]

Does anyone else who uses Dreamhost has problems with MySQL connection
going away?

The project I suggested hosting there is not even inproduction yet, but
we get MySQL server errors a lot (a few times a day). Sometimes it's
just Django error 500, but other times I get a Python traceback screen.

Is it just plain unlucky me and an unreliable MySQL server on Dreamhost
or I should have used something more robust than the setup described in
http://wiki.dreamhost.com/index.php/Django?

TIA,
Edgars

[Jamison Roberts]

I have nothing but problems with Dreamhost + Django.  More specifically Dreamhost + Python.  I hate that company and will be switching as soon as my year is up.

Can I threadjack and ask for a host recommendation?

[foot]

We've got our django project hosted over at Site5 and we have a similar
MySQL problem: A (2006, MySQL server has gone away) error, whenever the
site hasn't been accessed in a while. It seems to be related to your
mysql interactive timeout setting. There's a bit of a general
discussion about this problem here on the Site5 forums:
http://forums.site5.com/showthread.php?t=7281

I'm not sure who's 'fault' it is, Django's for not checking if it can
still make queries within the 'interactive session's timelimit', or
MySQL's for closing the connection after it's interactive timeout,
maybe you can stop it from closing an interative session at all? If you
figure out how please let me know...

At the moment we've just asked Site5 to set out interactive timeout a
bit higher, it means that if no one hits the site in ~5mins the next
person to will get a 'MySQL server has gone away' error... which they
will have to refresh but then they can browse normally... unless they
idle for 5mins.

It would be great to get this fixed tho...

- foot

[Eugene Lazutkin]

Edgars Jekabsons wrote:
> Does anyone else who uses Dreamhost has problems with MySQL connection
> going away?

I use Dreamhost without any problems, because I use MySQL patch (ticket
#463 --- http://code.djangoproject.com/ticket/463). Please use the last
version by nesh --- it was updated to use fair RW lock. I have no idea
why it didn't make to trunk yet --- I never heard complains about the patch.

Thanks,

Eugene

[Eugene Lazutkin]

foot wrote:
> We've got our django project hosted over at Site5 and we have a similar
> MySQL problem: A (2006, MySQL server has gone away) error, whenever the

[snip]

> It would be great to get this fixed tho...

It was fixed 5 months ago but didn't make it to trunk. Please try
http://code.djangoproject.com/ticket/463 (the new patch by nash) and see
if it helps you.

Thanks,

Eugene

[Eugene Lazutkin]

Jamison Roberts wrote:
> I have nothing but problems with Dreamhost + Django. More specifically
> Dreamhost + Python. I hate that company and will be switching as soon
> as my year is up.

What "more specifically" was your problem with Python?

> Can I threadjack and ask for a host recommendation?

http://code.djangoproject.com/wiki/DjangoFriendlyWebHosts

Thanks,

Eugene

[kmh]

Hi Eugene,

Your post reminded me about this bug which has occasionally caused me
problems too so I tried Nesh's patch but it no longer applies cleanly
to trunk. It also appears that the patch doesn't pass the
DATABASE_HOST or DATABASE_PASSWORD arguments in the connection request.

If you have a working patch against trunk could you please post it to
#463 for me to test because I'd also like to see a solution to this one
get committed.

Cheers,
Kieran

[Jeremy Jones]

I seemed to have a problem with a test environment I setup. But my
"main production" environment is doing better. I don't think it would
hurt anything to replicate your current database on a new DB which
you setup in the Dreamhost panel.

I wrote a little ping script which requests a URL in my app which
triggers a simple database lookup. I run the script on a cron job every
5 minutes on a server on my LAN at home. Over the past 8 days, it's
only reported back 4 errors. After I got the sense to store the
resulting error pages, it looks like one is a 500 error (which feels
like a database problem) and one is a a URL lookup failure. The latter
is a little weird. I'll try to beef up the error trapping and
reporting when I get a chance.

One of the reasons I did this, though, was on Dreamhost my app is
running under fastcgi. Since it's a low-traffic site, the response
time for a new request after no requests have happened in a while is
miserable. So, this script keeps the site alive and hopefully provides
users with a little better experience and provides information for when
the site was having problems. Anyway, that's not exactly what you were
asking, but I hope it's helpful nonetheless.

- jmj

[Edgars Jekabsons]

Hi Kieran,

I posted a working patch against Django release 0.91 on the same ticket
http://code.djangoproject.com/ticket/463

HTH,
Edgars

[kmh]

Thanks Edgars,

Unfortunately your patch can't get committed as-is either because it is
against 0.91 and there have been a few changes since then. The Django
devs are much wiser than to use MySQL ;) so it really needs to work
with the latest version of trunk in svn to stand a chance.

Anyone running a patch against svn?

Cheers,
Kieran

[Edgars Jekabsons]

OK, posted a patch agaist trunk (revision 2307)
http://code.djangoproject.com/attachment/ticket/463/mysql_rev2307.diff

--
Edgars

[kmh]

Hey thanks Edgars,

I'll run it for a while and post encouragement to the ticket if I have
no problems.

Kieran

[ska...@gmail.com]

I was also experiencing sql disconnect errors with Dreamhost. I have
just updated my site http://www.socialistsoftware.com/ to use this
patch.

We will see how it goes.

[Eugene Lazutkin]

Let me know how it goes. If there is a problem, I'll make my patch. I
use it for a long time now without problems.

Thanks,

Eugene

[kmh]

So I tried running this patch on a very low-volume production server
and MySQL (4.1/ Ubuntu Breezy) started chucking some "Too many
connections" errors.

I'm no expert on RDBMS. I did some googling and discovered the MySQL
"max_connections" setting. Mine is at the default of 100. I would
have thought that would be well-adequate in this particular case,
though the user-agent in the error message was a Yahoo-bot so it is
possible that there were many simultaneous requests at the time (my
access log was off so I can't say exactly - I'll turned it on if I try
again). I don't know how to tell if this is a problem with the patch
not cleaning up after itself or my database setup being piss-poor. Any
tips?

Various niggling problems with the MySQL/Django combination (together
with the knowledge that it doesn't get quite the same loving from the
core devs) are tempting me to switch to Postgresql, but the migration
process seems like a recipe for pain and suffering. Anyone been
through it? I have foggy notions about something involving Jacob and
an idea for a data migration tool...am I making this up?

Cheers,
Kieran

[Eugene Lazutkin]

kmh wrote:
>
> So I tried running this patch on a very low-volume production server
> and MySQL (4.1/ Ubuntu Breezy) started chucking some "Too many
> connections" errors.

:-) But no "lost connections" now?

> I'm no expert on RDBMS. I did some googling and discovered the MySQL
> "max_connections" setting. Mine is at the default of 100. I would
> have thought that would be well-adequate in this particular case,
> though the user-agent in the error message was a Yahoo-bot so it is
> possible that there were many simultaneous requests at the time (my
> access log was off so I can't say exactly - I'll turned it on if I try
> again). I don't know how to tell if this is a problem with the patch
> not cleaning up after itself or my database setup being piss-poor. Any
> tips?

You can try to reproduce the problem with openload (now it is Open Web
Load: http://sourceforge.net/projects/openwebload/). If you cannot
reproduce it, it maybe something else.

Some bots can overload the server. I never had problems with Yahoo-bot,
but some bot from Greece brought my web site down exactly once a week by
issuing a lot of requests per second, which triggered "too many
connections" error. Since then I switch to flup and restricted number of
connections and requests. My django.fcgi looks like this:

===========================
#!/usr/bin/env python2.4
if __name__ == '__main__':
import sys, os

sys.path += ['path-to-django']
sys.path += ['path-to-myproject']
os.environ['DJANGO_SETTINGS_MODULE'] = 'my.settings'

from flup.server.fcgi import WSGIServer
from django.core.handlers.wsgi import WSGIHandler
server = WSGIServer(WSGIHandler())

server.capability['FCGI_MAX_CONNS'] = 5
server.capability['FCGI_MAX_REQS'] = 50

server.run()
===========================

I don't know what numbers should be used. I guess you should experiment
a little. But anyway Greek bot doesn't bother me anymore. I don't know,
if bot's authors changed its behavior, or my settings actually helped. ;-)

Keep me posted.

Thanks,

Eugene

[ska...@gmail.com]

I have been running this patch for almost 2 days, and I just got my
first "lost connection"

Traceback (most recent call last):

File "/home/skabber/python/django/core/handlers/base.py", line 74, in
get_response
response = callback(request, *callback_args, **callback_kwargs)

File "/home/skabber/python/skabber/apps/blog/views.py", line 7, in
index
tags = blog.tags.get_list()

File "/home/skabber/django/trunk/django/utils/functional.py", line 3,
in _curried
return args[0](*(args[1:]+moreargs), **dict(kwargs.items() +
morekwargs.items()))

File "/home/skabber/python/django/core/meta/__init__.py", line 1395,
in function_get_list
return list(function_get_iterator(opts, klass, **kwargs))

File "/home/skabber/python/django/core/meta/__init__.py", line 1376,
in function_get_iterator
cursor = db.db.cursor()

File "/home/skabber/python/django/core/db/backends/mysql.py", line
106, in cursor
connection = self.__connect()

File "/home/skabber/python/django/core/db/backends/mysql.py", line 97,
in __connect
connection = Database.connect(**kwargs)

File "/usr/lib/python2.3/site-packages/MySQLdb/__init__.py", line 66,
in Connect
return Connection(*args, **kwargs)

File "/usr/lib/python2.3/site-packages/MySQLdb/connections.py", line
134, in __init__
super(Connection, self).__init__(*args, **kwargs2)

OperationalError: (2013, 'Lost connection to MySQL server during query')

[Eugene Lazutkin]

j...@skabber.com wrote:
> I have been running this patch for almost 2 days, and I just got my
> first "lost connection"

So far my "lost connections" could be traced to Dreamhost rebooting
servers. I suggest to follow up with them to trace a reason.

Thanks,

Eugene