memcache protocol errors in Django

[underco...@gmail.com]

I have googled around for this but haven't found much.

My environment:
Ubuntu 9.04
Django 1.1 beta1
cmemcache 0.95
memcached 1.2.2
python 2.6.2
Apache 2.2.11
mod_fastcgi 2.4.6

I am using the cached_db session setting in django to write through
memcache to a sqlite db which will be replaced with something better
later. I am benchmarking my app with a cluster using httperf and
other benchmarking scripts. When serving small numbers of requests
( ~ 10 req/s ) everything works just fine. When I load it up more to
60 req/s I start seeing 500s with these messages in the log files:

[FA...@1241739622.207433] mcm_fetch_cmd():1181: memcache(4) protocol
error: Operation now in progress: server sent data for key not in
request
[FA...@1241739622.341555] mcm_fetch_cmd():1310: memcache(4) protocol
error: �}qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs.


If I load it up with more req/s the main django process crashes and I
get one of these in my syslog:

kernel: [ 6769.234808] python[8118] trap stack segment ip:7fbb3b4eed50
sp:7fbb2a0551a0 error:0


and more of this in the fastcgi logs:

[FA...@1241735417.945646] mcm_fetch_cmd():1181: memcache(4) protocol
error: Operation now in progress: server sent data for key not in
request
[FA...@1241735418.067103] mcm_fetch_cmd():1310: memcache(4) protocol
error: �}qUcurrent_sessionqU$8094ad94-3b56-11de-96b9-001e4fe25c1fqs.
[FA...@1241738169.008296] mcm_fetch_cmd():1181: memcache(4) protocol
error: Operation now in progress: server sent data for key not in
request
[FA...@1241738169.286827] mcm_fetch_cmd():1310: memcache(4) protocol
error: �}qUcurrent_sessionqU$09646924-3b5d-11de-96ba-001e4fe25c1fqs.
[FA...@1241738169.327522] mcm_fetch_cmd():1181: memcache(4) protocol
error: server sent data for key not in request
[ER...@1241738170.887020] mcm_server_readable():2566
[IN...@1241738170.887205] mcm_get_line():1597
[FA...@1241738170.887231] mcm_server_send_cmd():2711: failed to send
command to the memcache server: Operation now in progress: Operation
now in progress
[FA...@1241739622.207433] mcm_fetch_cmd():1181: memcache(4) protocol
error: Operation now in progress: server sent data for key not in
request
[FA...@1241739622.341555] mcm_fetch_cmd():1310: memcache(4) protocol
error: �}qUcurrent_sessionqU$407b36b0-3b60-11de-96ba-001e4fe25c1fqs.


The code using the cache is very simple, I have a decorator that wraps
several views:

def uuid_session_decorator(request, *args, **kw):
my_uuid = request.session.get('current_session',False)
if not my_uuid:
my_uuid = uuid.uuid1().__str__()
request.session['current_session'] = my_uuid
return view(request, my_uuid, *args, **kw)


This is the only place that I am using sessions in my app. If I dont
use the decorator function I dont see any errors, and django can
handle a large amount more req/s before throwing 500s and it doesnt
crash. I changed the session to cache only to rule out issues with
sqlite and it has the same behavior and errors. Im only writing a
uuid string to the cache so I dont see why it should be erroring out
so quickly. Does anyone know what these errors mean and why it is
causing django to crash? The box is a quad core with 6GB ram by the
way and never hits swap during the tests.

[underco...@gmail.com]

The problem seemed to go away when switching to the pure python python-
memcached client. There must be some thread concurrency issue in the
cmemcache client that is appearing under heavy load.



On May 7, 5:00 pm, "undercoverid...@gmail.com"

> [I...@1241738170.887205] mcm_get_line():1597