python memcache collisions

[arashf]

is anybody else still running into this issue?
http://lists.danga.com/pipermail/memcached/2007-March/003737.html
i'm still getting key collisions (under heavy load, after a few days)
with the latest python-memcache. the threadlocal fix doesn't seem to
make a difference.

[arashf]

bump. anybody have ideas on who I can chat about this with? :)

On Jul 10, 3:51 pm, arashf <aferdo...@gmail.com> wrote:
> is anybody else still running into this issue?http://lists.danga.com/pipermail/memcached/2007-March/003737.html

[arashf]

last comment from me :)
I'm seeing the exact two problems outlined here from 2007:

hi, mark,
I'm having the same problem with key collisions with the python
binding, and I was wondering if you've made any progress. I haven't
been able to reproduce the problem in a test, but in production,
we'll encounter the problem all of a sudden after a while (the only
"solution" so far is to restart memcached). The server returns the
wrong value for the requested key. Not only that, but if you try a
get_many for key 'foo', you may get a response that has key 'bar'
and
its value.
Since I haven't been able to reproduce this behavior in a test, I
don't yet know if it's the python binding or memcached itself.
Does anyone have any ideas?

Dave



> Hi
> I am consistently getting key collisions. There are total of around
> 2000 key
> value pairs cached.
> This is how I prepare the key for memcached: md5.new(key).hexdigest
> () where
> key is the unique
> id for the database row that I want to cache.
> When I do a 'get' for a specific key, I get a value of a different
> key.
> I am using python-memcached as the client.
> This is causing lot of problems. If anyone has faced similar issues
> or has
> any suggestions do reply.
> cheers
> mark

On Jul 10, 3:51 pm, arashf <aferdo...@gmail.com> wrote:

> is anybody else still running into this issue?http://lists.danga.com/pipermail/memcached/2007-March/003737.html

[David Stanek]

I am using the latest client from tummy.com and not seeing an issue.
Is it possible that you really are using the same key for multiple
objects?

--
David
http://www.traceback.org

[arashf]

hey david,
i'm positive.
i'm getting tracebacks like this:

File '/srv/server/metaserver/metaserver/controllers/client.py', line
67 in list
cached = mc.get_multi(["chillout", "resync_%s" % h.id, "sendtrace_
%s" % h.id])
File '/srv/server/metaserver/metaserver/common/memcache.py', line 738
in get_multi
retvals[prefixed_to_orig_key[rkey]] = val # un-prefix returned
key.
KeyError: 'child-namespaces-567538'

i'm trying to fetch the keys 'chillout', 'resync_$somevar' and
'sendtrace_somevar' yet get back a totally random key. on some gets
there isn't even a KeyError and I get back a totally different value
(from what I stored) back for the key. several others ran into the
issue here:
http://lists.danga.com/pipermail/memcached/2007-February/003619.html.
the theory is that there were sync issues/data left on the socket, but
I'm establishing and closing a connection to memcache per request and
still running into the problem. I'm pretty stumped.

On Jul 15, 5:02 am, "David Stanek" <dsta...@dstanek.com> wrote:
> I am using the latest client from tummy.com and not seeing an issue.
> Is it possible that you really are using the same key for multiple
> objects?
>
>
>
> On Mon, Jul 14, 2008 at 11:41 PM, arashf <aferdo...@gmail.com> wrote:
>
> > last comment from me :)
> > I'm seeing the exact two problems outlined here from 2007:
>
> > hi, mark,
> > I'm having the same problem with key collisions with thepython
> > binding, and I was wondering if you've made any progress.  I haven't
> > been able to reproduce the problem in a test, but in production,
> > we'll encounter the problem all of a sudden after a while (the only
> > "solution" so far is to restart memcached).  The server returns the
> > wrong value for the requested key.  Not only that, but if you try a
> > get_many for key 'foo', you may get a response that has key 'bar'
> > and
> > its value.
> > Since I haven't been able to reproduce this behavior in a test, I

> > don't yet know if it's thepythonbinding or memcached itself.

> > Does anyone have any ideas?
>
> > Dave
>
> >> Hi
> >> I am consistently getting key collisions. There are total of around
> >> 2000 key
> >> value pairs cached.
> >> This is how I prepare the key for memcached: md5.new(key).hexdigest
> >> () where
> >> key is the unique
> >> id for the database row that I want to cache.
> >> When I do a 'get' for a specific key, I get a value of a different
> >> key.

> >> I am usingpython-memcached as the client.

> >> This is causing lot of problems. If anyone has faced similar issues
> >> or has
> >> any suggestions do reply.
> >> cheers
> >> mark
>
> > On Jul 10, 3:51 pm, arashf <aferdo...@gmail.com> wrote:
> >> is anybody else still running into this issue?http://lists.danga.com/pipermail/memcached/2007-March/003737.html
> >> i'm still getting key collisions (under heavy load, after a few days)

> >> with the latestpython-memcache. the threadlocal fix doesn't seem to

[arashf]

to add to the last email.

it generally takes a couple days to start seeing collisions, but
before that, I regularly get all sorts of errors from get, which look
like this:

File '/srv/server/metaserver/metaserver/common/util.py', line 581 in
get
ret = memcache.Client.get(self, key)
File '/srv/server/metaserver/metaserver/common/memcache.py', line 662
in get
rkey, flags, rlen, = self._expectvalue(server)
File '/srv/server/metaserver/metaserver/common/memcache.py', line 750
in _expectvalue
resp, rkey, flags, len = line.split()
ValueError: need more than 2 values to unpack

any help debugging appreciated :).

[Jehiah Czebotar]

>
> any help debugging appreciated :).
>

you are seeing errors so you have something to work with

the next step is to run memcached in verbose mode to see if there are
any errors on the server side that correlate with the time when errors
first occur on the client side, or to run a sniffer on the memcached
ports so you have all the data and can go back and see what is
happening on the wire at the time when you see errors.

also, what python memcached library version are you using, and what
memcached server version?

it looks like your code is odd in that your util.py file is calling
memache.Client.get(), so something could be odd there.

normally you use it like this

m = memcache.Client(['127.0.0.1:11211'])
print m.get(key)

so it could be a problem with how you are subclassing memcache.Client

--
Jehiah

[David Stanek]

You can contact me off list and I can help you debug. This is probably
not the right place for it. But for the record it seems that your key
error would come from this part of the code:
prefixed_to_orig_key[rkey]. As you stated it would seem that is pulled
something off the wire that is wasn't expecting.

--
David
http://www.traceback.org