Distributed resource locking using memcached

[yatagarasu]

Is it possible to implement distributed resource blocking using
memcached ?

Is it possible to implement them using following scheme (pseudocode)

lock(a)
1. if (add( key=>a, expiration=>5min )) {
2. if (cas( key=>a, expiration=>5min )) {
// resource locked by us
}
// Someone tried to lock resource the same time. Race condition
in add.
}
// resource is already locked

unlock(a)
1. delete(a)

What do you think?
Is race condition in add possible?
Will such double check work?
Or is there more simple way?


PS. Expiration in 5min is used to auto unlock resource in case of
client disconnection or other errors.

[Clint Webb]

Its possible... but really... its a cache.  Thats what it is designed to do, and that is what it is good for.  memcached is not optimized at all to be a lock server, why exactly are you wanting to use memcached for this instead of an actual lock server?

That being said, there are a number of issues when using memcached as a lock server that would make it far less than ideal. 

Also a default unlock after 5 minutes rather defeats the purpose of a lock server...   I wouldn't recommend this at all.  Far better off with an actual lock server that can defect if a client has disconnected.

Short answer:  No, do not use memcached as a lock server.

--
"Be excellent to each other"

[Adam Lee]

We actually use memcached as a lock server for certain very rare,
per-user transactions. It's less than ideal, but we scrambled to
write it in a pinch when our previous solution had problems and it's
held up thus far. It's something we've had on the "fix when we have
time" list since, but it does indeed work.

I still wouldn't recommend it, though. What kind of scale are you
looking for? Have you considered something like zookeeper, for
example?

--
awl

[Dustin]

On Apr 8, 7:07 am, Clint Webb <webb.cl...@gmail.com> wrote:
> Its possible... but really... its a cache.  Thats what it is designed to do,
> and that is what it is good for.  memcached is not optimized at all to be a
> lock server, why exactly are you wanting to use memcached for this instead
> of an actual lock server?
>
> That being said, there are a number of issues when using memcached as a lock
> server that would make it far less than ideal.
>
> Also a default unlock after 5 minutes rather defeats the purpose of a lock
> server...   I wouldn't recommend this at all.  Far better off with an actual
> lock server that can defect if a client has disconnected.

I wrote elockd for this. I generally want a lock released at the
very moment a client disconnects (though I have a feature that will
allow a lock to linger beyond a disconnect and allow sessions and all
of their locks to be resumed across connections).

http://github.com/dustin/elock

There are many other lock servers -- all better than memcached. :)

[Earl Cahill]

And I wrote IPC::Lock::Memcached for it.

http://search.cpan.org/~earl/IPC-Lock-0.14/lib/IPC/Lock/Memcached.pm

Earl

 

http://blog.spack.net
http://holaservers.com

---

From: Dustin <dsal...@gmail.com>
To: memcached <memc...@googlegroups.com>
Sent: Wednesday, April 8, 2009 10:19:42 AM
Subject: Re: Distributed resource locking using memcached

[Dustin]

On Apr 8, 4:37 pm, Earl Cahill <cahi...@yahoo.com> wrote:
> And I wrote IPC::Lock::Memcached for it.
>
> http://search.cpan.org/~earl/IPC-Lock-0.14/lib/IPC/Lock/Memcached.pm

Yes, but there are undesirable properties:

1) May release locks unexpectedly and without warning (and not just
when you're on the edge of memory availability).
2) Will not automatically release locks when a client crashes or
otherwise disconnects.
3) Has no ability to block for the availability of a lock (requiring
polling to check for lock availability).

It's the same sort of problems you run into when trying to use
memcached as a work queue. It's *possible* but it doesn't offer the
guarantees you should be building upon.

For example, it's quite likely you've given the same lock out to two
different processes without knowing it. It's the same class of bugs
as concurrency bugs -- you can have two threads occasionally write to
the same spot in memory without mutexes and have it actually work a
lot of the time. It will just stop working when you least expect it
and it will show up in strange ways.

[yatagarasu]

thanks everyone for the response.
i think we will look at zookeeper.