Suggestions for deferring DB write using memcached

[ehalpern]

We're building a system with heavy real-time write volume and looking
for a way to decouple db writes from the user request path.

We're exploring the approach of buffering updated entities in
memcached and writing them back to the database asynchronously. The
primary problem that we're concerned about is how to ensure that the
entity remains in the cache until the background process has a change
to write it.

Any advice and/or references would be greatly appreciated.

[dormando]

You really probably want to use a job server... http://gearman.org/ -
write updates async to gearman and update caches in memcached if you want,
then have workers write them back to the DB as fast as your system can
handle.

[Adam Lee]

You want a job/work queue for this or just a simple queue service.

I've built something that does this for us using Kestrel, a very simple, fast, persistent queue from Twitter that speaks memcached.

http://github.com/robey/kestrel

You can obviously make it as simple or complex as you need, but I'd say it's best to start out as simple as possible.  Since kestrel is fast and persistent, you should just be able to write your data to it and move on, with the assumption that a background service will pop items off of kestrel's queue and write them to the database as quickly as possible.  If you need the data to be immediately available, you can do something more complex like updating your cache at the same time, so that subsequent reads will get the new data as long as it's still available in the cache.  

--
awl

[Brian Moon]

+1 We use gearman for this.

Brian.
--------
http://brian.moonspot.net/

[Brian Moon]

Oh, and I wrote about one application that does this.

http://brian.moonspot.net/logging-with-mysql

Brian.
--------
http://brian.moonspot.net/

On 6/3/10 3:13 PM, dormando wrote:

[Vladimir Vuksan]

Others have suggested their favored tools. I would add beanstalkd to the
list

http://kr.github.com/beanstalkd/

I like it since it comes with simple (memcached inspired) protocol and can
persist jobs.

Vladimir

[Jay Paroline]

One more shop here that uses gearmand for this!

[ehalpern]

Thanks for the quick replies. I have two follow-up questions:

- Has anyone had experience using gearman with java bindings? The
only binding I've found is gearman-java and it does not look very
active.

- I'd like to keep our system as simple as possible and avoid
additional components if possible. Has anyone had success using
memcache itself for implementing work queues (a la memcachequeue)?

[Brian Moon]

> - I'd like to keep our system as simple as possible and avoid
> additional components if possible. Has anyone had success using
> memcache itself for implementing work queues (a la memcachequeue)?

See my slide from my recent memcached talk about queues.

http://screencast.com/t/MGE5NWU1Y

Brian.