Running memcached in a cluster

[sdeo]

Hi,
We are very excited to use a production cluster to host our
memcached servers for a web service. However I am a little bit unclear
as to the behavior in a real-life prod. environment where servers are
frequently being rebooted or swapped out and replaced (or new ones
added).

I know that you provide a identical list of servers to your clients so
that any keys are always mapped to the same server no matter which
client is performing the query. But I still have afew questions
regarding typical scenarios in our Prod environment:

1) what happens when a server to which a particular key hashes is
brought down, and restarted. Would the client (I am using the Java
client btw) be able to figure out that a particular server is back up
again transparently ?

2) In case one of the servers is brought down for a extended period,
would the keys that would have normally hashed continue to incur a DB
hit since the hashing function would always map to that machine for a
particular key or is there a way to have them start mapping to one of
the remaining server instances ?

3) Is there a way to make a live update to the list of servers that a
client can use without restarting the client ?

Thanks
sdeo.

[Dustin]

On Nov 3, 5:02 pm, sdeo <deo.shant...@gmail.com> wrote:

>   We are very excited to use a production cluster to host our
> memcached servers for a web service. However I am a little bit unclear
> as to the behavior in a real-life prod. environment where servers are
> frequently being rebooted or swapped out and replaced (or new ones
> added).

Such instability won't make for a very good environment of any kind.

> 1) what happens when a server to which a particular key hashes is
> brought down, and restarted. Would the client (I am using the Java
> client btw) be able to figure out that a particular server is back up
> again transparently ?

Which java client? Mine can, but it may very well be undesirable to
do that in your environment as it can introduce stale data.

> 2) In case one of the servers is brought down for a extended period,
> would the keys that would have normally hashed continue to incur a DB
> hit since the hashing function would always map to that machine for a
> particular key or is there a way to have them start mapping to one of
> the remaining server instances ?

No, they should map around the down servers quickly.

> 3) Is there a way to make a live update to the list of servers that a
> client can use without restarting the client ?

Not in my client. It'd be *possible*, but it's in no way
desirable. You're better off using guice to inject a client when you
need it and have it get the client from an AtomicReference that can be
swapped out with a complete client when stuff changes.

[sdeo]

Thanks for your response.

On Nov 3, 6:09 pm, Dustin <dsalli...@gmail.com> wrote:
> On Nov 3, 5:02 pm, sdeo <deo.shant...@gmail.com> wrote:
>
> >   We are very excited to use a production cluster to host our
> > memcached servers for a web service. However I am a little bit unclear
> > as to the behavior in a real-life prod. environment where servers are
> > frequently being rebooted or swapped out and replaced (or new ones
> > added).
>
>   Such instability won't make for a very good environment of any kind.
>
> > 1) what happens when a server to which a particular key hashes is
> > brought down, and restarted. Would the client (I am using the Java
> > client btw) be able to figure out that a particular server is back up
> > again transparently ?
>
>   Which java client?  Mine can, but it may very well be undesirable to
> do that in your environment as it can introduce stale data.

We use "memcached-1.4.jar" from (http://bleu.west.spy.net/~dustin/
projects/memcached/downloads.html)
which one do you use ?. We can handle stale data, what is important to
us is handling server restarts.

>
> > 2) In case one of the servers is brought down for a extended period,
> > would the keys that would have normally hashed continue to incur a DB
> > hit since the hashing function would always map to that machine for a
> > particular key or is there a way to have them start mapping to one of
> > the remaining server instances ?
>
>   No, they should map around the down servers quickly.

How so ? So you mean if key "foo" was mapping to memcached "server A",
and it goes down. The next time a client asks for "foo", it would hash
first to "server A" which fails, so would the client then
automatically remap to "Server B" ?
It would be really helpful if you can please explain it a bit more.

[Boris Partensky]

I think that's what "No, they should map around the down servers quickly." means :).

--
--Boris

[Boris Partensky]

Sorry, sent too quickly:

<< So you mean if key "foo" was mapping to memcached "server A",
<< and it goes down. The next time a client asks for "foo", it would hash
<< first to "server A" which fails, so would the client then
<< automatically remap to "Server B" ?

yes. This is essentially how memcache clients work.

--
--Boris

[Dustin]

On Nov 4, 11:07 am, sdeo <deo.shant...@gmail.com> wrote:

> We use  "memcached-1.4.jar" from (http://bleu.west.spy.net/~dustin/
> projects/memcached/downloads.html)
> which one do you use ?. We can handle stale data, what is important to
> us is handling server restarts.

That's my client, but a *really* old version of it. I'd recommend
upgrading to 2.2 (though I'm prepping a 2.2.1 with some minor
enhancements people have contributed or asked for since 2.2).

It'd be good to set up the scenario you want, and observe what
happens, and see what the difference is.

> >   No, they should map around the down servers quickly.
>
> How so ? So you mean if key "foo" was mapping to memcached "server A",
> and it goes down. The next time a client asks for "foo", it would hash
> first to "server A" which fails, so would the client then
> automatically remap to "Server B" ?
> It would be really helpful if you can please explain it a bit more.

When a server goes down, you want to pretend like it never existed.