memcached failover solution?

[Adi]

Hi,
I am using memcached on windows server 2003 in a web cluster
environment setup through NLB (Network load balancer) and using two
different memcached server for caching, using BeIT Memcached client.

I want to know if memcached doesn't provide failover explicitly than
is there any way to handle it?

I had also check out the faqs, let me know please which windows client
provide consistent hashing if one memcached node is dead? Can i handle
it manually buy identifying the dead node and remove it from the
server list.

Regards
Adeel Ahmed

[Henrik Schröder]

Hi Adi,

Why do you want failover? It's just a cache, so your application should be able to run ok if part of the cache cluster is unavailable, you would experience a slightly higher cache miss ratio.

If your application requires that all memcached instances are up and running all the time, it's pretty likely that you are doing something wrong, that you are using memcached in a way it was not intended for. Please tell us a bit about your application and we can probably help you.

That said, the BeITMemcached client supports consistent hashing, but there is no automatic failover, and no automatic recovery from failover, because those features would in the absolute majority of cases only cause subtle errors in the application, without offering any benefits whatsoever.


/Henrik

[dormando]

Anyone interested in getting one of the windows clients to support
libmemcached, or at least the same replication method that the windows
client uses?

However the answer is almost always to spread out your instances a little
bit. Either make it easy for a standby server to take over the IP of a
dead host, or make it easy to push an update to your server list.

-Dormando

[moses wejuli]

Hi Henrick,

When you say:

        "If your application requires that all memcached instances are up and running all the time, it's pretty likely that you are doing something wrong, that you are using memcached in a way it was not intended for."

Does this mean you cannot use memcache for session management in PHP, coz then you are relying on it for a key, facet of your application -- one you cannot do without!!

Please advise........... Adi's concerns particularly resonate with mine.

Cheers.

2009/10/16 Henrik Schröder <skr...@gmail.com>

[Dustin]

On Oct 17, 3:49 pm, moses wejuli <m.wej...@googlemail.com> wrote:

> Does this mean you cannot use memcache for session management in PHP, coz
> then you are relying on it for a key, facet of your application -- one you
> cannot do without!!

I recommend reading this: http://dormando.livejournal.com/495593.html

In summary: memcached is a cache -- it provides fast access to data
that may be stored in your primary (presumably slower) data store. If
you're using it for your primary data store, you are likely asking for
trouble.

[moses wejuli]

Thanks Dustin.

I guess what i'm really askin is: would you recommend using memcached for session management in PHP.. the PHP extension for memcache has got a facility for manging sessions. This behaviour (using memcache for session mgmt) can be turned on in the PHP ini file.

I know/believe C is your primary language of expression but if you are at all familiar with PHP, please let me know your thoughts. Sessions are pretty non-trivial in PHP. I would presume in case of a cache miss, PHP would look to the default session store: the filesystem!

Really looking for someone particularly adept with this topic to shed some much needed light on this...

Cheers all,

M/.

2009/10/18 Dustin <dsal...@gmail.com>

[Dustin]

On Oct 17, 4:22 pm, moses wejuli <m.wej...@googlemail.com> wrote:
> Thanks Dustin.
>
> I guess what i'm really askin is: would you recommend using memcached for
> session management in PHP.. the PHP extension for memcache has got a
> facility for manging sessions. This behaviour (using memcache for session
> mgmt) can be turned on in the PHP ini file.
>
> I know/believe C is your primary language of expression but if you are at
> all familiar with PHP, please let me know your thoughts. Sessions are pretty
> non-trivial in PHP. I would presume in case of a cache miss, PHP would look
> to the default session store: the filesystem!
>
> Really looking for someone particularly adept with this topic to shed some
> much needed light on this...

This isn't a language issue. It's about deciding how you want your
application to behave.

The article describes a lot of the trade offs. Was there a specific
part you disagreed with?

I would, in particular, *not* recommend filesystem based storage
unless it's well-abstracted and you've only got one web server. In
general, I like to pretend like filesystems don't exist when writing
application code.

[moses wejuli]

the article is very good, incisive and convincing -- in summary, it requires that you treat sessions as any other data -- stored in the DB, and cached in m/cache, blah blah blah

i suppose i'll have to implemnt it first before i can come up with better questions -- for the time being though, many thanks Dustin.........

M/.

2009/10/18 Dustin <dsal...@gmail.com>

[Ren]

On Oct 18, 12:22 am, moses wejuli <m.wej...@googlemail.com> wrote:
> Thanks Dustin.
>
> I guess what i'm really askin is: would you recommend using memcached for
> session management in PHP.. the PHP extension for memcache has got a
> facility for manging sessions. This behaviour (using memcache for session
> mgmt) can be turned on in the PHP ini file.
>
> I know/believe C is your primary language of expression but if you are at
> all familiar with PHP, please let me know your thoughts. Sessions are pretty
> non-trivial in PHP. I would presume in case of a cache miss, PHP would look
> to the default session store: the filesystem!
>

There is no fall back to the file system for a cache miss, when using
MemCache extension.
It'll just start a new (empty) session, I believe.

The PHP configuration setting is reused for both files & memcache
session save handlers.
If files, its a directory to save sessions too, if memcache it's a
list of memcache addresses.

So using sessions becomes dependant on a memcache.
Therefore unusable for anything serious.

Jared

[Ren]

Should read

The PHP session.save_path configuration setting is reused for both
files & memcache...

Jared

[Henrik Schröder]

On Sun, Oct 18, 2009 at 00:47, dormando <dorm...@rydia.net> wrote:

Anyone interested in getting one of the windows clients to support
libmemcached, or at least the same replication method that the windows
client uses?

What do you mean? What would be required for this?


/Henrik Schröder

[dormando]

http://blogs.sun.com/trond/date/20090625
^ client side replication.

I like this and feel it's more powerful, since tt scales past two severs
implicitly, and you can enable/disable it per key or key type. So instead
of halving your effective cache, you can make a decision that some bulk of
data is easier to recache than others.

-Dormando

[Henrik Schröder]

On Mon, Oct 19, 2009 at 03:34, dormando <dorm...@rydia.net> wrote:

http://blogs.sun.com/trond/date/20090625
^ client side replication.

I like this and feel it's more powerful, since tt scales past two severs
implicitly, and you can enable/disable it per key or key type. So instead
of halving your effective cache, you can make a decision that some bulk of
data is easier to recache than others.

Ah ok, I see. It's an improvement over failover with consistent hashing, because when you fail over, your data already exists on the failover server, on a per-key basis, at the cost of storing the item multiple times, every time. Hm, that would work extremely well together with parallel requests, which is also something I'd like to add to my client. And as Trond writes, he uses it for the binary protocol only so he can do quiet sets that work properly.

I'm still partially against failover because of the synchronization issues with automatic recovery from failover, but adding replication really only affects automatic failover and makes that case slightly nicer, so it's definitely something to consider if I start adding failover support. I'll add it to my todo-list in any case, there's a bunch of stuff I'd like to implement if I could find the time and the urge to do it. :-)


/Henrik

[Adi]

Thanks to All,

In a web farm where two memcached server is hosted separately, when
one server is down could all request served from server 2
automatically or we need to remove the dead node explicitly from the
client?

I have tested a scenario:

Key1, Data1 Cached on Server1
Key2, Data2 Cached on Server2

If Server1 dead, than data would get from database NOT from the
Server2, (The defualt behaviour should be, if Key1 not found in
Server1 OR server not respond than client should check Server2 for the
data correspond to Key1 and if not found then store that data on
Server2 so that in next request it would fetch from Server2.

As dormando said: "standby server to take over the IP of a

dead host, or make it easy to push an update to your server list."

if i update the server list manually doesn't it effect the cache?

[Henrik Schröder]

How it works depends on which client you use. If you use the BeITMemcached client, when one instance goes down it will internally mark it as dead, start writing about it in the error log, and all requests that would end up at that server will be cache misses, which means that if you only have two servers, half of your data will not be cached. Your application should be able to handle this, and you should have something monitoring the servers and the error log so that you can see that one instance is down, and then you have the choice of either bringing the server back up again, or removing it from the configuration of you application and restarting it.

If you have a client that supports automatic failover (BeITMemcached doesn't) then in the scenario above, all your data would be cached on the other instance instead, so you would still be fully cached while bringing the failing instance back up, or removing it from the configuration. However, you would still have to restart your application to reset the failover. This would be the best option, I'll try to add failover to the BeITMemcached client as soon as I have time for it. :-)

The third case is a client that supports automatic failover, and automatic recovery from failover. It's similar to the scenario above, except it won't need an application restart when the failing memcached server comes back up, HOWEVER this means that your cache will not be synchronized while your application gradually discovers that the memcached server is back up. Depending on your application, this can be catastrophic, or it can be inconsequential. I don't really want to add this to my client, and if I do, there will be lots of warnings about it.


/Henrik

[Adam Lee]

On Mon, Oct 19, 2009 at 6:00 AM, Henrik Schröder <skr...@gmail.com> wrote:

How it works depends on which client you use. If you use the BeITMemcached client, when one instance goes down it will internally mark it as dead, start writing about it in the error log, and all requests that would end up at that server will be cache misses, which means that if you only have two servers, half of your data will not be cached. Your application should be able to handle this, and you should have something monitoring the servers and the error log so that you can see that one instance is down, and then you have the choice of either bringing the server back up again, or removing it from the configuration of you application and restarting it.

If you have a client that supports automatic failover (BeITMemcached doesn't) then in the scenario above, all your data would be cached on the other instance instead, so you would still be fully cached while bringing the failing instance back up, or removing it from the configuration. However, you would still have to restart your application to reset the failover. This would be the best option, I'll try to add failover to the BeITMemcached client as soon as I have time for it. :-)

The third case is a client that supports automatic failover, and automatic recovery from failover. It's similar to the scenario above, except it won't need an application restart when the failing memcached server comes back up, HOWEVER this means that your cache will not be synchronized while your application gradually discovers that the memcached server is back up. Depending on your application, this can be catastrophic, or it can be inconsequential. I don't really want to add this to my client, and if I do, there will be lots of warnings about it.

An even better to solution is to have the client automatically remove the server on failure but to have server rediscovery be a manual and/or synchronous operation.  You don't want a server flapping between up and down states because of something like a faulty NIC and thus having your cache become horribly inconsistent because none of the clients agree on its state, so you only want it to go down if it's really down and you only want it to come back up if it's really up and every client can agree (or all be told at the same time) that it's back up.

We use memcached in a few different ways in our system and, therefore, have a few different ways of implementing this on our side.  In one instance, we use memcached as a deterministic cache where we want to guarantee that all data is always available-- we do this by using our own custom client that does client-side replication to every server in the pool.  It's a fairly small dataset (a few gigs) and fits in memory on every instance, so we can easily do this.  We ensure consistency, though, by writing a marker entry that indicates when the cache was last populated.  Our client code never writes this entry, so a new server will only ever be _actually_ added to the pool when a full-populate is run manually or the nightly crontab job executes.  This way, we know that we can add new servers to the pool and not have to worry about them missing or having inconsistent data-- they won't be read from until they're shown to have the proper dataset.

In the other instance, we use memcached in a more standard configuration.  Here, we don't do any sort of client side replication, though we do use ketama hashing (and a few other things that I won't get into like a hacked up NodeLocator and cache miss fall-through to a middle tier persistent cache)...  When a machine dies, we automatically take it out of the config.  To add a new machine, though, we have to push out the config we want and fire off an admin message (just a special class of message on our standard message queue) indicating that a new, valid memcached config is present and should be loaded.  This way, we can somewhat guarantee that configs stay consistent across all instances.

--
awl

[dormando]

Heh... There're a few use cases that get me. One is new users who're
bolting memcached on to deal with horrific backend responsetime/etc. If
they're not too huge they can reduce cache efficiency by half and not
bankrupt themselves. Then over time remove keys from the replicated
scenario.

For larger folks it'd hopefully be used sparingly.

-Dormando

[István]

well this is a bit like mixing together couple of things, i thing if you are having multiple memcached instance you have to handle it from your app or write a proxy what checks the memcached servers and distributes the load and removing the dead one from the pool like load balancers do with http servers, pretty much 

--
the sun shines for all

http://l1xl1x.blogspot.com

[Henrik Schröder]

All memcached clients handle the distribution of items between servers, you just need to tell it which machines are in the pool, and never worry about where specific items end up. Normally you really shouldn't worry about that sort of thing. Good memcached clients offer you ways to choose failover behaviour and server distribution behaviour and connection pooling.


/Henrik

[Steve Webb]

We have several MC instances that are under a HA-Linux setup, where if we
lose one machine, another active machine takes both IPs, so we only lose
1/nth of the total pool until the machine comes back up again. Consistant
hashing still works across the whole pool and the hit is very small in a
10+ server pool.

- Steve

On Tue, 20 Oct 2009, Henrik Schröder wrote:

> Date: Tue, 20 Oct 2009 14:13:23 +0200
> From: Henrik Schröder <skr...@gmail.com>
> Reply-To: memc...@googlegroups.com
> To: memc...@googlegroups.com
> Subject: Re: memcached failover solution?

--
Steve Webb - Lead System Administrator for Pronto.com
Email: sw...@pronto.com (Please send any work requests to: r...@pronto.com)
Cell: 303-564-4269, Office: 303-351-1312, YIM: scumola

[eeyore]

Hello Steve,
can you share what are you using for the HA-setup and how happy are
you with it?
Is it spread with wackamole? We are looking for such a transition and
an opinion would be greatly appreciated.
Thanks,
Ivo

On Oct 20, 5:50 pm, Steve Webb <sw...@pronto.com> wrote:
> We have several MC instances that are under a HA-Linux setup, where if we
> lose one machine, another active machine takes both IPs, so we only lose
> 1/nth of the total pool until the machine comes back up again.  Consistant
> hashing still works across the whole pool and the hit is very small in a
> 10+ server pool.
>
> - Steve
>
>
>
> On Tue, 20 Oct 2009, Henrik Schröder wrote:
> > Date: Tue, 20 Oct 2009 14:13:23 +0200

> > From: Henrik Schröder <skro...@gmail.com>

> > Reply-To: memc...@googlegroups.com
> > To: memc...@googlegroups.com
> > Subject: Re: memcached failover solution?
>
> > All memcached clients handle the distribution of items between servers, you
> > just need to tell it which machines are in the pool, and never worry about
> > where specific items end up. Normally you really shouldn't worry about that
> > sort of thing. Good memcached clients offer you ways to choose failover
> > behaviour and server distribution behaviour and connection pooling.
>
> > /Henrik
>

> > On Tue, Oct 20, 2009 at 02:24, István <lecc...@gmail.com> wrote:
>
> >> well this is a bit like mixing together couple of things, i thing if you
> >> are having multiple memcached instance you have to handle it from your app
> >> or write a proxy what checks the memcached servers and distributes the load
> >> and removing the dead one from the pool like load balancers do with http
> >> servers, pretty much
>

[Adi]

Hi Henrik,
could you please tell me how can i remove the dead node from the pool,
or can you provide me any documentation of BeIT memcached client so
that i understand it easily.

Actually, In BeIT client there is method to detech the status of the
host, i want to remove it from the socket pool if detect the server is
dead. let me know where can i change the code.

example:

public void RemoveDeadNode()
{
foreach (SocketPool pool in serverPool.HostList)
{
if (!serverPool.Execute<bool>(pool, false, delegate
{ return true; }))
{
pool.Remove(pool.Host); //This method not
exist

> > if i update the server list manually doesn't it effect the cache?- Hide quoted text -
>
> - Show quoted text -

[Henrik Schröder]

First, check SocketPool.cs, around line 160. That's the code that fires if it failed to open a socket. You want to check if deadEndPointSecondsUntilRetry is larger than some sort of threshhold (A minute or so, you don't want to remove servers the first time they fail, you can have short temporary network errors). If it is, you would do a call to owner.RemoveSocketPool(this).

Then, in ServerPool.cs, you would have to create that method, which removes a socket pool from the three data structures that are used for lookup. You first want to remove it from the array hostList. Then you want to find all entries in hostDictionary that have your socket pool as a value, and then remove those keys from the array hostKeys, and the entry from hostDictionary. And that should be it.


/Henrik