Memcached adding/removing nodes
mcevikce
other. I also understand that consistent hashing algorithm in client
libraries endure that data is distributed evenly between the servers.
What I am not clear about is:
Scenario 1:
Three server nodes running. key1 is stores on server #1. Server #1
goes down.
1-) Client makes a request on key1 stored on failed server. My
assumption is that cache will get a miss and value for key1 will be
retrieved from DB and stored in server #2.
2-) Now dead server is brought up. Client makes the request on a
key1 again. Does client get the key1 from server #2 then store it back
into server #1? How does re balancing occur?
Scenario 2:
Three nodes running. Another node is added. How does memcached
redistribute values to new node? I know that servers themselves do not
communicate with each other.
I hope my examples were clear. Your help will be appreciated.
Matt Ingenthron
The data isn't so much redistributed as it is re-populated from the
primary source to the new location in the consistent hash. The old
items in memory in the old positions will eventually expire or LRU out.
The key here is that the clients all have a consistent view and shared
hashing algorithm pointing to the same servers. If you add a new node,
for some portion of keys there will initally be misses, but then things
will balance out later.
Have a look at:
http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consistent_hashing_algo_for_memcache_clients
http://weblogs.java.net/blog/2007/11/27/consistent-hashing
Hope that helps,
- Matt
mcevikce
also raises some questions.
Lets use the example from that article.
Cache A contains values 1,4
Cache B contains values 2
Cache C contains values 3
Questions 1-) If cache A does down, would cache B will now have 4,1and
2? If so this is not equally distributed. cache B contains 3 values
(4,1,2) and Cache C has only one value (3). Lets assume if cache A
actually contained thousands of values, shifting these values to only
cache B will truly unbalance the distribution.
Questions-2) Lets use the same example as above for starters.
Cache A contains values 1,4
Cache B contains values 2
Cache C contains values 3
Lets say because of a network issue Cache A was not accessible but the
server is still up. Client will probably move values to cache B. Now
cache B has values 4,1,2. Lets say the content of values 1,4 changed.
Cache B has the new content for values. NOW the network problem is
fixed and Cache A again available. Will client get the values 1 and 4
from Cache A or Cache B? If it gets it from cache A, these values will
be stale because new content for these values are now in Cache B.
> Have a look at:http://www.last.fm/user/RJ/journal/2007/04/10/rz_libketama_-_a_consis...http://weblogs.java.net/blog/2007/11/27/consistent-hashing
Henrik Schröder
If synchronization is very important to your application, make sure you don't use automatic recovery from failover, and you won't get this problem. The flipside to that is that when you want to put back servers in the cluster, you need to restart your application so that all parts of it get the updated server list at the same time.
Another way of solving the problem is to not use failover at all. If your application is fine with more cache misses as long as one of your cache servers is down, then that solution is the best. You will never have synchronization problems, and you don't have to restart your application to bring back servers into the cluster.
In reality though, I have never seen a memcached server crash in production. It is very, very stable. Normally you don't have to worry about what happens if one server goes down, because they never do.
/Henrik
Dean Harding
> Three server nodes running. key1 is stores on server #1. Server #1
> goes down.
> 1-) Client makes a request on key1 stored on failed server. My
> assumption is that cache will get a miss and value for key1 will be
> retrieved from DB and stored in server #2.
Clients (that I know of) will not "automatically" redistribute keys if a
server goes down. It's usually only if you explicitly edit their
configuration to remove the server. If a server goes down, you'll just get
cache misses from that server until it comes back up again.
This works on the assumption that servers that have gone down "unexpectedly"
are only going to be temporary.
Dean.
Adam Lee
Yes, if you use automatic recovery from failover your cache can get unsynchronized as different parts of your application discover that a previous failing server is now back up at different points in time.
If synchronization is very important to your application, make sure you don't use automatic recovery from failover, and you won't get this problem. The flipside to that is that when you want to put back servers in the cluster, you need to restart your application so that all parts of it get the updated server list at the same time.
Another way of solving the problem is to not use failover at all. If your application is fine with more cache misses as long as one of your cache servers is down, then that solution is the best. You will never have synchronization problems, and you don't have to restart your application to bring back servers into the cluster.
In reality though, I have never seen a memcached server crash in production. It is very, very stable. Normally you don't have to worry about what happens if one server goes down, because they never do.
awl
Adam Lee
... we only use this mechanism to add new machines to the queue ...
awl