Fail over doesn't work?

451 views
Skip to first unread message

Kevin

unread,
May 25, 2010, 8:37:21 PM5/25/10
to spymemcached
I just set up two memcached server and configure my spy client to
connect these two memcached like below code snippet. However, it seems
not working as what I expect when I shut down one of the memcached.

client = new net.spy.memcached.MemcachedClient(cfb.build(), AddrUtil
.getAddresses("localhost:11211 localhost:11212"));

When I start the client with one server shutting down, many timeout
happen (it's OK because spy client will identify broken server), but
for the request come afterward, some of them can be operated
successfully, while some of them can not. Since the default failure
mode is Redistribute, I suppose the client should be clever enough to
route all request to the server which is available rather than go to
the broken memcached. Am I misunderstood? Any help is appreciated!

P.S. The behavior is similar to the startup when I shutdown one of the
two running memcahed in run time.

Matt Ingenthron

unread,
May 26, 2010, 1:42:56 AM5/26/10
to spymem...@googlegroups.com
Hi Kevin,

Kevin wrote:
> I just set up two memcached server and configure my spy client to
> connect these two memcached like below code snippet. However, it seems
> not working as what I expect when I shut down one of the memcached.
>
> client = new net.spy.memcached.MemcachedClient(cfb.build(), AddrUtil
> .getAddresses("localhost:11211 localhost:11212"));
>
> When I start the client with one server shutting down, many timeout
> happen (it's OK because spy client will identify broken server), but
> for the request come afterward, some of them can be operated
> successfully, while some of them can not. Since the default failure
> mode is Redistribute, I suppose the client should be clever enough to
> route all request to the server which is available rather than go to
> the broken memcached. Am I misunderstood? Any help is appreciated!
>

I'll try to help, but this is very hard to read.

If I understand what you're saying here, you are starting your client
program while only one of the two servers is up. Is this correct?
You're saying you're observing some timeouts, but after the server is
identified as down, you continue to see something unexpected?

What behavior are you seeing after spy marks the server as down?


> P.S. The behavior is similar to the startup when I shutdown one of the
> two running memcahed in run time.
>

Could you rephrase this ^^^ ? I don't really know what you're trying
to say.

- Matt

Kevin

unread,
May 26, 2010, 2:05:46 AM5/26/10
to spymemcached

On May 26, 1:42 pm, Matt Ingenthron <ingen...@cep.net> wrote:
> Hi Kevin,
>
>
>
> Kevin wrote:
> > I just set up two memcached server and configure my spy client to
> > connect these two memcached like below code snippet. However, it seems
> > not working as what I expect when I shut down one of the memcached.
>
> > client = new net.spy.memcached.MemcachedClient(cfb.build(), AddrUtil
> >                     .getAddresses("localhost:11211 localhost:11212"));
>
> > When I start the client with one server shutting down, many timeout
> > happen (it's OK because spy client will identify broken server), but
> > for the request come afterward, some of them can be operated
> > successfully, while some of them can not. Since the default failure
> > mode is Redistribute, I suppose the client should be clever enough to
> > route all request to the server which is available rather than go to
> > the broken memcached. Am I misunderstood? Any help is appreciated!
>
> I'll try to help, but this is very hard to read.
>
> If I understand what you're saying here, you are starting your client
> program while only one of the two servers is up.  Is this correct?  

Yes


> You're saying you're observing some timeouts, but after the server is
> identified as down, you continue to see something unexpected?

Yes, I found that the client still trying to send the request to the
server
which is already shut down

>
> What behavior are you seeing after spy marks the server as down?
>
> > P.S. The behavior is similar to the startup when I shutdown one of the
> > two running memcahed in run time.
>
> Could you rephrase this ^^^  ?  I don't really know what you're trying
> to say.

OK, I start the client with both server are running normally. then I
shutdown
one of the server. Some request still be routed to the fail server. I
thought they
should be routed to the other server because the failmode is set to
Redistribute

Thanks!

>
> - Matt

Kevin

unread,
May 26, 2010, 2:14:42 AM5/26/10
to spymemcached
Here "shut down" means killing the process (not turning off the
power),
sorry for making confuse.

Boris Partensky

unread,
May 26, 2010, 10:49:57 AM5/26/10
to spymem...@googlegroups.com
Hi Kevin,

Does this thread seem relevant? What hashing strategy are you using?

--
You received this message because you are subscribed to the Google Groups "spymemcached" group.
To post to this group, send email to spymem...@googlegroups.com.
To unsubscribe from this group, send email to spymemcached...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/spymemcached?hl=en.


Kevin

unread,
May 26, 2010, 9:00:47 PM5/26/10
to spymemcached
Hi Boris,

I found that when I use array_mod as locator in connection builder, it
can work well. (the spy client can re-route the request). But
consistent locator doesn't work for 2 nodes, I added one more node in
the cache server list, it turn out to be OK.
I guess there are something wrong inside the
MemcachedConnection.addOperation(), or there are some configurations
I'm missing?

Here is my code for initializing the memcacheClient

// fail over works for two nodes
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
cfb.setFailureMode(FailureMode.Redistribute);
cfb.setDaemon(true);
cfb.setTimeoutExceptionThreshold(100); // continuous timeout
cfb.setOpTimeout(200);
cfb.setProtocol(Protocol.BINARY);
cfb.setLocatorType(Locator.ARRAY_MOD);

// fail over works for 3 nodes (dosen't work for 2 nodes)
ConnectionFactoryBuilder cfb = new ConnectionFactoryBuilder();
cfb.setFailureMode(FailureMode.Redistribute);
cfb.setDaemon(true);
cfb.setTimeoutExceptionThreshold(100); // continuous timeout
cfb.setOpTimeout(200);
cfb.setProtocol(Protocol.BINARY);
cfb.setLocatorType(Locator.CONSISTENT);

Thanks!

On May 26, 10:49 pm, Boris Partensky <boris.parten...@gmail.com>
wrote:


> Hi Kevin,
>
> Does this thread seem relevant? What hashing strategy are you using?
>

> http://groups.google.com/group/spymemcached/browse_thread/thread/f7cc...

> > spymemcached...@googlegroups.com<spymemcached%2Bunsu...@googlegroups.com>

rhinmass

unread,
May 26, 2010, 5:28:45 PM5/26/10
to spymemcached
Kevin,
I'm not sure if it is the same problem, but I had something similar,
and what solved it was to turn off optimization.
Here are the settings that finally worked for me:


ConnectionFactoryBuilder builder = new
ConnectionFactoryBuilder();
builder.setHashAlg(HashAlgorithm.NATIVE_HASH);

builder.setLocatorType(ConnectionFactoryBuilder.Locator.CONSISTENT);
builder.setShouldOptimize(false);

builder.setProtocol(ConnectionFactoryBuilder.Protocol.BINARY);
builder.setFailureMode(FailureMode.Cancel);
builder.setDaemon(true);
builder.setTimeoutExceptionThreshold(100); // continuous
timeout

Reply all
Reply to author
Forward
0 new messages