Getting xmemcached to be reconnect to memcached

[Bryant]

Hi,
I'm exploring using xmemcached as my Java client to memcached. I
discovered this after experiencing poor performance with the whalin
implementation and seeing your performance comparison.

I've integrated with xmemcached but it seems as though it is unable
to heal it's connection if memcached is restarted. For that matter,
the client builder throws an exception while trying to create the
client if memcached has not yet started.

What settings would I use to create a client that can recover from
an unconnected state when memcached returns? Thanks.

btw, I have minimal setup. Any other recommendations on a
'production' caliber setup in addition to healing/reconnecting sockets
would be appreciated.

XMemcachedClientBuilder builder = XMemcachedClientBuilder
( AddrUtil.getAddresses ("localhost:1624" ) );
builder.setTranscoder( new MyTranscoder() );
Memcached client = builder.build();

Bryant

[dennis zhuang]

hi,bryant

  Yes,it is a bug when build xmemcached client trying to connect a memcached server that is not yet started.But xmemcached is able to heal it's connecton if memcached is restarted,only when the client is constructed succefully.So,you can build a client with no hosts,add server after constructed succesfully,it will also throw exception,but the client is constructed in proper state.

XMemcachedClientBuilder builder = XMemcachedClientBuilder( );

 builder.setTranscoder( new MyTranscoder() );
 Memcached client = builder.build();

//after construct,add server.

 client.addServer("localhost:1624");

2009/6/19 Bryant <publi...@myrete.com>

--
庄晓丹

[dennis zhuang]

Thanks for you report,it will be fixed in 1.1.2

2009/6/19 dennis zhuang <killm...@gmail.com>

--
庄晓丹

[Bryant]

Hi Dennis,
Finally got back to this and grabbed 1.1.3.

So a couple of things that I think are probably configuration
related. I've noticed that when I kill memcached that the calls to
xmemcached seem to block, I'd prefer they fail and I can handle the
downtime by moving on. It also seems that the healing of the
connections isn't very fast (if it all). I'd typically quit my
samples after a few minutes of restarting memcached and not seeing
execution resume.

What settings would you recommend to get the following behavior.

1. Fails quickly for reads/writes while memcached is unavailable (non
blocking)
2. Heals connection with memcached quickly for subsequent calls when
memcached is restored.

I tried to tinker with the yanf4j but there's not a lot of
documentation on how I'd get the desired behavior.

Thanks,

Bryant

[dennis zhuang]

1.xmemcached doesn't blocking when memcached is unavaiable,it will throw a MemcachedException with message:"There is no avriable session at this moment.

2.It's a problem,in fact,if a memcached is unavaible,xmemcached will add a reconnecting request to a waiting queue,then a monitor thread will poll the request from this queue and try to heal connection,it will try 3 times for one request,if heal fail then add this reconnecting request to queue yet,if heal succesfully then remove the request.Between these three times reconnecting,the interval is 2 seconds,i am sorry it can't be configured.Why choose these behavior?Because i think the reconnecting may be too frequently,and influenced performance. I will considard this problem,maybe change this interval can be configured.

2009/7/20 Bryant <publi...@myrete.com>

--
庄晓丹

[Bryant]

Okay my misunderstanding. It does indeed throw an exception as you
described.

How long would you expect it to take for the underlying connection to
heal itself? I'm testing in the following manner...

1. Start memcached
2. Launch program that is saving and reading from memcached
3. quitting memcached.
4. restarting memcached

What I'm seeing is the exception is thrown but I'm still never able to
recover and have my test program continue successfully without a
restart. This is after two to three minutes of continued testing.

I don't need instant reconnects, even 2-3 minutes to reconnect is
probably fine, but I'm not seeing it reconnect at all.

Bryant

On Jul 19, 6:51 pm, dennis zhuang <killme2...@gmail.com> wrote:
> 1.xmemcached doesn't blocking when memcached is unavaiable,it will throw a
> MemcachedException with message:"There is no avriable session at this
> moment.
>
> 2.It's a problem,in fact,if a memcached is unavaible,xmemcached will add a
> reconnecting request to a waiting queue,then a monitor thread will poll the
> request from this queue and try to heal connection,it will try 3 times for
> one request,if heal fail then add this reconnecting request to queue yet,if
> heal succesfully then remove the request.Between these three times
> reconnecting,the interval is 2 seconds,i am sorry it can't be configured.Why
> choose these behavior?Because i think the reconnecting may be too
> frequently,and influenced performance. I will considard this problem,maybe
> change this interval can be configured.
>

> 2009/7/20 Bryant <publish...@myrete.com>

[dennis zhuang]

In my test,xmemcached can heal the connection if you have restart memcached.You can find it in the log output when you have configured log4j.properties.I don't know what's the problem with your test program,maybe your test program have finished invoking functions  before xmemcached heal connection?You should configure log for more information. 

2009/7/21 Bryant <publi...@myrete.com>

--
庄晓丹

[Bryant]

So I have two simple programs, one that writes to memached, the other
that reads (skipping imports)

public class PutInCache {
public static void main(String[] args) throws Exception {
XMemcachedClientBuilder builder = new XMemcachedClientBuilder
(AddrUtil.getAddresses("localhost:1624"));
MemcachedClient client = builder.build();

for (int i=0; i<1000; i++) {
try {
client.set("foo", 0, "Bar" + i);
System.out.println("Updated cache" + i);
}
catch (Exception e) {
System.out.println("Skipping update" + i);
}
Thread.sleep(500);
}
client.shutdown();
}
}

And the other that reads

public class ReadFromCache {
public static void main(String[] args) throws Exception {
XMemcachedClientBuilder builder = new XMemcachedClientBuilder
(AddrUtil.getAddresses("localhost:1624"));
MemcachedClient client = builder.build();

for (int i=0; i<1000; i++) {
try {
Object value = client.get("foo");
System.out.println(value);
}
catch (Exception e) {
System.out.println("no value");
}

Thread.sleep(500);
}

client.shutdown();
}
}

So I start these two programs and you can see memcached getting
updated and the values getting read. Then I kill memcached, as
expected the output switches to the 'catch' block codepath. Then I
restart memcached and the output stays 'stuck' on the catch block code
path. I only see logging at the moment I kill memecached. After that
flurry of logging I no longer see any output suggesting any retries.
The logging output I get when memcached is first killed is as
follows...

ul 21, 2009 10:53:15 AM
com.google.code.yanf4j.nio.impl.AbstractController start
WARNING: The Controller started at port 0 ...
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
Jul 21, 2009 10:53:15 AM com.google.code.yanf4j.nio.impl.Reactor
dispatchEvent
SEVERE: java.io.IOException: Connect to localhost:1624 fail
java.io.IOException: Connect to localhost:1624 fail
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
... 3 more
Jul 21, 2009 10:53:15 AM net.rubyeye.xmemcached.XMemcachedClient
connect
SEVERE: connect to localhost:1624 error
java.util.concurrent.ExecutionException: java.net.ConnectException:
Connection refused
at net.rubyeye.xmemcached.impl.MemcachedConnector$ConnectFuture.get
(MemcachedConnector.java)
at net.rubyeye.xmemcached.impl.MemcachedConnector$ConnectFuture.get
(MemcachedConnector.java)
at net.rubyeye.xmemcached.XMemcachedClient.connect
(XMemcachedClient.java)
at net.rubyeye.xmemcached.XMemcachedClient.<init>
(XMemcachedClient.java)
at net.rubyeye.xmemcached.XMemcachedClientBuilder.build
(XMemcachedClientBuilder.java)
at memcached.example.PutInCache.main(PutInCache.java:14)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
Jul 21, 2009 10:53:15 AM net.rubyeye.xmemcached.XMemcachedClient
connect
SEVERE: Connect to localhost:1624 fail
java.util.concurrent.ExecutionException: java.net.ConnectException:
Connection refused
at net.rubyeye.xmemcached.impl.MemcachedConnector$ConnectFuture.get
(MemcachedConnector.java)
at net.rubyeye.xmemcached.impl.MemcachedConnector$ConnectFuture.get
(MemcachedConnector.java)
at net.rubyeye.xmemcached.XMemcachedClient.connect
(XMemcachedClient.java)
at net.rubyeye.xmemcached.XMemcachedClient.<init>
(XMemcachedClient.java)
at net.rubyeye.xmemcached.XMemcachedClientBuilder.build
(XMemcachedClientBuilder.java)
at memcached.example.PutInCache.main(PutInCache.java:14)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
Jul 21, 2009 10:53:15 AM com.google.code.yanf4j.nio.impl.Reactor
dispatchEvent
SEVERE: java.io.IOException: Connect to localhost:1624 fail
java.io.IOException: Connect to localhost:1624 fail
at net.rubyeye.xmemcached.impl.MemcachedConnector.onConnect
(MemcachedConnector.java)
at com.google.code.yanf4j.nio.impl.Reactor.dispatchEvent
(Reactor.java)
at com.google.code.yanf4j.nio.impl.Reactor.run(Reactor.java)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:
574)
... 3 more
Jul 21, 2009 10:53:15 AM net.rubyeye.xmemcached.impl.MemcachedConnector
$SessionMonitor run
WARNING: Try to reconnect to localhost:1624 for 1 times

these programs run for over 5 minutes each, so there's a fair amount
of time for the connection to reconnect...

Do you see anything in my configuration?

Thanks, for providing such great turn around support!

Bryant

On Jul 21, 8:36 am, dennis zhuang <killme2...@gmail.com> wrote:
> In my test,xmemcached can heal the connection if you have restart
> memcached.You can find it in the log output when you have configured
> log4j.properties.I don't know what's the problem with your test
> program,maybe your test program have finished invoking functions before
> xmemcached heal connection?You should configure log for more information.
>

> 2009/7/21 Bryant <publish...@myrete.com>

[dennis zhuang]

WARNING: Try to reconnect to localhost:1624 for 1 times

This message show that xmemcached is trying to heal the connection,and then if you restart the memcached,you will see a log output like :

add session localhost:1624

It means a new connection have been setup for xmemcached.

2009/7/22 Bryant <publi...@myrete.com>

--
庄晓丹

[Bryant]

Okay so I decided to look into this further. I used mvn to generate
eclipse projects..

mvn eclipse:eclipse

and I got both xmemcacheed and yanf4j to build in Eclipse.

Then I made my sample projects depend on these two new eclipse
projects instead of the jars. Like magic the healing worked fine when
running like this. I then tried jarring up the .class files generated
by Eclipse to see if using those class files in my own jars would
help.

Restoring the project to use jars and not depending on other projects
I once again do not see the retries occurring.

So when building your code with Eclipse and running in Eclipse I can
see it working. When I instead pull your classes from a jar it does
not.

In both cases I'm running my sample from Eclipse, it's just a matter
of whether or not I get the xmemcached classes from Eclipse or from a
jar. I'll probably write a test to see if this same issue exists on
Linux (I'm running on Max OS X).

Have you heard of anything like this before?

On Jul 21, 6:14 pm, dennis zhuang <killme2...@gmail.com> wrote:
> WARNING: Try to reconnect to localhost:1624 for 1 times
> This message show that xmemcached is trying to heal the connection,and then
> if you restart the memcached,you will see a log output like :
> add session localhost:1624
>
> It means a new connection have been setup for xmemcached.

> 2009/7/22 Bryant <publish...@myrete.com>

> ...
>
> read more »

[Nelz]

I may have seen something like this before, using a different client.

In some of our environments, the firewalls are set up to swallow
connections if there's no (actively running) service behind a port.

Basically, instead of failing fast by sending a response to the
client, it just hangs the client...

I don't know more of the specifics for your case, just thought I'd
throw out a bit of my experience.

- Nelz