Java.lang.RuntimeException(Could not create Memcached client) caused bu UnresolvedAddress exception

[Preeti]

I have two amazon ec2 linux instances. On each instance tomcat6 is installed and the two ec2 instances are behind one load balancer. I have configured the tomcats' context.xml for the memcached session manager. I have included necessary jar files into $CATALINA_HOME/lib. But still I am getting java.lang.RuntimeException (could not create memcached client ) and UnresolvedAddress Exception. I am new to this. Can anyone help me where I am going wrong?

I have attached log file. Please go through it and tell me what is wrong with configuration.

[Billy Bacon]

Preeti,

You shouldn't change the Tomcat context.xml. Your war files /META-INF/context.xml is where your MSM configuration should be placed. The only modification you need to make to the actual Tomcat container are the libs which you identified below as something you've already done.

No log files were attached to your email so maybe try sending those again. Based on the UnresolvedAddressException, I don't believe what I state above will actually fix your problem. Sounds like a DNS or security group issue, or the endpoint is incorrect in the MSM configuration. 

Try moving the configuration into the war file /META-INF/context.xml and verify the endpoint(s) are correct.

- Billy -

[Preeti]

Hi Billy,

 I am not able to attach log file. I am just pasting here the exception stack trace along with the configuration. Please check this and tell me.

Tomcat 1

Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:ec2-54-245-60-219.us-west-2.compute.amazonaws.com:11211,n2:ec2-54-245-98-228.us-west-2.compute.amazonaws.com:11211"
failoverNodes="n1"
sessionBackupAsync="false"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"

Tomcat 2

Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
memcachedNodes="n1:ec2-54-245-60-219.us-west-2.compute.amazonaws.com:11211,n2:ec2-54-245-98-228.us-west-2.compute.amazonaws.com:11211"
failoverNodes="n2"
sessionBackupAsync="false"
requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"

And the exception which is being thrown is

Oct 18, 2012 9:46:02 AM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
Oct 18, 2012 9:46:02 AM org.apache.coyote.http11.Http11Protocol destroy
INFO: Stopping Coyote HTTP/1.1 on http-8080
Oct 18, 2012 10:23:16 AM org.apache.catalina.core.AprLifecycleListener init
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64/server:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/lib/amd64:/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
Oct 18, 2012 10:23:16 AM org.apache.coyote.http11.Http11Protocol init
INFO: Initializing Coyote HTTP/1.1 on http-8080
Oct 18, 2012 10:23:16 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1660 ms
Oct 18, 2012 10:23:17 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Oct 18, 2012 10:23:17 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/6.0.35
Oct 18, 2012 10:23:17 AM org.apache.catalina.startup.HostConfig deployDescriptor
INFO: Deploying configuration descriptor host-manager.xml
Oct 18, 2012 10:23:17 AM de.javakaffee.web.msm.MemcachedSessionService startInternal
INFO: MemcachedSessionService starts initialization... (configured nodes definition n1:ec2-54.245.97.203.us-west-2.compute.amazonaws.com:11211,n2:ec2-54-245-98-229.us-west-2.compute.amazonaws.com:11211, failover nodes null)
Oct 18, 2012 10:23:17 AM org.apache.catalina.core.StandardContext start
SEVERE: Error manager.start()
java.lang.RuntimeException: Could not create memcached client
	at de.javakaffee.web.msm.MemcachedSessionService.createMemcachedClient(MemcachedSessionService.java:468)
	at de.javakaffee.web.msm.MemcachedSessionService.startInternal(MemcachedSessionService.java:404)
	at de.javakaffee.web.msm.MemcachedBackupSessionManager.startInternal(MemcachedBackupSessionManager.java:140)
	at de.javakaffee.web.msm.MemcachedBackupSessionManager.start(MemcachedBackupSessionManager.java:649)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4714)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.nio.channels.UnresolvedAddressException
	at sun.nio.ch.Net.checkAddress(Net.java:48)
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:505)
	at net.spy.memcached.MemcachedConnection.createConnections(MemcachedConnection.java:122)
	at net.spy.memcached.MemcachedConnection.<init>(MemcachedConnection.java:106)
	at net.spy.memcached.DefaultConnectionFactory.createConnection(DefaultConnectionFactory.java:153)
	at net.spy.memcached.MemcachedClient.<init>(MemcachedClient.java:186)
	at de.javakaffee.web.msm.MemcachedSessionService.createMemcachedClient(MemcachedSessionService.java:466)
	... 26 more
Oct 18, 2012 10:23:17 AM org.apache.catalina.core.StandardContext start
SEVERE: Context [/host-manager] startup failed due to previous errors
Oct 18, 2012 10:23:17 AM de.javakaffee.web.msm.MemcachedSessionService shutdown
INFO: Stopping services.
Oct 18, 2012 10:23:17 AM org.apache.catalina.core.StandardContext start
SEVERE: Exception during cleanup after start failed
java.lang.NullPointerException
	at de.javakaffee.web.msm.MemcachedSessionService.shutdown(MemcachedSessionService.java:380)
	at de.javakaffee.web.msm.MemcachedBackupSessionManager.stop(MemcachedBackupSessionManager.java:682)
	at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4882)
	at org.apache.catalina.core.StandardContext.start(StandardContext.java:4750)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
	at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:675)
	at org.apache.catalina.startup.HostConfig.deployDescriptors(HostConfig.java:601)
	at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
	at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
	at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
	at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
	at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
	at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
	at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
	at org.apache.catalina.core.StandardService.start(StandardService.java:525)
	at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Oct 18, 2012 10:23:17 AM org.apache.tomcat.util.modeler.Registry registerComponent



Can you tell me where should I need to change?

[Martin Grotzke]

Hi,

can you ping the configured IPs / hosts from the machines? You should be
able to connect via telnet on port 11211. To see if it's a name
resolution issue you can try to connect to the (private) IP. If you can
reach the host and only the port is blocked you should create
appropriate security policies, IIRC Billy has written about that before.

Cheers,
Martin

--
inoio gmbh - http://inoio.de
Breitenfelder Str. 13c, 20251 Hamburg
Amtsgericht Hamburg, HRB 123031
Geschäftsführer: Dennis Brakhane, Martin Grotzke, Ole Langbehn

[Preeti]

Hi Martin,

      As you suggested, I tried with private ip's. Now that exception is gone but it is not storing the session data. I am not understanding what is wrong with it. I have deployed simple program on both the tomcats which are behind ELB. I am using non sticky sessions. So whenever I will make second request to my program, the previous request data is lost. Can you tell me what is the mistake.

Regards,
Preeti

[Martin Grotzke]

You need to configure msm accordingly: add sticky="false" to the
configuration.

Cheers,
Martin

[Preeti]

Yes, I had already changed sticky="false" in my configuration. The configuration looks like this

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
    memcachedNodes="n1:10.252.61.60:11211,n2:10.252.172.225:11211"
    sticky="false"
    sessionBackupAsync="false"
    lockingMode="uriPattern:/path1|/path2"

    requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
    transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory"

    />

But still it is not storing the session data.

Regards,
Preeti

[Martin Grotzke]

Please activate debug logging and check/send logs.

https://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration#Configure_logging

You can also run memcached with verbose logging (-vv) and check
memcached output.

Cheers,
Martin

[Preeti]

HI Martin,

I have implemented memcached session manager for both sticky and non-sticky session.

1.       In case of non-sticky sessions, the session data is getting transferred between two tomcats running on two EC2  instances.

2.       In case of load balancer with sticky sessions, it is working fine. But when you stop one instance, after restarting,the private ip of the instance will change. But the other instance which was running initially, will store the session data into the memcached node of stopped instance. So after restarting the instance, the session won’t be available in the restarted instance as its private ip will be changed.

How can I overcome this issue?

[Martin Grotzke]

How does an ec2 instance know about other instances, how do you configure memcached nodes in the tomcats? Do you start the instances, check their ips and then configure memcachedNodes accordingly?

Cheers,
Martin

Am 26.10.2012 12:29 schrieb "Preeti" <preetikho...@gmail.com>:

HI Martin,

I have implemented memcached session manager for both sticky and non-sticky session.

1.       1. In case of non-sticky sessions, the session data is getting transferred between two tomcats running on two EC2  instances.

2.       2. In case of load balancer with sticky sessions, it is working fine. But when you stop one instance, after restarting,the private ip of the instance will change. But the other instance which was running initially, will store the session data into the memcached node of stopped instance. So after restarting the instance, the session won’t be available in the restarted instance as its private ip will be changed.

[Martin Grotzke]

One option would be to use ec2's describe-instances to find running
instances (maybe + filtering) when a new node starts. Then the new node
would have to determince its nodeId, so it would need to have some kind
of communication with running nodes. Also running nodes would have to
update their node list regularly.

Elasticsearch does a similar thing, you can read the Running the server
/ Discovery part of
http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html.

Cheers,
Martin

On 10/26/2012 12:29 PM, Preeti wrote:
>
>
> HI Martin,
>
> I have implemented memcached session manager for both sticky and
> non-sticky session.
>
> 1. 1. In case of non-sticky sessions, the session data is getting
> transferred between two tomcats running on two EC2 instances.
>
> 2. 2. In case of load balancer with sticky sessions, it is working
> fine. But when you stop one instance, after restarting,the private ip of
> the instance will change. But the other instance which was running
> initially, will store the session data into the memcached node of
> stopped instance. So after restarting the instance, the session won’t be
> available in the restarted instance as its private ip will be changed.
>
> How can I overcome this issue?
>
>

[Preeti]

Yes, I start the instances first  and then I will configure the memcachedNodes according to the private ip's of instances. Now I need to do the configuration automatically. I mean if we set auto-scaling of  EC2 instances, whenever new instance is created, the configuration of nodes should happen automatically. How can I achieve this? 

[Martin Grotzke]

Please my previous email that mentions ec2 describe-instances.

There's nothing available out-of-the-box but would have to be
added/implemented. I'm interested in this but don't have much time to
put into this.

Cheers,
Martin