Two EC2 Tomcat7s, ELB, Elasticache - One Tomcat works perfectly, one fails with "Could not create memcached client... Unresolved address exception"

[Scott Wood]

First of all, many thanks for this awesome piece of software and the detailed Wiki that has helped me set it up!

As mentioned in the title, my setup is an Amazon Elastic Load Balancer (ELB) fronting two EC2 instances running Tomcat7. I'm using a memcached backed Elasticache and have followed the configuration advice for it in the Wiki.

When I log in to my web app and tail logs for one of the Tomcats, everything seems to be working perfectly - I can see messages like "

Endpoint to use for configuration access in this poll NodeEndPoint - HostName:test.uishek.0001.use1.cache.amazonaws.com IpAddress:10.6.X.XXX Port:11211" and it switches between 0001 and 0002 (the two nodes in my Elasticache cluster) fairly regularly. I can see session logging that I've enabled, and all is well.

However, on the other box, all is not well. The error on the problem box is:

SEVERE: Error manager.start()

org.apache.catalina.LifecycleException: Failed to start component [de.javakaffee.web.msm.MemcachedBackupSessionManager[]]

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:154)

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5476)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)

        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)

        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632)

        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1229)

        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1875)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.RuntimeException: Could not create memcached client

        at de.javakaffee.web.msm.MemcachedClientFactory.createMemcachedClient(MemcachedClientFactory.java:55)

        at de.javakaffee.web.msm.MemcachedSessionService.createMemcachedClient(MemcachedSessionService.java:516)

        at de.javakaffee.web.msm.MemcachedSessionService.startInternal(MemcachedSessionService.java:437)

        at de.javakaffee.web.msm.MemcachedBackupSessionManager.startInternal(MemcachedBackupSessionManager.java:554)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

        ... 12 more

Caused by: java.nio.channels.UnresolvedAddressException

        at sun.nio.ch.Net.checkAddress(Net.java:107)

        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:649)

        at net.spy.memcached.MemcachedConnection.createConnections(MemcachedConnection.java:243)

        at net.spy.memcached.MemcachedConnection.<init>(MemcachedConnection.java:182)

I'm using the same context.xml on both Tomcat boxes, configured to point to my Elasticache which has two nodes. Here it is:

<?xml version='1.0' encoding='utf-8'?>

<Context>

<Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"

        memcachedNodes="test.uishek.cfg.use1.cache.amazonaws.com:11211"

        requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"

        sessionBackupAsync="true"

        sticky="true"

        copyCollectionsForSerialization="false"

>

    </Manager>

</Context>

I'm wondering if the error rings a bell to anyone, and if it might mean I should take a closer look at my security group settings or my Tomcat itself. As far as I can tell, both are using identical security groups, so I'm curious as to why one fails and one succeeds. For what it's worth, I can't curl the URL I am trying to reach on the problem box either, though I can see that my tomcat7 service is running on the problem box.

Thanks very much for any and all help provided! Let me know if there's anything else useful I can provide as a diagnostic.

[Billy Bacon]

That sounds like an AWS specific routing issue/subnet or security group issue which you touched on. Seems extremely odd that ElasticBeanstalk would botch the security groups for the two instances it's spinning up for you in this environment, that's all taken care of for you by the magic of EB and CloudFormation. 

Are you working within a VPC? I would check your AZ's, if you are running a multi-az environment (which is a good idea), you typically indicate that you want EB to select 'any 2' (in your case here) Availability Zones, so that it forces EB to put at least 1 instance in the AZ's you have configured with your subnets. I'd look at your subnet configurations and make sure that the routing tables can route to your ElastiCache server. At face value, this doesn't seem to be a configuration issue with MSM.

- Bacon -

--

---
You received this message because you are subscribed to the Google Groups "memcached-session-manager" group.
To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Scott Wood]

I am running inside a VPC. I tried some more miscellaneous fiddling with security groups and the like, and after restarting the Tomcat service on the box a couple more times, I actually restarted the box itself. For whatever reason, that fixed my problem. Thanks for the assistance!

To unsubscribe from this group and stop receiving emails from it, send an email to memcached-session-manager+unsub...@googlegroups.com.