Running in AWS/invalid serviceURL

TysonS

unread,

Jan 23, 2013, 12:41:51 PM1/23/13

to eureka_...@googlegroups.com

Trying to get a cluster going in AWS, and I can't get Eureka to bootstrap. As soon as I set the property eureka.datacenter=cloud, I get this:

Can anyone enlighten me? I've specified AWS credentials with the rights to read and assign EIPs, and I have set eureka.serviceUrl.us-east-1a and eureka.serviceUrl.us-east-1c to EIP URLs.

TysonS

unread,

Jan 23, 2013, 1:20:38 PM1/23/13

to eureka_...@googlegroups.com

Disregard. I'm running this in Windows and the Tomcat configuration dialog for the Java options owned me.

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 1:21:00 PM1/23/13

to eureka_...@googlegroups.com

Tyson,

Your configurations should be the following.

eureka.region=us-east-1(Default is us-east-1)

eureka.us-east-1.availabilityZones=us-east-1a,us-east-1c

and then followed by the EIP assignments

eureka.serviceUrl.us-east-1a=<MY_EIP>

eureka.serviceUrl.us-east-1c=<MY_EIP>

I will try to update the documentation to make it much more obvious.

TysonS

unread,

Jan 23, 2013, 3:24:01 PM1/23/13

to eureka_...@googlegroups.com

Karthik,

Yeah, my settings were correct, but the property -Deureka.datacenter was not being picked up by Tomcat.

I have a second configuration question: what all does the AWS user need access to? The config file says an account that "has access to EIP binding and looking up autoscaling information". I have given one the following policy. Am I missing anything? Could you document on the wiki a sample policy?

{

"Statement": [

{

"Sid": "Stmt",

"Action": [

"ec2:AssociateAddress"

],

"Effect": "Allow",

"Resource": [

"*"

]

},

{

"Sid": "Stmt",

"Action": [

"autoscaling:DescribeAdjustmentTypes",

"autoscaling:DescribeAutoScalingGroups",

"autoscaling:DescribeAutoScalingInstances",

"autoscaling:DescribeAutoScalingNotificationTypes",

"autoscaling:DescribeLaunchConfigurations",

"autoscaling:DescribeMetricCollectionTypes",

"autoscaling:DescribeNotificationConfigurations",

"autoscaling:DescribePolicies",

"autoscaling:DescribeScalingActivities",

"autoscaling:DescribeScalingProcessTypes",

"autoscaling:DescribeScheduledActions",

"autoscaling:DescribeTags",

"autoscaling:DescribeTriggers"

],

"Effect": "Allow",

"Resource": [

"*"

]

}

]

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 3:55:38 PM1/23/13

to eureka_...@googlegroups.com

Tyson,

Here is the sample policy file I generated using AWS policy generator with all access required. Please let me know if it works and I will update the docs accordingly.

{
  "Statement": [
    {
      "Sid": "Stmt1358974336152",
      "Action": [
        "ec2:AssociateAddress",
        "ec2:DisassociateAddress"
      ],
      "Effect": "Allow",
      "Resource": "*"
    },
    {
      "Sid": "Stmt1358974395291",
      "Action": [
        "autoscaling:DescribeAutoScalingGroups"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

TysonS

unread,

Jan 23, 2013, 4:04:11 PM1/23/13

to eureka_...@googlegroups.com

Karthik,

I keep getting an error. The first server in the cluster was able to bind right away. The second one keeps giving me this error:

2013-01-23 21:01:09,822 ERROR com.netflix.eureka.EurekaBootStrap:266 [http-bio-8080-exec-18] [bindEIP] Cannot bind to EIP

java.lang.NullPointerException

at java.lang.String.contains(Unknown Source)

at com.netflix.eureka.PeerAwareInstanceRegistry.isRegisterable(PeerAwareInstanceRegistry.java:697)

at com.netflix.eureka.util.EIPManager.getCandidateEIP(EIPManager.java:207)

at com.netflix.eureka.util.EIPManager.bindToEIP(EIPManager.java:108)

at com.netflix.eureka.EurekaBootStrap.bindEIP(EurekaBootStrap.java:263)

at com.netflix.eureka.EurekaBootStrap.handleEIPbinding(EurekaBootStrap.java:222)

at com.netflix.eureka.EurekaBootStrap.contextInitialized(EurekaBootStrap.java:111)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1256)

at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:714)

at org.apache.catalina.manager.HTMLManagerServlet.doPost(HTMLManagerServlet.java:219)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.catalina.filters.CsrfPreventionFilter.doFilter(CsrfPreventionFilter.java:213)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.catalina.filters.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:108)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:581)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)

at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)

at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

TysonS

unread,

Jan 23, 2013, 5:05:23 PM1/23/13

to eureka_...@googlegroups.com

Karthik,

Regarding that last error: I reduced the AZs down to just us-east-1a, and received this error:

I wondered if I've come across a bug, or if you can identify what I've done wrong.

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 5:22:42 PM1/23/13

to eureka_...@googlegroups.com

Can you pass in the system property -Deureka.region=us-east-1 and see if you can get around it. I will look into why that problem shows up in the second server.

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 5:48:19 PM1/23/13

to eureka_...@googlegroups.com

Looking at the code, the only thing I could think of is availability zones configuration might have a whitespace. Can you confirm?

eureka.us-east-1.availabilityZones

TysonS

unread,

Jan 23, 2013, 6:01:27 PM1/23/13

to eureka_...@googlegroups.com

Passing in the region didn't work. I commented out the property in eureka-client.properties, too, just to be sure. There is no whitespace in the lines. I retyped them to be sure it is only the characters expected.

This property should be set for all Eureka servers, right?

eureka.preferSameZone=false

Do the property files need to be in a particular format? Unicode? Windows or Unix line endings?

TysonS

unread,

Jan 23, 2013, 6:20:13 PM1/23/13

to eureka_...@googlegroups.com

Swapped servers so that the one in zone us-east-1a was alone when it started, and it associated the EIP perfectly. Then tried to start the one in zone us-east-1c with identical config files and it did the same thing as the other did before--it throws exceptions on the isRegisterable() call:

2013-01-23 23:17:36,454 ERROR com.netflix.eureka.PeerAwareInstanceRegistry:273 [localhost-startStop-1] [syncUp] During DS init copy

java.lang.NullPointerException

at java.lang.String.contains(Unknown Source)

at com.netflix.eureka.PeerAwareInstanceRegistry.isRegisterable(PeerAwareInstanceRegistry.java:697)

at com.netflix.eureka.PeerAwareInstanceRegistry.syncUp(PeerAwareInstanceRegistry.java:266)

at com.netflix.eureka.EurekaBootStrap.contextInitialized(EurekaBootStrap.java:106)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:633)

at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:977)

at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1655)

at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)

at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)

at java.util.concurrent.FutureTask.run(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

I only have 2 servers across 2 zones. That should work, right?

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 6:50:05 PM1/23/13

to eureka_...@googlegroups.com

Hmm.. Something else is going on. I will be able to look at in 3-4 hours.

Karthikeyan Ranganathan

unread,

Jan 23, 2013, 9:51:44 PM1/23/13

to eureka_...@googlegroups.com

Tyson,

Eureka should work with instances in the cluster from the same zone or multiple zones.

I identified a possible problem where the server tried to access a property from a dependent library directly instead of eureka itself. I fixed that. Please give it a try and let me know.

The preferSamezone should be false for eureka server and true for services/clients. So, the value you have (preferSamezone = false) is correct. The idea here is for the server to go across other zones to get information it wants and the client/services to prefer the same zone for latency and other things.

Regarding this error, when you reduced to one zone

ERROR com.netflix.discovery.DiscoveryClient:1146 [localhost-startStop-1] [getZoneOffset] DISCOVERY: invalid zone - us-east-1a defaulting to us-east-1a

Although the error is confusing, it did not have any negative impact. This is because you removed one zone from your list and since your settings (preferSameZone=false) is configured to go across zones, it confused the server a bit. Anyway the configuration you specified for testing(only one zone) is not an invalid use-case and hence I update it to a warning.

If everything goes fine, I would like to get your input on what was difficult and I will update the docs accordingly.

TysonS

unread,

Jan 24, 2013, 11:24:46 AM1/24/13

to eureka_...@googlegroups.com

Karthik,

Your fix appears to have fixed the problem; thank you! There are a couple issues with the second instance. It sees the primary instance as available and replicates from it (good!), but it lists itself as unavailable. Is that normal? After a few minutes of being up, it reported:

"EMERGENCY! EUREKA MAY BE INCORRECTLY CLAIMING INSTANCES ARE UP WHEN THEY'RE NOT. RENEWALS ARE LESSER THAN THRESHOLD AND HENCE THE INSTANCES ARE NOT BEING EXPIRED JUST TO BE SAFE."

The primary shows no such message. Is there a property I should be setting that I am not?

Right now, the only feedback I suggest for the doc is putting that property "eureka.datacenter=cloud" on the "Configuring Eureka in AWS Cloud" page. It took me a while to find it (it's in bold on the Getting Started page) when I moved the instances up to the cloud, and I was confused for a while about why the instances weren't registering themselves as AWS instances.

Thanks again for your help on this!

-Tyson

TysonS

unread,

Jan 24, 2013, 11:26:46 AM1/24/13

to eureka_...@googlegroups.com

Quick follow-up: the second server stopped reporting itself as unavailable after a few minutes. The emergency message is still showing up, however.

TysonS

unread,

Jan 24, 2013, 11:40:42 AM1/24/13

to eureka_...@googlegroups.com

Emergency message went away. I'd like to know what causes it.

Karthikeyan Ranganathan

unread,

Jan 24, 2013, 11:50:11 AM1/24/13

to eureka_...@googlegroups.com

Can you please attach the screen shot of both server status pages if possible?

Karthikeyan Ranganathan

unread,

Jan 24, 2013, 3:48:47 PM1/24/13

to eureka_...@googlegroups.com

Eureka has this feature to detect if it is getting heartbeats from servers that are registered. If it does not get heartbeats from 85% of servers at least (default ) it lets you know that it is not receiving heartbeats and hence it will try to keep the instances it has rather than expiring it.

More about that information here.

https://github.com/Netflix/eureka/wiki/Understanding-Eureka-Peer-to-Peer-Communication

TysonS

unread,

Jan 25, 2013, 1:52:31 AM1/25/13

to eureka_...@googlegroups.com

Karthik,

If I see it again, I'll capture it and post it. Thanks again for all of your help. We're pretty excited to be using it!

-Tyson

Karthikeyan Ranganathan

unread,

Jan 25, 2013, 11:10:00 AM1/25/13

to eureka_...@googlegroups.com

Absolutely. Let me know if you run into anything else.

Reply all

Reply to author

Forward