Using ICE outside of EC2

Stuart MacIntosh

unread,

Jul 25, 2013, 12:24:22 AM7/25/13

to iceu...@googlegroups.com

Hello

Just wonderinf if it's possible to use ICE outside of EC2?

In my testing, it fails with a java.net.ConnectException: Connection timed out attempting to contact port 80 of the Link-local address EC2 provides instance meta-data from.

We'd prefer to run ICE on our own infrastructure, if that's possible?

Thanks,

Stuart

Dominik Stadler

unread,

Jul 25, 2013, 12:30:43 AM7/25/13

to Stuart MacIntosh, iceu...@googlegroups.com

We have it running nicely outside of EC2, do you sit behind a proxy that prevents outgoing connections?

Dominik

--
You received this message because you are subscribed to the Google Groups "iceusers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to iceusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Stuart MacIntosh

unread,

Aug 1, 2013, 7:00:16 PM8/1/13

to iceu...@googlegroups.com, Stuart MacIntosh, dominik...@gmx.at

No we allow NAT-out the problem appears related to it trying to contact the EC2 metadata API hosted on the link-local address 169.254.169.254

I've since tried running on a t1.micro but a similar problem appears except, this time it's unable to contact http://169.254.169.254/latest/meta-data/iam/security-credentials/

Must admit this is the first grails-framework app I've tried to use.

Which paramaters control whether it attempts to contact the EC2 instance metadata API, or not?

f...@netflix.com

unread,

Aug 2, 2013, 10:16:54 PM8/2/13

to iceu...@googlegroups.com, Stuart MacIntosh, dominik...@gmx.at

Can you pull the latest code? The new code does not talk to http://169.254.169.254/latest/meta-data/iam/security-credentials/

f...@netflix.com

unread,

Aug 2, 2013, 10:20:18 PM8/2/13

to iceu...@googlegroups.com

To run Ice outside of ec2, you need to specify the following system properties at runtime. ice.s3AccessToken is optional.

ice.s3AccessKeyId=<accessKeyId>
  ice.s3SecretKey=<secretKey>
  ice.s3AccessToken=<accessToken>

-Fang

Stuart MacIntosh

unread,

Aug 4, 2013, 11:43:14 PM8/4/13

to iceu...@googlegroups.com

Thank you Fang

I performed 'git pull' and fast-forwarded to the latest code. Had already configured ice.properties based on the sample file, and with the two parameters you suggested set correctly (can mount the s3 bucket with them OK). Is src/java/ the correct location for the ice.properties file?

ubuntu@ip-10-000-000-000:~/ice$ ls -la src/java/ice.properties

-rw-r--r-- 1 ubuntu ubuntu 4082 Aug 1 23:12 src/java/ice.properties

Then I ran "./grailsw run-app" - it's the same result though. I also tried "./grailsw clean && ./grailsw compile" to be sure the output was fresh (although run-app seems to imply a compile).

Compilation succeeds with OpenJDK 1.6.0_27, it is run-app I have trouble with, on the Amazon Ubuntu 12.04 image. Please find exception details pasted below (RFC1918 IPv4 addresses redacted):

ubuntu@ip-10-000-000-000:~/ice$ ./grailsw run-app

| Running Grails application

2013-08-05 03:27:26,086 [main] INFO http11.Http11Protocol - Initializing ProtocolHandler ["http-bio-8080"]

2013-08-05 03:27:26,215 [main] INFO core.StandardService - Starting service Tomcat

2013-08-05 03:27:26,216 [main] INFO core.StandardEngine - Starting Servlet Engine: Apache Tomcat/7.0.30

2013-08-05 03:27:26,869 [localhost-startStop-1] INFO startup.ContextConfig - No global web.xml found

2013-08-05 03:27:27,227 [localhost-startStop-1] INFO [localhost].[/ice] - Initializing Spring root WebApplicationContext

2013-08-05 03:27:37,185 [localhost-startStop-1] INFO scaffolding.DefaultGrailsTemplateGenerator - Scaffolding template generator set to use resource loader org.codehaus.groovy.grails.commons.spring.GrailsWebApplicationContext@7da79447: startup date [Mon Aug 05 03:27:34 UTC 2013]; parent: Root WebApplicationContext

2013-08-05 03:27:40,318 [localhost-startStop-1] INFO context.GrailsConfigUtils - [GrailsContextLoader] Grails application loaded.

2013-08-05 03:27:40,453 [localhost-startStop-1] INFO BootStrap - Starting ice...

| Error com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service

| Error at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:99)

| Error at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:37)

| Error at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2969)

| Error at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:911)

| Error at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:801)

| Error at com.netflix.ice.common.AwsUtils.download(AwsUtils.java:313)

| Error at com.netflix.ice.common.AwsUtils.download(AwsUtils.java:308)

| Error at com.netflix.ice.common.AwsUtils.downloadFileIfNotExist(AwsUtils.java:296)

| Error at com.netflix.ice.basic.BasicReservationService.init(BasicReservationService.java:96)

| Error at com.netflix.ice.processor.ProcessorConfig.<init>(ProcessorConfig.java:86)

| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

| Error at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

| Error at java.lang.reflect.Constructor.newInstance(Constructor.java:532)

| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrConstructorNewInstance(ReflectiveInterceptor.java:975)

| Error at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)

| Error at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)

| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:57)

| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)

| Error at BootStrap$_closure1.doCall(BootStrap.groovy:174)

| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

| Error at java.lang.reflect.Method.invoke(Method.java:616)

| Error at org.springsource.loaded.ri.ReflectiveInterceptor.jlrMethodInvoke(ReflectiveInterceptor.java:1243)

| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)

| Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1082)

| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)

| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)

| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:956)

| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1106)

| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:906)

| Error at groovy.lang.Closure.call(Closure.java:412)

| Error at groovy.lang.Closure.call(Closure.java:406)

| Error at grails.util.Environment.evaluateEnvironmentSpecificBlock(Environment.java:308)

| Error at grails.util.Environment.executeForEnvironment(Environment.java:301)

| Error at grails.util.Environment.executeForCurrentEnvironment(Environment.java:277)

| Error at org.codehaus.groovy.grails.commons.DefaultGrailsBootstrapClass.callInit(DefaultGrailsBootstrapClass.java:60)

| Error at org.codehaus.groovy.grails.web.context.GrailsConfigUtils.executeGrailsBootstraps(GrailsConfigUtils.java:74)

| Error at org.codehaus.groovy.grails.web.context.GrailsContextLoader.initWebApplicationContext(GrailsContextLoader.java:108)

| Error at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:111)

| Error at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4791)

| Error at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5285)

| Error at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1559)

| Error at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1549)

| Error at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)

| Error at java.util.concurrent.FutureTask.run(FutureTask.java:166)

| Error at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)

| Error at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

| Error at java.lang.Thread.run(Thread.java:679)

| Error Caused by: java.io.FileNotFoundException: http://169.254.169.254/latest/meta-data/iam/security-credentials/

| Error at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1401)

| Error at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:106)

| Error at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:88)

| Error at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:53)

| Error at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:73)

| Error ... 51 more

| Error 2013-08-05 03:27:43,269 [localhost-startStop-1] ERROR BootStrap - Startup failed

Message: Unable to load credentials from Amazon EC2 metadata service

Line | Method

->> 99 | loadCredentials in com.amazonaws.auth.InstanceProfileCredentialsProvider

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

| 37 | getCredentials in ''

| 2969 | invoke . . . . . . . . . . . . . in com.amazonaws.services.s3.AmazonS3Client

| 911 | getObject in ''

| 801 | getObject . . . . . . . . . . . in ''

| 313 | download in com.netflix.ice.common.AwsUtils

| 308 | download . . . . . . . . . . . . in ''

| 296 | downloadFileIfNotExist in ''

| 96 | init . . . . . . . . . . . . . . in com.netflix.ice.basic.BasicReservationService

| 86 | <init> in com.netflix.ice.processor.ProcessorConfig

| 174 | doCall . . . . . . . . . . . . . in BootStrap$_closure1

| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment

| 301 | executeForEnvironment . . . . . in ''

| 277 | executeForCurrentEnvironment in ''

| 334 | innerRun . . . . . . . . . . . . in java.util.concurrent.FutureTask$Sync

| 166 | run in java.util.concurrent.FutureTask

| 1146 | runWorker . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor

| 615 | run in java.util.concurrent.ThreadPoolExecutor$Worker

^ 679 | run . . . . . . . . . . . . . . in java.lang.Thread

Caused by FileNotFoundException: http://169.254.169.254/latest/meta-data/iam/security-credentials/

->> 106 | readResponse in com.amazonaws.internal.EC2MetadataClient

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

| 88 | readResource in ''

| 53 | getDefaultCredentials . . . . . in ''

| 73 | loadCredentials in com.amazonaws.auth.InstanceProfileCredentialsProvider

| 37 | getCredentials . . . . . . . . . in ''

| 2969 | invoke in com.amazonaws.services.s3.AmazonS3Client

| 911 | getObject . . . . . . . . . . . in ''

| 801 | getObject in ''

| 313 | download . . . . . . . . . . . . in com.netflix.ice.common.AwsUtils

| 308 | download in ''

| 296 | downloadFileIfNotExist . . . . . in ''

| 96 | init in com.netflix.ice.basic.BasicReservationService

| 86 | <init> . . . . . . . . . . . . . in com.netflix.ice.processor.ProcessorConfig

| 174 | doCall in BootStrap$_closure1

| 308 | evaluateEnvironmentSpecificBlock in grails.util.Environment

| 301 | executeForEnvironment in ''

| 277 | executeForCurrentEnvironment . . in ''

| 334 | innerRun in java.util.concurrent.FutureTask$Sync

| 166 | run . . . . . . . . . . . . . . in java.util.concurrent.FutureTask

| 1146 | runWorker in java.util.concurrent.ThreadPoolExecutor

| 615 | run . . . . . . . . . . . . . . in java.util.concurrent.ThreadPoolExecutor$Worker

^ 679 | run in java.lang.Thread

On further investigation, the endpoint it is trying to contact I cannot from the virtual machine, either.

ubuntu@ip-10-000-000-000:~/ice$ wget -O - http://169.254.169.254/latest/meta-data/iam/security-credentials/

--2013-08-05 03:39:26-- http://169.254.169.254/latest/meta-data/iam/security-credentials/

Connecting to 169.254.169.254:80... connected.

HTTP request sent, awaiting response... 404 Not Found

2013-08-05 03:39:26 ERROR 404: Not Found.

The other EC2 metadata endpoints seem to work, I can retrieve instance ID for example:

ubuntu@ip-10-000-000-000:~/ice$ wget -q -O - http://169.254.169.254/latest/meta-data/instance-id

i-feedface

Hopefully this is just something simple I forgot to configure?

Thanks,

Stuart

Stuart MacIntosh

unread,

Aug 4, 2013, 11:50:39 PM8/4/13

to iceu...@googlegroups.com

Solved this!

I failed to read your answer carefully enough Fang :-)

After stipulating the s3 bucket credentials on the commandline (eg; ./grailsw -Dxxx) I now have the processor module working away!

Regards,

Stuart

Raymond Regnier

unread,

Aug 29, 2014, 1:50:36 PM8/29/14

to iceu...@googlegroups.com

Hey I'm having the same issue nearly a year later :)

BootStrap - Starting ice...
| Error com.amazonaws.AmazonClientException: The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/
--------------------------------------------------

I noticed Fang said the new code doesn't pull from this url http://169.254.169.254/latest/meta-data/iam/security-credentials/.. However I just downloaded from git yesterday so I would think the code is pretty current..

I've tried adding the -D and my credentials when I run ./grailsw but I'm stil getting the same error

Please advise

Reply all

Reply to author

Forward