Hazelcast discovery on AWS ECS Fargate

paul.car...@jini.guru

unread,

Jan 26, 2018, 12:53:49 AM1/26/18

to Hazelcast

Hi,

I'm trying to get discovery to work on Fargate tasks which only support awsvpc. Using version 3.9.2

My config:

<hazelcast xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://www.hazelcast.com/schema/config

http://www.hazelcast.com/schema/config/hazelcast-config-3.9.xsd"

xmlns="http://www.hazelcast.com/schema/config">

<group>

</group>

<property name="hazelcast.shutdownhook.enabled">false</property>

</properties>

<join>

<tcp-ip enabled="false"/>

<discovery-strategies>

<discovery-strategy enabled="true" class="com.hazelcast.aws.AwsDiscoveryStrategy">

<property name="iam-role">ecsTaskExecutionRole</property>

</properties>

</discovery-strategy>

</discovery-strategies>

</join>

</network>

</hazelcast>

The result:

Caused by: com.hazelcast.config.InvalidConfigurationException: Unable to lookup role in URI: http://169.254.169.254/latest/meta-data/placement/availability-zone/

at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:75)

at com.hazelcast.aws.AWSClient.getAvailabilityZone(AWSClient.java:61)

at com.hazelcast.aws.AwsDiscoveryStrategy.discoverLocalMetadata(AwsDiscoveryStrategy.java:120)

at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:91)

at com.hazelcast.instance.Node.mergeEnvironmentProvidedMemberMetadata(Node.java:487)

at com.hazelcast.instance.Node.start(Node.java:380)

at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:133)

at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:195)

at com.hazelcast.instance.HazelcastInstanceFactory.getOrCreateHazelcastInstance(HazelcastInstanceFactory.java:104)

at com.hazelcast.core.Hazelcast.getOrCreateHazelcastInstance(Hazelcast.java:166)

at guru.jini.arch.impl.hazelcast.HazelcastFactory.<init>(HazelcastFactory.java:64)

at guru.jini.arch.impl.hazelcast.HazelcastFactory.getHazelcastInstance(HazelcastFactory.java:35)

at guru.jini.arch.impl.map.HazelcastDistributedMap.getInstance(HazelcastDistributedMap.java:36)

at guru.jini.arch.impl.factory.JGImpl.getDistributedMap(JGImpl.java:187)

at guru.jini.arch.JG.<clinit>(JG.java:76)

... 13 more

Caused by: java.net.ConnectException: Invalid argument (connect failed)

at java.net.PlainSocketImpl.socketConnect(Native Method)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at sun.net.NetworkClient.doConnect(NetworkClient.java:175)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)

at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)

at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)

at sun.net.www.http.HttpClient.New(HttpClient.java:339)

at sun.net.www.http.HttpClient.New(HttpClient.java:357)

at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)

at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)

at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)

at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)

at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)

at com.hazelcast.aws.utility.MetadataUtil.retrieveMetadataFromURI(MetadataUtil.java:67)

Any ideas on what could be wrong

Guglielmo Nigri

unread,

Jan 26, 2018, 4:51:18 AM1/26/18

to Hazelcast

Looks like a connectivity issue.

What's the startup script that you're using?

Also, are you using a proxy?

You may try to add -Djava.net.preferIPv4Stack=true to JAVA_OPTS or directly to the start command line, if it's not already there.

paul.car...@jini.guru

unread,

Jan 26, 2018, 6:11:44 AM1/26/18

to Hazelcast

Sorry, and no I'm not using a proxy or anything. Very simple debian based container with default Fargate settings

İhsan Demir

unread,

Jan 29, 2018, 3:24:15 AM1/29/18

to haze...@googlegroups.com

Hello,

Can it be aws security issue? It says: Unable to lookup role in URI: http://169.254.169.254/latest/meta-data/placement/availability-zone/

Do you have the permission to query the roles from your machine?

Regards,

ihsan

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1456508a-c14a-4660-8e33-972a1396da3c%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Ihsan Demir
Software Engineer, Clients Team
Mahir İz Cad. No:35, Altunizade, İstanbul
ih...@hazelcast.com

skype: idemir

cell: +90-543-6512477

paul.car...@jini.guru

unread,

Jan 29, 2018, 5:35:20 PM1/29/18

to Hazelcast

I've cross posted to AWS forums as this is not specific to Hazelcast

https://forums.aws.amazon.com/thread.jspa?messageID=826994&#826994

Paul Carter-Brown

unread,

Jan 29, 2018, 5:35:20 PM1/29/18

to haze...@googlegroups.com

Hi Ihsan,

Further down in the trace it says "Caused by: java.net.ConnectException: Invalid argument (connect failed)"

I wondered if it could be permissions related but I tried all sorts of AWS roles and nothing changed.

Paul

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/tL9tuJ1Bkqo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+unsubscribe@googlegroups.com.

To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/CAH%2B3FmL94HQ-5hysGwWwz_E6HX4BtRfvzk_msEnVoDPt2Mt7fQ%40mail.gmail.com.

paul.car...@jini.guru

unread,

Jan 29, 2018, 5:35:20 PM1/29/18

to Hazelcast

I tried curl inside the container by adding a curl command to the script that runs inside the container when it starts:

echo "Getting availability zone"

curl -sv -4 http://169.254.169.254/latest/meta-data/placement/availability-zone 2>&1

echo "Got availability zone"

Getting availability zone
Trying 169.254.169.254...
TCP_NODELAY set
Immediate connect fail for 169.254.169.254: Invalid argument
Closing connection 0

Got availability zone

So yes it appears to be a networking issue but for the life of me I cannot see what could be causing "Invalid argument".

On Friday, January 26, 2018 at 11:51:18 AM UTC+2, Guglielmo Nigri wrote:

dougla...@fox.com

unread,

May 18, 2018, 7:53:19 PM5/18/18

to Hazelcast

The issue is caused by the changes in the metadata endpoint for aws fargate, seems like the fargate containers does not have access to the traditional metadata and the Hazelcast AWS SPI discovery method uses this endpoint.

The new endpoint is documented here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-metadata-endpoint.html

nor...@gmail.com

unread,

Oct 22, 2018, 10:26:02 AM10/22/18

to Hazelcast

Hello!

Does hazelcast work under Fargate? all the docs refer to ECS which is a bit different. For example, it says you need ec2DescribeInstance Role to get the Tags in order to discover other cluster members, however Fargate doesn't use instances...

Could you confirm you can deploy Hazelcast cluster on Fargate??

Cheers!

Mesut Celik

unread,

Oct 26, 2018, 11:39:02 AM10/26/18

to haze...@googlegroups.com

Hi,

Hazelcast - AWS Fargate support issue can be found here.

https://github.com/hazelcast/hazelcast-aws/issues/86

--

You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.

To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2f1ecf67-1ccb-4431-a283-49cc50eb76fd%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Mesut Celik
Services & Integration Team Lead
me...@hazelcast.com

@mesutcelik

dougla...@fox.com

unread,

Nov 1, 2018, 9:31:51 AM11/1/18

to Hazelcast

Hi, now is not working because the ec2-discovery strategy, these APIs are different than the used by fargate so the only way to make it work, is to handle the way were the nodes discover each other with another mechanism

rafal....@gmail.com

unread,

Nov 1, 2018, 9:31:51 AM11/1/18

to Hazelcast

Hi,

AFAIK Hazelcast will not work in Fargate out of the box. You would need to take care of the members discovery by yourself (e.g. using Eureka or Zookeeper). The Hazelcast AWS Plugin was never tested on Fargate and I believe it's not going to work as it is.

Regards,
Rafał

u...@revault.ch

unread,

Apr 26, 2019, 3:30:49 PM4/26/19

to Hazelcast

Hello

We wrote this discovery plugin, maybe it is helpful to others https://github.com/iKentoo/hazelcast-aws-ecs

Regards,

Urs

David Brimley

unread,

Apr 27, 2019, 5:32:53 AM4/27/19

to Hazelcast

Fargate support should be ready in about 2-3 weeks.

You can follow progress on the Github Issue here....

https://github.com/hazelcast/hazelcast-aws/issues/86

Reply all

Reply to author

Forward