Re: Hazlcast 4.2 not starting on machine with the hostname mapped on the loopback interface

[Josef Cacek]

Hello Aniket,

if I read the log correctly, you have the <tcp-ip> discovery enabled and the hostname DEV0008557.ORG.COM is listed in its member list.

I see 2 problems in the configuration.
* using hostnames instead of IP addresses in the configuration is not completely safe in Hazelcast versions before 5.1

* the default AddressPicker implementation only supports/picks addresses from the ones listed by network interfaces. (I guess your loopback interface only lists 127.0.0.1, but not the 127.0.1.1 to which the hostname is resolved).

So the safest way is to use the standard loopback address 127.0.0.1 instead of the hostname in the Hazelcast configuration.

Still, if you want to stick to this hostname, use it also in the <public-address> configuration. It could help in your scenario. Example:

<network>
    <public-address>DEV0008557.ORG.COM</public-address>
    <join>
        <tcp-ip enabled="true">
            <member-list>
                <member>DEV0008557.ORG.COM</member>
            </member-list>
        </tcp-ip>
    </join>
</network>

Regards,

-- Josef

On Wed, Jun 1, 2022 at 12:37 AM Aniket Patil <app...@asu.edu> wrote:

We upgraded to hazelcast 4.2 and hazelcast is not starting on machine with hostname mapped on the loopback interface. The same configuration ised to work for Hazelcast earlier version (3.12) 

I can see the following in the logs , Can I please get some pointers to the issue. 


31-May-2022 15:25:25.320 WARNING [main] com.hazelcast.instance.impl.HazelcastInstanceFactory.null Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
 --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
31-May-2022 15:25:25.400 WARNING [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [782c00c1-0c62-421c-973f-409370d15128] [4.2] You configured your member address as host name. Please be aware of that your dns can be spoofed. Make sure that your dns configurations are correct.
31-May-2022 15:25:25.400 INFO [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [782c00c1-0c62-421c-973f-409370d15128] [4.2] Resolving domain name 'DEV0008557.ORG.COM' to address(es): [127.0.1.1]
31-May-2022 15:25:25.400 INFO [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [782c00c1-0c62-421c-973f-409370d15128] [4.2] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [DEV0008557.ORG.COM/127.0.1.1]
31-May-2022 15:25:25.401 WARNING [main] com.hazelcast.instance.AddressPicker.null [LOCAL] [782c00c1-0c62-421c-973f-409370d15128] [4.2] Could not find a matching address to start with! Picking one of non-loopback addresses.
31-May-2022 15:25:25.438 INFO [main] com.hazelcast.system.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Hazelcast 4.2 (20210324 - 405cfd1) starting at [10.44.96.29]:5701
31-May-2022 15:25:26.059 INFO [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Using TCP/IP discovery
31-May-2022 15:25:26.061 WARNING [main] com.hazelcast.cp.CPSubsystem.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
31-May-2022 15:25:26.398 INFO [main] com.hazelcast.internal.diagnostics.Diagnostics.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
31-May-2022 15:25:26.553 INFO [main] com.hazelcast.core.LifecycleService.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [10.44.96.29]:5701 is STARTING
31-May-2022 15:25:26.608 INFO [hz.cool_moser.cached.thread-1] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5702 is added to the blacklist.
31-May-2022 15:25:26.608 INFO [hz.cool_moser.cached.thread-3] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5703 is added to the blacklist.
31-May-2022 15:25:26.617 INFO [hz.cool_moser.IO.thread-in-0] com.hazelcast.internal.server.tcp.TcpServerConnection.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Initialized new cluster connection between /127.0.1.1:5701 and /127.0.0.1:59921
31-May-2022 15:25:26.618 INFO [hz.cool_moser.IO.thread-in-0] com.hazelcast.internal.server.tcp.TcpServerConnection.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Initialized new cluster connection between /127.0.0.1:59921 and DEV0008557.ORG.COM/127.0.1.1:5701
31-May-2022 15:25:26.694 INFO [hz.cool_moser.cached.thread-1] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5703 is added to the blacklist.
31-May-2022 15:25:26.695 INFO [hz.cool_moser.cached.thread-1] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5702 is added to the blacklist.
31-May-2022 15:25:26.894 INFO [hz.cool_moser.cached.thread-2] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5703 is added to the blacklist.
31-May-2022 15:25:26.896 INFO [hz.cool_moser.cached.thread-2] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5702 is added to the blacklist.
31-May-2022 15:25:27.194 INFO [hz.cool_moser.cached.thread-2] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5703 is added to the blacklist.
31-May-2022 15:25:27.196 INFO [hz.cool_moser.cached.thread-2] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5702 is added to the blacklist.
31-May-2022 15:25:27.595 INFO [hz.cool_moser.cached.thread-1] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5703 is added to the blacklist.
31-May-2022 15:25:27.596 INFO [hz.cool_moser.cached.thread-1] com.hazelcast.internal.cluster.impl.TcpIpJoiner.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [DEV0008557.ORG.COM]:5702 is added to the blacklist.
31-May-2022 15:29:27.375 INFO [hz.cool_moser.InvocationMonitorThread] com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Invocations:1 timeouts:1 backup-timeouts:0
31-May-2022 15:31:37.372 INFO [hz.cool_moser.InvocationMonitorThread] com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Invocations:1 timeouts:1 backup-timeouts:0
31-May-2022 15:31:56.734 SEVERE [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Could not join cluster. Shutting down now!
31-May-2022 15:31:56.735 INFO [main] com.hazelcast.core.LifecycleService.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [10.44.96.29]:5701 is SHUTTING_DOWN
31-May-2022 15:31:56.738 WARNING [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Terminating forcefully...
31-May-2022 15:31:56.739 INFO [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Shutting down connection manager...
31-May-2022 15:31:56.740 INFO [main] com.hazelcast.internal.server.tcp.TcpServerConnection.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Connection[id=1, /127.0.0.1:59921->DEV0008557.ORG.COM/127.0.1.1:5701, qualifier=null, endpoint=[10.44.96.29]:5701, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: TcpServer is stopping
31-May-2022 15:31:56.741 INFO [hz.cool_moser.IO.thread-in-1] com.hazelcast.internal.server.tcp.TcpServerConnection.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Connection[id=2, /127.0.1.1:5701->/127.0.0.1:59921, qualifier=null, endpoint=[10.44.96.29]:5701, alive=false, connectionType=MEMBER, planeIndex=0] closed. Reason: Connection closed by the other side
31-May-2022 15:31:56.743 INFO [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Shutting down node engine...
31-May-2022 15:31:56.772 INFO [main] com.hazelcast.instance.impl.NodeExtension.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Destroying node NodeExtension.
31-May-2022 15:31:56.773 INFO [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Hazelcast Shutdown is completed in 35 ms.
31-May-2022 15:31:56.773 INFO [main] com.hazelcast.core.LifecycleService.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] [10.44.96.29]:5701 is SHUTDOWN
31-May-2022 15:31:56.789 WARNING [main] com.hazelcast.internal.util.phonehome.PhoneHome.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Could not schedule phone home task! Most probably Hazelcast failed to start.
31-May-2022 15:31:56.789 INFO [main] com.hazelcast.instance.impl.Node.null [10.44.96.29]:5701 [782c00c1-0c62-421c-973f-409370d15128] [4.2] Node is already shutting down... Waiting for shutdown process to complete...

--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/31b277b0-a6fd-4970-b1da-70c261977914n%40googlegroups.com.

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast