Hazelcast Java Client and number threads in Pool

349 views
Skip to first unread message

Rakesh Sharma

unread,
Nov 3, 2021, 1:09:16 AM11/3/21
to Hazelcast
There is no information on Hazelcast Java Client and Thread management on client side java code. How many number of threads  in Pool client side gets created. 
I do see 10 threads trying to connect to 3 node Hazelcast cluster if Hazelcast cluster is down and we have following reconnect settings for Hazelcast client:
 <connection-strategy async-start="false" reconnect-mode="ON">
           <connection-retry>
               <initial-backoff-millis>1000</initial-backoff-millis>
               <max-backoff-millis>30000</max-backoff-millis>
               <multiplier>2</multiplier>
               <cluster-connect-timeout-millis>60000</cluster-connect-timeout-millis>
               <jitter>0.0</jitter>
           </connection-retry>

Here is snippet from log which has 10 different threads trying to connect to cluster:
2021-11-02 11:32:52,231 c.h.c.i.c.ClientConnectionManager [hz.internalnotification.IO.thread-in-0] INFO  - internalnotification [EipDataGrid] [4.2.2] Removed connection to endpoint: [md2sn2nc.ad001.siemens.net]:6665:60980cc0-fafe-4a26-b897-016ad8558278, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/192.168.141.248:60478->md2sn2nc.ad001.siemens.net/192.168.141.248:6665}, remoteAddress=[md2sn2nc.ad001.siemens.net]:6665, lastReadTime=2021-11-02 11:32:52.223, lastWriteTime=2021-11-02 11:32:52.187, closedTime=2021-11-02 11:32:52.226, connected server version=4.2.2}
2021-11-02 11:32:52,232 c.h.c.LifecycleService [hz.internalnotification.IO.thread-in-0] INFO  - internalnotification [EipDataGrid] [4.2.2] HazelcastClient 4.2.2 (20210811 - c38011e) is CLIENT_DISCONNECTED
2021-11-02 11:32:52,232 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to cluster: EipDataGrid
2021-11-02 11:32:52,233 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:32:54,237 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:32:54,237 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:32:55,085 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:32:56,240 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:32:56,240 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:32:58,078 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:32:58,242 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:32:58,242 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 1000 ms, attempt: 1, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:32:59,243 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:33:01,080 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:01,247 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:01,247 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:33:03,250 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:33:03,250 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:33:04,086 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:05,252 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:33:05,252 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 2000 ms, attempt: 2, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:33:07,088 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:07,252 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:33:09,255 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:09,255 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:33:10,093 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:11,258 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:33:11,258 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:33:13,098 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:13,263 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:33:13,263 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 4000 ms, attempt: 3, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:33:16,110 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:17,265 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:33:19,105 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:19,268 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:19,269 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:33:21,269 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:33:21,269 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:33:22,108 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:23,275 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:33:23,275 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 8000 ms, attempt: 4, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:33:25,109 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:28,113 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:31,114 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:31,278 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:33:33,117 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:33,282 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:33,282 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:33:35,285 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:33:35,285 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:33:36,121 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:37,288 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:33:37,288 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 16000 ms, attempt: 5, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:33:39,124 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:42,130 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:45,133 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:48,135 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:51,139 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:53,292 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:33:54,153 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:55,297 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:55,297 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:33:57,148 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:33:57,303 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:33:57,303 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:33:59,305 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:33:59,306 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 30000 ms, attempt: 6, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:34:00,148 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:03,152 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:06,156 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:09,162 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:12,161 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:15,164 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:18,167 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:21,171 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:24,175 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:27,177 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:29,181 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:29,312 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:34:31,315 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:31,315 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:34:32,187 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:33,330 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:34:33,330 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:34:35,189 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:35,334 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:34:35,334 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 30000 ms, attempt: 7, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:34:38,193 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:41,196 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:44,201 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:47,200 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:49,215 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:52,207 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:54,212 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:34:57,217 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:00,220 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:02,223 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:05,229 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:05,340 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:35:07,344 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:07,345 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:35:08,233 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:09,348 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:35:09,348 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:35:11,234 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:11,351 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:35:11,351 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 30000 ms, attempt: 8, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:35:13,237 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:16,240 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:19,242 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:22,245 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:25,248 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:28,253 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:31,256 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:34,258 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:37,262 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:40,264 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:41,357 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:35:43,266 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:43,360 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:43,361 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:35:45,270 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:45,363 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:35:45,363 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:35:47,369 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:35:47,369 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 30000 ms, attempt: 9, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:35:48,285 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:51,279 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:54,281 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:35:57,284 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:00,288 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:03,298 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:06,294 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:09,296 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:12,299 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:15,302 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:17,375 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278
2021-11-02 11:36:18,306 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:19,378 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to Member [md2sn2nc.ad001.siemens.net]:6665 - 60980cc0-fafe-4a26-b897-016ad8558278: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:19,378 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6673
2021-11-02 11:36:21,310 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:21,383 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6673: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6673
2021-11-02 11:36:21,383 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] INFO  - internalnotification [EipDataGrid] [4.2.2] Trying to connect to [md2sn2nc.ad001.siemens.net]:6669
2021-11-02 11:36:23,386 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Exception during initial connection to [md2sn2nc.ad001.siemens.net]:6669: com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6669
2021-11-02 11:36:23,386 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-1] WARN  - internalnotification [EipDataGrid] [4.2.2] Unable to get live cluster connection, retry in 30000 ms, attempt: 10, cluster connect timeout: INFINITE, max backoff: 30000 ms
2021-11-02 11:36:24,315 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:27,318 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-10] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:30,320 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-8] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:33,324 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-5] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:36,326 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-2] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:39,338 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-7] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:42,332 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-4] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:45,335 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-3] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:48,350 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-9] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665
2021-11-02 11:36:51,342 c.h.c.i.c.ClientConnectionManager [internalnotification.internal-6] WARN  - internalnotification [EipDataGrid] [4.2.2] Could not connect to member 60980cc0-fafe-4a26-b897-016ad8558278, reason com.hazelcast.core.HazelcastException: java.io.IOException: Connection refused: no further information to address md2sn2nc.ad001.siemens.net/192.168.141.248:6665

Thanks,
Rakesh

Joe Sherwin

unread,
Nov 3, 2021, 1:40:41 AM11/3/21
to haze...@googlegroups.com
Rakesh, extensive documentation of the Java client can be found here...

To discover the defaults for different thread pools use by the client, jump to client properties here...
https://docs.hazelcast.com/imdg/4.2/clients/java#client-system-properties

If you desire even more granular implementation details, Hazelcast is open source you find the code on GitHub. This is also where you can contribute and report issues...
-zap


On Nov 3, 2021, at 1:09 AM, Rakesh Sharma <rsharm...@gmail.com> wrote:

There is no information on Hazelcast Java Client and Thread management on client side java code. How many number of threads  in Pool client side gets created. 
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2e468de7-190d-4396-a0b8-22813af0f4een%40googlegroups.com.

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

Rakesh Sharma

unread,
Nov 3, 2021, 12:05:33 PM11/3/21
to Hazelcast
Hi Zap,
            This is the only thread related information on Java Client docs ..That's also just mention about server side instead of client side

Hundreds or even thousands of clients can be connected to the cluster. By default, there are core count * 20 threads on the server side that handle all the requests, e.g., if the server has 4 cores, there will be 80 threads.
That's also just said about server side instead of client side.

Where as Properties is concerned I do see thread related properties: But there is no guideline on when to change it. . Since we use smart client I have mentioned default for it.
hazelcast.client.event.thread.count - 5 (default)
hazelcast.client.io.input.thread.count - 3 (smart client)
hazelcast.client.io.output.thread.count - 3 (smart client)

Will these setting is good enough for our multithreaded App using the same hazelcast client instance by different threads to make calls to Hazelcast cluster to data. What I am trying to understand how many concurrent call Single Java Hazlcast client with default Threads can handle .

Thanks,
Rakesh

M. Sancar Koyunlu

unread,
Nov 5, 2021, 3:59:09 AM11/5/21
to haze...@googlegroups.com
Hi Rakesh, 

Hazelcast Client tries to perform best by checking the processor count of the machine. There is no limit that the client can handle but for best performance you may want to stay close to the number of available processors for thread count that calls the hazelcast client.
There are some internal thread counts that you may tweak but increasing them does not always mean better performance. 
You can check the ClientProperty class. The javadocs should help you to understand what they do. I am listing the most relevant ones below. 
 Do your own test always, as this depends on the machine that you are using and your use case.

I hope this helps. 
/**
* Controls the number of IO input threads. Defaults to -1, so the system will decide.
* <p>
* If client is a smart client and processor count larger than 8, it will default to 3 otherwise it will default to 1.
*/
public static final HazelcastProperty IO_INPUT_THREAD_COUNT
= new HazelcastProperty("hazelcast.client.io.input.thread.count", -1);

/**
* Controls the number of IO output threads. Defaults to -1, so the system will decide.
* <p>
* If client is a smart client and processor count larger than 8 , it will default to 3 otherwise it will default to 1.
*/
public static final HazelcastProperty IO_OUTPUT_THREAD_COUNT
= new HazelcastProperty("hazelcast.client.io.output.thread.count", -1);
/**
* The number of response threads.
* <p>
* By default there are 2 response threads; this gives stable and good performance.
* <p>
* If set to 0, the response threads are bypassed and the response handling is done
* on the IO threads. Under certain conditions this can give a higher throughput, but
* setting to 0 should be regarded an experimental feature.
* <p>
* If set to 0, the IO_OUTPUT_THREAD_COUNT is really going to matter because the
* inbound thread will have more work to do. By default when TLS isn't enable,
* there is just 1 inbound thread.
*/
public static final HazelcastProperty RESPONSE_THREAD_COUNT
= new HazelcastProperty("hazelcast.client.response.thread.count", 2);
/**
* Enabled dynamic switching between processing responses on the io threads
* and offloading the response threads.
* <p>
* Under certain conditions (single threaded clients) processing on the io
* thread can increase performance because useless handover to the response
* thread is removed. Also the response thread isn't created until it is needed
* and especially for ephemeral clients reducing threads can lead to
* increased performance and reduced memory usage.
*/
public static final HazelcastProperty RESPONSE_THREAD_DYNAMIC
= new HazelcastProperty("hazelcast.client.response.thread.dynamic", true);



--

Sancar Koyunlu

Software Engineer

san...@hazelcast.com

Hazelcast

2 W 5th Ave, Ste 300 | San Mateo, CA 94402 | USA

+1 (650) 521-5453 | hazelcast.com

Rakesh Sharma

unread,
Nov 5, 2021, 12:29:38 PM11/5/21
to Hazelcast
Hi Sancar,
                  I have 4 cores (8 processors) in my laptop and when I run following simple code to see number of threads added by Hazelcast client and server. Every time I add a new Hazelcast client I see 24 new threads added to current count: I like to know exact formula to determine or document on how many threads will be created on client and server side. It will wonderful to know exactly with different category of threads with their corresponding count. I need this information so that I can justify why there are so many new threads in our Enterprise Application.

I hope I am clear what and why I need this information. I have attached simple code I am using to get Server side threads and client side threads added.

Thanks,
Rakesh
HazelcastLotsOfThreads.java
HazelcastLotsOfThreadsServer.java

M. Sancar Koyunlu

unread,
Nov 8, 2021, 8:09:46 AM11/8/21
to haze...@googlegroups.com
Ok, fair enough. Here are the details:
Note that if not specifically mentioned, they are not configurable. 

There are `internal` threads with a `ScheduledThreadPoolExecutor`. Its core pool size is 13(3+10).  Since the maximum pool size of ScheduledThreadPoolExecutor is MAX_INT. They are created on demand and they are configured to be removed when idle.
3 threads are mainly used for internal  tasks like retry, heartbeat pings, cleanup tasks etc. 10 threads are for opening connections/authentication etc.

There is a `eventRegistration` thread to make event registration logic run on a single thread for simplicity.

There are `MetricsRegistry` threads with a `ScheduledThreadPoolExecutor`. Its core pool size is 2. Since the maximum pool size of ScheduledThreadPoolExecutor is MAX_INT. They are created on demand and they are configured to be removed when idle.
These are for gathering statistics periodically.

There are `IO.thread-out-#` threads
If the client is a smart client and processor count larger than 8, it will default to 3 otherwise it will default to 1.
Configurable via "hazelcast.client.io.input.thread.count"

There are `IO.thread-in-#` threads.
If the client is a smart client and processor count larger than 8, it will default to 3 otherwise it will default to 1.
Configurable via "hazelcast.client.io.output.thread.count"

There is a single `lifecycle` thread. Its job is to make sure lifecycle events are sent in the same order.

There are `event` threads. Their max count is 5 by default. They start lazily.  Configurable via "hazelcast.client.event.thread.count"

There are `responseThreads`. Their max size is 2 by default.
Configurable via "hazelcast.client.response.thread.count" and "hazelcast.client.response.thread.dynamic"

Reply all
Reply to author
Forward
0 new messages