Members dropping from the Views on Kubernetes
62 views
Skip to first unread message
Chintan Mohan Rohila
Feb 6, 2024, 9:39:56 PM2/6/24
to jgroups-dev
Hello there,
We are running JGroups 4.2.10 with KUBE_PING 1.0.15 with following protocol stack.
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP_NIO2 bind_port="8800" diagnostics_port="10599"
port_range="950" recv_buf_size="${tcp.recv_buf_size:230k}"
send_buf_size="${tcp.send_buf_size:230k}" max_bundle_size="64K"
sock_conn_timeout="300" thread_pool.min_threads="0"
thread_pool.max_threads="980" thread_pool.keep_alive_time="70000" />
<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${KUBERNETES_NAMESPACE}" labels="${KUBERNETES_LABELS}" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="80000" interval="18000" timeout_check_interval="18000" />
<VERIFY_SUSPECT />
<BARRIER />
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="8000"
leave_timeout="18000" view_ack_collection_timeout="18000" />
<UFC />
<MFC />
<FRAG2 />
<pbcast.STATE_TRANSFER />
</config>
xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP_NIO2 bind_port="8800" diagnostics_port="10599"
port_range="950" recv_buf_size="${tcp.recv_buf_size:230k}"
send_buf_size="${tcp.send_buf_size:230k}" max_bundle_size="64K"
sock_conn_timeout="300" thread_pool.min_threads="0"
thread_pool.max_threads="980" thread_pool.keep_alive_time="70000" />
<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${KUBERNETES_NAMESPACE}" labels="${KUBERNETES_LABELS}" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="80000" interval="18000" timeout_check_interval="18000" />
<VERIFY_SUSPECT />
<BARRIER />
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="8000"
leave_timeout="18000" view_ack_collection_timeout="18000" />
<UFC />
<MFC />
<FRAG2 />
<pbcast.STATE_TRANSFER />
</config>
We are running 19 JChannels on each pod, and on scaling our app to18 pods, there is a drop of members in views (19x18=342) of random channels on random pods across the cluster.
Due to this, we are seeing intermittent failures in traffic.
On probing with following command all the pods every 5 seconds to check the view count on all the channels, we saw that view count on random channels on random pods are dropping from 342.
java -cp /opt/vs/bundles/jgroups-4.2.10.Final.jar org.jgroups.tests.Probe -port 10599 -bind_addr <pod dns>
Has anyone observed similar issue or suggest any changes on the protocol stack config to resolve this issue?
Thanks and regards.
Chintan Rohila
Bela Ban
Feb 7, 2024, 2:12:27 AM2/7/24
to jgrou...@googlegroups.com
A few thoughts:
* A port_range of 980 is generating a LOT of traffic: a new member
connects to every pod (18) on 980 ports = 17640 connection attempts!
port_range should be 19 *max*!
* KUBE_PING should also have a port_range of 19
* FD_SOCK establishes a destination cache of pingable members. This is
costly and requires communication. FD_SOCK2 would be better, but that's
not available in 4.x. I suggest experiment with FD_SOCK being commented
* Before doing this, find out who's suspecting members: 'probe
jmx=FD_SOCK.num_' and `probe jmx=FD_ALL.num_` and look at num_suspect_events
* Check if KUBE_PING correctly reads all members from Kubernetes: 'probe
op=KUBE_PING.fetchFromKube[]' when all members are running: this should
print a list of all members
* Have you sized your resources sufficiently? 19 channels on a pod
requires memory, CPU and resources (threads. sockets). You want to avoid
that Kubernetes kills a pod because that pod is using more resources
than allotted.
Hope this helps as a start!
On 07.02.2024 03:39, Chintan Mohan Rohila wrote:
> Hello there,
>
> We are running JGroups 4.2.10 with KUBE_PING 1.0.15 with following
> protocol stack.
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> <http://www.w3.org/2001/XMLSchema-instance%22>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/41a8cb96-5727-481b-ad4c-f0dc07b594afn%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/41a8cb96-5727-481b-ad4c-f0dc07b594afn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Bela Ban | http://www.jgroups.org
* A port_range of 980 is generating a LOT of traffic: a new member
connects to every pod (18) on 980 ports = 17640 connection attempts!
port_range should be 19 *max*!
* KUBE_PING should also have a port_range of 19
* FD_SOCK establishes a destination cache of pingable members. This is
costly and requires communication. FD_SOCK2 would be better, but that's
not available in 4.x. I suggest experiment with FD_SOCK being commented
* Before doing this, find out who's suspecting members: 'probe
jmx=FD_SOCK.num_' and `probe jmx=FD_ALL.num_` and look at num_suspect_events
* Check if KUBE_PING correctly reads all members from Kubernetes: 'probe
op=KUBE_PING.fetchFromKube[]' when all members are running: this should
print a list of all members
* Have you sized your resources sufficiently? 19 channels on a pod
requires memory, CPU and resources (threads. sockets). You want to avoid
that Kubernetes kills a pod because that pod is using more resources
than allotted.
Hope this helps as a start!
On 07.02.2024 03:39, Chintan Mohan Rohila wrote:
> Hello there,
>
> We are running JGroups 4.2.10 with KUBE_PING 1.0.15 with following
> protocol stack.
> <config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups
> http://www.jgroups.org/schema/jgroups.xsd">
> <http://www.jgroups.org/schema/jgroups.xsd%22>>
> http://www.jgroups.org/schema/jgroups.xsd">
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/41a8cb96-5727-481b-ad4c-f0dc07b594afn%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/41a8cb96-5727-481b-ad4c-f0dc07b594afn%40googlegroups.com?utm_medium=email&utm_source=footer>.
--
Bela Ban | http://www.jgroups.org
Chintan Mohan Rohila
Feb 12, 2024, 6:08:18 AM2/12/24
to jgroups-dev
Hi Bela Ban,
Thanks for your inputs.
Limiting the port ranges did help to stabilize the views. On a 5 second Probe interval, we didn't see Members drop from any of the view across all the channels.
But despite of that we did see traffic failure with the huge following thread dump (more than 30K lines). Please suggest what else can be done to avoid this?
"message" : "[org.jgroups.protocols.TCP_NIO2]eric-bss-ec-vs50-evarmee-vs-17-25152: thread pool is full (max=10, active=10); thread dump (dumped once, until thread_dump is reset):
jgroups-int-8,TRAFFIC,eric-bss-ec-vs50-evarmee-vs-17-25152:
at sun.misc.Unsafe.park(Unsafe.java:-2)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
jgroups-int-7,TRAFFIC,eric-bss-ec-vs50-evarmee-vs-17-25152:
at sun.misc.Unsafe.park(Unsafe.java:-2)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
NioConnection.Reader [192.168.87.47:9805]-1835,TRAFFIC,eric-bss-ec-vs50-evarmee-vs-17-25152:
at sun.misc.Unsafe.park(Unsafe.java:-2)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.jgroups.util.CondVar.waitFor(CondVar.java:63)
at org.jgroups.blocks.cs.NioConnection$Reader._run(NioConnection.java:490)
at org.jgroups.blocks.cs.NioConnection$Reader.run(NioConnection.java:463)
at java.lang.Thread.run(Thread.java:750)
Bela Ban
Feb 12, 2024, 6:13:32 AM2/12/24
to jgrou...@googlegroups.com
Your thread pool is only 10 (?) and full. The max size should be <number
of cluster members> + delta, where delta is 2-4 for timer threads. This
holds if you only use *regular* message; for *OOB* messages you'll need
more threads.
> https://groups.google.com/d/msgid/jgroups-dev/b427ffde-94be-4f11-8d65-7787f0d9da01n%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/b427ffde-94be-4f11-8d65-7787f0d9da01n%40googlegroups.com?utm_medium=email&utm_source=footer>.
of cluster members> + delta, where delta is 2-4 for timer threads. This
holds if you only use *regular* message; for *OOB* messages you'll need
more threads.
> <https://groups.google.com/d/msgid/jgroups-dev/b427ffde-94be-4f11-8d65-7787f0d9da01n%40googlegroups.com?utm_medium=email&utm_source=footer>.
Chintan Mohan Rohila
Jul 12, 2024, 2:54:29 AM7/12/24
to jgroups-dev
The issue still persist. In a kubernetes cluster channels from one pod are unable to join the cluster when the coordinator channel is running on another pod due to failure in decryption of the joining message.
send_buf_size="${tcp.send_buf_size:230k}" max_bundle_size="64K"
sock_conn_timeout="300" thread_pool.min_threads="0"
<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${KUBERNETES_NAMESPACE}" labels="${KUBERNETES_LABELS}" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="80000" interval="18000" timeout_check_interval="18000" />
<VERIFY_SUSPECT />
<BARRIER />
/>
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="8000"
leave_timeout="18000" view_ack_collection_timeout="18000" />
<UFC />
<MFC />
<FRAG2 />
<pbcast.STATE_TRANSFER />
</config>
It seems like a bug in v4.2.x; can someone confirm and help with the way forward? We don't have an option to move to v5.x as we are running java 8 and can't move to java 11 due to other dependencies.
The protocol stack, the application used to reproduce this issue are as follows:
Protocol stack:
xmlns="urn:org:jgroups" xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP_NIO2 bind_port="9800" diagnostics_port="20599"
port_range="50" recv_buf_size="${tcp.recv_buf_size:230k}"
port_range="50" recv_buf_size="${tcp.recv_buf_size:230k}"
send_buf_size="${tcp.send_buf_size:230k}" max_bundle_size="64K"
sock_conn_timeout="300" thread_pool.min_threads="0"
thread_pool.max_threads="100" thread_pool.keep_alive_time="70000" />
<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${KUBERNETES_NAMESPACE}" labels="${KUBERNETES_LABELS}" />
<MERGE3 min_interval="10000" max_interval="30000" />
<FD_SOCK />
<FD_ALL timeout="80000" interval="18000" timeout_check_interval="18000" />
<VERIFY_SUSPECT />
<BARRIER />
<SYM_ENCRYPT
keystore_name="${jgroups.keystore_name}"
store_password="${jgroups.store_password}"
alias="${jgroups.alias}"
keystore_name="${jgroups.keystore_name}"
store_password="${jgroups.store_password}"
alias="${jgroups.alias}"
/>
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true" />
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000" max_bytes="4M" />
<pbcast.GMS print_local_addr="true" join_timeout="8000"
leave_timeout="18000" view_ack_collection_timeout="18000" />
<AUTH
auth_class="org.jgroups.auth.MD5Token"
auth_value="${jgroups.auth_value}"
token_hash="MD5"/>
auth_class="org.jgroups.auth.MD5Token"
auth_value="${jgroups.auth_value}"
token_hash="MD5"/>
<UFC />
<MFC />
<FRAG2 />
<pbcast.STATE_TRANSFER />
</config>
JGroupChat application:
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.List;
import org.jgroups.Address;
import org.jgroups.JChannel;
import org.jgroups.Message;
import org.jgroups.ReceiverAdapter;
import org.jgroups.View;
public class JGroupChat extends ReceiverAdapter {
JChannel channel;
String name, room = "default";
boolean isSecure = false;
List<Address> members;
private String protocol;
private String keystore_name;
public void viewAccepted(View new_view) {
members = new_view.getMembers();
System.out.println("\nChat room: " + room + " | Members: " + new_view.getMembers());
System.out.print(name + "> ");
}
public void receive(Message msg) {
System.out.println("\n[ Message from " + msg.getSrc() + "]: " + msg.getObject());
System.out.print(name + "> "); System.out.flush();
}
private void start() throws Exception {
System.out.println("Creating secure jgroup channel..");
if (isSecure)
channel=new JChannel(protocol);
else
channel=new JChannel(protocol);
if(name != null)
channel.name(name);
channel.setReceiver(this);
channel.setDiscardOwnMessages(true);
channel.connect(room);
eventLoop();
channel.close();
}
private void eventLoop() {
BufferedReader in=new BufferedReader(new InputStreamReader(System.in));
while(true) {
try {
System.out.print(name + "> "); System.out.flush();
String line=in.readLine();
if(line.startsWith("quit") || line.startsWith("exit")) {
break;
}
Message msg=new Message(null, line);
if (line.startsWith("@"))
{
String target = line.split(" ")[0];
target = target.replace("@", "");
boolean isValidTarget = false;
Address targetAddress = null;
for (Address address : members) {
if (address.toString().equals(target))
{
isValidTarget = true;
targetAddress = address;
}
}
if (isValidTarget)
channel.send(targetAddress, line.replaceFirst(line.split(" ")[0]+" ", ""));
else
{
System.out.println(target + " is not a valid member.");
}
}
else
channel.send(msg);
}
catch(Exception e) {
}
}
}
public static void main(String[] args) throws Exception {
JGroupChat chat = new JGroupChat();
for(int i=0; i < args.length; i++) {
if(args[i].equals("-name")) {
chat.name=args[++i];
continue;
}
if(args[i].equals("-room")) {
chat.room=args[++i];
continue;
}
if(args[i].equals("-secure")) {
chat.isSecure=new Boolean(args[++i]);
continue;
}
if(args[i].equals("-protocol")) {
chat.protocol=args[++i];
continue;
}
if(args[i].equals("-keystore_name")) {
chat.keystore_name=args[++i];
continue;
}
if(args[i].equals("-bind_addr")) {
System.setProperty("jgroups.bind_addr", args[++i]);
continue;
}
help();
return;
}
System.setProperty("jgroups.keystore_name", chat.keystore_name);
System.setProperty("jgroups.store_password", "Password123456");
System.setProperty("jgroups.alias", "mykey");
System.setProperty("jgroups.auth_value", "AuthT3amR8D");
chat.start();
}
protected static void help() {
System.out.println("Chat [-name name] [-room room] [-secure true/false] [-protocol protocol_path] [-keystore_name keystore_name] [-bind_addr bind_addr]");
}
}
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.util.List;
import org.jgroups.Address;
import org.jgroups.JChannel;
import org.jgroups.Message;
import org.jgroups.ReceiverAdapter;
import org.jgroups.View;
public class JGroupChat extends ReceiverAdapter {
JChannel channel;
String name, room = "default";
boolean isSecure = false;
List<Address> members;
private String protocol;
private String keystore_name;
public void viewAccepted(View new_view) {
members = new_view.getMembers();
System.out.println("\nChat room: " + room + " | Members: " + new_view.getMembers());
System.out.print(name + "> ");
}
public void receive(Message msg) {
System.out.println("\n[ Message from " + msg.getSrc() + "]: " + msg.getObject());
System.out.print(name + "> "); System.out.flush();
}
private void start() throws Exception {
System.out.println("Creating secure jgroup channel..");
if (isSecure)
channel=new JChannel(protocol);
else
channel=new JChannel(protocol);
if(name != null)
channel.name(name);
channel.setReceiver(this);
channel.setDiscardOwnMessages(true);
channel.connect(room);
eventLoop();
channel.close();
}
private void eventLoop() {
BufferedReader in=new BufferedReader(new InputStreamReader(System.in));
while(true) {
try {
System.out.print(name + "> "); System.out.flush();
String line=in.readLine();
if(line.startsWith("quit") || line.startsWith("exit")) {
break;
}
Message msg=new Message(null, line);
if (line.startsWith("@"))
{
String target = line.split(" ")[0];
target = target.replace("@", "");
boolean isValidTarget = false;
Address targetAddress = null;
for (Address address : members) {
if (address.toString().equals(target))
{
isValidTarget = true;
targetAddress = address;
}
}
if (isValidTarget)
channel.send(targetAddress, line.replaceFirst(line.split(" ")[0]+" ", ""));
else
{
System.out.println(target + " is not a valid member.");
}
}
else
channel.send(msg);
}
catch(Exception e) {
}
}
}
public static void main(String[] args) throws Exception {
JGroupChat chat = new JGroupChat();
for(int i=0; i < args.length; i++) {
if(args[i].equals("-name")) {
chat.name=args[++i];
continue;
}
if(args[i].equals("-room")) {
chat.room=args[++i];
continue;
}
if(args[i].equals("-secure")) {
chat.isSecure=new Boolean(args[++i]);
continue;
}
if(args[i].equals("-protocol")) {
chat.protocol=args[++i];
continue;
}
if(args[i].equals("-keystore_name")) {
chat.keystore_name=args[++i];
continue;
}
if(args[i].equals("-bind_addr")) {
System.setProperty("jgroups.bind_addr", args[++i]);
continue;
}
help();
return;
}
System.setProperty("jgroups.keystore_name", chat.keystore_name);
System.setProperty("jgroups.store_password", "Password123456");
System.setProperty("jgroups.alias", "mykey");
System.setProperty("jgroups.auth_value", "AuthT3amR8D");
chat.start();
}
protected static void help() {
System.out.println("Chat [-name name] [-room room] [-secure true/false] [-protocol protocol_path] [-keystore_name keystore_name] [-bind_addr bind_addr]");
}
}
System Info. and debug logs of coordinator channel, channel running on the coordinator channel pod and channel running on a pod other than the coordinator channel pod.
OS info.
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # uname -a
Linux eric-bss-ec-vs50-satyjgr-0 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Kuberneters cluster info.:
kubectl cluster-info dump
"kernelVersion": "5.14.21-150500.55.44-default",
"osImage": "SUSE Linux Enterprise Server 15 SP5",
"containerRuntimeVersion": "containerd://1.7.8",
"kubeletVersion": "v1.29.1",
"kubeProxyVersion": "v1.29.1",
"operatingSystem": "linux",
"architecture": "amd64"
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Java Info.
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -version
java version "1.8.0_381"
Java(TM) SE Runtime Environment (build 1.8.0_381-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.381-b09, mixed mode)
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Keystore creation command:
/opt/Java8/1.8.0.381/1/fast/jdk1.8.0.381/jre/bin/keytool -genseckey -alias mykey -keypass Password123456 -storepass Password123456 -keyalg Blowfish -keysize 56 -keystore /opt/jgroups/jgroup.keystore -storetype JCEKS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coordinator channel on pod 1:
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod11 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-0
Creating secure jgroup channel..
14:29:15.585 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:29:15.597 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:29:15.601 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:29:15.611 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:29:15.671 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod11, cluster=mycluster, physical address=192.168.184.219:9800
-------------------------------------------------------------------
14:29:16.122 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:29:19.920 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed
14:29:24.127 [main] INFO org.jgroups.protocols.pbcast.GMS - pod11: no members discovered after 8221 ms: creating cluster as coordinator
14:29:24.128 [main] DEBUG org.jgroups.protocols.pbcast.NAKACK2 -
[pod11 setDigest()]
existing digest: []
new digest: pod11: [0 (0)]
resulting digest: pod11: [0 (0)]
14:29:24.129 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|0] (1) [pod11] (pod11 joined)
14:29:24.130 [main] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resuming message garbage collection
Chat room: mycluster | Members: [pod11]
pod11> 14:29:24.140 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: created cluster (first member). My view is [pod11|0], impl is CoordGmsImpl
pod11> 14:29:32.681 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:30:38.755 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed (received 3 identical messages from cc85814a-1d12-4293-9062-3c44886307ed in the last 78835 ms)
14:30:39.505 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:30:51.437 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980 (received 4 identical messages from 01c0a045-94a7-4a97-b564-e20425db4980 in the last 78756 ms)
14:31:10.889 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: suspending message garbage collection
14:31:10.890 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resume task started, max_suspend_time=33000
14:31:10.891 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|1] (2) [pod11, pod12] (pod12 joined)
14:31:10.892 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: pingable_mbrs=[pod11, pod12], ping_dest=pod12
Chat room: mycluster | Members: [pod11, pod12]
pod11> 14:31:10.912 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resuming message garbage collection
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:45.577 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed (received 3 identical messages from cc85814a-1d12-4293-9062-3c44886307ed in the last 66822 ms)
14:31:55.546 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:56.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:56.814 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:57.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:57.814 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:58.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:58.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:59.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:59.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:00.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:00.639 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:00.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:01.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:01.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:02.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:02.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:03.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:03.630 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:04.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:04.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:05.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:05.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:06.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:06.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:07.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:07.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:08.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:08.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:09.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:09.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:10.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:10.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:11.102 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980 (received 3 identical messages from 01c0a045-94a7-4a97-b564-e20425db4980 in the last 79665 ms)
14:32:11.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:11.711 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:12.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:12.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:13.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:13.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:14.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:14.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:15.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:15.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:16.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:16.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:17.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:17.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:18.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:18.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:19.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:19.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:20.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:20.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:21.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:21.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:22.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:22.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:23.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:23.821 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:24.321 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:24.821 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:04.326 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:10.479 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.UNICAST3 - pod11: closing expired connection for pod12 (120059 ms old) in send_table
14:33:10.479 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.UNICAST3 - pod11: closing expired connection for pod12 (120059 ms old) in recv_table
14:33:13.923 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: pod12 closed socket (eof)
14:33:13.924 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: broadcasting suspect(pod12)
14:33:13.925 [jgroups-24,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: suspecting [pod12]
14:33:15.927 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.VERIFY_SUSPECT - [pod12] is dead (passing up SUSPECT event)
14:33:15.929 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|2] (1) [pod11] (pod12 left)
14:33:15.929 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.NAKACK2 - pod11: removed pod12 from xmit_table (not member anymore)
Chat room: mycluster | Members: [pod11]
pod11> 14:33:21.506 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:33:32.440 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed
c14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2nd channel on pod 1:
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod12 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-0
Creating secure jgroup channel..
14:31:10.345 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:31:10.358 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:31:10.362 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:31:10.372 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:31:10.432 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod12, cluster=mycluster, physical address=192.168.184.219:9802
-------------------------------------------------------------------
14:31:10.868 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:10.884 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod12: sending JOIN(pod12) to pod11
14:31:10.901 [main] DEBUG org.jgroups.protocols.pbcast.NAKACK2 -
[pod12 setDigest()]
existing digest: []
new digest: pod11: [0 (0)], pod12: [0 (0)]
resulting digest: pod11: [0 (0)], pod12: [0 (0)]
14:31:10.901 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod12: installing view [pod11|1] (2) [pod11, pod12] ([pod11, pod12] joined)
14:31:10.906 [FD_SOCK pinger-13,mycluster,pod12] DEBUG org.jgroups.protocols.FD_SOCK - pod12: pingable_mbrs=[pod11, pod12], ping_dest=pod11
Chat room: mycluster | Members: [pod11, pod12]
pod12> pod12> 14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:52.890 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:10.105 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:11.276 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.UNICAST3 - pod12: closing expired connection for pod11 (120400 ms old) in send_table
14:33:11.276 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.UNICAST3 - pod12: closing expired connection for pod11 (120400 ms old) in recv_table
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
New channel on pod 2:
eric-bss-ec-vs50-satyjgr-1:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod21 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-1
Creating secure jgroup channel..
14:31:54.997 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:31:55.010 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:31:55.015 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:31:55.025 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:31:55.085 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod21, cluster=mycluster, physical address=192.168.134.119:9801
-------------------------------------------------------------------
14:31:55.530 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:55.544 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:03.546 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 0
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:03.628 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:11.103 [jgroups-11,mycluster,pod21] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:32:11.629 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 1
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:11.710 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:19.710 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 2
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:19.819 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
eric-bss-ec-vs50-satyjgr-1:/opt/jgroups #
Linux eric-bss-ec-vs50-satyjgr-0 5.14.21-150500.55.44-default #1 SMP PREEMPT_DYNAMIC Mon Jan 15 10:03:40 UTC 2024 (cc7d8b6) x86_64 x86_64 x86_64 GNU/Linux
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Kuberneters cluster info.:
kubectl cluster-info dump
"kernelVersion": "5.14.21-150500.55.44-default",
"osImage": "SUSE Linux Enterprise Server 15 SP5",
"containerRuntimeVersion": "containerd://1.7.8",
"kubeletVersion": "v1.29.1",
"kubeProxyVersion": "v1.29.1",
"operatingSystem": "linux",
"architecture": "amd64"
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Java Info.
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -version
java version "1.8.0_381"
Java(TM) SE Runtime Environment (build 1.8.0_381-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.381-b09, mixed mode)
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Keystore creation command:
/opt/Java8/1.8.0.381/1/fast/jdk1.8.0.381/jre/bin/keytool -genseckey -alias mykey -keypass Password123456 -storepass Password123456 -keyalg Blowfish -keysize 56 -keystore /opt/jgroups/jgroup.keystore -storetype JCEKS
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Coordinator channel on pod 1:
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod11 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-0
Creating secure jgroup channel..
14:29:15.585 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:29:15.597 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:29:15.601 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:29:15.611 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:29:15.671 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod11, cluster=mycluster, physical address=192.168.184.219:9800
-------------------------------------------------------------------
14:29:16.122 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:16.124 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:29:19.920 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed
14:29:24.127 [main] INFO org.jgroups.protocols.pbcast.GMS - pod11: no members discovered after 8221 ms: creating cluster as coordinator
14:29:24.128 [main] DEBUG org.jgroups.protocols.pbcast.NAKACK2 -
[pod11 setDigest()]
existing digest: []
new digest: pod11: [0 (0)]
resulting digest: pod11: [0 (0)]
14:29:24.129 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|0] (1) [pod11] (pod11 joined)
14:29:24.130 [main] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resuming message garbage collection
Chat room: mycluster | Members: [pod11]
pod11> 14:29:24.140 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: created cluster (first member). My view is [pod11|0], impl is CoordGmsImpl
pod11> 14:29:32.681 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:29:55.133 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:30:38.755 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed (received 3 identical messages from cc85814a-1d12-4293-9062-3c44886307ed in the last 78835 ms)
14:30:39.505 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:30:39.506 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:30:51.437 [jgroups-11,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980 (received 4 identical messages from 01c0a045-94a7-4a97-b564-e20425db4980 in the last 78756 ms)
14:31:10.889 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: suspending message garbage collection
14:31:10.890 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resume task started, max_suspend_time=33000
14:31:10.891 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|1] (2) [pod11, pod12] (pod12 joined)
14:31:10.892 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: pingable_mbrs=[pod11, pod12], ping_dest=pod12
Chat room: mycluster | Members: [pod11, pod12]
pod11> 14:31:10.912 [jgroups-11,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.STABLE - pod11: resuming message garbage collection
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:44.342 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:45.577 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed (received 3 identical messages from cc85814a-1d12-4293-9062-3c44886307ed in the last 66822 ms)
14:31:55.546 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:56.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:56.814 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:57.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:57.814 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:58.314 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:58.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:59.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:31:59.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:00.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:00.639 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:00.640 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:00.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:01.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:01.815 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:02.315 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:02.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:03.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:03.630 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:04.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:04.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:05.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:05.816 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:06.316 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:06.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:07.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:07.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:08.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:08.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:09.317 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:09.817 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:10.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:10.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:11.102 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980 (received 3 identical messages from 01c0a045-94a7-4a97-b564-e20425db4980 in the last 79665 ms)
14:32:11.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:11.711 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:12.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:12.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:13.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:13.818 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:14.318 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:14.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:15.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:15.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:16.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:16.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:17.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:17.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:18.319 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:18.819 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:19.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:19.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:20.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:20.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:21.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:21.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:22.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:22.820 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:23.320 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:23.821 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:24.321 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:24.821 [jgroups-24,mycluster,pod11] ERROR org.jgroups.protocols.SYM_ENCRYPT - pod11: rejected decryption of unicast message from non-member pod21
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:37.974 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:04.326 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:04.327 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:10.479 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.UNICAST3 - pod11: closing expired connection for pod12 (120059 ms old) in send_table
14:33:10.479 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.UNICAST3 - pod11: closing expired connection for pod12 (120059 ms old) in recv_table
14:33:13.923 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: pod12 closed socket (eof)
14:33:13.924 [FD_SOCK pinger-22,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: broadcasting suspect(pod12)
14:33:13.925 [jgroups-24,mycluster,pod11] DEBUG org.jgroups.protocols.FD_SOCK - pod11: suspecting [pod12]
14:33:15.927 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.VERIFY_SUSPECT - [pod12] is dead (passing up SUSPECT event)
14:33:15.929 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.GMS - pod11: installing view [pod11|2] (1) [pod11] (pod12 left)
14:33:15.929 [VERIFY_SUSPECT.TimerThread-45,mycluster,pod11] DEBUG org.jgroups.protocols.pbcast.NAKACK2 - pod11: removed pod12 from xmit_table (not member anymore)
Chat room: mycluster | Members: [pod11]
pod11> 14:33:21.506 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:33:32.440 [jgroups-24,mycluster,pod11] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was cc85814a-1d12-4293-9062-3c44886307ed
c14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:36.276 [Timer runner-3,mycluster,pod11] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2nd channel on pod 1:
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod12 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-0
Creating secure jgroup channel..
14:31:10.345 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:31:10.358 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:31:10.362 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:31:10.372 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:31:10.432 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod12, cluster=mycluster, physical address=192.168.184.219:9802
-------------------------------------------------------------------
14:31:10.868 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:10.870 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:10.884 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod12: sending JOIN(pod12) to pod11
14:31:10.901 [main] DEBUG org.jgroups.protocols.pbcast.NAKACK2 -
[pod12 setDigest()]
existing digest: []
new digest: pod11: [0 (0)], pod12: [0 (0)]
resulting digest: pod11: [0 (0)], pod12: [0 (0)]
14:31:10.901 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod12: installing view [pod11|1] (2) [pod11, pod12] ([pod11, pod12] joined)
14:31:10.906 [FD_SOCK pinger-13,mycluster,pod12] DEBUG org.jgroups.protocols.FD_SOCK - pod12: pingable_mbrs=[pod11, pod12], ping_dest=pod11
Chat room: mycluster | Members: [pod11, pod12]
pod12> pod12> 14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:32.223 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:52.890 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:52.891 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:33.770 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:10.105 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:33:10.106 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:33:11.276 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.UNICAST3 - pod12: closing expired connection for pod11 (120400 ms old) in send_table
14:33:11.276 [Timer runner-3,mycluster,pod12] DEBUG org.jgroups.protocols.UNICAST3 - pod12: closing expired connection for pod11 (120400 ms old) in recv_table
eric-bss-ec-vs50-satyjgr-0:/opt/jgroups #
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
New channel on pod 2:
eric-bss-ec-vs50-satyjgr-1:/opt/jgroups # java -cp /opt/jgroups:/opt/jgroups/jgroups-4.2.26.Final-E002.jar:/opt/jgroups/ch.qos.logback.core_1.2.6.jar:/opt/jgroups/ch.qos.logback.classic_1.2.6.jar:/opt/jgroups/log4j.over.slf4j_1.7.2.jar:/opt/jgroups/slf4j.api_1.7.36.jar:/opt/jgroups/log4j2.xml JGroupChat -name pod21 -room mycluster -secure true -protocol /opt/jgroups/secure_tcpnio2.xml -keystore_name /opt/jgroups/jgroup.keystore -bind_addr eric-bss-ec-vs50-satyjgr-1
Creating secure jgroup channel..
14:31:54.997 [main] DEBUG org.jgroups.stack.Configurator - set property TCP_NIO2.diagnostics_addr to default value /224.0.75.75
14:31:55.010 [main] DEBUG org.jgroups.protocols.TCP_NIO2 - thread pool min/max/keep-alive: 0/100/70000 use_fork_join=false, internal pool: 0/6/30000 (6 cores available)
14:31:55.015 [main] INFO org.jgroups.protocols.kubernetes.KUBE_PING - namespace ec-system set; clustering enabled
14:31:55.025 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - KubePING configuration: KubePing{namespace='ec-system', labels='app.kubernetes.io/name=eric-bss-ec-vs50-satyjgr'}
14:31:55.085 [main] DEBUG org.jgroups.protocols.SYM_ENCRYPT - null: key_password used is same as store_password
-------------------------------------------------------------------
GMS: address=pod21, cluster=mycluster, physical address=192.168.134.119:9801
-------------------------------------------------------------------
14:31:55.530 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:31:55.532 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:31:55.544 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:03.546 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 0
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:03.628 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:03.628 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:11.103 [jgroups-11,mycluster,pod21] WARN org.jgroups.protocols.TCP_NIO2 - JGRP000012: discarded message from different cluster NG-protocol (our cluster is mycluster). Sender was 01c0a045-94a7-4a97-b564-e20425db4980
14:32:11.629 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 1
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:11.709 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:11.710 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
14:32:19.710 [main] WARN org.jgroups.protocols.pbcast.GMS - pod21: JOIN(pod21) sent to pod11 timed out (after 8000 ms), on try 2
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-0, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-1, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-2, group eric-bss-ec-vs50-satyjgr-85c5d7f5fd
14:32:19.818 [main] DEBUG org.jgroups.protocols.kubernetes.KUBE_PING - pod eric-bss-ec-vs50-satyjgr-cronjob-28677661-f77wg, group null
14:32:19.819 [main] DEBUG org.jgroups.protocols.pbcast.GMS - pod21: sending JOIN(pod21) to pod11
eric-bss-ec-vs50-satyjgr-1:/opt/jgroups #
Reply all
Reply to author
Forward
0 new messages