stateMachine.set(...) is hanging

Martin Tauber

unread,

May 6, 2025, 3:09:19 PMMay 6

to jgroups-raft

The version is: 1.0.14.Final

Hi José,

I am posting this again since my answers keep being deleted :(

The raft configuration is:

RAFT raft = new RAFT() // Raft protocol
.members(membersList)
.setValue("raft_id", nodeId)
.setValue("log_dir", uvuyoHome + File.separator + "data")
.setValue("log_prefix", "memdb." + nodeId)
.setValue("max_log_size", maxLogSize)
// .setValue("send_commits_immediately", true)
.setValue("log_class", "org.jgroups.protocols.raft.FileBasedLog");

The node is NOT the leader.

After adding the commit configuration the leader sumps the state permanently and the other two nodes do nothing ...

José Bolina

unread,

May 6, 2025, 4:21:34 PMMay 6

to Martin Tauber, jgroups-raft

Thanks for giving more details!

Hrmmm, I don't have a solution ready, so I will need your help to investigate a little more.

Since the sender is not the leader, it might be something getting in the way of completing the message redirection in `<raft.REDIRECT />`. So a few more questions:

* Can you give more details about the protocols you have in your stack? I am interested in which position the REDIRECT protocol is.

* Are you able to confirm the command is applied in the leader's state machine? Can you also confirm this method does not hang and returns? Does the operation complete successfully if you send it directly from the leader without redirection?

* Can you enable trace level for REDIRECT and share the logs when it hangs? Something like `.level("trace")` should do it. Ideally, we want the log from the sender that hangs and the leader. This should show us if the redirection is correctly happening.

Thanks for the help!

Cheers,

--
You received this message because you are subscribed to the Google Groups "jgroups-raft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-raft...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/46197a42-4f6d-4f59-84f5-ae7bf8a4b550n%40googlegroups.com.

--

José Bolina

Bela Ban

unread,

May 7, 2025, 3:03:36 AMMay 7

to jgroup...@googlegroups.com

Note that it is better to not use setValue(), but rather use the stronly typed equivalents, e.g. `raftId(id)` etc. This has nothing to do with your issue, though...

Do you have `REDIRECT` above `RAFT`? Can you post your entire programmatic config?

If you replace programmatic configuration with simply using `raft.xml`, does this work?

--
You received this message because you are subscribed to the Google Groups "jgroups-raft" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-raft...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/46197a42-4f6d-4f59-84f5-ae7bf8a4b550n%40googlegroups.com.

-- 
Bela Ban | http://www.jgroups.org

Bela Ban

unread,

May 7, 2025, 3:05:08 AMMay 7

to jgroup...@googlegroups.com

Also note that you could use `bin/probe.sh RAFT` to dump information from all members

To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/CADBykT7U5wEGQvLs0wrcpiHFkHa7NuJhCKBExapT4cMAQ2b7mw%40mail.gmail.com.

Martin Tauber

unread,

May 7, 2025, 4:19:08 AMMay 7

to jgroups-raft

Hi everybody,

I think I got it working last night ... the problem seamed to be in my messy code :( I am using react in my application and the call to the state machine function was wrapped in a Mono<Object>. The idea is to have a function create the Mono and then subscribe to the Mono and receive the result of the state function (the setAsync(...) call).

Here is the code that did not work properly:

Mono<Object> execute(String providerId, String name, Object... args) {

.....

CompletableFuture<byte[]> result = raftHandle.setAsync(bytes, 0, bytes.length);

return Mono.fromFuture(result).flatMap(a -> {
try {
String resultString = new String(a);
StateMachineResultMessage resultMessage = objectMapper.readValue(resultString, StateMachineResultMessage.class);
if (resultMessage.getResult() == null)
return Mono.empty();

return Mono.just(resultMessage.getResult());
} catch (JsonProcessingException e) {
return Mono.error(e);
} catch (Exception e) {
return Mono.error(e);
}

}

The issue with this code is that setAsync(...) obviously executes when the Mono is created and not when it is subscribed to ... That is the wrong time but, I still wonder why it caused the hick ups ... still testing to see if it is really gone ...

Here is the code that seams to work ...

public Mono<Object> execute(String providerId, String name, Object... args) {
return Mono.create(sink -> {
....

CompletableFuture<byte[]> result = null;
try {
result = raftHandle.setAsync(bytes, 0, bytes.length);
} catch (Exception e) {
sink.error(new RuntimeException("Error executing state provider method '" + name + "'", e));
return;
}

result.whenComplete((b, error) -> {
if (error != null) {
log.error("Error executing state provider method '" + name + "'", error);
sink.error(error);
} else {
try {
String resultString = new String(b);
StateMachineResultMessage resultMessage = objectMapper.readValue(resultString, StateMachineResultMessage.class);
if (resultMessage.getResult() == null) {
sink.error(new RuntimeException("State provider method '" + name + "' returned null"));
return;
}

sink.success(resultMessage.getResult());
} catch (JsonProcessingException e) {
sink.error(new RuntimeException("Error deserializing state provider result", e));
}
}
});
});
}

José Bolina

unread,

May 7, 2025, 9:37:31 AMMay 7

to Martin Tauber, jgroups-raft

Hey, Martin, thanks for letting us know.

By submitting the operation outside of Mono, maybe the threads are blocking each other(?). I guess a thread dump could shed some light if you need to investigate that, but glad it is working now.

Feel free to reach out if you have any questions.

Cheers,

To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/b49a6b31-b1b1-4871-9cb9-03c01362be3dn%40googlegroups.com.

--

José Bolina

Martin Tauber

unread,

May 8, 2025, 4:33:38 AMMay 8

to jgroups-raft

Ah to bad I'm running into the same issue again .... :( but a :) because I want to get to the root cause of this ...

here is the protocol stack

private Protocol[] getProtocolStack() {
log.info("Using cluster protocol '{}'.", this.protocol);

List<String> membersList;
if ("".equals(members) || members == null) {
membersList = List.of(nodeId);
} else {
membersList = List.of(members.split(","));
}

log.info("Using cluster members '{}'.", membersList);

TP tp;
Protocol ping;

if (protocol.equals("tcp")) {
log.info("Using port '{}'.", this.port);
log.info("Using port range '{}'.", portRange);
log.info("Using bootstrap server '{}'.", bootstrapServers);

List<IpAddress> initialHosts = getInitialHosts(bootstrapServers);

for (IpAddress ipAddress : initialHosts) {
log.info("Contacting node using address '{}'.", ipAddress.printIpAddress());
}

tp = new TCP();
tp.setBindPort(port);

ping = new TCPPING()
.setValue("initial_hosts", initialHosts)
.setValue("port_range", portRange);
} else {
tp = new UDP();
ping = new PING();
}

if (!"".equals(externalAddress)) {
log.info("Using external address '{}'.", externalAddress);

try {
tp.setExternalAddr(InetAddress.getByName(externalAddress));
} catch (UnknownHostException e) {
log.error("External address '{}' was not found.", externalAddress);

}
}

RAFT raft = new RAFT() // Raft protocol
.members(membersList)
.setValue("raft_id", nodeId)
.setValue("log_dir", uvuyoHome + File.separator + "data")
.setValue("log_prefix", "memdb." + nodeId)
.setValue("max_log_size", maxLogSize)
// .setValue("send_commits_immediately", true)
.setValue("log_class", "org.jgroups.protocols.raft.FileBasedLog");

return new Protocol[]{
tp,
ping, // Discovery protocol
new MERGE3(), // Handles cluster splits
new FD_SOCK(), // Failure detection based on nodes connecting to neighbors
new FD_ALL() // Failure detection based on heartbeats
.setValue("timeout_check_interval", failureDetectionTimeout),
new VERIFY_SUSPECT(), // After detecting Failure check if node is really gone
new NAKACK2(), // Reliable FIFO message transfer
new UNICAST3(), // Reliable FIFO message transfer
new STABLE(), // Garbage Collect message
new GMS(), // Handle Group Membership
new UFC(), // Unicast Flow Control
new MFC(), // Multicast Flow Control
new FRAG2(), // Fragment large messages
new BARRIER(), // Needed by state transfer
new STATE_TRANSFER(), // Allow states to be transferred between nodes
new ELECTION(),
raft,
new REDIRECT(),
new CLIENT()
};
}

Martin Tauber

unread,

May 8, 2025, 8:21:07 AMMay 8

to jgroups-raft

Hi everybody,

adding some more information. In the code I am doing the setAsync, which returns. but the completable future never completes. I have added some debuggin code do verify this.

CompletableFuture<byte[]> result = null;
try {

log.trace("Invoking setAsync on raft handle ...");

result = raftHandle.setAsync(bytes, 0, bytes.length);

log.trace("setAsync on raft handle returned.");

log.trace("Fork join size" + ForkJoinPool.commonPool().getPoolSize());
log.trace("Fork join count" + ForkJoinPool.commonPool().getActiveThreadCount());

Map<Thread, StackTraceElement[]> threadMap = Thread.getAllStackTraces();
for (Thread thread : threadMap.keySet()) {
log.trace("Thread: " + thread.getName() + " (State: " + thread.getState() + ")");
}

result.whenComplete((b, error) -> {
log.trace("State provider method '" + name + "' completed.");
log.trace("Running on thread: " + Thread.currentThread().getName() + " (State: " + Thread.currentThread().getState() + ")");

if (error != null) {
log.error("Error executing state provider method '" + name + "'", error);
sink.error(error);
} else {
try {
String resultString = new String(b);

log.trace("State provider method '" + name + "' returned: " + resultString);

StateMachineResultMessage resultMessage = objectMapper.readValue(resultString, StateMachineResultMessage.class);
if (resultMessage.getResult() == null) {
sink.error(new RuntimeException("State provider method '" + name + "' returned null"));
return;
}

log.trace("Sinking state provider method '" + name + "' returned: " + resultMessage.getResult());

sink.success(resultMessage.getResult());
} catch (JsonProcessingException e) {
sink.error(new RuntimeException("Error deserializing state provider result", e));
}
}
});

} catch (Exception e) {

I also switched on tracing on jgroups ... to see what is happening .... I see a lot of resends to my leader node (uvuyo1) from uvuyo2 the node which hangs .. I checked the logs on the leader there i see that the message was received applyed and returned result ... I am really wondering how the problem should be in my code since the async call should run in it's own thread and therfor be able to complete ... I am very happy if i missed something ;)

Log from the leader showing that the message was received ...

2025-05-08T14:11:16.970+02:00 TRACE 23775 --- [ runner-1] n.the2yetis.uvuyo.cluster.StateMachine : Applying state machine message: {"type":"execute","providerId":"repository","name":"databaseExists","args":["2yetis","endpoints"]}
2025-05-08T14:11:16.970+02:00 TRACE 23775 --- [ runner-1] n.the2yetis.uvuyo.cluster.StateMachine : Applying state provider 'repository' with method 'databaseExists' and args [2yetis, endpoints]
2025-05-08T14:11:16.970+02:00 TRACE 23775 --- [ runner-1] n.the2yetis.uvuyo.cluster.StateMachine : Searching for method 'databaseExists' with args [2yetis, endpoints] in class class net.the2yetis.uvuyo.memory.MemoryRepository with types [class java.lang.String, class java.lang.String]
2025-05-08T14:11:16.970+02:00 TRACE 23775 --- [ runner-1] n.the2yetis.uvuyo.cluster.StateMachine : Found method 'databaseExists' in state provider 'repository' with args [2yetis, endpoints] and types [class java.lang.String, class java.lang.String]
2025-05-08T14:11:16.970+02:00 TRACE 23775 --- [ runner-1] n.the2yetis.uvuyo.cluster.StateMachine : Returning result: {"result":true}

Log from the follower (uvuyo1) which hangs ....

2025-05-08T13:52:45.284+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Executing state provider 'repository' with method 'databaseExists' and args [2yetis, endpoints]

Here is my setAsync(...)
2025-05-08T13:52:45.284+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Invoking setAsync on raft handle ...
2025-05-08T13:52:45.284+02:00 TRACE 28991 --- [ NODE-1] org.jgroups.protocols.raft.REDIRECT : uvuyo2: redirecting request 11 to leader uvuyo1
2025-05-08T13:52:45.285+02:00 TRACE 28991 --- [ NODE-1] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: DATA(#31, conn_id=0)
2025-05-08T13:52:45.285+02:00 TRACE 28991 --- [ NODE-1] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=225, hdrs: REDIRECT: REQ, corr_id=11, exception=false, UNICAST3: DATA, seqno=31, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:45.285+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : setAsync on raft handle returned.
2025-05-08T13:52:45.285+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Fork join size9
2025-05-08T13:52:45.285+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Fork join count0
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received message batch of 1 messages from uvuyo1
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2 <-- uvuyo1: #329-329 (1 messages)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.MFC : uvuyo1 used 111 credits, 4998350 remaining
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ runner-1] org.jgroups.protocols.raft.RAFT : uvuyo2: from uvuyo1, [uvuyo1 to <all>, 111 bytes], obj: 1 entries header AppendEntriesRequest: current_term=123, leader=uvuyo1, entry_term=123, prev_log_index=5921, prev_log_term=123, leader_commit=5921
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ runner-1] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: DATA(#32, conn_id=0)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ runner-1] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis

Dumping all threads ...
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-5 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-3 (State: TIMED_WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: boundedElastic-evictor-1 (State: TIMED_WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-4 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-8 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-2 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: runner-1 (State: TIMED_WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: MC-0 (State: RUNNABLE)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Timer runner-3,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: DestroyJavaVM (State: RUNNABLE)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-12,uvuyo-gateway-2yetis,uvuyo2 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: logback-44 (State: TERMINATED)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-7 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-9 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-3 (State: WAITING)
2025-05-08T13:52:45.286+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-11,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-4 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-7 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: MC-1 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-2 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Notification Thread (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Signal Dispatcher (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-1 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-admin-client-thread | adminclient-1 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Monitor Ctrl-Break (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: logback-45 (State: TERMINATED)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Reference Handler (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-6 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: TQ-Bundler-7,uvuyo-gateway-2yetis,uvuyo2 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-coordinator-heartbeat-thread | uvuyo (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-6,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-13,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Connection.Receiver [192.168.2.42:7801 - 192.168.2.42:64056]-15,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Finalizer (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Connection.Receiver [192.168.2.42:7801 - 192.168.2.42:64049]-9,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK pinger-14,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-8 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-5,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: reactor-http-nio-1 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK acceptor-8,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-5 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: CLIENT.Acceptor (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: server (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-coordinator-heartbeat-thread | single-consumer (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-6 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Common-Cleaner (State: TIMED_WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-1 (State: WAITING)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: TcpServer.Acceptor[7801]-2,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK conn-handler-16,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)
2025-05-08T13:52:45.287+02:00 TRACE 28991 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: NODE-1 (State: RUNNABLE)
2025-05-08T13:52:45.404+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: ACK(#17)
2025-05-08T13:52:45.404+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=115, hdrs: UNICAST3: ACK, seqno=17, conn_id=14, ts=6, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:45.905+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:45.911+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:45.953+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.MERGE3 : uvuyo2: found no inconsistent views: [uvuyo1|20]: [(3) uvuyo1, uvuyo3, uvuyo2]

2025-05-08T13:52:45.961+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to null, src=uvuyo2, size=72, hdrs: FD_ALL: heartbeat, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:45.962+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.FD_ALL : uvuyo2: sent heartbeat

Is ths a problem that it keeps on resending?
2025-05-08T13:52:46.419+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:46.419+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:46.508+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to <all>, 0 bytes, flags=OOB|NO_FC], headers are NAKACK2: [HIGHEST_SEQNO, seqno=329], TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:46.921+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:46.925+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:47.431+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:47.432+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:47.743+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.STABLE : uvuyo2: sending stable msg to uvuyo1: uvuyo1: [329], uvuyo3: [0], uvuyo2: [0]
2025-05-08T13:52:47.744+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=267, hdrs: STABLE: [STABLE_GOSSIP] view-id= [uvuyo1|20], TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:47.746+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to <all>, 119 bytes, flags=OOB|NO_FC|NO_RELIABILITY|NO_RELAY], obj: uvuyo1: [326 (329)], uvuyo3: [0 (0)], uvuyo2: [0 (0)], headers are STABLE: [STABILITY] view-id= [uvuyo1|20], TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:47.747+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.STABLE : uvuyo2: received stability msg from uvuyo1: uvuyo1: [326], uvuyo3: [0], uvuyo2: [0]
2025-05-08T13:52:47.747+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: received stable digest uvuyo1: [326 (329)], uvuyo3: [0 (0)], uvuyo2: [0 (0)]
2025-05-08T13:52:47.747+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 326 from uvuyo1
2025-05-08T13:52:47.747+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 0 from uvuyo3
2025-05-08T13:52:47.747+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 0 from uvuyo2
2025-05-08T13:52:47.937+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:47.938+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:48.442+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:48.444+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:48.947+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:48.949+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:49.453+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:49.454+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:49.957+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:49.958+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:50.017+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received message batch of 1 messages from uvuyo3
2025-05-08T13:52:50.017+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.FD_ALL : uvuyo2: received heartbeat from uvuyo3
2025-05-08T13:52:50.464+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:50.465+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:50.490+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo3 to 192.168.2.42:7801, 83 bytes, flags=OOB|DONT_BUNDLE], headers are TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:50.491+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 127.0.0.1:7802, src=uvuyo2, size=161, hdrs: MERGE3: INFO: view_id=[uvuyo1|20], logical_name=MacBook-Pro-von-Martin-2-32767, physical_addr=192.168.2.42:7801, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:50.492+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: received GET_MBRS_REQ from uvuyo3, sending response uvuyo2, name=MacBook-Pro-von-Martin-2-32767, addr=192.168.2.42:7801, server
2025-05-08T13:52:50.492+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo3, src=uvuyo2, size=199, hdrs: TCPPING: GET_MBRS_RSP cluster=null initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:50.494+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received message batch of 1 messages from uvuyo3
2025-05-08T13:52:50.971+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:50.972+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:51.474+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:51.475+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:51.916+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to 192.168.2.42:7801, 84 bytes, flags=OOB|DONT_BUNDLE], headers are TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:51.917+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 192.168.2.42:7800, src=uvuyo2, size=161, hdrs: MERGE3: INFO: view_id=[uvuyo1|20], logical_name=MacBook-Pro-von-Martin-2-32767, physical_addr=192.168.2.42:7801, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:51.917+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: received GET_MBRS_REQ from uvuyo1, sending response uvuyo2, name=MacBook-Pro-von-Martin-2-32767, addr=192.168.2.42:7801, server
2025-05-08T13:52:51.917+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=199, hdrs: TCPPING: GET_MBRS_RSP cluster=null initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:51.980+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:51.982+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.486+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:52.488+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.658+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: sending discovery request to 192.168.2.42:7800
2025-05-08T13:52:52.658+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 192.168.2.42:7800, src=uvuyo2, size=193, hdrs: TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.658+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: sending discovery request to 192.168.2.42:7802
2025-05-08T13:52:52.658+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 192.168.2.42:7802, src=uvuyo2, size=193, hdrs: TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.659+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: sending discovery request to 127.0.0.1:7802
2025-05-08T13:52:52.659+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 127.0.0.1:7802, src=uvuyo2, size=193, hdrs: TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.659+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : 192.168.2.42:7801: connecting to 192.168.2.42:7802
2025-05-08T13:52:52.660+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to uvuyo2, 84 bytes, flags=OOB|DONT_BUNDLE], headers are TCPPING: GET_MBRS_RSP cluster=null initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.660+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: received GET_MBRS_RSP from uvuyo1: uvuyo1, name=MacBook-Pro-von-Martin-2-42819, addr=192.168.2.42:7800, coord
2025-05-08T13:52:52.660+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 192.168.2.42:7800, src=uvuyo2, size=161, hdrs: MERGE3: INFO: view_id=[uvuyo1|20], logical_name=MacBook-Pro-von-Martin-2-32767, physical_addr=192.168.2.42:7801, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.661+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : 192.168.2.42:7801: removed connection to 192.168.2.42:7802
2025-05-08T13:52:52.661+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received message batch of 1 messages from uvuyo3
2025-05-08T13:52:52.661+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo3 to uvuyo2, 83 bytes, flags=OOB|DONT_BUNDLE], headers are TCPPING: GET_MBRS_RSP cluster=null initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.661+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: received GET_MBRS_RSP from uvuyo3: uvuyo3, name=MacBook-Pro-von-Martin-2-3644, addr=127.0.0.1:7802, server
2025-05-08T13:52:52.661+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 127.0.0.1:7802, src=uvuyo2, size=161, hdrs: MERGE3: INFO: view_id=[uvuyo1|20], logical_name=MacBook-Pro-von-Martin-2-32767, physical_addr=192.168.2.42:7801, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:52.993+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:52.994+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:53.500+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:53.500+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:53.966+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to null, src=uvuyo2, size=72, hdrs: FD_ALL: heartbeat, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:53.966+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.FD_ALL : uvuyo2: sent heartbeat
2025-05-08T13:52:54.003+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:54.003+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:54.508+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:54.509+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:55.014+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:55.015+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:55.520+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:55.521+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:56.027+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:56.028+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:56.072+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.STABLE : uvuyo2: sending stable msg to uvuyo1: uvuyo1: [329], uvuyo3: [0], uvuyo2: [0]
2025-05-08T13:52:56.073+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=267, hdrs: STABLE: [STABLE_GOSSIP] view-id= [uvuyo1|20], TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to <all>, 119 bytes, flags=OOB|NO_FC|NO_RELIABILITY|NO_RELAY], obj: uvuyo1: [329 (329)], uvuyo3: [0 (0)], uvuyo2: [0 (0)], headers are STABLE: [STABILITY] view-id= [uvuyo1|20], TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.STABLE : uvuyo2: received stability msg from uvuyo1: uvuyo1: [329], uvuyo3: [0], uvuyo2: [0]
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: received stable digest uvuyo1: [329 (329)], uvuyo3: [0 (0)], uvuyo2: [0 (0)]
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 329 from uvuyo1
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 0 from uvuyo3
2025-05-08T13:52:56.079+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.pbcast.NAKACK2 : uvuyo2: deleting msgs <= 0 from uvuyo2
2025-05-08T13:52:56.532+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:56.532+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.034+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:57.034+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.263+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received [uvuyo1 to 192.168.2.42:7801, 84 bytes, flags=OOB|DONT_BUNDLE], headers are TCPPING: GET_MBRS_REQ cluster=uvuyo-gateway-2yetis initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.263+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to 192.168.2.42:7800, src=uvuyo2, size=161, hdrs: MERGE3: INFO: view_id=[uvuyo1|20], logical_name=MacBook-Pro-von-Martin-2-32767, physical_addr=192.168.2.42:7801, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.264+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCPPING : uvuyo2: received GET_MBRS_REQ from uvuyo1, sending response uvuyo2, name=MacBook-Pro-von-Martin-2-32767, addr=192.168.2.42:7801, server
2025-05-08T13:52:57.264+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=199, hdrs: TCPPING: GET_MBRS_RSP cluster=null initial_discovery=false, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.536+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:57.536+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:57.756+02:00 INFO 28991 --- [ionShutdownHook] o.s.b.w.embedded.netty.GracefulShutdown : Commencing graceful shutdown. Waiting for active requests to complete
2025-05-08T13:52:57.760+02:00 INFO 28991 --- [ netty-shutdown] o.s.b.w.embedded.netty.GracefulShutdown : Graceful shutdown complete
2025-05-08T13:52:58.018+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: received message batch of 1 messages from uvuyo3
2025-05-08T13:52:58.018+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.FD_ALL : uvuyo2: received heartbeat from uvuyo3
2025-05-08T13:52:58.037+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:58.038+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:58.544+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:58.545+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:59.050+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:59.051+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:59.555+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.UNICAST3 : uvuyo2 --> uvuyo1: resending(#32)
2025-05-08T13:52:59.556+02:00 TRACE 28991 --- [y-2yetis,uvuyo2] org.jgroups.protocols.TCP : uvuyo2: sending msg to uvuyo1, src=uvuyo2, size=129, hdrs: RAFT: AppendEntriesResponse: current_term=123, result: true, index=5922, commit-index=5921, UNICAST3: DATA, seqno=32, TP: cluster=uvuyo-gateway-2yetis
2025-05-08T13:52:59.853+02:00 INFO 28991 --- [ionShutdownHook] n.t.uvuyo.gateway.GatewayStarter : Shutdown of the application initiated ...

José Bolina

unread,

May 8, 2025, 8:47:37 AMMay 8

to Martin Tauber, jgroups-raft

Thanks for the data, Martin!

I'll take a look at it and get back to you. Hopefully we can get to the bottom of this 🤞

Can you generate the thread dump with jcmd? This would include the stack traces which would greatly help.

Regarding the previous comment with the stack, I have a few suggestions:

* You shouldn't need `STATE_TRANSFER` and `BARRIER`.

* FD_ALL -> FD_ALL3

* FRAG2 -> FRAG4

* raft.NO_DUPES below GMS in the stack. This avoids misconfigurations as adding two nodes with same raft_id.

* CLIENT is only really necessary if you utilize the `bin/client.sh` script to perform membership changes.

Cheers,

To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/c81b6b78-91c3-4a7f-8c9f-be98b836b986n%40googlegroups.com.

--

José Bolina

Martin Tauber

unread,

May 8, 2025, 8:50:21 AMMay 8

to José Bolina, jgroups-raft

Hi José,

thanks for your help it is really appreciated!

Ok I’ll do the changes … I’ll also try to add the stack traces …

Kind Regards

Martin

Martin Tauber

unread,

May 8, 2025, 9:33:31 AMMay 8

to José Bolina, jgroups-raft

Hi José,

Here ist are the stacks oft he threads:

2025-05-08T15:31:59.242+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Invoking setAsync on raft handle ...

2025-05-08T15:31:59.242+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : setAsync on raft handle returned.

2025-05-08T15:31:59.242+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Fork join size9

2025-05-08T15:31:59.242+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Fork join count0

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Timer runner-3,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.DelayQueue.take(DelayQueue.java:229)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.DelayQueue.take(DelayQueue.java:77)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.util.TimeScheduler3.run(TimeScheduler3.java:197)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: reactor-http-nio-1 (State: RUNNABLE)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueue.poll(Native Method)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:146)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68)

2025-05-08T15:31:59.243+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: TQ-Bundler-7,uvuyo-gateway-2yetis,uvuyo2 (State: WAITING)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:420)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.TransferQueueBundler.run(TransferQueueBundler.java:122)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: MC-0 (State: RUNNABLE)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueue.poll(Native Method)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.select(Selector.java:878)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.poll(Selector.java:469)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)

2025-05-08T15:31:59.244+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.pollForFetches(LegacyKafkaConsumer.java:687)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:618)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:591)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//net.the2yetis.uvuyo.message.kafka.KafkaMessageConsumer$1.run(KafkaMessageConsumer.java:92)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-2 (State: WAITING)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: MC-1 (State: RUNNABLE)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueue.poll(Native Method)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.select(Selector.java:878)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.poll(Selector.java:469)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:281)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:252)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.pollForFetches(LegacyKafkaConsumer.java:687)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:618)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.LegacyKafkaConsumer.poll(LegacyKafkaConsumer.java:591)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:874)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//net.the2yetis.uvuyo.message.kafka.KafkaMessageConsumer$1.run(KafkaMessageConsumer.java:92)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-1 (State: WAITING)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.245+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-coordinator-heartbeat-thread | single-consumer (State: TIMED_WAITING)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Object.wait(Native Method)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1526)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-8 (State: WAITING)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-4 (State: WAITING)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Reference Handler (State: RUNNABLE)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.Reference.waitForReferencePendingList(Native Method)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.Reference.processPendingReferences(Reference.java:253)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:215)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-3 (State: WAITING)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.246+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK acceptor-8,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.Net.accept(Native Method)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:675)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.platformImplAccept(ServerSocket.java:641)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:617)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:574)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.accept(ServerSocket.java:532)

2025-05-08T15:31:59.247+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FD_SOCK$ServerSocketHandler.run(FD_SOCK.java:1049)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Connection.Receiver [192.168.2.42:52994 - 192.168.2.42:7800]-9,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read0(Native Method)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:966)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedInputStream.read(BufferedInputStream.java:263)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.DataInputStream.readInt(DataInputStream.java:393)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:292)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-admin-client-thread | adminclient-1 (State: RUNNABLE)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueue.poll(Native Method)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.select(Selector.java:878)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.common.network.Selector.poll(Selector.java:469)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:595)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1524)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1455)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: kafka-coordinator-heartbeat-thread | uvuyo (State: TIMED_WAITING)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Object.wait(Native Method)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.apache.kafka.clients.consumer.internals.AbstractCoordinator$HeartbeatThread.run(AbstractCoordinator.java:1526)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-6 (State: WAITING)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Finalizer (State: WAITING)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Object.wait(Native Method)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)

2025-05-08T15:31:59.248+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:172)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Common-Cleaner (State: TIMED_WAITING)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Object.wait(Native Method)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:140)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:162)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: boundedElastic-evictor-1 (State: TIMED_WAITING)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-1 (State: WAITING)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.249+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Monitor Ctrl-Break (State: RUNNABLE)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read0(Native Method)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:966)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:270)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:313)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:188)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.InputStreamReader.read(InputStreamReader.java:177)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedReader.fill(BufferedReader.java:162)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedReader.readLine(BufferedReader.java:329)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedReader.readLine(BufferedReader.java:396)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:31)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-7 (State: WAITING)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Notification Thread (State: RUNNABLE)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-5,uvuyo-gateway-2yetis,uvuyo2 (State: WAITING)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.250+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//reactor.core.publisher.Mono.block(Mono.java:1779)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//net.the2yetis.uvuyo.gateway.GatewayNode.lambda$postConstruct$1(GatewayNode.java:208)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//net.the2yetis.uvuyo.gateway.GatewayNode$$Lambda$874/0x000000080068b508.apply(Unknown Source)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//net.the2yetis.uvuyo.cluster.ClusterManager._callRemoteFunction(ClusterManager.java:481)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.reflect.Method.invoke(Method.java:568)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.MethodCall.invoke(MethodCall.java:100)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:199)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:421)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:375)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.RequestCorrelator.iterate(RequestCorrelator.java:293)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.RequestCorrelator.receiveMessageBatch(RequestCorrelator.java:276)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:506)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.JChannel.up(JChannel.java:642)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:942)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.REDIRECT.up(REDIRECT.java:125)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.RAFT.up(RAFT.java:631)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.election.BaseElection.up(BaseElection.java:159)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FRAG2.up(FRAG2.java:159)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FlowControl.up(FlowControl.java:319)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.pbcast.GMS.up(GMS.java:797)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.NO_DUPES.up(NO_DUPES.java:56)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:255)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1188)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:996)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:959)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:585)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:684)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:146)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.stack.Protocol.up(Protocol.java:392)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.MERGE3.up(MERGE3.java:286)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.Discovery.up(Discovery.java:316)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.TP.passBatchUp(TP.java:1183)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:266)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK conn-handler-12,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read0(Native Method)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:966)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:961)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FD_SOCK$ClientConnectionHandler.run(FD_SOCK.java:1111)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.251+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Signal Dispatcher (State: RUNNABLE)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-7 (State: WAITING)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-8 (State: WAITING)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: runner-1 (State: TIMED_WAITING)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.RAFT.processQueue(RAFT.java:791)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.raft.RAFT$$Lambda$990/0x000000080071f548.run(Unknown Source)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.util.Runner.run(Runner.java:80)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.252+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-10,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-5 (State: WAITING)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-6,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-2 (State: WAITING)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.253+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: parallel-3 (State: WAITING)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:506)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3463)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3434)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1623)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1170)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1062)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: DestroyJavaVM (State: RUNNABLE)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-5 (State: WAITING)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: TcpServer.Acceptor[7801]-2,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.Net.accept(Native Method)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:755)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:675)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.platformImplAccept(ServerSocket.java:641)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:617)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.implAccept(ServerSocket.java:574)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.ServerSocket.accept(ServerSocket.java:532)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:102)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: FD_SOCK pinger-13,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read0(Native Method)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:47)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.tryRead(NioSocketImpl.java:261)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:312)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:966)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:961)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.protocols.FD_SOCK.run(FD_SOCK.java:464)

2025-05-08T15:31:59.254+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-4 (State: WAITING)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: NODE-1 (State: RUNNABLE)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.lang.Thread.getStackTrace(Thread.java:1610)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at net.the2yetis.uvuyo.cluster.StateMachine.lambda$execute$1(StateMachine.java:112)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:61)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4576)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribeWith(Mono.java:4642)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4542)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4478)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4425)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at net.the2yetis.uvuyo.repository.RepositoryManager.lambda$initDatabase$2(RepositoryManager.java:58)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:61)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4576)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.Mono.subscribe(Mono.java:4576)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.publisher.MonoSubscribeOn$SubscribeOnSubscriber.run(MonoSubscribeOn.java:126)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java.base/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-9 (State: TIMED_WAITING)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkUntil(LockSupport.java:410)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1726)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: jgroups-11,uvuyo-gateway-2yetis,uvuyo2 (State: TIMED_WAITING)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.255+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:401)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:903)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1061)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1122)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: server (State: WAITING)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//reactor.core.publisher.Mono.block(Mono.java:1779)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.springframework.boot.web.embedded.netty.NettyWebServer$1.run(NettyWebServer.java:214)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: ForkJoinPool.commonPool-worker-6 (State: WAITING)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.awaitWork(ForkJoinPool.java:1724)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1623)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : Thread: Connection.Receiver [192.168.2.42:7801 - 192.168.2.42:53019]-14,uvuyo-gateway-2yetis,uvuyo2 (State: RUNNABLE)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.Net.poll(Native Method)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:181)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.park(NioSocketImpl.java:190)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:314)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.net.Socket$SocketInputStream.read(Socket.java:966)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedInputStream.fill(BufferedInputStream.java:244)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.BufferedInputStream.read(BufferedInputStream.java:263)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.io.DataInputStream.readInt(DataInputStream.java:393)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at app//org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:292)

2025-05-08T15:31:59.256+02:00 TRACE 35834 --- [ NODE-1] n.the2yetis.uvuyo.cluster.StateMachine : at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

Von: José Bolina <jbo...@redhat.com>
Datum: Donnerstag, 8. Mai 2025 um 14:47
An: Martin Tauber <martin...@2yetis.net>
Cc: jgroups-raft <jgroup...@googlegroups.com>
Betreff: Re: [jgroups-raft] stateMachine.set(...) is hanging

Thanks for the data, Martin!

Martin Tauber

unread,

May 9, 2025, 2:11:29 AMMay 9

to jgroups-raft

Good morning,

just some more thoughts on this ....

The problem occurs when i want to "warm" restart the node. This means that the node receives the message to initialize itself again. The initialization is only to the level of the application meaning the cluster setup is not touched during the warm restart. Nevertheless the trigger to warm restart is send to the node via a rpcDispatcher ... could this be the issue?

Kind Regards

Martin

Martin Tauber

unread,

May 9, 2025, 2:27:07 AMMay 9

to jgroups-raft

Another good Morning ;)

I have just changed the code so that the remote function will not wait until the restart completed it will just initiate the restart. et voila it works! (again fingers crossed) Any explanation why?

Thanks

Martin

Bela Ban

unread,

May 9, 2025, 2:37:32 AMMay 9

to jgroup...@googlegroups.com

I can see that RAFT.runner is waiting for work; this means that no requests are received by this member, this is fine.

Thread: runner-1 (State: TIMED_WAITING)
    at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    at java...@17.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:252)
    at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1672)
    at java...@17.0.2/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:435)
    at app//org.jgroups.protocols.raft.RAFT.processQueue(RAFT.java:791)
    at app//org.jgroups.protocols.raft.RAFT$$Lambda$990/0x000000080071f548.run(Unknown Source)
    at app//org.jgroups.util.Runner.run(Runner.java:80)
    at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

Also: I see NO_DUPES in the stack trace, but *not* in your config. Have you changed this meanwhile?

The following stack trace looks suspicious:
* You receive a unicast message batch, pass it all the way up into the application and then *block on the same thread* (on a latch).
* This will prevent delivery of further unicast messages until you unblock!
* So you're using an RpcDispatcher on the same channel as jgroups-raft runs on. What's the point? What do you want to achieve? This is DANGEROUS! Can't you use a second JChannel for RpcDispatcher?

Thread: jgroups-5,uvuyo-gateway-2yetis,uvuyo2 (State: WAITING)
    at java...@17.0.2/jdk.internal.misc.Unsafe.park(Native Method)
    at java...@17.0.2/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)
    at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)
    at java...@17.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1047)
    at java...@17.0.2/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:230)
    at app//reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:91)
    at app//reactor.core.publisher.Mono.block(Mono.java:1779)
    at app//net.the2yetis.uvuyo.gateway.GatewayNode.lambda$postConstruct$1(GatewayNode.java:208)
    at app//net.the2yetis.uvuyo.gateway.GatewayNode$$Lambda$874/0x000000080068b508.apply(Unknown Source)
    at app//net.the2yetis.uvuyo.cluster.ClusterManager._callRemoteFunction(ClusterManager.java:481)
    at java...@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java...@17.0.2/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java...@17.0.2/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java...@17.0.2/java.lang.reflect.Method.invoke(Method.java:568)
    at app//org.jgroups.blocks.MethodCall.invoke(MethodCall.java:100)
    at app//org.jgroups.blocks.RpcDispatcher.handle(RpcDispatcher.java:199)
    at app//org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:421)
    at app//org.jgroups.blocks.RequestCorrelator.dispatch(RequestCorrelator.java:375)
    at app//org.jgroups.blocks.RequestCorrelator.iterate(RequestCorrelator.java:293)
    at app//org.jgroups.blocks.RequestCorrelator.receiveMessageBatch(RequestCorrelator.java:276)
    at app//org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:506)
    at app//org.jgroups.JChannel.up(JChannel.java:642)
    at app//org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:942)
    at app//org.jgroups.protocols.raft.REDIRECT.up(REDIRECT.java:125)
    at app//org.jgroups.protocols.raft.RAFT.up(RAFT.java:631)
    at app//org.jgroups.protocols.raft.election.BaseElection.up(BaseElection.java:159)
    at app//org.jgroups.protocols.FRAG2.up(FRAG2.java:159)
    at app//org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
    at app//org.jgroups.protocols.FlowControl.up(FlowControl.java:319)
    at app//org.jgroups.protocols.pbcast.GMS.up(GMS.java:797)
    at app//org.jgroups.protocols.raft.NO_DUPES.up(NO_DUPES.java:56)
    at app//org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:255)
    at app//org.jgroups.protocols.UNICAST3.deliverBatch(UNICAST3.java:1188)
    at app//org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:996)
    at app//org.jgroups.protocols.UNICAST3.handleBatchReceived(UNICAST3.java:959)
    at app//org.jgroups.protocols.UNICAST3.up(UNICAST3.java:585)
    at app//org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:684)
    at app//org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:146)
    at app//org.jgroups.protocols.FailureDetection.up(FailureDetection.java:193)
    at app//org.jgroups.stack.Protocol.up(Protocol.java:392)
    at app//org.jgroups.protocols.MERGE3.up(MERGE3.java:286)
    at app//org.jgroups.protocols.Discovery.up(Discovery.java:316)
    at app//org.jgroups.protocols.TP.passBatchUp(TP.java:1183)
    at app//org.jgroups.util.MaxOneThreadPerSender$BatchHandlerLoop.run(MaxOneThreadPerSender.java:266)
    at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java...@17.0.2/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java...@17.0.2/java.lang.Thread.run(Thread.java:833)

Bela Ban

unread,

May 9, 2025, 2:40:00 AMMay 9

to jgroup...@googlegroups.com

Yes. I do something similar in IspnPerfTest (jgroups-extras repo), but use a *second channel* (control channel). You seem to use the same channel as RAFT, and *block* on it, which prevents delivery of further requests.

Either use a second (control) channel, or make your control messages non-blocking and OOB.

To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/c2f4874f-a705-45a6-adc8-6fd48a25a07dn%40googlegroups.com.

Bela Ban

unread,

May 9, 2025, 2:40:37 AMMay 9

to jgroup...@googlegroups.com

On 09.05.2025 08:27, Martin Tauber wrote:

Another good Morning ;)

I have just changed the code so that the remote function will not wait until the restart completed it will just initiate the restart. et voila it works! (again fingers crossed) Any explanation why?

Yes, see my 2 emails this morning which just crossed your... :-)

To view this discussion visit https://groups.google.com/d/msgid/jgroups-raft/0e83beb8-d15a-410b-94c5-515153b4d2f1n%40googlegroups.com.

Reply all

Reply to author

Forward