com.hazelcast.spi.exception.PartitionMigratingException: Partition is migrating!

[Alexander K]

Hello, we have faced issue in production. We are running application in k8s cluster with 5 replicas. The situation occurred when we deployed our app with new features and application started to throw the following exception:

com.hazelcast.spi.exception.PartitionMigratingException: Partition is migrating! this: [10.233.109.204]:4447, partitionId: 194, operation: com.hazelcast.map.impl.operation.PutWithExpiryOperation, service: hz:impl:mapService
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.ensureNoPartitionProblems(OperationRunnerImpl.java:384)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:241)
    at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139)
    at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123)
    at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102)
    at ------ submitted from ------.()
    at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:279)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112)
    at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100)
    at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617)
    at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:479)
    at com.hazelcast.map.impl.proxy.MapProxySupport.putInternal(MapProxySupport.java:417)
    at com.hazelcast.map.impl.proxy.MapProxyImpl.put(MapProxyImpl.java:138) 

We replaced Hazelcast with other cache implementation until we know how to resolve this issue.

Hazelcast version - 4.2.5

Our application uses two instances of the cache that use different ports. for replication. 

Thank you for any recommendations or assistance.

[Neil Stevenson]

 Does it do this with the latest version (5.2.0) ?
 It should be a trivial upgrade from 4.2.5 to 5.2.0, and it might be a known & fixed issue

[Alexander K]

Haven't checked it since the app is running on Java 8 and after that behavior of 4.2.5 will be quite hard to prove to the management that we can trust it and try again.

Anyway, thank you for the recommendation. 

вторник, 1 ноября 2022 г. в 13:37:34 UTC+2, ne...@hazelcast.com:

[Neil Stevenson]

Are you able to share more of the logs ? The clue may be elsewhere

It would be helpful to clarify when you say " two instances of the cache that use different ports" what that exactly means

Is it two Hazelcast clusters ? Or one cluster with two `IMap` in use ?

If it is two Hazelcast clusters, be sure to set at least the cluster name differently to stop them accidentally joining

Neil

[Alexander K]

Unfortunately no, I don't have more logs.

"two instances means" the following:

HazelcastInstance backupCacheInstance = Hazelcast.newHazelcastInstance(hzBackupConfig);

HazelcastInstance mainCacheInstance = Hazelcast.newHazelcastInstance(hzMainConfig);

...

hzConfig.setClusterName(groupName);
hzConfig.getNetworkConfig().setPort(port);

...

hzConfig.getNetworkConfig()
.getJoin()
.getKubernetesConfig()
.setProperty("service-dns", serviceDns)
.setProperty("service-port", port.toString())

...

where  groupName and port are the following:

backupCacheInstance  -  groupName = hz_backup,  port = 4446

mainCacheInstance -  groupName = hz_main,  port = 4445

вторник, 1 ноября 2022 г. в 18:21:36 UTC+2, ne...@hazelcast.com:

[Neil Stevenson]

In your log I see "Partition is migrating! this: [10.233.109.204]:4447" so it's using port 4447

Given you've configured for 4445 and 4446 that suggests something has gone wrong earlier in the logs.

On Kubernetes, all pods would normally be on the same port.

[Alexander K]

I am working on getting logs before that exception occurred.

чт, 3 лист. 2022, 15:06 користувач 'Neil Stevenson' via Hazelcast <haze...@googlegroups.com> пише:

This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast

--
You received this message because you are subscribed to a topic in the Google Groups "Hazelcast" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hazelcast/Ve25-eWXSgk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hazelcast+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/96ac6020-3d6c-4e04-886f-d1549fa53363n%40googlegroups.com.