Updating clustered app in Payara Micro Cluster: getting BLOCKED: tenant not available

[Andrew GR]

Hello everyone,

can someone clarify please on the proper approach of Payara Micro cluster gradual upgrade?

Scenario is the following:

1. Let's say we have two Payara Micro (5.2021.4 Community) uber jar running joined the same hazelcast datagrid. Both host an app called app-1.0.2 (app-1.0.2.war built with property set to: <version>1.0.2</version> in pom.xml)
   
   
2. We would like to upgrade it to app-1.0.3: so we shutting down 1st node, replacing with the updated uber jar and launching it, expecting that Payara Micro Uber jar carrying app-1.0.3 successfully joins the cluster (datagrid), replicates sessions, etc., so we can proceed with updating the 2nd node.
   At least it worked like this before (at least I do not remember any issues with this approach).
   
   
3. But as of now it behaves a bit different: once we are starting updated node carrying newer version of app (app-1.0.3) which should join datagrid where app of previous version (app-1.0.2) still runs - that new node hangs during boot with printing the following messages to console:
   
   [2021-08-23T10:57:07.583+0800] [] [INFO] [] [fish.payara.nucleus.hazelcast.PayaraHazelcastTenant] [tid: _ThreadID=51 _ThreadName=hz.serene_lalande.partition-operation.thread-3] [timeMillis: 1629687427583] [levelValue: 800] BLOCKED: tenant not available: app-1.0.2_/, module app-1.0.2, Operation: com.hazelcast.map.impl.operation.PutOperation
   
   
4. So seems it can not acquire lock due to module version mismatch:
   https://github.com/payara/Payara/blob/3005876833ccd348317a6210f459345a2b467766/nucleus/payara-modules/hazelcast-bootstrap/src/main/java/fish/payara/nucleus/hazelcast/PayaraHazelcastTenant.java#L145

Any suggestions are welcome how this is to be addressed. Is this a feature or bug?

Thank you!

[Andrew GR]

I would add that when I mentioned "At least it worked like this before" - I think it is related to Payara Server, and not Payara Micro. So maybe with Payara Micro it works in a different way?

[Rudy De Busscher]

Hi,

Can you try version 5.2021.5 or 5.2021.6 as we have updated some of the Hazelcast with Tenant Control functionality?

Best Regards

Rudy

[Andrew G10i]

Hello,

thank you for quick response!

Seems can not try (5.2021.5 or 5.2020.6) right now as app won't deploy due to another issue:

Bug Report: payara-micro-5.2021.5 fails to deploy WAR with "ServletConfig has not been initialized" Exception/FISH-5660 #5363

Will try once that issue is resolved and provide feedback here.

Thank you!

[Andrew G10i]

Hello everyone,

I have retested with Payara Micro 5.2021.7 and confirm that initially posted issue has gone and both applications (prev. version and new version) are starting fine and joining the same cluster.

BUT unfortunately replication doesn't work smoothly anyway and even thought both apps (prev. version and new version) has joined the same cluster - in-memory replication throws exceptions like:

[2021-09-16T08:58:26.234+0800] [] [SEVERE] [AS-WEB-CORE-00037] [javax.enterprise.web.core] [tid: _ThreadID=142 _ThreadName=http-thread-pool::http-listener(1)] [timeMillis: 1631753906234] [levelValue: 1000] [[
  An exception or error occurred in the container during the request processing
org.jboss.weld.exceptions.WeldException: WELD-001500: Failed to deserialize proxy object with beanId WELD%ManagedBean%App-b2319.war.bda|App-b2319.war/WEB-INF/lib/App-b2319|ua.app.beans.ConfigBean|null|0
        at org.jboss.weld.bean.proxy.util.SerializableClientProxy.readResolve(SerializableClientProxy.java:60)

I will create another topic for this case in order to discuss it further.

Thank you!

--

Best regards,

Andrew

On Monday, August 23, 2021 at 5:19:14 PM UTC+8 Andrew G10i wrote:

Hello,

thank you for quick response!

Seems can not try (5.2021.5 or 5.2020.6) right now as app won't deploy due to another issue:

Bug Report: payara-micro-5.2021.5 fails to deploy WAR with "ServletConfig has not been initialized" Exception/FISH-5660 #5363

Will try once that issue is resolved and provide feedback here.

Thank you!

On Monday, August 23, 2021 at 3:27:20 PM UTC+8 Rudy De Busscher wrote:

Hi,

Can you try version 5.2021.5 or 5.2021.6 as we have updated some of the Hazelcast with Tenant Control functionality?

Best Regards

Rudy