Jgroups Upgrade procedure

[tarun]

Hi,

I was going through http://www.jgroups.org/ site,but i could not find any link or material related to Jgroups update.The only thing that i found was https://github.com/jgroups-extras/RollingUpgrades but that is relevant to kubernetes.

I am looking for Jgroups documentation which talks about Jgroups version upgrade lets say from 2.0.X to 3.0.X.Can you please provide the upgrade link ?

Thanks & Regards

Tarun

[Bela Ban]

On 10.08.20 07:34, tarun wrote:
> Hi,
>
> I was going through http://www.jgroups.org/ site,but i could not find
> any link or material related to Jgroups update.The only thing that i
> found was https://github.com/jgroups-extras/RollingUpgrades but that is
> relevant to kubernetes.

No, this can be used to upgrade a running system from one version to
another.
(I need to update the RollingUpgrades project though to include version
5.0.0)

> I am looking for Jgroups documentation which talks about Jgroups version
> upgrade lets say from 2.0.X to 3.0.X.Can you please provide the upgrade
> link ?

Excluding rolling upgrades: upgrade the version, compile and fix the
compilation errors. Also read the manual for 2.x and 3.x to figure out
what to change.

If questions are left: post them to the mailing list.

> Thanks & Regards
> Tarun
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>.

--
Bela Ban | http://www.jgroups.org

[tarun]

Thanks Bela for your reply.
I saw a earlier post about the upgrade mentioning about migration of 3.4.3 to 4.0.10 Jgroups version migration https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ where it is mentioned the backward comaptibility is not supported.

 

So does this upgrade incompatiblity persist in every release (post 4.0.10) or it is only incompatible before 4.0.10 version ?

 

Currently we are using 4.0.10 and want to upgrade Jgroup version, Is the backward comaptibility supported ,if we migrate from 4.0.10 to higher version ? 

 

Also ,is https://github.com/jgroups-extras/RollingUpgrades required for every post version 4.0.10 ?

[Bela Ban]

Different versions of JGroups are *not* guaranteed to be compatible with
each other, except for micro versions. So 1.2.3 and 1.2.4 (fantasy
versions, for illustration) may be compatible, whereas 1.2.3 and 1.3.0
are not.

RollingUpgrade is work in progress, but I've successfully upgraded 3.6.x
to 4.0.x using it. I need to beef it up to include 5.x though...

> https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[bel...@gmail.com]

Note that I'm addingb support for JGroups 5.x to RollingUpgrades. Unless something else comes up, this features should be available shortly.

Cheers,

[Dattaram Porob]

Hi,

I am working on an upgrade from Keycloak 17 (Wildfly with jgroups-4.2.15.Final & infinispan-core-12.1.7.Final) to Keycloak 21 (Quarkus with jgroups-5.2.10.Final & infinispan-core-14.0.4.Final). Since the jgroups versions are not compatible I am trying to see whether I can use https://github.com/jgroups-extras/RollingUpgrades. 

I see that RollingUpgrades has a project for jgroups-5 so looks like Jgroups 5 is supported and should work but it doesn't seem to work for me.

Before trying the actual mixed version cluster, I thought to just try to start individual keycloak (and hence jgroups) instances and see whether they are able to connect to the upgrade-server and get the view.

When I try to startup keycloak 17 (with jgroups 4.x with the UPGRADE element), things seem to work fine; infinispan starts up fine and gets the node in the view etc.

But when I try to startup Keycloak 21 (i.e. jgroups version 5.x with the UPGRADE element added to the stack), infinispan fails to start which looks like is because it is does not get the local address (may be because it does not get the global cluster view in time?). The stack definition looks like below:

<stack name="mytest">
      <UDP bind_port="55200" mcast_group_addr="230.0.0.4" mcast_port="45688"/>
      <PING/>
      <MERGE3 />
      <FD_SOCK start_port="54200"/>
      <FD_ALL />
      <VERIFY_SUSPECT/>
      <pbcast.NAKACK2 />
      <UNICAST3 />
      <pbcast.STABLE />
      <pbcast.GMS />
      <UFC />
      <MFC />
      <FRAG3 />
      <upgrade.UPGRADE server_address="135.9.77.19" active="true"/>
</stack>

The infinispan exception is below and looking at the source code it is throwing a NULL pointer since the local address is null:

2023-08-10 11:32:43,582 DEBUG [org.jgroups.protocols.upgrade.UPGRADE] (client-reconnector) null: joining cluster ejb
2023-08-10 11:32:43,624 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ejb` local address is `null`, physical addresses are `[135.9.77.15:55200]`
2023-08-10 11:32:43,625 ERROR [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000660: DefaultCacheManager start failed, stopping any running components: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component org.infinispan.metrics.impl.MetricsCollector failed to start
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:357)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
        at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:774)
        at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:742)
        at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:406)
        at org.keycloak.quarkus.runtime.storage.legacy.infinispan.CacheManagerFactory.startCacheManager(CacheManagerFactory.java:96)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException
        at org.infinispan.metrics.impl.MetricsCollector.start(MetricsCollector.java:79)
        at org.infinispan.metrics.impl.CorePackageImpl$1.start(CorePackageImpl.java:41)
        at org.infinispan.metrics.impl.CorePackageImpl$1.start(CorePackageImpl.java:34)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
        ... 15 more

Can I provide any more logs or traces to get to the root of this issue ? 

Could this be just because infinispan is relying on getting the cluster view to figure out the transport address and since the cluster view is not received in time (before some  infinispan code tries to access the transport address and use it) from the upgrade server, we face this issue ?

Any thoughts/suggestions and how to get past this issue ?

Thanks,

Dattaram.

[Bela Ban]

It could be because Protocol.local_addr is set via setAddress() in
5.2.x, but because UPGRADE 5.x extends UpgradeBase (4.2.18), this
method is not called.

Looking into this; it is unclear to me at first glance why UpgradeBase
is tied to a version...

> `*null*`, physical addresses are `[135.9.77.15:55200]`

> https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/>

> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >
> >
> > --
> > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "jgroups-dev" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send
> > an email to jgroups-dev...@googlegroups.com
> > <mailto:jgroups-dev...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com>
> >

> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Bela Ban]

OK, so I moved UpgradeBase to UpgradeBase5 and am using JGroups
5.2.10.Final now (in module jgroups-5).

I committed and pushed my changes, so you can try it out now. Let me
know how it does.

I'm going to create a module jgroups-5.2 which will use JGroups 5.2.18.
Cheers,

On 15.08.23 19:18, Dattaram Porob wrote:

> `*null*`, physical addresses are `[135.9.77.15:55200]`

> https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/>

> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> >
> >
> > --
> > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

> >
> > --
> > You received this message because you are subscribed to the
> Google
> > Groups "jgroups-dev" group.
> > To unsubscribe from this group and stop receiving emails from
> it, send
> > an email to jgroups-dev...@googlegroups.com
> > <mailto:jgroups-dev...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com>
> >

> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Dattaram Porob]

Thank you so much for the quick turnaround; after the change the address is getting set in the UPGRADE protocol but infinispan still fails with the same exception:

2023-08-16 18:13:38,308 INFO  [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) ghosh11-dev9-cdom-53623: no members discovered after 2003 ms: creating cluster as coordinator
2023-08-16 18:13:38,319 DEBUG [org.jgroups.protocols.pbcast.NAKACK2] (keycloak-cache-init)
[ghosh11-dev9-cdom-53623 setDigest()]
existing digest:  []
new digest:       ghosh11-dev9-cdom-53623: [0 (0)]
resulting digest: ghosh11-dev9-cdom-53623: [0 (0)]
2023-08-16 18:13:38,327 DEBUG [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) ghosh11-dev9-cdom-53623: installing view [ghosh11-dev9-cdom-53623|0] (1) [ghosh11-dev9-cdom-53623] (ghosh11-dev9-cdom-53623 joined)
2023-08-16 18:13:38,330 DEBUG [org.jgroups.protocols.pbcast.STABLE] (keycloak-cache-init) ghosh11-dev9-cdom-53623: resuming message garbage collection
2023-08-16 18:13:38,343 DEBUG [org.jgroups.protocols.pbcast.GMS] (keycloak-cache-init) ghosh11-dev9-cdom-53623: created cluster (first member). My view is [ghosh11-dev9-cdom-53623|0], impl is CoordGmsImpl
2023-08-16 18:13:38,352 DEBUG [org.jgroups.protocols.upgrade.UPGRADE] (keycloak-cache-init) ghosh11-dev9-cdom-53623: joining cluster ejb
2023-08-16 18:13:38,384 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000079: Channel `ejb` local address is `null`, physical addresses are `[135.9.77.15:55200]`

2023-08-16 18:13:38,459 ERROR [org.infinispan.CONFIG] (keycloak-cache-init) ISPN000660: DefaultCacheManager start failed, stopping any running components: org.infinispan.commons.CacheConfigurationException: ISPN000659: Component org.infinispan.topology.ClusterTopologyManager failed to start

        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:585)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startDependencies(BasicComponentRegistryImpl.java:635)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:599)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl$ComponentWrapper.running(BasicComponentRegistryImpl.java:808)
        at org.infinispan.factories.AbstractComponentRegistry.internalStart(AbstractComponentRegistry.java:357)
        at org.infinispan.factories.AbstractComponentRegistry.start(AbstractComponentRegistry.java:250)
        at org.infinispan.manager.DefaultCacheManager.internalStart(DefaultCacheManager.java:774)
        at org.infinispan.manager.DefaultCacheManager.start(DefaultCacheManager.java:742)
        at org.infinispan.manager.DefaultCacheManager.<init>(DefaultCacheManager.java:406)
        at org.keycloak.quarkus.runtime.storage.legacy.infinispan.CacheManagerFactory.startCacheManager(CacheManagerFactory.java:96)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException

        at org.infinispan.topology.TopologyManagementHelper.executeOnCoordinator(TopologyManagementHelper.java:83)
        at org.infinispan.topology.ClusterTopologyManagerImpl.fetchRebalancingStatusFromCoordinator(ClusterTopologyManagerImpl.java:156)
        at org.infinispan.topology.ClusterTopologyManagerImpl.start(ClusterTopologyManagerImpl.java:147)
        at org.infinispan.topology.CorePackageImpl$3.start(CorePackageImpl.java:74)
        at org.infinispan.topology.CorePackageImpl$3.start(CorePackageImpl.java:58)

        at org.infinispan.factories.impl.BasicComponentRegistryImpl.invokeStart(BasicComponentRegistryImpl.java:617)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.doStartWrapper(BasicComponentRegistryImpl.java:608)
        at org.infinispan.factories.impl.BasicComponentRegistryImpl.startWrapper(BasicComponentRegistryImpl.java:577)

        ... 19 more

2023-08-16 18:13:38,461 INFO  [org.infinispan.CLUSTER] (keycloak-cache-init) ISPN000080: Disconnecting JGroups channel `ejb`
2023-08-16 18:13:38,464 DEBUG [org.jgroups.protocols.upgrade.UPGRADE] (keycloak-cache-init) ghosh11-dev9-cdom-53623: leaving cluster ejb

Any other suggestions/ideas ?

[Bela Ban]

As you can see, UPGRADE *has* a non-null local_addr:
ghosh11-dev9-cdom-53623:

2023-08-16 18:13:38,352 DEBUG [org.jgroups.protocols.upgrade.UPGRADE]
(keycloak-cache-init) ghosh11-dev9-cdom-53623: joining cluster ejb

However, the next log statement has the local address as null:

2023-08-16 18:13:38,384 INFO [org.infinispan.CLUSTER]
(keycloak-cache-init) ISPN000079: Channel `ejb` local address is `null`,
physical addresses are `[135.9.77.15:55200]`

TBH, I don't know which component org.infinispan.CLUSTER is, either
Infinispan itself or Keycloak. I suggest ping the Infinispan or Keycloak
teams to clarify this.
Cheers

On 17.08.23 00:43, Dattaram Porob wrote:
> Thank you so much for the quick turnaround; after the change the address
> is getting set in the UPGRADE protocol but infinispan still fails with
> the same exception:
>
> 2023-08-16 18:13:38,308 INFO  [org.jgroups.protocols.pbcast.GMS]
> (keycloak-cache-init) ghosh11-dev9-cdom-53623: no members discovered
> after 2003 ms: creating cluster as coordinator
> 2023-08-16 18:13:38,319 DEBUG [org.jgroups.protocols.pbcast.NAKACK2]
> (keycloak-cache-init)
> [ghosh11-dev9-cdom-53623 setDigest()]
> existing digest:  []
> new digest:       ghosh11-dev9-cdom-53623: [0 (0)]
> resulting digest: ghosh11-dev9-cdom-53623: [0 (0)]
> 2023-08-16 18:13:38,327 DEBUG [org.jgroups.protocols.pbcast.GMS]
> (keycloak-cache-init) ghosh11-dev9-cdom-53623: installing view
> [ghosh11-dev9-cdom-53623|0] (1) [ghosh11-dev9-cdom-53623]
> (ghosh11-dev9-cdom-53623 joined)
> 2023-08-16 18:13:38,330 DEBUG [org.jgroups.protocols.pbcast.STABLE]
> (keycloak-cache-init) ghosh11-dev9-cdom-53623: resuming message garbage
> collection
> 2023-08-16 18:13:38,343 DEBUG [org.jgroups.protocols.pbcast.GMS]
> (keycloak-cache-init) ghosh11-dev9-cdom-53623: created cluster (first
> member). My view is [ghosh11-dev9-cdom-53623|0], impl is CoordGmsImpl
> 2023-08-16 18:13:38,352 DEBUG [org.jgroups.protocols.upgrade.UPGRADE]

> (keycloak-cache-init) *ghosh11-dev9-cdom-53623*: joining cluster ejb

> 2023-08-16 18:13:38,384 INFO  [org.infinispan.CLUSTER]
> (keycloak-cache-init) ISPN000079: Channel `ejb` local address is

> `*null*`, physical addresses are `[135.9.77.15:55200]`

> <https://github.com/jgroups-extras/RollingUpgrades>.

> https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/> <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/>>

> https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com>>
> > >
> > > >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>>>.

> > >
> > >
> > > --
> > > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>
> > >
> > > --
> > > You received this message because you are subscribed to the
> > Google
> > > Groups "jgroups-dev" group.
> > > To unsubscribe from this group and stop receiving emails from
> > it, send
> > > an email to jgroups-dev...@googlegroups.com
> > > <mailto:jgroups-dev...@googlegroups.com>.
> > > To view this discussion on the web visit
> > >
> >

> https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com>>
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>>>.

> >
> > --
> > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "jgroups-dev" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to jgroups-dev...@googlegroups.com
> > <mailto:jgroups-dev...@googlegroups.com>.
> > To view this discussion on the web visit
> >

> https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --

> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Dattaram Porob]

Thanks Bela. 

That log ( ISPN000079: Channel `ejb` local address is `null`) is from the infinispan code specifically https://github.com/infinispan/infinispan/blob/14.0.4.Final/core/src/main/java/org/infinispan/remoting/transport/jgroups/JGroupsTransport.java#L632. 

The 'address' in this JGroupsTransport class is only being set in receiveClusterView which in turn is invoked on the 'up' event (implements org.jgroups.UpHandler). So the start method is assuming that the 'up' event will get invoked before it returns and hence the address will be set. When we have UPGRADE in the stack, this is not happening (since it does not send the UP event to the application?) and hence the address stays as null and then other infinispan modules get the null address from the transport component and fail to start.

I see a comment in the infinispan code at https://github.com/infinispan/infinispan/blob/14.0.4.Final/core/src/main/java/org/infinispan/remoting/transport/jgroups/JGroupsTransport.java#L785 which states "The first view is installed before returning from JChannel.connect. So we need to set the local address here".

So, it looks to me as if there is an underlying assumption on which this code is based and that assumption is not true when using UPGRADE protocol. Does this mean that UPGRADE protocol will not work with infinispan ?

I can check on this with infinispan team as well but I wanted to understand from you on whether this assumption in the infinispan code is the right way for it to work with Jgroups ? If not, does infinispan need to fix something here OR does UPGRADE need to change so that it can work with infinispan.

Thanks.

[Bela Ban]

OK, I see that UPGRADE.up(VIEW_CHANGE) does *not* send the event further
up when 'active==true'. IIRC, this is done on purpose, because when
UPGRADE is active, we don't want to pass up the local view to an
application, but only the global view (created via UPGRADE).

IMO, the Infinispan code should *not* fetch the local address in the
view handler callback, but in startJGroupsChannelIfNeeded(), after
channel.connect().

@Pedro: your thoughts?

You can experiment with changing up(VIEW_CHANGE) in your local repo, to
pass the view up the stack. But IMO this changes the semantics of
UPGRADE, and I don't konw what ramifications this change has.
Cheers,


On 17.08.23 19:20, Dattaram Porob wrote:
> Thanks Bela.
>
> That log (ISPN000079: Channel `ejb` local address is `null`) is from the

> https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/> <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/>> <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/> <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/ <https://sourceforge.net/p/javagroups/discussion/134002/thread/14d44517/>>>

> https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com>> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com>>>
> > > >
> > > > >
> > > >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/0d3a7c13-a9e0-4ab7-97da-39e64b2e8296o%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.

> > > >
> > > >
> > > > --
> > > > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>
> > <http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>>
> > > >
> > > > --
> > > > You received this message because you are subscribed to the
> > > Google
> > > > Groups "jgroups-dev" group.
> > > > To unsubscribe from this group and stop receiving emails from
> > > it, send
> > > > an email to jgroups-dev...@googlegroups.com
> > > > <mailto:jgroups-dev...@googlegroups.com>.
> > > > To view this discussion on the web visit
> > > >
> > >
> >

> https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com>> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com>>>
> > > >
> > >
> >
> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/25e7d6ff-9297-49a0-8fc8-0a8bc32263bcn%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.

> > >
> > > --
> > > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>
> > <http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>>
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "jgroups-dev" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > send
> > > an email to jgroups-dev...@googlegroups.com
> > > <mailto:jgroups-dev...@googlegroups.com>.
> > > To view this discussion on the web visit
> > >
> >

> https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com>> <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer> <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/b91cb47c-a402-49e5-b45c-39bef0c26fe9n%40googlegroups.com?utm_medium=email&utm_source=footer>>>.

> >
> > --
> > Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
> <http://www.jgroups.org <http://www.jgroups.org>>
> >
> > --
> > You received this message because you are subscribed to the Google
> > Groups "jgroups-dev" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send
> > an email to jgroups-dev...@googlegroups.com
> > <mailto:jgroups-dev...@googlegroups.com>.
> > To view this discussion on the web visit
> >

> https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/3c858799-1b4a-4326-8426-1d6c29b0e341n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --
> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/e4da9be6-eba8-4a72-b30c-44a508fddf76n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/e4da9be6-eba8-4a72-b30c-44a508fddf76n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Pedro Ruivo]

I've created https://issues.redhat.com/browse/ISPN-15117 to track the Infinispan changes.

Also, even with the UPGRADE protocol, Infinispan 12 and Infinispan 14 are unable to talk to each other since the wire protocol is different. 

[Bela Ban]

So, just to illustrate: you mean that (e.g.) a PUT command in 12
marshalls to a byte array that cannot be unmarshalled to a PUT command
in 14, correct?

That means, even if version 12 nodes could talk to version 14 nodes at
the JGroups level (UPGRADE), the different Infinispan versions would not
be able to understand each other at the Infinispan level.

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/2f641bd8-401b-4f00-8e06-e8c9fd1f26a0n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/2f641bd8-401b-4f00-8e06-e8c9fd1f26a0n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Pedro Ruivo]

Yes, that is correct.

[Dattaram Porob]

So, are we essentially saying that there is no way to achieve a rolling upgrade of Keycloak 17 (using infinispan 12.x) to keycloak 21 (using infinispan 14.x) without losing the data cached in infinispan ?

Any application (like Keycloak) that uses infinispan would have the same issue when the infinispan version in the application is updated in the next major version of the application isn't it ? 

Do you know of any other solutions that can be used to achieve this without losing that data in infinispan cache ? 

Thanks.

[Pedro Ruivo]

We (Infinispan) do not have support for rolling upgrades in embedded caches. What does the Keycloak documentation say about the upgrades? Do you have persistence configured? 

[Dattaram Porob]

Looks like Keycloak does not support rolling upgrades stating DB schema compatibility issues. But, I looked at the schema changes and tried a small experiment to check whether Keycloak 17 and Keycloak 21 could point to the same DB and it seemed to work fine.

So, I believe the biggest hurdle is the possible loss of session data which is held in distributed (infinispan) cache. Hence I was trying to see whether I can get a Keycloak 17 (infinispan 12.x) and Keycloak 21(infinispan 14.x) instance to share the infinispan cache which you are saying is not possible, since the versions are different, correct ?

Thanks.

[Bela Ban]

On 21.08.23 21:30, Dattaram Porob wrote:
> Looks like Keycloak does not support rolling upgrades stating DB schema
> compatibility issues. But, I looked at the schema changes and tried a
> small experiment to check whether Keycloak 17 and Keycloak 21 could
> point to the same DB and it seemed to work fine.

Hmm, in this case you could start the new (14.x) instances as
non-distributed/local instances with a cache store pointing to the
shared DB. Data would be loaded into the caches as needed from the DB,
and changes would be written back to the DB.

When all 14.x instances are running, configure the 14.x instances as
distributed, and restart them one by one.

This would slow down things a bit, since the upgrade makes the system
non distributed and the DB a SPOF. However, it might work.. *if* the
schema is really compatible.

> So, I believe the biggest hurdle is the possible loss of session data
> which is held in distributed (infinispan) cache. Hence I was trying to
> see whether I can get a Keycloak 17 (infinispan 12.x) and Keycloak
> 21(infinispan 14.x) instance to share the infinispan cache which you are
> saying is not possible, since the versions are different, correct ?

I think this is what Pedro was saying. While the data might be
compatible, e.g. byte[] arrays, the commands to replicate the data might
differ. Example: a put(k,v) command in 12.x marshals to a byte[] array
that cannot be read by a 14.x instance.

[Dattaram Porob]

Bela, Thanks for the suggestion.

But, as far as I know, Keycloak stores 'session' data only in the distributed cache (and not in the DB); so the only way to get access to this cache is by joining the distributed cache infinispan cluster.

And looks like this is something that is not possible since the new keycloak instances (new infinispan version) will not be able to join the existing cluster (old infinispan version).

-Datta.

[Bela Ban]

So let's assume we have a version-12 in-memory only cluster.

You could, one by one, restart the cluster with a changed configuration,
in which all members also have a *cache store* configured.

Once this is done, all members should flush their contents to DB (not
sure if such a call exists).

Then you change the config again, and make all members LOCAL (e.g. from
DIST_SYNC), and restart.

Now start the version-14 members, also LOCAL: they'll fetch the data via
the cache loader/store from the common (shared) DB. According to your
tests, this should work.

Once we only have version-14 instances, change the config to DIST_SYNC
and restart one by one.

Once this is done, restart again, this time without a cachestore configured.

This would allow you to upgrade from 12 -> 14 without the 12 instances
needing to talk to the 14 instances.

Of course, the devil lies in the details! :-)
Cheers

> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>

>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Dattaram Porob]

Thanks Bela for the suggestion. But I am not sure whether there is any API to flush contents to DB.

I was thinking of an alternate approach (hopefully simple) to get this to work based on the assumption that a Keycloak cluster (with embedded infinispan 12.x) allows a remote infinispan cluster (version 12.x as well and with the same distributed/replicated caches configured)) to join in. 

@Bela / @Pedro, do you think this will work ? 

I tried to start a single Keycloak 17 node (with embedded infinispan 12.x) and a single infinispan node (version 12.x) but they are failing at the rebalancing step:

2023-09-21 12:59:31,185 INFO  (main) [org.infinispan.CLUSTER] ISPN000078: Starting JGroups channel ejb with stack mytest
-------------------------------------------------------------------
GMS: address=porobd-dev16-smgr-28497, cluster=ejb, physical address=135.9.77.19:7600
-------------------------------------------------------------------
2023-09-21 12:59:31,473 INFO  (main) [org.infinispan.CLUSTER] ISPN000094: Received new cluster view for channel ejb: [ghosh11-dev9-smgr|1] (2) [ghosh11-dev9-smgr, porobd-dev16-smgr-28497]
2023-09-21 12:59:31,488 INFO  (main) [org.infinispan.CLUSTER] ISPN000079: Channel ejb local address is porobd-dev16-smgr-28497, physical addresses are [135.9.77.19:7600]
2023-09-21 12:59:55,542 WARN  (timeout-thread--p4-t1) [org.infinispan.topology.ClusterTopologyManagerImpl] ISPN000329: Unable to read rebalancing status from coordinator ghosh11-dev9-smgr java.util.concurrent.CompletionException: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1 from ghosh11-dev9-smgr
        at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source)
        at org.infinispan.remoting.transport.AbstractRequest.completeExceptionally(AbstractRequest.java:74)
        at org.infinispan.remoting.transport.impl.SingleTargetRequest.onTimeout(SingleTargetRequest.java:85)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87)
        at org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 1 from ghosh11-dev9-smgr
        ... 8 more

Could this be a problem with some configuration OR is something like this impossible to achieve ? 

If it is possible to get something like this to work, I can send the configuration that I have tried so that we can figure out whether there is some issue on that front.

Thanks

[Bela Ban]

THis is more into the realm of Infinispan; why don't you re-post on the
Infinispan ML or Zulip chat?

On 21.09.23 19:28, Dattaram Porob wrote:
> Thanks Bela for the suggestion. But I am not sure whether there is any
> API to flush contents to DB.
>
> I was thinking of an alternate approach (hopefully simple) to get this
> to work based on the assumption that a Keycloak cluster (with embedded
> infinispan 12.x) allows a remote infinispan cluster (version 12.x as
> well and with the same distributed/replicated caches configured)) to
> join in.
>
> @Bela / @Pedro, do you think this will work ?
>
> I tried to start a single Keycloak 17 node (with embedded infinispan
> 12.x) and a single infinispan node (version 12.x) but they are failing
> at the rebalancing step:
>

> /2023-09-21 12:59:31,185 INFO  (main) [org.infinispan.CLUSTER]

>         ... 8 more/

> https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com> <https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com?utm_medium=email&utm_source=footer <https://groups.google.com/d/msgid/jgroups-dev/cc9ad09d-f01a-4992-a80e-e5a252a53796n%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> --

> Bela Ban | http://www.jgroups.org <http://www.jgroups.org>
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/8ab1e5db-a3e3-4117-ae39-44dc34b24d9en%40googlegroups.com <https://groups.google.com/d/msgid/jgroups-dev/8ab1e5db-a3e3-4117-ae39-44dc34b24d9en%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Pedro Ruivo]

Embedded Infinispan is different from Infinispan server. Without code changes, it is impossible to make it work. Also, Infinispan is used as cache so your date is safe in the database. Why are you going through all this trouble? 

[Dattaram Porob]

Thanks for your replies Bela and Pedro.

In case of keycloak, the embedded infinispan distributed cache is storing the state of active user sessions and I want to retain that state while upgrading keycloak (from a version that uses infinispan 12.x to a version that uses infinispan 14.x) so that the user sessions are active and working while keycloak is getting upgraded.

My hope was that I would move keycloak from using embedded infinispan to a remote infinispan while retaining the session cache and for that I wanted to see whether I can have a cluster formed between the existing embedded infinispan cluster and a new standalone infinispan cluster. 

I see that the embedded infinispan in keycloak uses fork channels. Is there a way I can force the standalone infinispan to also use fork channels ? If I could have the standalone infinispan also use fork channels, should they not be able to communicate ? 

Thanks.

[Pedro Ruivo]

Talked to the Keycloak team and they do not have persistent session support. Losing a session is not a problem; it just makes the active user authenticate again. I'm not sure if it is worth all this work.

Back to the original question, the short answer is no. Long answer, the server is not the same as embedded and you'll have to change the server code to create the ForkChannel. Also, the distributed cache in the server is different from the distributed caches in the embedded, so this is another change that needs to be made. 

My suggestion would be to change the Keycloak configuration to include a remote store. In more detail, you would start an Infinispan server cluster somewhere accessible to your Keycloak cluster. Then update the configuration one by one. For example, create a new Keycloak server instance with the remote store, wait for the state transfer, and shut down one of the existing Keycloak servers. Do that one by one until all Keycloak servers have the remote store configured. 

You still need downtime, but after your sessions are stored in the external Infinispan cluster, you can shut down the Keycloak cluster and start a new one. Be aware that the remote store brings some response time penalty.