Re: [payara-forum] JGroups cache coordination fails on Payara 5

[Hans Pikkemaat]

Hi,

I don’t have any experience with jgroups so can’t help you with that.

But if you want to use eclipselink cache coordination using hazelcast that works well in payara.

If you run payara in AWS and have different EC2 instances in separated networks then you need instances to find each other

so hazelcast can connect them.

See here how you can do that:

https://github.com/hazelcast/hazelcast-aws

See here for more general info about hazelcast and using cache coordination using hazelcast

https://docs.payara.fish/enterprise/docs/5.24.0/documentation/payara-server/hazelcast/README.html

https://blog.payara.fish/setting-up-cache-jpa-coordination-with-the-payara-platform-using-eclipselink-and-jms/hazelcast

https://docs.payara.fish/community/docs/5.2020.7/documentation/payara-server/jpa-cache-coordination.html

If you would run payara in docker and e.g. have two docker containers running in the same network then I think they will

find eachother without a problem, but this is probably not what you are doing in AWS now.

gr. Hans

Op 5 feb. 2021, om 16:39 heeft 'Dimitri' via Payara Forum <payara...@googlegroups.com> het volgende geschreven:

Hi all,

I used JGroups on Glassfish 4 without a problem, but have some trouble using it with Payara 5.2020.7.

First of all, I enable JPA cache coordination in persistence.xml

   <shared-cache-mode>DISABLE_SELECTIVE</shared-cache-mode>
   <properties>
      <property name="eclipselink.cache.coordination.protocol" value="jgroups" />
      <property name="eclipselink.cache.coordination.jgroups.config" value="jgroups.xml" />
   </properties>

That alone is not enough as deploying my web app gives a java.lang.NoClassDefFoundError: org/jgroups/JChannel

So I assume that jgroups.jar is not included with Payara 5.2020.7

I then add JGroups 4.2.11.Final.jar in /glassfish/lib and try again. However now I get

 

Exception [EclipseLink-28019] (Eclipse Persistence Services - 2.7.7.payara-p3): org.eclipse.persistence.exceptions.EntityManagerSetupException

Exception Description: Deployment of PersistenceUnit [PictabiteMaster] failed. Close all factories for this PersistenceUnit.

Internal Exception: java.lang.NoSuchMethodError: org.jgroups.JChannel.connect(Ljava/lang/String;)V|#]

I contacted the JGroups folks and they are telling me that this is very likely a jar incompatibility. 

I'd like to confirm two things:

1) Do I really need to supply my own jgroups.jar or is there one included with Payara, and if so, then why do I get a java.lang.NoClassDefFoundError: org/jgroups/JChannel

2) If jgroups.jar is not included in Payara 5, then which version should I supply to fix the Exception: java.lang.NoSuchMethodError: org.jgroups.JChannel.connect(Ljava/lang/String;)V|#] error?

Btw:

Enabling Hazelcast as described here does not raise any errors, but also does not seem to perform any cache synchronization either - perhaps because my cluster is an AWS Elasticbeanstalk environment, so EC2 instances start and stop automatically depending on load (therefore the manual configuration described here does not make much sense )

Thank you!

--
You received this message because you are subscribed to the Google Groups "Payara Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to payara-forum...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/218fba4d-03d6-43eb-9c33-98b29990cc08n%40googlegroups.com.

[Hans Pikkemaat]

See my test project here

https://github.com/thehpi/cache-coordination

It uses docker so you probably cannot use it in AWS like this but you might get it working with the discovery plugin I mentioned.

good luck and let us know if (and how) you got it working :)

greetz, Hans

[Ondro Mihályi]

Hi Dimitri,

Are you sure that your GlassFish 4 installation contained a JGroups JAR? GlassFish 4 doesn't include JGroups so you probably added a JGroups JAR in the GlassFish 4 and then forgot about it. Payara Server also doesn't include JGroups JAR. Payara Server 5.2020.7 includes EclipseLink 2.7.7, which declares JGroups 4.1.0 as a n optional dependency, so I'd try adding that version to Payara Server.

However, I recommend you to give Hazelcast another try. As Hans confirms, it works out of the box in Payara Server. The only catch is that the Payara instances need to connect together in the same Data Grid. By default, the Data Grid uses the "domain" discovery, which assumes there's a single DAS and all Payara instances know its IP address. From what I know about AWS EB, you're probably spinning up one or more DAS servers. Therefore the "domain" discovery won't work and the DAS servers will create separate Data Grids. Hans switched to the "multicast" discovery for his example, but multicast doesn't work in AWS. You can either use the external Hazelcast AWS plugin, or use either the TCP/IP or DNS discovery supported by Payara Server out of the box: https://docs.payara.fish/community/docs/5.2020.7/documentation/payara-server/hazelcast/discovery.html

Using the DNS discovery is very simple, you'd just need to point Payara Server to a domain name, which would resolve to the IP of all Payara instances. But you'd also need to configure a DNS server and register all your EC2 instances created by AWS EB to it under the same DNS name, which is quite a lot of work. 

The TCP/IP discovery is more flexible but requires some scripting. You can store IP addresses of each Payara Instance into a shared S3 bucket or a DB and then read them into an environment variable. You will need to use a post boot commands file with a set-hazelcast-configuration command that can reference the environment variable. In Payara Docker, you can add the shell scripts into the ${SCRIPT_DIR} directory and place the post boot commands file to ${POSTBOOT_COMMANDS} , see the Payara Docker docs.


I hope that helps,

Ondro

so 6. 2. 2021 o 16:53 Hans Pikkemaat <the...@gmail.com> napísal(a):

To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/49C5456B-44AA-44CC-9BD9-31F8B1FDED80%40gmail.com.

[Dimitri]

Thanks everyone for the suggestions!

It turns out I had to supply a jgroups.3.x.jar - For some reason  jgroups 4.x and jgroups 5.x didn't work for me.

I'll stick with JGroups for now as it was quite complex to set it up and so far has worked well. I expect Hazelcast will likely be as complex to get it to work. The difficulty we had was getting nodes to discover each other dynamically. Manually maintaining a list of running EC2 instances (on a DNS server or elsewhere) would probably not work well in my case because EC2 instances are started and terminated automatically by AWS Beanstalk as load conditions fluctuate. JGroups offers some features whereby new nodes register themselves in an S3 bucket or DB, and also check for the registration of any pre-existing nodes, to discover cluster participants. It was a lot of work and scripting to get it to work, especially since my architecture is AWS EBS > EC2 > Multi-Container Docker > Payara > web app, and the deployment on AWS is via automated CI / CD scripts.

I may try Hazelcast in the future though as it has some nice features. I'll post here a working solution If I can get it to work.

Thanks everyone!

[Ondro Mihályi]

Hi Dimitri,

Thank you for sharing the information on how your clustering with JGroups works. When you mentioned you had JGroups working with EB it immediately caught my attention because I had a hard time setting Hazelcast clustering with Payara Server on AWS EB when I tried to do it some time ago. I suspected that for the same reasons it's not easy to set it up with JGroups and according to what you say I was right.

I also thought about using an S3 bucket to distribute the IP addresses of Payara instances so that the Payara's TCP joiner can use it to discover other instances and join them. It's nice to see that a similar solution works for you with JGroups. I'd like to spend some time to see how to do it with Payara Server and then how to add support for it in the official Payara Docker images to simplify it for you and anybody who would like to use Payara Server in AWS EB or another cloud environment.

All the best,

Ondro

so 6. 2. 2021 o 23:55 'Dimitri' via Payara Forum <payara...@googlegroups.com> napísal(a):

To view this discussion on the web, visit https://groups.google.com/d/msgid/payara-forum/30e48c02-7384-4c8b-9e44-9769c90ff965n%40googlegroups.com.

[Dimitri]

That would be great, thanks!