Wildfly Remote EJB in kubernetes clustered environment

[Peter Andreus]

Hi. 

Im working at progect that uses Remote EJB invocation in kubernetes environment.

I have 3 pods of one deployment that is Server side of EJB, which are connected to cluster via JGroups. Deployment is exposed to cluster via ClusterIP service.

Then there is one client, i want to call remote ejb on server.

Everything works fine, remote EJBs are called and so, BUT its always called only on one of three backends.

My jndi properties are:

jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory");
        jndiProperties.put(Context.PROVIDER_URL, actorConfig.getProvider()); //http-remoting://backend:80
        jndiProperties.put(Context.SECURITY_PRINCIPAL, actorConfig.getPrincipal());
        jndiProperties.put(Context.SECURITY_CREDENTIALS, actorConfig.getCredentials());

from Wildfly Naming Client (https://github.com/wildfly/wildfly-naming-client) i read:

....all consumers of the same remote URL will share a connection. The connection lifecycle is independent of any Context instances which reference it.

How should cluster support work then? Is there any client side approach to call EJBs on different server on cluster, or im missing some server side configuration? We are using WildFly 18 server (i know its old, but cannot upgrade higher)

Thanks for any suggestion

[Bartosz Baranowski]

I did not play with remoting for quite a bit, but Im tad confused or just miss some part here.

First of all you highlighted part about connection - how do you think it relates to cluster? AFAIR it indicates caching done in naming, essentially every call to the same URL - regardless of initiating party, will go through the same channel/connection.

As to kubernetes - Im almost sure PODs have distinct IP addresses( AFAIR Kubernetes is somewhat weird when it comes to notion of cluster/deployment/networking). So if you hit single, specific IP you will target one Pod?

https://kubernetes.io/docs/concepts/services-networking/service/

[Peter Andreus]

Well im targeting Service, not pod itself. So in normal behaviour Service decides which pod he access. 

You are correct, each pod has separate IP address, and when i target specific pod IP i reach remote EJB from given pod. But pods are ephemeral so i cannot rely on pod IP and thus must use Service.

And there is my problem (maybe misunderstanding how things works). Each time i call Service URL i should get random pod, not always same one.

[Richard Achmatowicz]

Test (please ignore).

[Richard Achmatowicz]

Hi Peter

There are a few points to clarify before continuing:

- how are you deploying your WildFly deployment into the k8s cluster (using a standard k8s Deployment definition or the Wildfly operator (https://github.com/wildfly/wildfly-operator))
- is your EJB client application internal to the Kubernetes cluster on which the deployment is deployed or external to the cluster (I assume internal as you mention using a ClusterIP service)

- which EJB client transport protocol is your client using to communicate with the deployment (remote, remote+http, http)?

Once these points are cleared up, we an look into this further.

The recommendation for internal clients is to use the "remote" or "remote+http" protocol combined with the WildFly operator (to provide persistence of Pod IPs and storage references (among other things).

Richard 

On Tuesday, May 16, 2023 at 6:02:55 AM UTC-4 Peter Andreus wrote:

[Peter Andreus]

Hi Richard,

Wildfly is deployed in standard way via Helm package, so simple Deployment definition

Yes, remote EJB is called from another (Springboot) app thats in same kubernetes/namespace.. etc..

As default we use remote+http and i tried also http-remoting protocol

[Richard Achmatowicz]

Hi Peter

Using remote+http with a cluster-internal service (representing the clustered deployment) will not result in load balanced requests to the cluster.
This is because the remote and remote+http protocols establish persistent connections from a client to a server, unlike http which will reconnect with the on each HTTP request.
So if your client invokes on the service using remote+http, that initial connection attempt will be directed to a (randomly chosen) backend server and ,once established, the
connection between the client and the server will persist and be available for subsequent invocations. Consequently, all requests from that client to that specific service will

reuse the same connection and you will see all requests going to the same backend server, which is what is happening.

If you want to make use of a cluster-internal EJB client application with a WildFly deployment, the recommendation is to configure your application client
as you would in a bare metal network: in your case, point your PROVIDER_URL  to at least one of the cluster member IPs. This will cause the first invocation to
target that specified server, a cluster topology update will be returned to the client with the IP addresses of the other Pods in the cluster, and load balancing and failover
will work as expected. The problem is that because you are in a k8s cluster, Pod IP addresses are not persistent, and if Pods are restarted for some reason, the IP addresses
that the client obtained from the topology update will become stale. Deploying your deployment using the Wildfly operator will cause the Pod IPs to be persistent (among other
benefits) and you won't have a problem with stale IPs being held by your client.

Richard 

--
You received this message because you are subscribed to the Google Groups "WildFly" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wildfly+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/4cf720e9-bdc4-4d9f-91bd-5dc3284b9c38n%40googlegroups.com.

[Peter Andreus]

Hi Richard.

Can you enlighten me the last section of your message? You say that i should target one of pods internal IP where wildfly server is, and then wildfly server will return cluster topology with IPs and client (Wildfly naming client?) will choose one of these IPs and create connection with given one?

Shouldn't it be working exactly like this with Kubernetes Service? I contact service, service redirects to one Pod, Pod should return cluster topology and client should connect directly to (maybe) different Pod.

Is there problem that Service is in the way of communication client -> Pod?

[Richard Achmatowicz]

Hi Peter

Comments inline...

On 5/17/23 08:11, Peter Andreus wrote:

Hi Richard.

Can you enlighten me the last section of your message? You say that i should target one of pods internal IP where wildfly server is, and then wildfly server will return cluster topology with IPs and client (Wildfly naming client?) will choose one of these IPs and create connection with given one?

Yes. This is the way the EJB client is conventionally used on bare metal. If you make use of the remote or remote+http protocols, the EJB client *itself* manages many of the clustering-related features, such as load balancing and fail-over. The process I described previously is part of the EJB client's discovery mechanism, which it uses to locate servers and deployments on servers. When you specify a PROVIDER_URL in your JNDI context, you are giving the EJB client a hint as to where your servers are located, a starting point to allow it to "discover" any others. So if you set the PROVIDER_URL to point to a cluster member IP address, it will find the others in the way that I mentioned. Why do we need this? Because on bare metal, users often did not have an intervening load balancer between the client and the cluster and there needed to be an easy way to discover the members of a cluster.

Shouldn't it be working exactly like this with Kubernetes Service? I contact service, service redirects to one Pod, Pod should return cluster topology and client should connect directly to (maybe) different Pod.

Is there problem that Service is in the way of communication client -> Pod?

Yes, it should. In order to use the EJB client with a load balancer (e.g. a k8s Service), you need to use the http protocol, which under the covers delegates load balancing and fail-over to a load balancer and does not make use of persistent connections, as I mentioned previously. The problem is that there is an issue with the EJB client http protocol working with K8s services and it is being addressed at the moment. Once it is fixed, the behaviour you expect will be available. But for now, the recommendation is to use the remote or remote+http protocol.

Richard

To view this discussion on the web visit https://groups.google.com/d/msgid/wildfly/89334cca-4ae4-4a18-83c6-9522ea422313n%40googlegroups.com.