Clustering Wildfly/Keycloak on Kubernetes

1,791 views
Skip to first unread message

msa...@redhat.com

unread,
Mar 5, 2018, 5:51:10 AM3/5/18
to jgroups-dev
Hello everyone,
I need to enable JGroup clustering feature with Kubernetes so that my Keycloak or wildfly instances can form a cluster when deployed on Kubernetes.
    As expained here [1], I will need to enable the KUBE_PING variable and the KUBE_NAMESPACE. 
    The KUBE_PING Protocol beeing in charge to ask to kubernetes for the list of Pods IP_ADRESSES.
    I'm looking for a working example that could help me doing it.
    I'm wondering where to set the KUBE_PING variable.
    [1]=https://github.com/jgroups-extras/jgroups-kubernetes

Meissa

Bela Ban

unread,
Mar 5, 2018, 9:02:59 AM3/5/18
to jgrou...@googlegroups.com
What do you mean by 'KUBE_PING variable'? KUBE_PING is a discovery
protocol that needs to be placed directly on top of a transport (such as
TCP), e.g.

<TCP..../>
<KUBE_PING.../>
<MERGE3.../>
...

You can take your existing TCP-based configuration and simply replace
TCPPING with KUBE_PING
> <https://github.com/jgroups-extras/jgroups-kubernetes>
>
> Meissa
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To post to this group, send email to jgrou...@googlegroups.com
> <mailto:jgrou...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Bela Ban | http://www.jgroups.org

Bela Ban

unread,
Mar 5, 2018, 9:05:51 AM3/5/18
to jgrou...@googlegroups.com
Below is the config I used last time with minikube. This requires the
JGroups and jgroups-kubernetes JARs to be on the classpath.



<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:org:jgroups"
xsi:schemaLocation="urn:org:jgroups
http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_addr="loopback,match-interface:en2,match-interface:en0"
sock_conn_timeout="500"
bind_port="7800"
recv_buf_size="${tcp.recv_buf_size:130k}"
send_buf_size="${tcp.send_buf_size:130k}"
max_bundle_size="64K"

thread_pool.enabled="true"
thread_pool.min_threads="0"
thread_pool.max_threads="500"
thread_pool.keep_alive_time="30000"/>

<org.jgroups.protocols.kubernetes.KUBE_PING
namespace="${kubernetes.namespace:default}"
label="cluster=nyc"
/>
<!--TCPPING initial_hosts="192.168.1.105[7800]" port_range="1"/-->

<!--org.jgroups.aws.s3.NATIVE_S3_PING
region_name="${S3_REGION:us-east-1}"
bucket_name="${S3_BUCKET:belaban}"
/-->

<MERGE3 min_interval="10000" max_interval="30000"/>
<FD_SOCK/>
<FD timeout="3000" max_tries="3" />
<VERIFY_SUSPECT timeout="1500" />
<BARRIER />
<pbcast.NAKACK2 use_mcast_xmit="false"
discard_delivered_msgs="true"/>
<UNICAST3 />
<pbcast.STABLE desired_avg_gossip="50000"
max_bytes="4M"/>
<pbcast.GMS print_local_addr="true" join_timeout="2000"
view_bundling="true"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG2 frag_size="60K" />
<pbcast.STATE_TRANSFER/>
</config>


On 05/03/18 11:51, msa...@redhat.com wrote:

msa...@redhat.com

unread,
Mar 5, 2018, 11:09:32 AM3/5/18
to jgroups-dev
I'm using the wildfly docker image that rely on wildfly 11 where the default stack is udp.
Below is an extract of my jgroup subsystem configuration.

<subsystem xmlns="urn:jboss:domain:jgroups:5.0">
            <channels default="ee">
                <channel name="ee" stack="udp" cluster="ejb"/>
            </channels>
            <stacks>
                <stack name="udp">
                    <transport type="UDP" socket-binding="jgroups-udp"/>
                    <protocol type="PING"/>
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG2"/>
                </stack>

               
If I understand what you're saying, I will have to add the KUBE_PING discovery protocol in replacement of the PING protocol?
Would this be enough? Do I have to change the default stack from udp to tcp and replace the TCPPING by KUBE_PING discovery protocol?
Where would I add jgroups-kubernetes jars since jgroups libraries are loaded as modules on wildfly?

thanks,
Meissa

Bela Ban

unread,
Mar 6, 2018, 3:18:41 AM3/6/18
to jgrou...@googlegroups.com


On 05/03/18 17:09, msa...@redhat.com wrote:
> I'm using the wildfly docker image that rely on wildfly 11 where the
> default stack is udp.

Do you have a URL to the image?

Using UDP won't work in a cloud where IP multicast is not available...

> Below is an extract of my jgroup subsystem configuration.
>
> <subsystem xmlns="urn:jboss:domain:jgroups:5.0">
>             <channels default="ee">
>                 <channel name="ee" stack="udp" cluster="ejb"/>
>             </channels>
>             <stacks>
>                 <stack name="udp">
>                     <transport type="UDP" socket-binding="jgroups-udp"/>
>                     <protocol type="PING"/>
>                     <protocol type="MERGE3"/>
>                     <protocol type="FD_SOCK"/>
>                     <protocol type="FD_ALL"/>
>                     <protocol type="VERIFY_SUSPECT"/>
>                     <protocol type="pbcast.NAKACK2"/>
>                     <protocol type="UNICAST3"/>
>                     <protocol type="pbcast.STABLE"/>
>                     <protocol type="pbcast.GMS"/>
>                     <protocol type="UFC"/>
>                     <protocol type="MFC"/>
>                     <protocol type="FRAG2"/>
>                 </stack>
>
>
> If I understand what you're saying, I will have to add the KUBE_PING
> discovery protocol in replacement of the PING protocol?

Yes, but I suggest start with the TCP config, not the UDP config

> Would this be enough? Do I have to change the default stack from udp to
> tcp and replace the TCPPING by KUBE_PING discovery protocol?

Yes

> Where would I add jgroups-kubernetes jars since jgroups libraries are
> loaded as modules on wildfly?

I think you need to add some descriptor under ./modules that points to a
local maven deployment, but better ask the Wildfly folks for details.
> > an email to jgroups-dev...@googlegroups.com <javascript:>
> > <mailto:jgroups-dev...@googlegroups.com <javascript:>>.
> > To post to this group, send email to jgrou...@googlegroups.com
> <javascript:>
> > <mailto:jgrou...@googlegroups.com <javascript:>>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
> >
> <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
> --
> Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To post to this group, send email to jgrou...@googlegroups.com
> <mailto:jgrou...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/74044df5-13bc-4976-945f-cef476626535%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/74044df5-13bc-4976-945f-cef476626535%40googlegroups.com?utm_medium=email&utm_source=footer>.

msa...@redhat.com

unread,
Mar 6, 2018, 4:29:26 AM3/6/18
to jgroups-dev
I've switched from wildfly to Keycloak which also run on wildfly 11. I't exactly the same purpose.
The keycloak image is below:

I've extended the image to run the standalone-ha.xml.
below is the content of my Dockerfile. very short. 

FROM jboss/keycloak:3.4.3.Final

CMD ["-b", "0.0.0.0", "--server-config", "standalone-ha.xml"]

multicast UDP is available in my cloud environment, but I can change the configuration to swtich to TCP.
The only thing I'm not sure about how to handle is the jgroups-kubernetes jars.
I think I will have to add them as modules librairies.
In Keycloak (or widlfly or even JBoss EAP7) the jgroups librairies is configured as a module in :

/KEYCLOAK_HOME/modules/system/layers/base/org/jboss/as/clustering/jgroups/main 
with the module.xml file 
<module xmlns="urn:jboss:module:1.5" name="org.jboss.as.clustering.jgroups">
    <properties>
        <property name="jboss.api" value="private"/>
    </properties>

    <exports>
        <exclude path="org/jboss/as/clustering/jgroups/logging"/>
    </exports>

    <resources>
        <resource-root path="wildfly-clustering-jgroups-extension-11.0.0.Final.jar"/>
    </resources>

    <dependencies>
        <module name="javax.api"/>
        <module name="org.jboss.as.clustering.common"/>
        <module name="org.jboss.as.connector"/>
        <module name="org.jboss.as.controller"/>
        <module name="org.jboss.as.naming"/>
        <module name="org.jboss.as.network"/>
        <module name="org.jboss.as.server"/>
        <module name="org.jboss.logging"/>
        <module name="org.jboss.marshalling"/>
        <module name="org.jboss.modules"/>
        <module name="org.jboss.msc"/>
        <module name="org.jboss.staxmapper"/>
        <module name="org.jgroups"/>
        <module name="org.wildfly.clustering.jgroups.spi"/>
        <module name="org.wildfly.clustering.service"/>
        <module name="org.wildfly.clustering.spi"/>
        <module name="org.wildfly.common"/>
        <module name="org.wildfly.security.elytron-private"/>
    </dependencies>
</module>

I'm thinking about creating a module named jgroups-kubernetes and adding the module dependencie in the dependencies above.
What do you think about that.

In Openshift, we use the OPENSHIFT_KUBE_PING_LABELS, OPENSHIFT_KUBE_PING_NAMESPACE environment variable for EAP pods to enable jgroup clustering. But behind the sceene I guess , jgroups-kubernetes librairies should have  been added.
Do you know about it?

Meissa
>      > <mailto:jgroups-dev+unsub...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com

Alan Field

unread,
Mar 6, 2018, 1:12:35 PM3/6/18
to msa...@redhat.com, jgroups-dev
Hey Meissa,

You might need to try and build your image from this PR:


Thanks,
Alan

>      > <mailto:jgroups-dev...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
--
You received this message because you are subscribed to the Google Groups "jgroups-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-dev...@googlegroups.com.
To post to this group, send email to jgrou...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jgroups-dev/062db7db-3ff2-4a5a-9f84-3ad9aa5abeff%40googlegroups.com.

Meissa M'baye Sakho

unread,
Mar 6, 2018, 2:04:59 PM3/6/18
to Alan Field, jgroups-dev
Hey Alan,
That's exactly what I was looking for. 
I will try it and let you know.
I was very surprised nobody seems to know what I was talking about.
I've made some post in our internal rhsso mailing list, talked to some keycloak developers in vain.

How does other folks do when they need to integrate Keycloak or wildfly in other cloud environment with kubernetes with jgroups clustering enabled?

Meissa

>      > <mailto:jgroups-dev+unsub...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com


--
You received this message because you are subscribed to the Google Groups "jgroups-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-dev+unsubscribe@googlegroups.com.

Alan Field

unread,
Mar 6, 2018, 2:13:43 PM3/6/18
to Meissa M'baye Sakho, jgroups-dev
Hey Meissa,

Ideally I think that the Wildfly image that Keycloak is based on should include the KUBE_PING JARs and a <stack> in the configuration that uses it, so that any projects layered on top can use it for clustering. I'll also have to check this in the Infinispan images.

Thanks,
Alan

>      > <mailto:jgroups-dev...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com


--
You received this message because you are subscribed to the Google Groups "jgroups-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-dev...@googlegroups.com.

Meissa M'baye Sakho

unread,
Mar 7, 2018, 4:03:37 AM3/7/18
to Alan Field, jgroups-dev, Baroux, Patrick, Thorgersen, Stian
Alan,
To be honest, I was thinking that the Wildlfy image or the Keycloak image included the KUBE_PING JARS as module and the stack in it's configuration. I discover that it's not the case.
I understand now why the blog post [1] on is not really working. I mean keycloak nodes are not forming a cluster on kubernetes.
I've added Stian a keycloak team member. He might be interested on our discussion.

I'm currently working for a customer who needs this feature on keycloak. We are deploying it in a cloud environment which is no Openshift (Apprenda).


Thanks,
Meissa


>      > <mailto:jgroups-dev+unsub...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com


--
You received this message because you are subscribed to the Google Groups "jgroups-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jgroups-dev+unsubscribe@googlegroups.com.

Meissa M'baye Sakho

unread,
Mar 7, 2018, 10:33:25 AM3/7/18
to Stian Thorgersen, Alan Field, jgroups-dev, Baroux, Patrick
The customer can't use rhsso because he is deploying keycloak on Apprenda which is an Openshift competitor.
We have accepted the engagment to help them doing so because we want to keep doing business with that customer.
I can undestand your point of view as as I undestand the same point of view of many of us. But we could not refuse to work with the customer because we need to stay around.
I also think that it's an opportunity for Keycloak because we will be able to challenge it's behavior in a cloud environnement other than OCP for free (the customer will not open case, or complain when something goes wrong).
One of my objectif is to deploy Keycloak with cluster enabled on apprenda.

Is there a chance that the pull request could be accepted?



On Wed, Mar 7, 2018 at 4:17 PM, Stian Thorgersen <stho...@redhat.com> wrote:
We don't have clustering enabled Keycloak images now, but there are PRs you could look. We don't support Keycloak though, so why is the customer not using RH-SSO xpaas images?

>      > <mailto:jgroups-dev+unsubscribe...@googlegroups.com <javascript:>>.
>      > To post to this group, send email to jgrou...@googlegroups.com
>     <javascript:>
>      > <mailto:jgrou...@googlegroups.com <javascript:>>.
>      > To view this discussion on the web visit
>      >
>     https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com>
>
>      >
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer
>     <https://groups.google.com/d/msgid/jgroups-dev/120f4ae5-d35d-4179-8f1c-bc08b2762300%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
>      > For more options, visit https://groups.google.com/d/optout
>     <https://groups.google.com/d/optout>.
>
>     --
>     Bela Ban | http://www.jgroups.org
>
> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages