FILE_PING - large number of stale entries during continuous failure

[Vikrant Singh]

Hi,

I am running keycloak 4.3.1 using JGroups 3.6.13.Final on OpenShift platform. I have 3 keycloak nodes running in clustered mode and FILE_PING is used as discovery protocol, below is configuration.

<subsystem xmlns="urn:jboss:domain:jgroups:5.0">

<channels default="ee">

<channel name="ee" stack="file_ping"/>

</channels>

<stacks>

<stack name="file_ping">

<transport type="TCP" socket-binding="jgroups-tcp" site="${env.DATA_CENTER}">

<property name="external_addr">

${jgroups.tcp.external_addr:}

</property>

</transport>

<protocol type="FILE_PING">

<property name="location">

/data

</property>

<property name="remove_all_files_on_view_change">

true

</property>

<property name="remove_old_coords_on_view_change">

true

</property>

</protocol>

<protocol type="MERGE3"/>

<protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>

<protocol type="FD_ALL"/>

<protocol type="VERIFY_SUSPECT"/>

<protocol type="pbcast.NAKACK2"/>

<protocol type="UNICAST3"/>

<protocol type="pbcast.STABLE"/>

<protocol type="pbcast.GMS"/>

<protocol type="MFC"/>

<protocol type="FRAG2"/>

</stack>

</stacks>

</subsystem>

This configuration works well in normal situations but when there is continuous failure during keycloak startup e.g. backend DB is down and openshift keeps killing unhealthy keycloak nodes and brings new one, if DB is down for couple of hours then there will be a huge number of stale entries(in thousands) in file_ping file resulting in continuous keycloak failure even if DB comes online since during startup keycloak tries to find each member stores in file_ping file.

Is there anyway to avoid having hundreds of stale entries during continuous startup failure?

[Bela Ban]

I'm not sure I understand the issue you're having. First off, why do you
use FILE_PING? For Openshift, without (AFAIUI) persistent memory, I'd
prefer either KUBE_PING or DNS_PING...

Re the many entries in the file: have you adjusted
logical_addr_cache_max_size / logical_addr_cache_expiration /
logical_addr_cache_reaper_interval in TCP?

On 05/03/18 12:52, Vikrant Singh wrote:
> Hi,
>

> I am running keycloak 4.3.1 using JGroups 3.6.13.Finalon OpenShift

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To post to this group, send email to jgrou...@googlegroups.com
> <mailto:jgrou...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jgroups-dev/3b4748cd-8566-4393-b7c4-23e0fc520592%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/3b4748cd-8566-4393-b7c4-23e0fc520592%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

--
Bela Ban | http://www.jgroups.org

[Vikrant Singh]

Thanks for reply.

Keycloak is running on wildfly 11 which comes with JGroups 3.x. Latest kube_ping 1.x doesn't support Jgroups 3.x and kube_ping 0.9 have an memory leak issue, also dns_ping seems to have introduces in jgroups 4.x that is why we are using file_ping instead of kube_ping/dns_ping.

We have a persistent mount to store file_ping file.

Let me try if "logical_addr_cache_max_size / logical_addr_cache_expiration / 
logical_addr_cache_reaper_interval" helps in my scenario.

-Vikrant

[Vikrant Singh]

Below is the configuration I tried

<subsystem xmlns="urn:jboss:domain:jgroups:5.0">

<channels default="ee">

<channel name="ee" stack="file_ping" statistics-enabled="true"/>

</channels>

<stacks>

<stack name="file_ping">

<transport type="TCP" socket-binding="jgroups-tcp" site="${env.DATA_CENTER}">

<property name="external_addr">

${jgroups.tcp.external_addr:}

</property>

<property name="logical_addr_cache_expiration">

10

</property>

<property name="logical_addr_cache_max_size">

10

</property>

<property name="logical_addr_cache_reaper_interval">

10

But still same result, entries in file_ping file still grows beyond 10 and doesn't reduces even after keycloak is running for couple of hours. Any insight or suggestion would be really helpful

[Bela Ban]

I see there was a related issue [1] which was fixed in 4.0.10.

Is there a more recent version of keycloak/wildfly which uses JGroups
4.0.x? If so, I suggest use it and replace the JGroups version it ships
with with 4.0.10.Final, to see if the issue disappears.

[1] https://issues.jboss.org/browse/JGRP-2232

> --
> You received this message because you are subscribed to the Google
> Groups "jgroups-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jgroups-dev...@googlegroups.com
> <mailto:jgroups-dev...@googlegroups.com>.
> To post to this group, send email to jgrou...@googlegroups.com
> <mailto:jgrou...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jgroups-dev/9e8ecf90-f84f-42be-86bc-812f0456a254%40googlegroups.com
> <https://groups.google.com/d/msgid/jgroups-dev/9e8ecf90-f84f-42be-86bc-812f0456a254%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Vikrant Singh]

Unfortunately no :( 

we are on latest keycloak version which uses 3.6.x and I don't see any near future plan where keycloak is being upgraded to use wildfly 12.

[Vikrant Singh]

Is there any plan to back-port JGRP-2232 fix to JGroups 3.x? Is it possible to include JDRP-2232 fixes directly to JGroups 3.x code base. Or do you have any other suggestions to avoid ever growing stale entries in file and keep using JGroups 3.x

Thanks,

Vikrant

> <mailto:jgroups-dev+unsub...@googlegroups.com>.