Keycloak 11.0.2 cann't expose cache statistics
715 views
Skip to first unread message
Ilya Benderskiy
Oct 15, 2020, 1:52:34 AM10/15/20
to Keycloak User
Hallo,
I have problems wirh exposing cache statistics (datasource, jgroups statistics are exposed without problems). My configuration:
<subsystem xmlns="urn:jboss:domain:infinispan:10.0">
<cache-container name="keycloak" module="org.keycloak.keycloak-model-infinispan" statistics-enabled="true">
<transport lock-timeout="60000"/>
<local-cache name="realms">
<object-memory size="10000"/>
</local-cache>
<local-cache name="users">
<object-memory size="10000"/>
</local-cache>
<local-cache name="authorization">
<object-memory size="10000"/>
</local-cache>
<local-cache name="keys">
<object-memory size="1000"/>
<expiration max-idle="3600000"/>
</local-cache>
<replicated-cache name="work" statistics-enabled="true"/>
<distributed-cache name="sessions" statistics-enabled="true" owners="3"/>
<distributed-cache name="authenticationSessions" statistics-enabled="true" owners="3"/>
<distributed-cache name="offlineSessions" statistics-enabled="true" owners="3"/>
<distributed-cache name="clientSessions" statistics-enabled="true" owners="3"/>
<distributed-cache name="offlineClientSessions" statistics-enabled="true" owners="3"/>
<distributed-cache name="loginFailures" statistics-enabled="true" owners="1"/>
<distributed-cache name="actionTokens" statistics-enabled="true" owners="3">
<object-memory size="-1"/>
<expiration interval="300000" max-idle="-1"/>
</distributed-cache>
</cache-container>
....
Any suggestions?
Regards,
Ilya.
Benjamin Weimer
Dec 17, 2020, 3:06:37 AM12/17/20
to Keycloak User
Hi @keycloak-user,
we have the same issue as described in https://groups.google.com/g/keycloak-user/c/hrFLykT4FdY/m/iTdCy6xGAQAJ
After upgrading from Keycloak 10 (Wildfly 19) to Keycloak 11 (Wildfly 20) we do not get any infinispan metrics anymore with a configuration like this:
<cache-container name="keycloak" module="org.keycloak.keycloak-model-infinispan">
<local-cache name="realms" statistics-enabled="true">
<object-memory size="10000"/>
</local-cache>
… </local-cache>
</cache-container>
</subsystem>
Are there any known issues about this? Or do have have to adapt the configuration somehow for Keycloak 11? This configuration works fine with Keycloak 10, infinispan subsystem version is 9.0 in Keycloak 10.
Also found another issue about this on stackoverflow: https://stackoverflow.com/questions/63136969/how-to-enable-infinispan-smallrye-metrics-in-wildfly-20
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer
Dec 17, 2020, 3:49:04 AM12/17/20
to Keycloak User
Hi @keycloak-user,
may this be the issue? https://issues.redhat.com/browse/WFLY-14063
If that's the case it seems that the fix is provided in Wildfly 22?
Mit
freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer
Message has been deleted
Benjamin Weimer
Dec 17, 2020, 3:52:17 AM12/17/20
to Keycloak User
Hi @keycloak-user,
Keycloak 12 uses Wildfly 21, so I expect this is still an issue with Keycloak 12 and will most likely be fixed in Keycloak 13?
Mit
freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer
Todor Kazakov
Dec 17, 2020, 11:03:00 AM12/17/20
to Keycloak User
You can use this jmx metrics exporter https://github.com/tidepool-org/keycloak-extensions/tree/master/jmx-metrics
This is an example configuration file:
---
startDelaySeconds: 30
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
- "jboss.as:subsystem=infinispan,cache-container=keycloak,distributed-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,replicated-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,local-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,cache=*"
rules:
- pattern: "^jboss.as<subsystem=infinispan, cache-container=keycloak, (distributed|replicated|local)?-?cache=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_infinispan_cache_$3
labels:
type: $1
name: $2
- pattern: "^jboss.as<subsystem=infinispan, statistics=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_infinispan_$1_$2
startDelaySeconds: 30
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
- "jboss.as:subsystem=infinispan,cache-container=keycloak,distributed-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,replicated-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,local-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,cache=*"
rules:
- pattern: "^jboss.as<subsystem=infinispan, cache-container=keycloak, (distributed|replicated|local)?-?cache=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_infinispan_cache_$3
labels:
type: $1
name: $2
- pattern: "^jboss.as<subsystem=infinispan, statistics=(.+)><>(.+):"
attrNameSnakeCase: true
name: wildfly_infinispan_$1_$2
Matysiak Joerg (IOC/PAU1)
Jan 8, 2021, 2:39:03 AM1/8/21
to Todor Kazakov, Keycloak User
Unfortunatly that doesn’t help since the MBean “org.wildfly.clustering.infinispan” is no longer available from keycloak 11 on.
Is there a configuration setting to get it back?
(Issue also exists in the official image from quay.io: in version 10 the MBean is available, in versions 11 and 12 it’s gone.)
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen |
GERMANY | www.bosch.io
Tel. +49 711 811-58484 | Telefax +49 711 811-58200 |
Threema / Threema Work: XB9B9CXU |
Joerg.M...@bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
keycloak-use...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/keycloak-user/5ca7131d-4af2-423f-a429-c97f289741d6n%40googlegroups.com.
Thomas Darimont
Jan 10, 2021, 8:13:20 PM1/10/21
to Matysiak Joerg (IOC/PAU1), Todor Kazakov, Keycloak User
Hello,
This is the code in wildfly 21.0.1.Final (WildFly Core 13.0.3.Final) which is used by Keycloak 12.0.1:
https://github.com/wildfly/wildfly/blob/21.0.1.Final/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java
https://github.com/wildfly/wildfly/blob/a2378b501858ee590669c685d61680b902e03c09/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java#L134
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
This should be:
```
For comparison in 19.1.0.Final (WildFly Core 11.1.1.Final) as used by Keycloak 10.0.1.
https://github.com/wildfly/wildfly/blob/19.1.0.Final/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java#L154
```
builder.globalJmxStatistics()
.enabled(this.statisticsEnabled) // <---- this was not ported!
.cacheManagerName(this.name)
.mBeanServerLookup(new MBeanServerProvider((this.server != null) ? this.server.get() : null))
.jmxDomain("org.wildfly.clustering.infinispan")
.allowDuplicateDomains(true);
I think I found the reason why the mbean is not there anymore.
It seems that in wildfly 21.0.1 the org.wildfly.clustering.infinispan is no longer exposed, because enabled wildfly statistics are not propagated to the CacheContiner JMX component.
This is the code in wildfly 21.0.1.Final (WildFly Core 13.0.3.Final) which is used by Keycloak 12.0.1:
https://github.com/wildfly/wildfly/blob/21.0.1.Final/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java
https://github.com/wildfly/wildfly/blob/a2378b501858ee590669c685d61680b902e03c09/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java#L134
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
This should be:
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
.enabled(this.statisticsEnabled) // this is IMHO missing here!!!
;
```
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
.enabled(this.statisticsEnabled) // this is IMHO missing here!!!
;
```
As I could not find a call to enabled(...) in the wildfly / wildfly-core project, I think this is an oversight when the infinispan code was changed in wildfly and should be fixed.
For comparison in 19.1.0.Final (WildFly Core 11.1.1.Final) as used by Keycloak 10.0.1.
https://github.com/wildfly/wildfly/blob/19.1.0.Final/clustering/infinispan/extension/src/main/java/org/jboss/as/clustering/infinispan/subsystem/GlobalConfigurationServiceConfigurator.java#L154
```
builder.globalJmxStatistics()
.enabled(this.statisticsEnabled) // <---- this was not ported!
.cacheManagerName(this.name)
.mBeanServerLookup(new MBeanServerProvider((this.server != null) ? this.server.get() : null))
.jmxDomain("org.wildfly.clustering.infinispan")
.allowDuplicateDomains(true);
```
If I set a breakpoint in the call to mBeanServerLookup(..) and invoke this.enabled(true), then I can see the mbean in visualvm.
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
See:
Cheers,
Thomas
To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/AM0PR10MB192321AC19FD9511058BFB8BEDAE0%40AM0PR10MB1923.EURPRD10.PROD.OUTLOOK.COM.
Jan Lieskovsky
Jan 11, 2021, 4:38:37 AM1/11/21
to Thomas Darimont, Matysiak Joerg (IOC/PAU1), Todor Kazakov, Keycloak User, Paul Ferraro
Hello Thomas,
thank you for your research / analysis!
1) Cc-ing Paul for his awareness / possibility to comment on this -- @Paul Ferraro PTAL at the issue below
2) @Thomas Darimont Since the issue is still present in latest Wildfly master code too, did you possibly
report corresponding upstream WFLY issue yet? (so it can be corrected in some next Wildfly version,
possibly even in 22.0.1.Final yet)
Thank you && Regards,
Jan
--
Don't see the problems. See their solutions!To view this discussion on the web visit https://groups.google.com/d/msgid/keycloak-user/CAK-7U1i%3DLV20tscN9RAt8NaxyBg5tYwomJM%3DzzdANsUUD1XRGA%40mail.gmail.com.
Todor Kazakov
Jan 11, 2021, 4:44:30 AM1/11/21
to Keycloak User
Hi Thomas,
I think you're looking at the wrong mbean - "jboss.as.infinispan" is available in keycloak 12 and works with the metrics exporter and config that I provided earlier in the thread.
Best,
Todd
Matysiak Joerg (IOC/PAU1)
Jan 11, 2021, 4:52:52 AM1/11/21
to Keycloak User
Hi all,
I think there is already a ticket for that and even a solution for Wildfly 22, see these links
https://issues.redhat.com/browse/WFLY-14063
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen |
GERMANY | www.bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Matysiak Joerg (IOC/PAU1)
Jan 12, 2021, 5:45:50 AM1/12/21
to Keycloak User
Hi all,
thank you very much for your support on this.
Since we’re currently on the way to solve the issue in our environment I’d like to summarize and share our results:
There are two kind of metrics that share statistics about infinispan: JMX and smallrye metrics.
Unfortunately none of them is working out of the box in Keycloak 11.
Smallrye metrics are broken because of the issue addressed in https://issues.redhat.com/browse/WFLY-14063.
Replacing the related jar file with a patched version makes infinispan metrics available as smallrye metrics on the wildfly management interface.
To fix JMX metrics the code change suggest by Thomas Darimont is required.
Replacing the jar file in the wildfly installation with a patched version makes the infinispan statistics mbeans available again.
We've tested this with Keycloak 11 based on the images provided by quay.io.
Since the wildfly bugs haven't been fixed in the version used by Keycloak 12, it's likely that the same patches need to be apply there.
However, according to Todd the MBean implementation has probably changed with Keycloak 12 and different solutions may be required for this version.
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen | GERMANY | www.bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
2) mailto:thomas....@googlemail.com Since the issue is still present in latest Wildfly master code too, did you possibly
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwildfly%2Fwildfly%2Fblob%2Fa2378b501858ee590669c685d61680b902e03c09%2Fclustering%2Finfinispan%2Fextension%2Fsrc%2Fmain%2Fjava%2Forg%2Fjboss%2Fas%2Fclustering%2Finfinispan%2Fsubsystem%2FGlobalConfigurationServiceConfigurator.java%23L134&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767628164%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6XD7pALq3YgzNqbumtcP4Ff4lS4hpRddZoNiNuSE6hE%3D&reserved=0
```
```
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen | GERMANY | https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bosch.io%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767648151%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DuthHoXBxXxgfYOhecpbKk5YcRouCFVEKD7QvzFOMYs%3D&reserved=0
Tel. +49 711 811-58484 | Telefax +49 711 811-58200 | Threema / Threema Work: XB9B9CXU | mailto:Joerg.M...@bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Von: mailto:keyclo...@googlegroups.com <mailto:keyclo...@googlegroups.com> Im Auftrag von Todor Kazakov
This is an example configuration file:
---
startDelaySeconds: 30
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
- "jboss.as:subsystem=infinispan,cache-container=keycloak,distributed-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,replicated-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,local-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,cache=*"
rules:
- pattern: "^https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fjboss.as%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767658144%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DObXByNbRi3yIJwAvb0I0si235Z1iWuXlRQSrlm6Ikw%3D&reserved=0<subsystem=infinispan, cache-container=keycloak, (distributed|replicated|local)?-?cache=(.+)><>(.+):"
If that's the case it seems that the fix is provided in Wildfly 22?
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer schrieb am Donnerstag, 17. Dezember 2020 um 09:06:37 UTC+1:
Hi @keycloak-user,
we have the same issue as described in https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fg%2Fkeycloak-user%2Fc%2FhrFLykT4FdY%2Fm%2FiTdCy6xGAQAJ&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767678129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=bjtChxQyoFVaQqruGxTnUNmuJHm3%2FNZWIsrI4NyMMqU%3D&reserved=0
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
To unsubscribe from this group and stop receiving emails from it, send an email to mailto:keycloak-use...@googlegroups.com.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2F5ca7131d-4af2-423f-a429-c97f289741d6n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767688125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SA5BIJf456lfEErs4aParENmhCm8uZIzXya5%2FlEZjE4%3D&reserved=0.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FAM0PR10MB192321AC19FD9511058BFB8BEDAE0%2540AM0PR10MB1923.EURPRD10.PROD.OUTLOOK.COM%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767698119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2FmiJwIOdvXn0Btc8OsTVjkruCAkDUEI8UP2V7wqZjc%3D&reserved=0.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FCAK-7U1i%253DLV20tscN9RAt8NaxyBg5tYwomJM%253DzzdANsUUD1XRGA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767698119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gZRC%2Bq3MlGFpFUXHsZf5LV%2BKg%2B5rEw1VDw2jYbFzw9M%3D&reserved=0.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FAM0PR10MB19234950F6BC3DB76493F297EDAB0%2540AM0PR10MB1923.EURPRD10.PROD.OUTLOOK.COM%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767708111%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DM0F76Ol0bYOpT%2BYVhLlhWEarDHcKolCztkThcUl%2B0w%3D&reserved=0.
thank you very much for your support on this.
Since we’re currently on the way to solve the issue in our environment I’d like to summarize and share our results:
There are two kind of metrics that share statistics about infinispan: JMX and smallrye metrics.
Unfortunately none of them is working out of the box in Keycloak 11.
Smallrye metrics are broken because of the issue addressed in https://issues.redhat.com/browse/WFLY-14063.
Replacing the related jar file with a patched version makes infinispan metrics available as smallrye metrics on the wildfly management interface.
To fix JMX metrics the code change suggest by Thomas Darimont is required.
Replacing the jar file in the wildfly installation with a patched version makes the infinispan statistics mbeans available again.
We've tested this with Keycloak 11 based on the images provided by quay.io.
Since the wildfly bugs haven't been fixed in the version used by Keycloak 12, it's likely that the same patches need to be apply there.
However, according to Todd the MBean implementation has probably changed with Keycloak 12 and different solutions may be required for this version.
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen | GERMANY | www.bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Von: 'Matysiak Joerg (IOC/PAU1)' via Keycloak User <keyclo...@googlegroups.com>
Gesendet: Montag, 11. Januar 2021 10:53
An: Keycloak User <keyclo...@googlegroups.com>
Betreff: AW: [keycloak-user] Re: Keycloak 11.0.2 cann't expose cache statistics
Hi all,
I think there is already a ticket for that and even a solution for Wildfly 22, see these links
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.redhat.com%2Fbrowse%2FWFLY-14063&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767608170%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hWDTvNJlbtyxh0QS%2BSWQozu3wO9WM77IameCUghLZ64%3D&reserved=0
Gesendet: Montag, 11. Januar 2021 10:53
An: Keycloak User <keyclo...@googlegroups.com>
Betreff: AW: [keycloak-user] Re: Keycloak 11.0.2 cann't expose cache statistics
Hi all,
I think there is already a ticket for that and even a solution for Wildfly 22, see these links
https://stackoverflow.com/questions/63136969/how-to-enable-infinispan-smallrye-metrics-in-wildfly-20/65406403#65406403
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Bosch.IO GmbH | Stuttgarter Straße 130 | 71332 Waiblingen | GERMANY | https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.bosch.io%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767618169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=oYVu4hIsUbCPXZf83075fYATRgR78gA%2Bxb2VnkzdQBo%3D&reserved=0
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Von: Jan Lieskovsky <mailto:jlie...@redhat.com>
Gesendet: Montag, 11. Januar 2021 10:38
An: Thomas Darimont <mailto:thomas....@googlemail.com>
Cc: Matysiak Joerg (IOC/PAU1) <mailto:Joerg.M...@bosch.io>; Todor Kazakov <mailto:todd.k...@gmail.com>; Keycloak User <mailto:keyclo...@googlegroups.com>; Paul Ferraro <mailto:pfer...@redhat.com>
Betreff: Re: [keycloak-user] Re: Keycloak 11.0.2 cann't expose cache statistics
Hello Thomas,
thank you for your research / analysis!
1) Cc-ing Paul for his awareness / possibility to comment on this -- mailto:pfer...@redhat.com PTAL at the issue below
Gesendet: Montag, 11. Januar 2021 10:38
An: Thomas Darimont <mailto:thomas....@googlemail.com>
Cc: Matysiak Joerg (IOC/PAU1) <mailto:Joerg.M...@bosch.io>; Todor Kazakov <mailto:todd.k...@gmail.com>; Keycloak User <mailto:keyclo...@googlegroups.com>; Paul Ferraro <mailto:pfer...@redhat.com>
Betreff: Re: [keycloak-user] Re: Keycloak 11.0.2 cann't expose cache statistics
Hello Thomas,
thank you for your research / analysis!
2) mailto:thomas....@googlemail.com Since the issue is still present in latest Wildfly master code too, did you possibly
report corresponding upstream WFLY issue yet? (so it can be corrected in some next Wildfly version,
possibly even in 22.0.1.Final yet)
possibly even in 22.0.1.Final yet)
On Mon, Jan 11, 2021 at 2:13 AM 'Thomas Darimont' via Keycloak User <mailto:keyclo...@googlegroups.com> wrote:
Hello,
I think I found the reason why the mbean is not there anymore.
It seems that in wildfly 21.0.1 the org.wildfly.clustering.infinispan is no longer exposed, because enabled wildfly statistics are not propagated to the CacheContiner JMX component.
This is the code in wildfly 21.0.1.Final (WildFly Core 13.0.3.Final) which is used by Keycloak 12.0.1:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwildfly%2Fwildfly%2Fblob%2F21.0.1.Final%2Fclustering%2Finfinispan%2Fextension%2Fsrc%2Fmain%2Fjava%2Forg%2Fjboss%2Fas%2Fclustering%2Finfinispan%2Fsubsystem%2FGlobalConfigurationServiceConfigurator.java&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767618169%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=jfJn7H2CEkftjLvCtGKB%2BljGzOB%2FxhD1vQ8YGJiOpCk%3D&reserved=0
Hello,
I think I found the reason why the mbean is not there anymore.
It seems that in wildfly 21.0.1 the org.wildfly.clustering.infinispan is no longer exposed, because enabled wildfly statistics are not propagated to the CacheContiner JMX component.
This is the code in wildfly 21.0.1.Final (WildFly Core 13.0.3.Final) which is used by Keycloak 12.0.1:
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwildfly%2Fwildfly%2Fblob%2Fa2378b501858ee590669c685d61680b902e03c09%2Fclustering%2Finfinispan%2Fextension%2Fsrc%2Fmain%2Fjava%2Forg%2Fjboss%2Fas%2Fclustering%2Finfinispan%2Fsubsystem%2FGlobalConfigurationServiceConfigurator.java%23L134&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767628164%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=6XD7pALq3YgzNqbumtcP4Ff4lS4hpRddZoNiNuSE6hE%3D&reserved=0
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
This should be:
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
.enabled(this.statisticsEnabled) // this is IMHO missing here!!!
;
```
As I could not find a call to enabled(...) in the wildfly / wildfly-core project, I think this is an oversight when the infinispan code was changed in wildfly and should be fixed.
For comparison in 19.1.0.Final (WildFly Core 11.1.1.Final) as used by Keycloak 10.0.1.
https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fwildfly%2Fwildfly%2Fblob%2F19.1.0.Final%2Fclustering%2Finfinispan%2Fextension%2Fsrc%2Fmain%2Fjava%2Forg%2Fjboss%2Fas%2Fclustering%2Finfinispan%2Fsubsystem%2FGlobalConfigurationServiceConfigurator.java%23L154&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767638152%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=EAHHBVVVFX%2BNV9PcPJPCCi3gZrOdJihKLDS0umfUjfk%3D&reserved=0
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
This should be:
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
.enabled(this.statisticsEnabled) // this is IMHO missing here!!!
;
```
As I could not find a call to enabled(...) in the wildfly / wildfly-core project, I think this is an oversight when the infinispan code was changed in wildfly and should be fixed.
For comparison in 19.1.0.Final (WildFly Core 11.1.1.Final) as used by Keycloak 10.0.1.
```
builder.globalJmxStatistics()
.enabled(this.statisticsEnabled) // <---- this was not ported!
.cacheManagerName(https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fthis.name%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767638152%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=iVitINDhHphFbifTCzr295uMAaOmiI7KuSEccv5FuXA%3D&reserved=0)
.enabled(this.statisticsEnabled) // <---- this was not ported!
.mBeanServerLookup(new MBeanServerProvider((this.server != null) ? this.server.get() : null))
.jmxDomain("org.wildfly.clustering.infinispan")
.allowDuplicateDomains(true);
```
If I set a breakpoint in the call to mBeanServerLookup(..) and invoke this.enabled(true), then I can see the mbean in visualvm.
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
See:
.jmxDomain("org.wildfly.clustering.infinispan")
.allowDuplicateDomains(true);
```
If I set a breakpoint in the call to mBeanServerLookup(..) and invoke this.enabled(true), then I can see the mbean in visualvm.
```
builder.jmx().domain("org.wildfly.clustering.infinispan")
.mBeanServerLookup(new MBeanServerProvider((this.server != null) && this.statisticsEnabled ? this.server.get() : null))
;
```
See:
Cheers,
Thomas
Thank you && Regards,
Jan
--
Don't see the problems. See their solutions!
On Fri, 8 Jan 2021 at 08:39, 'Matysiak Joerg (IOC/PAU1)' via Keycloak User <mailto:keyclo...@googlegroups.com> wrote:
Unfortunatly that doesn’t help since the MBean “org.wildfly.clustering.infinispan” is no longer available from keycloak 11 on.
Is there a configuration setting to get it back?
(Issue also exists in the official image from https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fquay.io%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767648151%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dGQy3Llq42zatRBPgGmAoPC64ZrNxEteBce3wBYcj%2BA%3D&reserved=0: in version 10 the MBean is available, in versions 11 and 12 it’s gone.)
Thomas
Thank you && Regards,
Jan
--
Don't see the problems. See their solutions!
On Fri, 8 Jan 2021 at 08:39, 'Matysiak Joerg (IOC/PAU1)' via Keycloak User <mailto:keyclo...@googlegroups.com> wrote:
Unfortunatly that doesn’t help since the MBean “org.wildfly.clustering.infinispan” is no longer available from keycloak 11 on.
Is there a configuration setting to get it back?
Mit freundlichen Grüßen / Best regards
Jörg Matysiak
Product Area User Management (IOC/PAU)
Tel. +49 711 811-58484 | Telefax +49 711 811-58200 | Threema / Threema Work: XB9B9CXU | mailto:Joerg.M...@bosch.io
Sitz: Berlin, Registergericht: Amtsgericht Charlottenburg; HRB 148411 B
Aufsichtsratsvorsitzender: Dr.-Ing. Thorsten Lücke; Geschäftsführung: Dr. Stefan Ferber, Dr. Aleksandar Mitrovic, Yvonne Reckling
Gesendet: Donnerstag, 17. Dezember 2020 17:03
An: Keycloak User <mailto:keyclo...@googlegroups.com>
Betreff: [keycloak-user] Re: Keycloak 11.0.2 cann't expose cache statistics
You can use this jmx metrics exporter https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftidepool-org%2Fkeycloak-extensions%2Ftree%2Fmaster%2Fjmx-metrics&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767658144%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pryCwhXQbaZ1Y7YL%2Fkfu%2Fp0uY2BidOsRAo7gVTf6%2Btc%3D&reserved=0
This is an example configuration file:
---
startDelaySeconds: 30
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
- "jboss.as:subsystem=infinispan,cache-container=keycloak,distributed-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,replicated-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,local-cache=*"
- "jboss.as:subsystem=infinispan,cache-container=keycloak,cache=*"
rules:
attrNameSnakeCase: true
name: wildfly_infinispan_cache_$3
labels:
type: $1
name: $2
- pattern: "^https://eur03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fjboss.as%2F&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767668138%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=w%2FBdVshghUweBA5QtrxW1QBHJSS4bl1D24tTtPJMS2s%3D&reserved=0<subsystem=infinispan, statistics=(.+)><>(.+):"
name: wildfly_infinispan_cache_$3
labels:
type: $1
name: $2
attrNameSnakeCase: true
name: wildfly_infinispan_$1_$2
On Thursday, 17 December 2020 at 10:52:17 UTC+2 Benjamin Weimer wrote:
Hi @keycloak-user,
Keycloak 12 uses Wildfly 21, so I expect this is still an issue with Keycloak 12 and will most likely be fixed in Keycloak 13?
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer schrieb am Donnerstag, 17. Dezember 2020 um 09:49:04 UTC+1:
Hi @keycloak-user,
may this be the issue? https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.redhat.com%2Fbrowse%2FWFLY-14063&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767678129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=j9Ge2S7pvdwF2CdEHuVXG8Z5cZ62u%2F%2F%2BqUCFbSd24h4%3D&reserved=0
name: wildfly_infinispan_$1_$2
On Thursday, 17 December 2020 at 10:52:17 UTC+2 Benjamin Weimer wrote:
Hi @keycloak-user,
Keycloak 12 uses Wildfly 21, so I expect this is still an issue with Keycloak 12 and will most likely be fixed in Keycloak 13?
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer schrieb am Donnerstag, 17. Dezember 2020 um 09:49:04 UTC+1:
Hi @keycloak-user,
If that's the case it seems that the fix is provided in Wildfly 22?
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
Benjamin Weimer schrieb am Donnerstag, 17. Dezember 2020 um 09:06:37 UTC+1:
Hi @keycloak-user,
After upgrading from Keycloak 10 (Wildfly 19) to Keycloak 11 (Wildfly 20) we do not get any infinispan metrics anymore with a configuration like this:
<subsystem xmlns="urn:jboss:domain:infinispan:10.0">
<cache-container name="keycloak" module="org.keycloak.keycloak-model-infinispan">
<transport lock-timeout="60000"/>
<local-cache name="realms" statistics-enabled="true">
<object-memory size="10000"/>
</local-cache>
…
</cache-container>
</subsystem>
Are there any known issues about this? Or do have have to adapt the configuration somehow for Keycloak 11? This configuration works fine with Keycloak 10, infinispan subsystem version is 9.0 in Keycloak 10.
Also found another issue about this on stackoverflow: https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fquestions%2F63136969%2Fhow-to-enable-infinispan-smallrye-metrics-in-wildfly-20&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767688125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=D0Qjxjy7f2nwZV7o2w3W1MBizUtIF8N1kIVFPIeYFBk%3D&reserved=0
<subsystem xmlns="urn:jboss:domain:infinispan:10.0">
<cache-container name="keycloak" module="org.keycloak.keycloak-model-infinispan">
<transport lock-timeout="60000"/>
<local-cache name="realms" statistics-enabled="true">
<object-memory size="10000"/>
</local-cache>
…
</cache-container>
</subsystem>
Are there any known issues about this? Or do have have to adapt the configuration somehow for Keycloak 11? This configuration works fine with Keycloak 10, infinispan subsystem version is 9.0 in Keycloak 10.
Mit freundlichen Grüßen / Best regards
Benjamin Weimer
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2F5ca7131d-4af2-423f-a429-c97f289741d6n%2540googlegroups.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767688125%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=SA5BIJf456lfEErs4aParENmhCm8uZIzXya5%2FlEZjE4%3D&reserved=0.
--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mailto:keycloak-use...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FAM0PR10MB192321AC19FD9511058BFB8BEDAE0%2540AM0PR10MB1923.EURPRD10.PROD.OUTLOOK.COM%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767698119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=2FmiJwIOdvXn0Btc8OsTVjkruCAkDUEI8UP2V7wqZjc%3D&reserved=0.
--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mailto:keycloak-use...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FCAK-7U1i%253DLV20tscN9RAt8NaxyBg5tYwomJM%253DzzdANsUUD1XRGA%2540mail.gmail.com%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767698119%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gZRC%2Bq3MlGFpFUXHsZf5LV%2BKg%2B5rEw1VDw2jYbFzw9M%3D&reserved=0.
--
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mailto:keycloak-use...@googlegroups.com.
You received this message because you are subscribed to the Google Groups "Keycloak User" group.
To view this discussion on the web visit https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgroups.google.com%2Fd%2Fmsgid%2Fkeycloak-user%2FAM0PR10MB19234950F6BC3DB76493F297EDAB0%2540AM0PR10MB1923.EURPRD10.PROD.OUTLOOK.COM%3Futm_medium%3Demail%26utm_source%3Dfooter&data=04%7C01%7Cjoerg.matysiak%40bosch.io%7C34127cc7cf3f4e5ecad008d8b616aa1e%7C0ae51e1907c84e4bbb6d648ee58410f4%7C0%7C0%7C637459555767708111%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=DM0F76Ol0bYOpT%2BYVhLlhWEarDHcKolCztkThcUl%2B0w%3D&reserved=0.
Jan Lieskovsky
Jan 13, 2021, 5:13:48 AM1/13/21
to Paul Ferraro, Thomas Darimont, Matysiak Joerg (IOC/PAU1), Todor Kazakov, Keycloak User
On Tue, Jan 12, 2021 at 9:07 PM Paul Ferraro <paul.f...@redhat.com> wrote:
Jan - I've opened https://issues.redhat.com/browse/WFLY-14286. I'll make sure this gets ported to WF 22.0.1.Final.
@Paul Ferraro - Wonderful, TYVM!
Jan
--
Don't see the problems. See their solutions!Reply all
Reply to author
Forward
0 new messages