very slow ticket delivery on CAS 6.6 & redis ticket registry

[Pascal Rigaux]

Hi,

In 6.6.x Redis ticket registry key is suffixed with userid (since 6.6.0-RC4)

This is great to know who owns a TGT or a ST.

Alas, this means getting a TGT from Redis now requires a "SCAN"... which is much more costly.
Example: full "SCAN" is ~100 times slower then "GET" on our production Redis (dbsize ~100k, because we have 1 month rememberMe TGT)


For the record, getting a ST triggers
- on 5.3 : 8 redis "GET" on the TGT
- on 6.5 : 17 redis "GET" on the TGT
- on 6.6 : 15 redis "SCAN" + "GET" on the TGT on a small redis db



PS: "cas.ticket.registry.core.enable-locking=false" fails on redis ticket registry with error
> Could not find a destroy method named 'destroy' on bean with name 'casTicketRegistryRedisLockRegistry'

[Jérôme LELEU]

Hi,

Thanks for raising the point.

It's always hard to find a good balance between a generic design and performance.

It seems to me that performing scans to get a ticket is not the best thing to do in terms of performance.

The Redis ticket registry is commonly used and we should try to avoid any performance degradation.

I have a few ideas in mind, but I'm not a Redis specialist: what do you propose?

Thanks.

Best regards,

Jérôme

--
- Website: https://apereo.github.io/cas
- Gitter Chatroom: https://gitter.im/apereo/cas
- List Guidelines: https://goo.gl/1VRrw7
- Contributions: https://goo.gl/mh7qDG
---
You received this message because you are subscribed to the Google Groups "CAS Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cas-user+u...@apereo.org.
To view this discussion on the web visit https://groups.google.com/a/apereo.org/d/msgid/cas-user/ca6a06e7-88cd-8f5f-1f26-232238eb6d5b%40univ-paris1.fr.

[Pascal Rigaux]

Solutions I can think of:

- add a memory cache [ ticketId => redisKey ]
(it should help a lot, even if it will still be slower than before in case of load balancing)

- revert suffixing redis key with userid
(easy change in RedisTicketRegistry.java)

- and possibly add userid suffix in a UniqueTicketIdGenerator, the way HostNameBasedUniqueTicketIdGenerator suffixes with hostname
(but it may be hard to do...)

cu

[Mike Rokitka]

FWIW,

We just had to roll back our CAS 6.6.1 deployment in production due to this performance regression.  P99 latencies ranged between 35 seconds and 9 minutes for some requests under load.

~Mike

[Robin Dupret]

Chiming in just to say that we are sticking with 6.5 as well because of this performance issue.

Thanks

[Henry Heikkinen]

Have you given up on this Redis ticket registry?

[Pascal Rigaux]

There is work in progress on this subject, as can be seen on cas-dev mailing list: https://www.mail-archive.com/cas...@apereo.org/msg00880.html

On 15/11/2022 14:37, Henry Heikkinen wrote:
> Have you given up on this Redis ticket registry?
>
> tiistai 8. marraskuuta 2022 klo 12.39.33 UTC+2 robin....@gmail.com kirjoitti:
>
> Chiming in just to say that we are sticking with 6.5 as well because of this performance issue.
>
> Thanks
> Le vendredi 4 novembre 2022 à 19:43:06 UTC+1, mike.r...@gmail.com a écrit :
>
> FWIW,
>
> We just had to roll back our CAS 6.6.1 deployment in production due to this performance regression.  P99 latencies ranged between 35 seconds and 9 minutes for some requests under load.

> [...]

[Henry Heikkinen]

Thank you for sharing the discussion on cas-dev mailing list and great to see the issue being solved.