CAS 7.0: RedisTicketRegistry Metaspace memory leak

[Nathan Cailbourdin]

Hello,

Since version 7 of CAS, I have noticed an abnormal and progressive increase in the memory consumed by the metaspace when the Redis ticket registry is used. This issue is due to accessor classes that are dynamically generated by Spring Data to access the attributes of RedisTicketDocument.  

The increase can easily be observed by monitoring the evolution of the metaspace as soon as connections are made: https://postimg.cc/21p7znzd
Here, the example involves approximately 2000 users connecting to CAS, using it to obtain STs, and then disconnecting. We can see the increase in memory consumption during the connection phase (from 10:52 AM to 11:02 AM), and especially, we do not observe any memory release even during the disconnection phase (starting at 11:02 AM).  

In comparison, with the same number of connections but with the default ticket registry, no metaspace increase is observed: https://postimg.cc/zL8wh5HV  
With CAS version 6 and the Redis ticket registry, we do not observe any increase, so the problem only appears starting from version 7.  

We can observe the class loading by launching the CAS server with the argument -Xlog:class+load=info. The following message is displayed during each new connection to CAS (with a different class name each time):  
[info][class,load] org.apereo.cas.ticket.registry.RedisTicketDocument_Accessor_ss8dk8 source: __JVM_LookupDefineClass__

If we look at the loaded classes with the command jcmd PID VM.classes | grep org.apereo.cas.ticket.registry.RedisTicketDocument_Accessor, we can see that for each new connection, a new accessor class is created:  
0x00007f03c1e0f218    73  fully_initialized     W        org.apereo.cas.ticket.registry.RedisTicketDocument_Accessor_ss8dk8
0x00007f03c1cbdab8    73  fully_initialized     W        org.apereo.cas.ticket.registry.RedisTicketDocument_Accessor_3aa8i2
and the list goes on...  


Debugging step by step, I found that the problem comes from RedisTicketRegistry.java, specifically from the addOrUpdateTicket method: https://github.com/apereo/cas/blob/v7.0.4/support/cas-server-support-redis-ticket-registry/src/main/java/org/apereo/cas/ticket/registry/RedisTicketRegistry.java#L417

Tracing the call chain progressively, I found that the following method in Spring Data is invoked: https://github.com/spring-projects/spring-data-commons/blob/main/src/main/java/org/springframework/data/mapping/model/ClassGeneratingPropertyAccessorFactory.java#L97

This call results in creating a new class (not a new instance, a new class) because the PersistentPropertyAccessorFactory responsible for creating accessor classes associated with the mapping context does not know that a class has already been generated for this purpose.

A map (propertyAccessorClasses) is used to save the class type (here RedisTicketDocument) to its associated accessor class, but in our case, it becomes empty for each new ticket! (Whereas it should only be empty for the first one, see https://github.com/spring-projects/spring-data-commons/blob/main/src/main/java/org/springframework/data/mapping/model/ClassGeneratingPropertyAccessorFactory.java#L186)

This issue is due to a new RedisKeyValueAdapter being created for each call to addOrUpdateTicket, resulting in a new RedisMappingContext being created each time. When a new RedisMappingContext is created, it also creates a new PersistentPropertyAccessorFactory, which does not have knowledge of the previously created accessor class and will create a new one when it tries to access the ticket attributes.

RedisMappingContext instances are created in this method: https://github.com/apereo/cas/blob/v7.0.4/support/cas-server-support-redis-ticket-registry/src/main/java/org/apereo/cas/ticket/registry/RedisTicketRegistry.java#L439
 
They seem to depend on a KeyspaceConfiguration that changes based on the TGT. We cannot directly modify the MappingConfiguration or KeyspaceConfiguration as they are final attributes (which explains why a new MappingConfiguration is recreated each time).  

My question is: what would be the best way to solve this problem?


A potential fix would be to create only one RedisMappingContext, then dynamically change its KeyspaceSettings in the buildRedisKeyValueAdapter method. In other words, this would involve this change in the buildRedisKeyValueAdapter method:
this.redisMappingContext.getMappingConfiguration().getKeyspaceConfiguration().addKeyspaceSettings(new KeyspaceConfiguration.KeyspaceSettings(RedisTicketDocument.class, redisKeyPattern));  
val adapter = new RedisKeyValueAdapter(casRedisTemplates.getTicketsRedisTemplate(), this.redisMappingContext)  

With this modification, the metaspace memory consumption returns to normal, but maybe there is a better way to solve the problem.

 

Thanks in advance for your assistance.  
Best regards.