Wildfly Infinispan custom store attribute in the standalone.xml config

[Ronny Gunawan]

Hi All,

I have a custom Infinispan store implementation where I need to pass additional custom attribute/ parameter specified in the xml so the default value can be overridden anytime.

I followed this guide to create a custom store (https://infinispan.org/docs/stable/titles/getting_started/getting_started.html#maven_archetypes) and now that I have a custom parser class, how can I link this parser with the xml element so the value of collection in following config can be read from the builder class?

                <distributed-cache name="dist">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                    <expiration interval="0"/>
                    <store class="org.infinispan.persistence.mongodb.configuration.MongoDBStoreConfigurationBuilder" passivation="false">
                        <property name="collection">new-collection</property>
                    </store>
</distributed-cache>

Really appreciate any idea/ thought from you guys. 

Thanks and regards,

Ronny

[Ronny Gunawan]

Do I need a custom schema config to be added to support this new property?

[Paul Ferraro]

By default, WildFly will only be able to locate cache store classes that are visible to the infinispan subsystem's module.

To use a custom cache store such as MongoDB, you need to either:

1. create a custom module containing the MongoDB library and configure your cache-container with that module

2. add the infinispan-persistence-mongodb.jar to the existing org.infinispan module.

I suggest option #2, by copying the jar to $WILDFLY_HOME/modules/system/layers/base/org/infinispan/main/.

and editing $WILDFLY_HOME/modules/system/layers/base/org/infinispan/main/module.xml to reference this jar.

[Ronny Gunawan]

Hi Paul,

The wildfly is able to locate the jar and use it it as a session store. 

However  I'm currently hardcoding the collection name. Is it possible to pass the collection name from xml, either through custom attribute within <store> or custom property under <store> ?

Thanks and regards,

Ronny

[Paul Ferraro]

On Wednesday, April 12, 2023 at 7:50:05 PM UTC-4 Ronny Gunawan wrote:

Hi Paul,

The wildfly is able to locate the jar and use it it as a session store. 

Are you trying to use the above configuration as an HttpSession store?  What role is MongoDB meant to play in this use case?

I'm a little configured by the configuration in your initial post - and how you intend for MongoDB to contribute to a session store.

Using the configuration you attached, MongoDB behaves as a write-only store, effectively forcing every request to incur the cost of writing to MongoDB, even though the contents will never be read.

I don't imagine this is what you intended.  Can you elaborate?

However  I'm currently hardcoding the collection name. Is it possible to pass the collection name from xml, either through custom attribute within <store> or custom property under <store> ?

Where is the collection name hardcoded?  The configuration XML snippet above already contains a collection name, no?

Perhaps I misunderstood the question?

Paul

[Ronny Gunawan]

Hi Paul, yes it is meant to be an external HttpSession store.

- I would prefer it to be read store as well as it will be used as distributed cache. Are you seeing any problem that I missed?

  So far I have only tested with passivation="false", I will need to try again with passivation="true".

- The configuration xml for property element that I provided is not working at the moment so this would be my main concern for now. 

  As I would need to find a way not to hardcode it.

 - The collection name is currently hardcoded inside MongoDBStoreConfigurationBuilder. Following is the code snippet for the hardcoded value:

private String connectionURI = "mongodb+srv://localhost:27017";

private String collection = "jstore";

public MongoDBStoreConfigurationBuilder(PersistenceConfigurationBuilder builder) {

   super(builder, MongoDBStoreConfiguration.attributeDefinitionSet());

}

Thanks and regards,

Ronny

[Paul Ferraro]

On Friday, April 14, 2023 at 5:34:10 AM UTC-4 Ronny Gunawan wrote:

Hi Paul, yes it is meant to be an external HttpSession store.

- I would prefer it to be read store as well as it will be used as distributed cache. Are you seeing any problem that I missed?

I think you may have some misconceptions about the role of the cache store within Infinispan [1].

In general, Infinispan's distributed-cache stores all cache entries within the collective heap of the cluster.

Infinispan will only attempt to read a given cache entry from the cache store if it is not found in memory (either locally or remotely).

A cache store operates in 2 modes:

* Persistence, i.e. passivation="false"

  + Cache entries are persisted to the store for every write operation, removed from the store for every remove operation, and read from the store if the entry was not found in memory (locally or remotely).

  + Since a distributed (or replicated) cache stores all cache entries in memory (either locally or remotely), Infinispan will never read from a persistent cache.

  + Using a cache with a persistent store, the cache store will always contain a redundant copy of any entry in memory

* Passivation, i.e. passivation="true"

  + Cache entries are persisted to the store for every eviction operation, and removed from the store for every get operation

  + In a passivation-enabled cache, a given entry exists either in memory (locally or remotely) or in the store, but never both.

Consequently, the only cache types with reasonable use cases for a persistent cache store are local-cache and invalidation-cache.

In other words, you *could* configure WildFly's distributed HttpSession manager to use an invalidation-cache with a *shared* persistent store using the MongoDB cache store.

However, the only tangible benefit to this type of configuration over the default HA configuration is the potential to scale the number of active sessions independently from (and beyond that of) your WildFly cluster.

However, most users find the scaling characteristics of a passivation-enabled distributed cache to be sufficient such that it is more appealing than the lower performance of a shared persistence solution.

Out of curiosity, what is the motivation for using MongoDB in this way?

  So far I have only tested with passivation="false", I will need to try again with passivation="true".

It does not make sense to use a shared cache store for passivation.

 

- The configuration xml for property element that I provided is not working at the moment so this would be my main concern for now. 

  As I would need to find a way not to hardcode it.

 - The collection name is currently hardcoded inside MongoDBStoreConfigurationBuilder. Following is the code snippet for the hardcoded value:

I assume you are using a fork of the upstream MongoDBStoreConfigurationBuilder.java [2]?

Assuming that your builder implementation also extends AbstractStoreConfigurationBuilder [3] you should have built-in support for generic properties.

See the configuration for SingleFileStore [4] and its builder [5] as an example.

[1] https://infinispan.org/docs/stable/titles/configuring/configuring.html#persistence

[2] https://github.com/infinispan/infinispan-cachestore-mongodb/blob/main/src/main/java/org/infinispan/persistence/mongodb/configuration/MongoDBStoreConfigurationBuilder.java

[3] https://github.com/infinispan/infinispan/blob/main/core/src/main/java/org/infinispan/configuration/cache/AbstractStoreConfigurationBuilder.java

[4] https://github.com/infinispan/infinispan/blob/main/core/src/main/java/org/infinispan/configuration/cache/SingleFileStoreConfiguration.java

[5] https://github.com/infinispan/infinispan/blob/main/core/src/main/java/org/infinispan/configuration/cache/SingleFileStoreConfigurationBuilder.java

Paul

[Ronny Gunawan]

Hi Paul, please find my comments below.

- This is the behavior that we would like to have where it will read from cache store when it's not found in memory locally. Our objectives is to test distributed cache on Wildfly.

Currently we are on default behavior where session is stored in memory and once an instance go down then there is no where to continue the session in the other running instance.

Please correct me if distributed-cache will allow the user to continue their session in other instance within the same server group.

- We have got existing approved MongoDB so we could leverage it. Otherwise, it will need to go through another approval for a new relational database system

Thanks,

Ronny

On Friday, April 14, 2023 at 11:07:54 PM UTC+8 Paul Ferraro wrote:

On Friday, April 14, 2023 at 5:34:10 AM UTC-4 Ronny Gunawan wrote:

Hi Paul, yes it is meant to be an external HttpSession store.

- I would prefer it to be read store as well as it will be used as distributed cache. Are you seeing any problem that I missed?

I think you may have some misconceptions about the role of the cache store within Infinispan [1].

In general, Infinispan's distributed-cache stores all cache entries within the collective heap of the cluster.

Infinispan will only attempt to read a given cache entry from the cache store if it is not found in memory (either locally or remotely).

This is the behavior that we would like to have where it will read from cache store when it's not found in memory locally. Our objectives is to test distributed cache on Wildfly.

Currently we are on default behavior where session is stored in memory and once an instance go down then there is no where to continue the session in the other running instance.

Please correct me if distributed-cache will allow the user to continue their session in other instance within the same server group.

 

A cache store operates in 2 modes:

* Persistence, i.e. passivation="false"

  + Cache entries are persisted to the store for every write operation, removed from the store for every remove operation, and read from the store if the entry was not found in memory (locally or remotely).

  + Since a distributed (or replicated) cache stores all cache entries in memory (either locally or remotely), Infinispan will never read from a persistent cache.

  + Using a cache with a persistent store, the cache store will always contain a redundant copy of any entry in memory

* Passivation, i.e. passivation="true"

  + Cache entries are persisted to the store for every eviction operation, and removed from the store for every get operation

  + In a passivation-enabled cache, a given entry exists either in memory (locally or remotely) or in the store, but never both.

Consequently, the only cache types with reasonable use cases for a persistent cache store are local-cache and invalidation-cache.

In other words, you *could* configure WildFly's distributed HttpSession manager to use an invalidation-cache with a *shared* persistent store using the MongoDB cache store.

However, the only tangible benefit to this type of configuration over the default HA configuration is the potential to scale the number of active sessions independently from (and beyond that of) your WildFly cluster.

However, most users find the scaling characteristics of a passivation-enabled distributed cache to be sufficient such that it is more appealing than the lower performance of a shared persistence solution.

Out of curiosity, what is the motivation for using MongoDB in this way?

We have got existing approved MongoDB so we could leverage it. Otherwise, it will need to go through another approval for a new relational database system

[Paul Ferraro]

On Sunday, April 16, 2023 at 10:01:07 PM UTC-4 Ronny Gunawan wrote:

Hi Paul, please find my comments below.

- This is the behavior that we would like to have where it will read from cache store when it's not found in memory locally. Our objectives is to test distributed cache on Wildfly.

A distributed-cache does not use the cache store in this way. See: https://infinispan.org/docs/stable/titles/configuring/configuring.html#cache-modes_caches

 

Currently we are on default behavior where session is stored in memory and once an instance go down then there is no where to continue the session in the other running instance.

This is the default behavior for non-distributable web applications.

A web application indicates its need for distributed HttpSession via the <distributable/> element in web.xml

See: https://jakarta.ee/specifications/servlet/6.0/jakarta-servlet-spec-6.0.html#distributed-environments

https://docs.wildfly.org/27/High_Availability_Guide.html#Distributable_Web_Applications

When WildFly is started using either the "ha" or "full-ha" profile, WildFly will ensure that every session is accessible by every member of the cluster backed by a configurable number of replicas.

All of this is instrumented via Infinispan, where cluster discovery, group management, and message QoS is handled via JGroups.

 

Please correct me if distributed-cache will allow the user to continue their session in other instance within the same server group.

Correct.  However, this works out of the box (provided your socket-bindings are correctly configured).  You don't need any special cache store for this.

Perhaps you should trying using the default HA behavior before trying to invent something else?

 

- We have got existing approved MongoDB so we could leverage it. Otherwise, it will need to go through another approval for a new relational database system

WildFly's distributed session support does not require the use of a relational database.

As I said previously, you *could* configure a distributed session manager using an invalidation-cache with a shared persistent cache store, but this type of configuration will always be less performant.

Paul

[Ronny Gunawan]

Thank you Paul for your suggestion and appreciate your advice.

I will follow your suggestion to use the default behavior and observe it before implementing a shared persistent cache store using an invalidation cache.

By the way, a side track question, what is the difference of running ha profile in domain mode vs standalone mode?

ha profile in domain mode is quite obvious but how could a ha profile used in standalone mode? Does it mean if I have few instances of standalone wildfly then they can behave like it would be in a domain mode?

Best Regards,

Ronny

[Paul Ferraro]

On Tuesday, April 18, 2023 at 9:30:42 AM UTC-4 Ronny Gunawan wrote:

Thank you Paul for your suggestion and appreciate your advice.

I will follow your suggestion to use the default behavior and observe it before implementing a shared persistent cache store using an invalidation cache.

By the way, a side track question, what is the difference of running ha profile in domain mode vs standalone mode?

ha profile in domain mode is quite obvious but how could a ha profile used in standalone mode? Does it mean if I have few instances of standalone wildfly then they can behave like it would be in a domain mode?

From the perspective of the server instance, there is no difference.

Domain mode and WildFly's high-availability features are completely orthogonal.

Domain mode lets you *manage* multiple server instances, to perform operational things like deploy an application to a group of servers, etc.

Whereas deploying an application to a set of standalone servers requires individual operations per server.

I recommend reviewing https://docs.wildfly.org/27/Admin_Guide.html#Domain_Setup for details.

Whether a given server supports high-availability or not depends only on its configuration.

Therefore, you can start multiple standalone servers using one of the HA profiles (e.g. standalone-ha.xml or standalone-full-ha.xml) to leverage things like a distributed session manager, distributed @Stateful EJBs, etc.

[Ronny Gunawan]

Hi Paul,

I was testing the default behavior of distributable session without any external session store and encounter  this error:

writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(HQ7J-bvWdL8eN8Q9nmhFm7Z3fzIXjSdK3rzYkJj4) and requestor GlobalTransaction{id=14, addr=wildflydev1:wildflydev1-08, remote=false, xid=null, internalId=-1}. Lock is held by GlobalTransaction{id=12, addr=wildflydev1:wildflydev1-08, remote=false, xid=null, internalId=-1}
[Server:wildflydev1-08]         at org.inf...@14.0.2.Final//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:299)
[Server:wildflydev1-08]         at org.inf...@14.0.2.Final//org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.get(DefaultLockManager.java:229)
[Server:wildflydev1-08]         at org.inf...@14.0.2.Final//org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.checkState(InfinispanLock.java:440)
[Server:wildflydev1-08]         at org.inf...@14.0.2.Final//org.infinispan.util.concurrent.locks.impl.InfinispanLock$LockPlaceHolder.lambda$toInvocationStage$3(InfinispanLock.java:416)
[Server:wildflydev1-08]         at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646)
[Server:wildflydev1-08]         at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
[Server:wildflydev1-08]         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[Server:wildflydev1-08]         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[Server:wildflydev1-08]         at org.wildfly.clu...@27.0.1.Final//org.wildfly.clustering.context.ContextReferenceExecutor.execute(ContextReferenceExecutor.java:49)
[Server:wildflydev1-08]         at org.wildfly.clu...@27.0.1.Final//org.wildfly.clustering.context.ContextualExecutor$1.run(ContextualExecutor.java:78)
[Server:wildflydev1-08]         at java.base/java.lang.Thread.run(Thread.java:833)

Was there any other configuration needed on top of the default config of distributed-cache?

                    <distributed-cache name="dist">
                        <locking isolation="REPEATABLE_READ"/>
                        <transaction mode="BATCH"/>
                        <expiration interval="0"/>

                        <file-store passivation="true" purge="true"/>
                    </distributed-cache>

Thanks,

Ronny

[Paul Ferraro]

This exception indicates that a second request thread is attempting to access a session that is currently being used by a different request thread.

See the documentation: https://docs.wildfly.org/28/High_Availability_Guide.html#session_concurrency

... for details, and how to modify the default configuration to accommodate concurrent requests for a given session.