Proposal: Infinispan-based persistent cache module for Gerrit

117 views
Skip to first unread message

Yash Chaturvedi

unread,
Mar 27, 2026, 4:24:12 PM (9 days ago) Mar 27
to Repo and Gerrit Discussion
Hi all,
 
We are exploring a new Gerrit module at Qualcomm that would provide a persistent cache implementation based on Infinispan as an alternative to the current H2-backed persistent cache. I wanted to share the idea early and get feedback on both the approach and the proposed module naming.
 
Our interest in this comes from operational issues we have seen with the current persistent cache implementation in Gerrit v3.11, which relies on H2 v1 for local cache persistence. In our environment, H2 has not provided sufficiently reliable expiration and eviction behavior on disk, and it does not consistently enforce the configured disk limits. Over time, this can lead to uncontrolled cache growth and, eventually, request timeouts.
 
Moving to Gerrit v3.12 is not, by itself, enough to address this for our use case, as H2 v2 has also proven unreliable in similar scenarios.
 
A second concern is the overall efficiency of the current model. Because persistent caches are maintained locally by each Gerrit process, cache population work is duplicated across the cluster, and disk usage grows with each additional node. For expensive caches, this results in avoidable recomputation and poor scaling characteristics.
 
Infinispan is a distributed in-memory key/value data store with optional schema support, available under the Apache License 2.0. We think it could be a good fit for a few reasons:
 
  • single-primary Gerrit setups can use Infinispan in a way that is similar to how H2 is used today
  • optionally, a remote or distributed Infinispan cache can be used to de-duplicate cache work across a cluster
  • persistence options beyond file-based storage, such as RocksDB and JDBC
  • features such as passivation, preload, clustering, and transactional guarantees
  • per-cache expiration and eviction policies
  • better observability via JMX/Micrometer
  • richer cache APIs and management capabilities
For naming, following the existing module conventions in Gerrit, one possible option would be:
 
modules/cache-infinispan
 
We also have a couple of proof-of-concept changes that illustrate the direction:
 
https://gerrit-review.googlesource.com/c/plugins/examples/+/566042
https://gerrit-review.googlesource.com/c/plugins/examples/+/566121

Feedback on the overall direction, scope, and naming would be very welcome.
 
Thanks,
Yash

Luca Milanesio

unread,
Mar 27, 2026, 4:47:39 PM (9 days ago) Mar 27
to Repo and Gerrit Discussion, Luca Milanesio, Yash Chaturvedi
Hi Yash,

Thanks for sharing your work at Qualcomm about Infinispan caches, that looks really interesting.

Do you have some comparison graphs with H2v1, H2v2 and cache-chroniclemap?
See for example the benchmarks and data shared by Tony last year at GerritMeets [1].

> On 27 Mar 2026, at 17:07, 'Yash Chaturvedi' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
>
> Hi all,
>
> We are exploring a new Gerrit module at Qualcomm that would provide a persistent cache implementation based on Infinispan as an alternative to the current H2-backed persistent cache. I wanted to share the idea early and get feedback on both the approach and the proposed module naming.
>
> Our interest in this comes from operational issues we have seen with the current persistent cache implementation in Gerrit v3.11, which relies on H2 v1 for local cache persistence. In our environment, H2 has not provided sufficiently reliable expiration and eviction behavior on disk, and it does not consistently enforce the configured disk limits. Over time, this can lead to uncontrolled cache growth and, eventually, request timeouts.
>
> Moving to Gerrit v3.12 is not, by itself, enough to address this for our use case, as H2 v2 has also proven unreliable in similar scenarios.

Agreed, also Tony highlighted the challenges in terms of memory consumption with H2 v2 at [1].

>
> A second concern is the overall efficiency of the current model. Because persistent caches are maintained locally by each Gerrit process, cache population work is duplicated across the cluster, and disk usage grows with each additional node. For expensive caches, this results in avoidable recomputation and poor scaling characteristics.

True, I believe you’re referring to a shared-everything model, where all the repositories are on common volume.
When using multi-site though, having a shared cache isn’t suitable nor desirable.

>
> Infinispan is a distributed in-memory key/value data store with optional schema support, available under the Apache License 2.0. We think it could be a good fit for a few reasons:
>
> • single-primary Gerrit setups can use Infinispan in a way that is similar to how H2 is used today
> • optionally, a remote or distributed Infinispan cache can be used to de-duplicate cache work across a cluster

That looks cool; how does it compare to other shared-caches like Redis?

> • persistence options beyond file-based storage, such as RocksDB and JDBC

Have you tried them?
Would they involve maintaining yet another system though?

> • features such as passivation, preload, clustering, and transactional guarantees
> • per-cache expiration and eviction policies

Does Infinispan manage automatic expiration and LRU policies?


> • better observability via JMX/Micrometer
> • richer cache APIs and management capabilities

I saw the code-base and APIs, that’s a really huge project indeed!

> For naming, following the existing module conventions in Gerrit, one possible option would be:
>
> modules/cache-infinispan

The name makes sense to me.

> We also have a couple of proof-of-concept changes that illustrate the direction:
>
> https://gerrit-review.googlesource.com/c/plugins/examples/+/566042
> https://gerrit-review.googlesource.com/c/plugins/examples/+/566121
>
> Feedback on the overall direction, scope, and naming would be very welcome.

Have you used it in staging with some workload?
Do you have any data to share with us?

Thanks again for sharing it.

Luca.

[1] https://youtu.be/T2ApofhyF10

>
> Thanks,
> Yash
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/30cfd6ef-1864-402e-aa1f-644f87381510n%40googlegroups.com.

Clark Boylan

unread,
Mar 27, 2026, 4:51:38 PM (9 days ago) Mar 27
to Repo and Gerrit Discussion
On Fri, Mar 27, 2026 at 1:24 PM 'Yash Chaturvedi' via Repo and Gerrit
Discussion <repo-d...@googlegroups.com> wrote:
>
> Hi all,
>
> We are exploring a new Gerrit module at Qualcomm that would provide a persistent cache implementation based on Infinispan as an alternative to the current H2-backed persistent cache. I wanted to share the idea early and get feedback on both the approach and the proposed module naming.
>
> Our interest in this comes from operational issues we have seen with the current persistent cache implementation in Gerrit v3.11, which relies on H2 v1 for local cache persistence. In our environment, H2 has not provided sufficiently reliable expiration and eviction behavior on disk, and it does not consistently enforce the configured disk limits. Over time, this can lead to uncontrolled cache growth and, eventually, request timeouts.
>
> Moving to Gerrit v3.12 is not, by itself, enough to address this for our use case, as H2 v2 has also proven unreliable in similar scenarios.

We have struggled with the H2 caches as well. An open source
alternative that can be used would be great.

>
> A second concern is the overall efficiency of the current model. Because persistent caches are maintained locally by each Gerrit process, cache population work is duplicated across the cluster, and disk usage grows with each additional node. For expensive caches, this results in avoidable recomputation and poor scaling characteristics.
>
> Infinispan is a distributed in-memory key/value data store with optional schema support, available under the Apache License 2.0. We think it could be a good fit for a few reasons:
>
>
> single-primary Gerrit setups can use Infinispan in a way that is similar to how H2 is used today
> optionally, a remote or distributed Infinispan cache can be used to de-duplicate cache work across a cluster
> persistence options beyond file-based storage, such as RocksDB and JDBC
> features such as passivation, preload, clustering, and transactional guarantees
> per-cache expiration and eviction policies
> better observability via JMX/Micrometer
> richer cache APIs and management capabilities
>
> For naming, following the existing module conventions in Gerrit, one possible option would be:
>
> modules/cache-infinispan
>
> We also have a couple of proof-of-concept changes that illustrate the direction:
>
> https://gerrit-review.googlesource.com/c/plugins/examples/+/566042
> https://gerrit-review.googlesource.com/c/plugins/examples/+/566121
>
> Feedback on the overall direction, scope, and naming would be very welcome.

It looks like your examples cover embedding the cache system within
Gerrit itself or speaking to a remote service running the infinispan
cache. I think for those of us running a single Gerrit instance
avoiding additional services is nice if possible. Is the idea that for
a simple Gerrit deployment we wouldn't need any additional services?
Then the other question I have is how is persistence done? An in
memory cache is not persistent. 566042 refers to SIFS which does write
cache data to disk. It looks like SIFS stores data in append only
files and it only reclaims disk space when a file drops below 50%
utilization (it copies the valid records into a new smaller file then
deletes the old file).

With H2 we have observed two major problems. The first is that the
backing files grow forever. We have been mitigating that with
occasional restarts of Gerrit where we delete the caches entirely
while Gerrit is not running. The second is that H2 performs cleanup
routines during Gerrit shutdown which causes shutdowns to take several
minutes at least. This is problematic when we 're going to immediately
delete the backing cache file as soon as Gerrit actually stops. I'd be
curious if SIFS performs better than H2 when it comes to pruning and
managing the disk consumption. I'm also curious if there are shutdown
impacts when using SIFS. Does Infinispan need to flush data to disk on
shutdown and will that make shutdown slow like H2?

In any case I think the problems with H2 have made it difficult to
operate Gerrit and alternatives would be great. Ideally any
alternative would address the disk consumption problem and slow
shutdown process slowness that H2 creates.

>
> Thanks,
> Yash
>

Luca Milanesio

unread,
Mar 27, 2026, 5:45:18 PM (9 days ago) Mar 27
to Repo and Gerrit Discussion, Luca Milanesio, Clark Boylan
+1, that’s the reason why we use cache-chroniclemap [2] as alternative to H2. No additional services, just a faster and more reliable storage.

> Is the idea that for
> a simple Gerrit deployment we wouldn't need any additional services?
> Then the other question I have is how is persistence done? An in
> memory cache is not persistent. 566042 refers to SIFS which does write
> cache data to disk. It looks like SIFS stores data in append only
> files and it only reclaims disk space when a file drops below 50%
> utilization (it copies the valid records into a new smaller file then
> deletes the old file).

From what I read at [3], it is basically an in-memory cache of a B+ tree stored in a set of flat file on disk with the garbage recollection logic you’ve mentioned.
I would be interested to see in practice if the garbage recollection causes hiccups or not to Gerrit.

With cache-chroniclemap [2] there is no garbage recollection at all: it’s a pre-allocated file, that’s it!
The disk space is allocated at start and never reclaimed, but always reused in fragments.

>
> With H2 we have observed two major problems. The first is that the
> backing files grow forever. We have been mitigating that with
> occasional restarts of Gerrit where we delete the caches entirely
> while Gerrit is not running.

Yes, that’s what we experienced as well. Gerrit with a large H2 v1 (or v2) cache file is so slow, that removing the caches altogether makes it faster, which is a paradox.
H2 has been good as quick Open-Source alternative to Google’s native cache storage, however, the choice has been unfortunate IMHO as it showed very soon all its limitations.

> The second is that H2 performs cleanup
> routines during Gerrit shutdown which causes shutdowns to take several
> minutes at least.

... and at startup the cache pruning to enforce the disk limits.

> This is problematic when we 're going to immediately
> delete the backing cache file as soon as Gerrit actually stops. I'd be
> curious if SIFS performs better than H2 when it comes to pruning and
> managing the disk consumption. I'm also curious if there are shutdown
> impacts when using SIFS. Does Infinispan need to flush data to disk on
> shutdown and will that make shutdown slow like H2?

Yep, some data would be useful, even if preliminary.

>
> In any case I think the problems with H2 have made it difficult to
> operate Gerrit and alternatives would be great. Ideally any
> alternative would address the disk consumption problem and slow
> shutdown process slowness that H2 creates.

I believe there are already alternatives, including a traditional JDBC storage.
One more alternative is always very welcomed: more choice is better than less choice.

Luca.

[2] https://gerrit.googlesource.com/modules/cache-chroniclemap/
[3] https://infinispan.org/blog/2014/10/31/soft-index-file-store


>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/CAAcH74gqj5d3GJ_8W2vO2yM7kb5RSece9b5X9Vsh1kEtxBXynA%40mail.gmail.com.


Nasser Grainawi

unread,
Apr 1, 2026, 12:14:40 PM (4 days ago) Apr 1
to Luca Milanesio, Repo and Gerrit Discussion, Clark Boylan
Agreed. That's why I liked Infinispan as a solution when I came across it. It's a set of solutions vs just one specific way to use a cache.
 

> Is the idea that for
> a simple Gerrit deployment we wouldn't need any additional services?
> Then the other question I have is how is persistence done? An in
> memory cache is not persistent. 566042 refers to SIFS which does write
> cache data to disk. It looks like SIFS stores data in append only
> files and it only reclaims disk space when a file drops below 50%
> utilization (it copies the valid records into a new smaller file then
> deletes the old file).

From what I read at [3], it is basically an in-memory cache of a B+ tree stored in a set of flat file on disk with the garbage recollection logic you’ve mentioned.
I would be interested to see in practice if the garbage recollection causes hiccups or not to Gerrit.

Our limited testing so far hasn't shown any issues, but I'm not sure that testing would surface those. If it becomes that SIFS itself is a problem, we would try the RocksDB persisted backend next (still embedded and not an extra service).
 

With cache-chroniclemap [2] there is no garbage recollection at all: it’s a pre-allocated file, that’s it!
The disk space is allocated at start and never reclaimed, but always reused in fragments.

>
> With H2 we have observed two major problems. The first is that the
> backing files grow forever. We have been mitigating that with
> occasional restarts of Gerrit where we delete the caches entirely
> while Gerrit is not running.

Yes, that’s what we experienced as well. Gerrit with a large H2 v1 (or v2) cache file is so slow, that removing the caches altogether makes it faster, which is a paradox.
H2 has been good as quick Open-Source alternative to Google’s native cache storage, however, the choice has been unfortunate IMHO as it showed very soon all its limitations.

> The second is that H2 performs cleanup
> routines during Gerrit shutdown which causes shutdowns to take several
> minutes at least.

... and at startup the cache pruning to enforce the disk limits.

> This is problematic when we 're going to immediately
> delete the backing cache file as soon as Gerrit actually stops. I'd be
> curious if SIFS performs better than H2 when it comes to pruning and
> managing the disk consumption. I'm also curious if there are shutdown
> impacts when using SIFS. Does Infinispan need to flush data to disk on
> shutdown and will that make shutdown slow like H2?

Yep, some data would be useful, even if preliminary.

We are hoping this Infinispan solution addresses both of those problems. However, we're going to be more focused on the remote cache solution and gathering data for that as we very much want the cache de-duplication functionality. We'll do some testing with local SIFS/RocksDB, but probably only enough to progress to the remote cache solution. We'd love it if others in the community interested in the local solution can help test some of these specific use cases further.
 

>
> In any case I think the problems with H2 have made it difficult to
> operate Gerrit and alternatives would be great. Ideally any
> alternative would address the disk consumption problem and slow
> shutdown process slowness that H2 creates.

I believe there are already alternatives, including a traditional JDBC storage.
One more alternative is always very welcomed: more choice is better than less choice.

It doesn't sound like there's any concerns then with creating the cache-infinispan repo on gerrit-review?
 

Luca.

[2] https://gerrit.googlesource.com/modules/cache-chroniclemap/
[3] https://infinispan.org/blog/2014/10/31/soft-index-file-store


>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/CAAcH74gqj5d3GJ_8W2vO2yM7kb5RSece9b5X9Vsh1kEtxBXynA%40mail.gmail.com.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

Matthias Sohn

unread,
Apr 1, 2026, 5:34:04 PM (4 days ago) Apr 1
to Nasser Grainawi, Luca Milanesio, Repo and Gerrit Discussion, Clark Boylan
I am interested and can help testing the new cache solution in the type of deployment we use (k8s-gerrit on GCP).
The remote cache would be the perfect addition to our plan to centralize indexes using the OpenSearch module.
This should facilitate introducing auto-scaling in k8s-gerrit.
 
>
> In any case I think the problems with H2 have made it difficult to
> operate Gerrit and alternatives would be great. Ideally any
> alternative would address the disk consumption problem and slow
> shutdown process slowness that H2 creates.

I believe there are already alternatives, including a traditional JDBC storage.
One more alternative is always very welcomed: more choice is better than less choice.

It doesn't sound like there's any concerns then with creating the cache-infinispan repo on gerrit-review?

+1, sounds good to me 

-Matthias

Luca Milanesio

unread,
Apr 1, 2026, 5:35:40 PM (4 days ago) Apr 1
to Repo and Gerrit Discussion, Luca Milanesio, Nasser Grainawi, Clark Boylan, Matthias Sohn
Likewise, +1 to have it and experimenting it.

Luca.

Nasser Grainawi

unread,
Apr 1, 2026, 5:46:24 PM (4 days ago) Apr 1
to Luca Milanesio, Repo and Gerrit Discussion, Clark Boylan, Matthias Sohn
Yes, this is definitely a use case we have in mind too.
 
 
> 
> In any case I think the problems with H2 have made it difficult to
> operate Gerrit and alternatives would be great. Ideally any
> alternative would address the disk consumption problem and slow
> shutdown process slowness that H2 creates.

I believe there are already alternatives, including a traditional JDBC storage.
One more alternative is always very welcomed: more choice is better than less choice.

It doesn't sound like there's any concerns then with creating the cache-infinispan repo on gerrit-review?

+1, sounds good to me 

Likewise, +1 to have it and experimenting it.

Reply all
Reply to author
Forward
0 new messages