After switching to chroniclemap, some cache shows worse result.

[Yingchun Li]

Hi,

    After switching to cache-chroniclemap, some caches(mostly the diff_summary)

shows worse result than H2-V1.

    The show-caches command output:

ssh -p 29418 localhost gerrit show-caches
Gerrit Code Review        3.12.3                    now    08:57:53   UTC
                                                 uptime    19 hrs 34 min

  Name                          |Entries              |  AvgGet |Hit Ratio|
                                |   Mem   Disk   Space|         |Mem  Disk|
--------------------------------+---------------------+---------+---------+
  adv_bases                     |                     |         |         |
  changeid_project              |   168               |         | 83%     |
  changes_by_project            |                     |         |  0%     |
  default_preferences           |                     |         |         |
  external_ids_map              |     1               |         | 99%     |
  groups                        |                     |         |         |
  groups_bymember               |     3               |   1.2ms | 25%     |
  groups_byname                 |     1               |  12.0ms | 99%     |
  groups_bysubgroup             |     7               | 766.0us | 58%     |
  groups_byuuid                 |    41               |   6.9ms | 99%     |
  groups_external               |     1               |  67.3ms | 75%     |
  groups_external_persisted     |                     |  64.9ms |  0%     |
  ldap_group_existence          |                     |         |         |
  ldap_groups                   |    15               |   2.9ms | 50%     |
  ldap_groups_byinclude         |                     |         |         |
  ldap_usernames                |   159               | 244.2us | 67%     |
  permission_sort               |  1024               |  21.4us | 63%     |
  plugin_resources              |     6               |         | 93%     |
  project_list                  |     1               |  40.3ms | 99%     |
  projects                      |  1024               | 649.4us | 99%     |
  prolog_rules                  |                     |         |         |
  soy_sauce_compiled_templates  |     1               |  92.6ms | 99%     |
  sshkeys                       |   244               |  20.9ms | 99%     |
  static_content                |    28               |   4.6ms | 79%     |
  lfs-lfs_project_locks         |     8               | 208.8us | 95%     |
  plugin-manager-plugins_list   |                     |  130.7s |  0%     |
  uploadvalidator-patternCache  |                     |         |         |
D accounts                      |   711    711 444.00k|   1.3ms | 99%     |
D change_kind                   | 13838  13838   9.96m|  39.2ms | 91%     |
D change_notes                  | 12415  19555   1.25g|   1.5ms | 95%     |
D comment_context               |  2011   2011  10.01m|   1.9ms | 69%     |
D conflicts                     | 18055  18055  30.39m| 146.1ms | 70%     |
D diff_intraline                |  1924   1924  36.44m|   3.8ms | 26%     |
D diff_summary                  |  2379   2379  69.37m|    5.4s | 85%     |
D gerrit_file_diff              |394557 394557   3.70g|  46.7ms | 54%     |
D git_file_diff                 |388019 388019   2.44g|  47.3ms |  0%     |
D git_modified_files            |  3019   3019 674.57m|   2.1ms |  2%     |
D git_tags                      |     2      2 160.00k|   2.4ms | 98%   0%|
D groups_byuuid_persisted       |           41  52.00k|         |         |
D mergeability                  |  1359   1359   4.13m|  59.1ms | 50%   0%|
D modified_files                |  1767   3806 175.48m|   2.5ms | 88%   0%|
D oauth_tokens                  |                2.52m|         |         |
D persisted_projects            |         1047   7.70m|         |         |
D pure_revert                   |     8      8  32.00k|  56.5ms | 88%     |
D quota-repo_size               |  1024   1047   2.66m|   5.0ms |  0%     |
D web_sessions                  |  1505   1505 168.26m|         | 76%   0%|

SSH:      7  users, oldest session started   0 ms ago
Tasks:    9  total =    4 running +      0 ready +    5 sleeping
Mem: 38.41g total = 22.99g used + 7.42g free + 8.00g buffers
     56.00g max
         432 open files

Threads: 16 CPUs available, 230 threads.

Here the diff_summy AvgGet is 5.4s seconds. this is much higher than before.

just before it is a few milliseconds.

 Here is result before the switching, the H2-v1 cache backend,

Gerrit Code Review        3.10.6                    now    03:47:41   UTC
                                                 uptime   13 days 16 hrs

  Name                          |Entries              |  AvgGet |Hit Ratio|
                                |   Mem   Disk   Space|         |Mem  Disk|
--------------------------------+---------------------+---------+---------+
  adv_bases                     |                     |         | 99%     |
  change_notes                  |  6217               |   1.3ms | 96%     |
  changeid_project              |  1024               |         | 92%     |
  changes_by_project            |                     |         |  0%     |
  default_preferences           |                     |         |         |
  external_ids_map              |     1               |         | 99%     |
  groups                        |                     |         |         |
  groups_bymember               |   358               |   1.2ms | 99%     |
  groups_byname                 |     1               |   9.2ms | 99%     |
  groups_bysubgroup             |    53               |   2.0ms | 99%     |
  groups_byuuid                 |   460               | 627.2us | 99%     |
  groups_external               |     1               | 537.0ms | 99%     |
  groups_external_persisted     |                     | 535.9ms |  0%     |
  ldap_group_existence          |                     |         |         |
  ldap_groups                   |    63               |  12.3ms | 99%     |
  ldap_groups_byinclude         |                     |         |         |
  ldap_usernames                |  1024               | 223.9us | 99%     |
  permission_sort               |  1024               |  22.8us | 89%     |
  plugin_resources              |     9               |         | 76%     |
  project_list                  |     1               | 183.5ms | 99%     |
  projects                      |  1024               |   2.3ms | 99%     |
  prolog_rules                  |                     |         |         |
  soy_sauce_compiled_templates  |     1               |  91.5ms | 99%     |
  sshkeys                       |   378               |  24.5ms | 99%     |
  static_content                |    30               |   3.1ms | 85%     |
  lfs-lfs_project_locks         |    17               | 302.4us | 98%     |
  plugin-manager-plugins_list   |                     |  131.3s |  0%     |
  uploadvalidator-patternCache  |                     |         |         |
D accounts                      |   779   2053 974.50k|   1.6ms | 99% 100%|
D change_kind                   | 17064 141951  17.54m|   3.4ms | 97%  99%|
D comment_context               |  8773  39416  23.97m|   5.9ms | 92%  99%|
D conflicts                     | 37400 942956 128.04m|  66.9ms | 45%  99%|
D diff_intraline                | 19995  99457 128.04m|  22.0ms | 30%  97%|
D diff_summary                  |  1417 288083 218.75m|   6.8ms | 99%  99%|
D gerrit_file_diff              |299000 299358 345.42m|  20.7ms | 97%  39%|
D git_file_diff                 |150425 150725 172.73m|  25.6ms | 59%  17%|
D git_modified_files            |  7040  14009 127.83m|   3.7ms |  4%  49%|
D git_tags                      |     2      3 138.15k|  38.1ms | 98% 100%|
D groups_byuuid_persisted       |          136  86.31k|         |     100%|
D mergeability                  |  8738 131366  20.07m|  63.6ms | 53%  84%|
D modified_files                |  7451  26625 124.02m|   2.4ms | 95%  98%|
D oauth_tokens                  |                0.00k|         |         |
D persisted_projects            |         2365   4.34m|         |     100%|
D pure_revert                   |    79   2929 341.33k|  28.6ms | 99% 100%|
D web_sessions                  |  2048  17062   7.10m|         | 99% 100%|

SSH:      4  users, oldest session started   0 ms ago
Tasks:   24  total =    2 running +      0 ready +   22 sleeping
Mem: 30.16g total = 17.57g used + 4.58g free + 8.00g buffers
     56.00g max
         219 open files

Threads: 16 CPUs available, 260 threads.

 I adjusted the setting according to the result of

ssh -p 29418 localhost cache-chroniclemap analyze-h2-caches

 And below is my cache settings:

[cache "web_sessions"]
        maxAge = 4w
        memoryLimit = 2048

[cache "change_notes"]
        memoryLimit = 512m
diskLimit = 2g

[cache "git_modified_files"]
        memoryLimit = 1024m
diskLimit = 5g

[cache "git_file_diff"]
        memoryLimit = 1024m
diskLimit = 5g

[cache "gerrit_file_diff"]
        memoryLimit = 768m
diskLimit = 4g

[cache "diff_intraline"]
        memoryLimit = 512m

[cache "diff_summary"]
    memoryLimit = 512m

[cache "mergeability"]
        memoryLimit = 256m
diskLimit = 2g

[cache "conflicts"]
        memoryLimit = 256m
diskLimit = 2g

############################
#* Chronicle-map template **
############################

[cache "persisted_projects"]
maxEntries = 3980
avgKeySize = 114
avgValueSize = 1438
[cache "mergeability"]
maxEntries = 18200
avgKeySize = 152
avgValueSize = 20
[cache "pure_revert"]
maxEntries = 123
avgKeySize = 119
avgValueSize = 20
[cache "git_modified_files"]
maxEntries = 29331
avgKeySize = 124
avgValueSize = 20415
[cache "git_file_diff"]
maxEntries = 1786337
avgKeySize = 336
avgValueSize = 888
[cache "accounts"]
maxEntries = 817
avgKeySize = 52
avgValueSize = 413
[cache "web_sessions"]
maxEntries = 311197
avgKeySize = 68
avgValueSize = 376
[cache "git_tags"]
maxEntries = 2
avgKeySize = 21
avgValueSize = 74183
[cache "groups_byuuid_persisted"]
maxEntries = 56
avgKeySize = 128
avgValueSize = 528
[cache "modified_files"]
maxEntries = 4483
avgKeySize = 125
avgValueSize = 28211
[cache "conflicts"]
maxEntries = 150377
avgKeySize = 134
avgValueSize = 20
[cache "change_kind"]
maxEntries = 52341
avgKeySize = 110
avgValueSize = 32
[cache "comment_context"]
maxEntries = 11923
avgKeySize = 161
avgValueSize = 547
[cache "change_notes"]
maxEntries = 108211
avgKeySize = 83
avgValueSize = 10904
[cache "diff_intraline"]
maxEntries = 23363
avgKeySize = 1006
avgValueSize = 361
[cache "gerrit_file_diff"]
maxEntries = 2602456
avgKeySize = 336
avgValueSize = 921
[cache "diff_summary"]
maxEntries = 28149
avgKeySize = 192
avgValueSize = 2004

 Also the "git_file_diff" and "git_modified_files" suffer a very very low cache hits,

almost zero, (but the H2-V1 also have a low cache hits).

Br,

Yingchun

[Z]

Is there a solution to this problem? Thank you.

Yingchun Li 在 2025年12月30日 星期二下午5:11:18 [UTC+8] 的信中寫道：

[Luca Milanesio]

> On 30 Dec 2025, at 09:11, Yingchun Li <sword.l...@gmail.com> wrote:
>
> Hi,
> After switching to cache-chroniclemap, some caches(mostly the diff_summary)
> shows worse result than H2-V1.

Can you share the P95 and P99 of the Gerrit REST-API with H2v1 vs. cache-chroniclemap?
In our tests, it shows way better results and more stable performance.

The AvgGet isn’t executed in the cache but in the same code in Gerrit.
That shows how much it costs to load an entry in the cache.

HTH

Luca.

> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/180030ae-0444-4505-b426-afab7a82faf1n%40googlegroups.com.

[Yingchun Li]

On Tuesday, January 6, 2026 at 2:01:25 PM UTC+8 Luca Milanesio wrote:

> On 30 Dec 2025, at 09:11, Yingchun Li <sword.l...@gmail.com> wrote:
>
> Hi,
> After switching to cache-chroniclemap, some caches(mostly the diff_summary)
> shows worse result than H2-V1.

Can you share the P95 and P99 of the Gerrit REST-API with H2v1 vs. cache-chroniclemap?
In our tests, it shows way better results and more stable performance.

Could you give some methods to check the result of P95 and P99 ? I have installed the prometheus

plugin, but it is hard to install the gerrit-monitor plugin, there are too many version mismatch issues

and the some links in the doc  are invalid.

https://github.com/GerritCodeReview/gerrit-monitoring

The AvgGet value h2-v2 is not  so large, now both of my servers shows very large value

one is 

D diff_summary                  |   282    282  22.97m|   19.3s | 95%     |

and another one is :

D diff_intraline                | 16268  18433  36.44m|   7.8ms | 41%     |
D diff_summary                  | 19652  21690  69.37m|    9.6s | 92%     |

But strange thing is  I don't suffer a long broswer loading when going into my dashboard or changs,  and also there is  no other one complaining.

[Luca Milanesio]

> On 12 Jan 2026, at 07:47, Yingchun Li <sword.l...@gmail.com> wrote:
>
>
>
> On Tuesday, January 6, 2026 at 2:01:25 PM UTC+8 Luca Milanesio wrote:
>
>
> > On 30 Dec 2025, at 09:11, Yingchun Li <sword.l...@gmail.com> wrote:
> >
> > Hi,
> > After switching to cache-chroniclemap, some caches(mostly the diff_summary)
> > shows worse result than H2-V1.
>
> Can you share the P95 and P99 of the Gerrit REST-API with H2v1 vs. cache-chroniclemap?
> In our tests, it shows way better results and more stable performance.
>
> Could you give some methods to check the result of P95 and P99 ?

I would just crunch the Gerrit’s httpd_log, extract the latencies and use tools such as datamash [1] to calculate the P95 and P99.
You may well use other ways to calculate those values.

HTH

Luca.

[1] https://www.gnu.org/software/datamash/

> To view this discussion visit https://groups.google.com/d/msgid/repo-discuss/6f38643c-0739-4bf9-ba99-c41fcc1d3d96n%40googlegroups.com.