Upgrading to Gerrit 3.7.0, fails to start with "No index versions for index 'changes' ready"

635 views
Skip to first unread message

Chung Wu

unread,
Nov 17, 2022, 8:07:49 PM11/17/22
to Repo and Gerrit Discussion
Hi all,

We just upgraded from Gerrit 3.5.2 to 3.7.0. However, Gerrit no longer starts up.  Any help would be much appreciated!

We're using Docker Gerrit, and ran first with the command init.  However, upon starting up Gerrit, we see 

gerrit_1  | Running Gerrit ...
gerrit_1  | Running Gerrit Code Review:
gerrit_1  | Nov 18, 2022 12:59:11 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
gerrit_1  | WARNING: AssistedInject factory com.google.gerrit.server.api.changes.ChangeApiImpl$Factory will be slow because class com.google.gerrit.server.api.changes.ChangeApiImpl has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.server.api.changes.ChangeApiImpl.<init>()[48]]
gerrit_1  | [2022-11-18 00:59:14,042] [main] INFO  com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache
gerrit_1  | [2022-11-18 00:59:14,349] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue
gerrit_1  | [2022-11-18 00:59:14,427] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue
gerrit_1  | [2022-11-18 00:59:14,429] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue
gerrit_1  | [2022-11-18 00:59:14,494] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue
gerrit_1  | [2022-11-18 00:59:14,496] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue
gerrit_1  | [2022-11-18 00:59:15,290] [main] INFO  com.google.gerrit.server.rules.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000
gerrit_1  | [2022-11-18 00:59:15,297] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "gc".
gerrit_1  | [2022-11-18 00:59:15,297] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "changeCleanup".
gerrit_1  | Nov 18, 2022 12:59:15 AM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
gerrit_1  | WARNING: AssistedInject factory com.google.gerrit.sshd.DispatchCommand$Factory will be slow because class com.google.gerrit.sshd.DispatchCommand has assisted Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.sshd.BaseCommand.injector]
gerrit_1  | [2022-11-18T00:59:15.891Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SshCommandStart' queue
gerrit_1  | [2022-11-18T00:59:16.528Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Stream-Worker' queue
gerrit_1  | [2022-11-18T00:59:16.530Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Interactive-Worker' queue
gerrit_1  | [2022-11-18T00:59:16.531Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Batch-Worker' queue
gerrit_1  | [2022-11-18T00:59:16.532Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'MigrateExternalIdCase' queue
gerrit_1  | [2022-11-18T00:59:16.548Z] [main] WARN  com.google.gerrit.server.config.GitwebCgiConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)
gerrit_1  | [2022-11-18T00:59:17.530Z] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @8506ms to org.eclipse.jetty.util.log.Slf4jLog
gerrit_1  | [2022-11-18T00:59:17.698Z] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config
gerrit_1  | [2022-11-18T00:59:17.703Z] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 977m
gerrit_1  | [2022-11-18T00:59:20.789Z] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
gerrit_1  | com.google.inject.ProvisionException: Unable to provision, see the following errors:
gerrit_1  |
gerrit_1  | 1) No index versions for index 'changes' ready; run java -jar /var/gerrit/bin/gerrit.war reindex --index changes
gerrit_1  |
gerrit_1  | 1 error
gerrit_1  |     at com.google.gerrit.server.index.VersionManager.initIndex(VersionManager.java:172)
gerrit_1  |     at com.google.gerrit.server.index.VersionManager.start(VersionManager.java:95)
gerrit_1  |     at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:95)
gerrit_1  |     at com.google.gerrit.pgm.Daemon.start(Daemon.java:386)
gerrit_1  |     at com.google.gerrit.pgm.Daemon.run(Daemon.java:285)
gerrit_1  |     at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
gerrit_1  |     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
gerrit_1  |     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
gerrit_1  |     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
gerrit_1  |     at java.base/java.lang.reflect.Method.invoke(Method.java:566)
gerrit_1  |     at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
gerrit_1  |     at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
gerrit_1  |     at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
gerrit_1  |     at Main.main(Main.java:29)
gerrit_gerrit_1 exited with code 1


===

We did try to reindex as well, and saw:

gerrit_1  | WARNING: An illegal reflective access operation has occurred
gerrit_1  | WARNING: Illegal reflective access by com.google.gerrit.common.IoUtil (file:/var/gerrit/.gerritcodereview/tmp/gerrit_17071708961186393414_app/com_google_gerrit_common_libserver.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
gerrit_1  | WARNING: Please consider reporting this to the maintainers of com.google.gerrit.common.IoUtil
gerrit_1  | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
gerrit_1  | WARNING: All illegal access operations will be denied in a future release
gerrit_1  | [2022-11-18 00:57:01,774] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config
gerrit_1  | [2022-11-18 00:57:01,802] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 977m
gerrit_1  | [2022-11-18 00:57:03,083] [main] INFO  com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache
gerrit_1  | [2022-11-18 00:57:03,156] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue
gerrit_1  | [2022-11-18 00:57:03,164] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue
gerrit_1  | [2022-11-18 00:57:03,169] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue
gerrit_1  | [2022-11-18 00:57:03,339] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue
gerrit_1  | [2022-11-18 00:57:03,341] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue
gerrit_1  | [2022-11-18 00:57:03,390] [main] INFO  com.google.gerrit.server.rules.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000
gerrit_1  | [2022-11-18 00:57:04,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /var/gerrit/plugins
gerrit_1  | [2022-11-18 00:57:04,226] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin avatars-gravatar, version 4d45f66597
gerrit_1  | [2022-11-18 00:57:04,248] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin codemirror-editor, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,268] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,288] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin delete-project, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,306] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,347] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gerrit-oauth-provider, version 0f807f6
gerrit_1  | [2022-11-18 00:57:04,378] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,402] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,426] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,450] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,473] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,496] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,520] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin uploadvalidator, version v3.0.0-rc1-310-g7bf50a9ab1
gerrit_1  | [2022-11-18 00:57:04,544] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin webhooks, version v3.7.0
gerrit_1  | [2022-11-18 00:57:04,561] [main] WARN  com.google.gerrit.server.project.PeriodicProjectListCacheWarmer : project_list cache warmer is disabled
Reindexing changes: changes: 100% (11277/11277), project-slices: 100% (35/35), Slicing projects: 100% (38/38), done    
gerrit_1  | Reindexed 11277 documents in changes index in 50.7s (222.6/s)
gerrit_1  | Index changes in version 79 is ready
gerrit_1  | Cache Statistics at the end of reindexing
gerrit_1  |   Name                          |Entries              |  AvgGet |Hit Ratio|
gerrit_1  |                                 |   Mem   Disk   Space|         |Mem  Disk|
gerrit_1  | --------------------------------+---------------------+---------+---------+
gerrit_1  |   groups_external               |                     |         |         |
gerrit_1  |   groups_bysubgroup             |                     |         |         |
gerrit_1  |   change_notes                  |   221               |   3.8ms |  0%     |
gerrit_1  |   groups_byuuid                 |                     |         |         |
gerrit_1  |   default_preferences           |                     |         |         |
gerrit_1  |   prolog_rules                  |                     |         |         |
gerrit_1  |   groups                        |                     |         |         |
gerrit_1  |   projects                      |    26               | 266.3ms | 99%     |
gerrit_1  |   groups_byname                 |                     |         |         |
gerrit_1  |   groups_external_persisted     |                     |         |         |
gerrit_1  |   project_list                  |     1               |  12.4ms |  0%     |
gerrit_1  |   groups_bymember               |                     |         |         |
gerrit_1  |   permission_sort               |                     |         |         |
gerrit_1  |   external_ids_map              |                     |         |         |
gerrit_1  | D diff_summary                  |  7896  21773  13.05m|   2.4ms |  0% 100%|
gerrit_1  | D pure_revert                   |   100    160  18.01k|  25.2ms |  0% 100%|
gerrit_1  | D groups_byuuid_persisted       |           27  11.29k|         |         |
gerrit_1  | D change_kind                   |        14201   1.72m|         |         |
gerrit_1  | D mergeability                  |        61437   9.28m|         |         |
gerrit_1  | D conflicts                     |        14834   2.00m|         |         |
gerrit_1  | D diff_intraline                |        65504  89.14m|         |         |
gerrit_1  | D git_tags                      |                0.00k|         |         |
gerrit_1  | D gerrit_file_diff              |        23860  41.70m|         |         |
gerrit_1  | D git_file_diff                 |        21175  38.67m|         |         |
gerrit_1  | D accounts                      |          341 164.88k|         |         |
gerrit_1  | D persisted_projects            |           38  78.72k|         |     100%|
gerrit_1  | D git_modified_files            |                0.00k|         |         |
gerrit_1  | D modified_files                |                0.00k|         |         |
gerrit_1  |
gerrit_1  | [2022-11-18 00:57:56,325] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin replication, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,327] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin codemirror-editor, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,327] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin gitiles, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,327] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin reviewnotes, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,328] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin download-commands, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,328] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin commit-message-length-validator, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,328] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin webhooks, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,328] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin avatars-gravatar, version 4d45f66597
gerrit_1  | [2022-11-18 00:57:56,329] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin gerrit-oauth-provider, version 0f807f6
gerrit_1  | [2022-11-18 00:57:56,329] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin delete-project, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,329] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin plugin-manager, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,329] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin singleusergroup, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,330] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin hooks, version v3.7.0
gerrit_1  | [2022-11-18 00:57:56,330] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin uploadvalidator, version v3.0.0-rc1-310-g7bf50a9ab1
gerrit_1  | [2022-11-18 00:57:56,557] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_replication_221118_0057_12800627632096684444.jar
gerrit_1  | [2022-11-18 00:57:56,558] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_codemirror-editor_221118_0057_572299070437956224.jar
gerrit_1  | [2022-11-18 00:57:56,558] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_gitiles_221118_0057_5629862977742709338.jar
gerrit_1  | [2022-11-18 00:57:56,558] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_reviewnotes_221118_0057_9055715645535632836.jar
gerrit_1  | [2022-11-18 00:57:56,558] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_download-commands_221118_0057_422855622554322795.jar
gerrit_1  | [2022-11-18 00:57:56,559] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_commit-message-length-validator_221118_0057_8135329960398565749.jar
gerrit_1  | [2022-11-18 00:57:56,559] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_webhooks_221118_0057_8532048684970754343.jar
gerrit_1  | [2022-11-18 00:57:56,559] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_avatars-gravatar_221118_0057_5890827554712529255.jar
gerrit_1  | [2022-11-18 00:57:56,559] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_gerrit-oauth-provider_221118_0057_9476845543859991953.jar
gerrit_1  | [2022-11-18 00:57:56,560] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_delete-project_221118_0057_7853976825348025118.jar
gerrit_1  | [2022-11-18 00:57:56,560] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_plugin-manager_221118_0057_11551173459039953926.jar
gerrit_1  | [2022-11-18 00:57:56,561] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_singleusergroup_221118_0057_4504760207368603079.jar
gerrit_1  | [2022-11-18 00:57:56,561] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_hooks_221118_0057_6609462589917232378.jar
gerrit_1  | [2022-11-18 00:57:56,561] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_uploadvalidator_221118_0057_16547060919853405600.jar

gerrit_gerrit_1 exited with code 0

Chung Wu

unread,
Nov 17, 2022, 8:16:02 PM11/17/22
to Repo and Gerrit Discussion
Hmm I also noticed in my /var/gerrit/index/gerrit_index.config, that:

[index "changes_0079"]
    ready = false

Even though when I ran the re-index, it did print out:

gerrit_1  | Reindexed 11277 documents in changes index in 50.7s (222.6/s)
gerrit_1  | Index changes in version 79 is ready

Is the changes index ready or not 🤔

Chung Wu

unread,
Nov 17, 2022, 8:28:05 PM11/17/22
to Repo and Gerrit Discussion
Alright, I blew away /var/gerrit/index, and ran java -jar gerrit.war reindex, and it re-created the indexes with ready set to true. Now things are working.

Sorry for the spam; hope it helps somebody 😂

Luca Milanesio

unread,
Nov 18, 2022, 5:42:03 AM11/18/22
to Repo and Gerrit Discussion, Luca Milanesio, Chung Wu

On 18 Nov 2022, at 01:28, Chung Wu <chu...@gmail.com> wrote:

Alright, I blew away /var/gerrit/index, and ran java -jar gerrit.war reindex, and it re-created the indexes with ready set to true. Now things are working.

By looking at [1] I believe I’ve mentioned upgrading from v3.6.x, whilst you’re upgrading from v3.5.2 isn’t it?

I’ll add a note that any upgrade from versions *earlier* than v3.6 requires:
- offline upgrade
- offline reindexing

Luca.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/8f45cac9-fe58-42dd-9398-b4d67f714325n%40googlegroups.com.

Chung Wu

unread,
Nov 18, 2022, 1:01:49 PM11/18/22
to Repo and Gerrit Discussion
Yup, though I did perform the upgrade/reindex offline.

Luca Milanesio

unread,
Nov 18, 2022, 5:56:01 PM11/18/22
to Repo and Gerrit Discussion, Luca Milanesio, Chung Wu

On 18 Nov 2022, at 18:01, Chung Wu <chu...@gmail.com> wrote:

Yup, though I did perform the upgrade/reindex offline.

On Friday, November 18, 2022 at 2:42:03 AM UTC-8 lucamilanesio wrote:

On 18 Nov 2022, at 01:28, Chung Wu <chu...@gmail.com> wrote:

Alright, I blew away /var/gerrit/index, and ran java -jar gerrit.war reindex, and it re-created the indexes with ready set to true. Now things are working.

By looking at [1] I believe I’ve mentioned upgrading from v3.6.x, whilst you’re upgrading from v3.5.2 isn’t it?

I’ll add a note that any upgrade from versions *earlier* than v3.6 requires:
- offline upgrade
- offline reindexing

Reply all
Reply to author
Forward
0 new messages