3.9.2 reindex -- all changes disappear

353 views
Skip to first unread message

Aaron Smith

unread,
Mar 13, 2024, 6:34:20 PM3/13/24
to Repo and Gerrit Discussion
I upgraded our production Gerrit server from 3.9.1 to 3.9.2 this morning and all is well. I used a somewhat manual process that I've successfully used in the past.

In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

I did the upgrade in the sandbox by running the container with the argument 'init' to invoke the upgrade, something I hadn't tried before. I also manually ran the commands from the entrypoint script. Both scenarios resulted in all changes disappearing from the web UI. Changes are also not found when using gerrit query over SSH.

I determined that the step that causes changes to disappear is the reindex after init.

if [ ! -d /var/gerrit/git/All-Projects.git ] || [ "$1" == "init" ]
then
  echo "Initializing Gerrit site ..."
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war init --batch --install-all-plugins -d /var/gerrit
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "-Djava.security.egd=file:/dev/./urandom"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.net=ALL-UNNAMED"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.lang.invoke=ALL-UNNAMED"
fi

The git repositories and the postgres database are still there and are about the same size as before the reindex, so the changes haven't been removed from disk. Gerrit just isn't finding them.

I would appreciate any insight into why this happened, whether it can be fixed in-place, if I did something incorrect, and if whether is a bug.

Here is the console output from the reindex and Gerrit startup.

gerrit@63754ad91d4e:/$ export JAVA_OPTS='--add-opens java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang.invoke=ALL-UNNAMED'                                            [115/90344]
gerrit@63754ad91d4e:/$   java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
[2024-03-13 21:51:18,804] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config
[2024-03-13 21:51:18,821] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 972m
[2024-03-13 21:51:19,662] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue
[2024-03-13 21:51:19,669] [main] INFO  com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache
[2024-03-13 21:51:19,704] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue
[2024-03-13 21:51:19,707] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue
[2024-03-13 21:51:19,851] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue
[2024-03-13 21:51:19,852] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue
[2024-03-13 21:51:20,027] [main] INFO  com.google.gerrit.server.rules.prolog.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000
[2024-03-13 21:51:20,431] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /var/gerrit/plugins
[2024-03-13 21:51:20,509] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin codemirror-editor, version v3.9.2
[2024-03-13 21:51:20,529] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v3.9.2
[2024-03-13 21:51:20,554] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin delete-project, version v3.9.2
[2024-03-13 21:51:20,575] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v3.9.2
[2024-03-13 21:51:20,604] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.9.2
[2024-03-13 21:51:20,624] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v3.9.2
[2024-03-13 21:51:20,646] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v3.9.2
[2024-03-13 21:51:20,666] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin readonly, version fd0d01b1f8
[2024-03-13 21:51:20,689] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v3.9.2
[2024-03-13 21:51:20,709] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v3.9.2
[2024-03-13 21:51:20,741] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v3.9.2
[2024-03-13 21:51:20,760] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin uploadvalidator, version v3.0.0-rc1-316-gd83c4085af
[2024-03-13 21:51:20,778] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin webhooks, version v3.9.2
[2024-03-13 21:51:20,781] [main] WARN  com.google.gerrit.server.project.PeriodicProjectListCacheWarmer : project_list cache warmer is disabled
Collecting accounts:    598
Reindexing accounts:    100% (598/598)
Reindexed 598 documents in accounts index in 4.3s (140.7/s)
Index accounts in version 13 is ready
Reindexing groups:      100% (29/29)
Reindexed 29 documents in groups index in 0.3s (100.7/s)
Index groups in version 10 is ready
[2024-03-13 21:51:25,516] [main] INFO  com.google.gerrit.server.git.MultiProgressMonitor : Reindexing changes: Slicing projects: 2% (11/548) [CONTEXT ratelimit_period="1 MINUTES" ]
Reindexing changes: changes: 62% (47849/77079), project-slices: 82% (201/244), Slicing projects: 100% (548/548) (/)[2024-03-13 21:52:25,896] [main] INFO  com.google.gerrit.server.git.MultiPr
ogressMonitor : Reindexing changes: changes: 62% (47977/77079), project-slices: 82% (201/244), Slicing projects: 100% (548/548) (-) [CONTEXT ratelimit_period="1 MINUTES [skipped: 215]" ]
Reindexing changes: changes: 88% (68527/77079), project-slices: 95% (232/244), Slicing projects: 100% (548/548) (/)[2024-03-13 21:53:25,994] [main] INFO  com.google.gerrit.server.git.MultiPr
ogressMonitor : Reindexing changes: changes: 89% (69199/77079), project-slices: 95% (234/244), Slicing projects: 100% (548/548) (-) [CONTEXT ratelimit_period="1 MINUTES [skipped: 119]" ]
Reindexing changes: changes: 100% (77079/77079), project-slices: 100% (244/244), Slicing projects: 100% (548/548), done
Reindexed 77079 documents in changes index in 130.0s (592.8/s)
Index changes in version 84 is ready
Reindexing projects:    100% (548/548)
Reindexed 548 documents in projects index in 0.6s (996.4/s)
Index projects in version 8 is ready
[2024-03-13 21:53:36,122] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin replication, version v3.9.2                                                 [71/90344]
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin codemirror-editor, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin gitiles, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin reviewnotes, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin download-commands, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin commit-message-length-validator, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin readonly, version fd0d01b1f8
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin webhooks, version v3.9.2
[2024-03-13 21:53:36,123] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin delete-project, version v3.9.2
[2024-03-13 21:53:36,124] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin plugin-manager, version v3.9.2
[2024-03-13 21:53:36,124] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin singleusergroup, version v3.9.2
[2024-03-13 21:53:36,124] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin hooks, version v3.9.2
[2024-03-13 21:53:36,124] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Unloading plugin uploadvalidator, version v3.0.0-rc1-316-gd83c4085af
[2024-03-13 21:53:36,124] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_replication_240313_2151_13677347049733325583.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_codemirror-editor_240313_2151_7175847933928749595.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_gitiles_240313_2151_8903372032764808185.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_reviewnotes_240313_2151_13029319072669980925.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_download-commands_240313_2151_10699123469168645564.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_commit-message-length-validator_240313_2151_16267772777653005929.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_readonly_240313_2151_778813578405501516.jar
[2024-03-13 21:53:36,125] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_webhooks_240313_2151_11269364659768755851.jar
[2024-03-13 21:53:36,126] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_delete-project_240313_2151_7622641239250458000.jar
[2024-03-13 21:53:36,126] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_plugin-manager_240313_2151_6133754582356559867.jar
[2024-03-13 21:53:36,126] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_singleusergroup_240313_2151_11503232738250626258.jar
[2024-03-13 21:53:36,126] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_hooks_240313_2151_11774524149775124935.jar
[2024-03-13 21:53:36,126] [main] INFO  com.google.gerrit.server.plugins.CleanupHandle : Cleaned plugin plugin_uploadvalidator_240313_2151_14847182623560441692.jar

8<------------8<------------8<------------8<------------8<

gerrit@63754ad91d4e:/$ /entrypoint.sh
Running Gerrit ...
Running Gerrit Code Review:
Mar 13, 2024 9:53:49 PM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
WARNING: AssistedInject factory com.google.gerrit.server.api.changes.ChangeApiImpl$Factory will be slow because class com.google.gerrit.server.api.changes.ChangeApiImpl has assisted Provider
 dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending d
eps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.google.gerrit.server.api.changes.ChangeApiImpl.<init>()[48]]
[2024-03-13 21:53:51,450] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'WorkQueue' queue
[2024-03-13 21:53:51,485] [main] INFO  com.google.gerrit.server.cache.PersistentCacheBaseFactory : Enabling disk cache /var/gerrit/cache
[2024-03-13 21:53:51,709] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Interactive' queue
[2024-03-13 21:53:51,711] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'Index-Batch' queue
[2024-03-13 21:53:51,753] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'ReceiveCommits' queue
[2024-03-13 21:53:51,754] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SendEmail' queue
[2024-03-13 21:53:53,290] [main] INFO  com.google.gerrit.server.rules.prolog.PrologEnvironment : reductionLimit: 100000, compileLimit: 1000000
[2024-03-13 21:53:53,294] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "gc".
[2024-03-13 21:53:53,295] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "changeCleanup".
[2024-03-13 21:53:53,295] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "attentionSet".
Mar 13, 2024 9:53:53 PM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
WARNING: AssistedInject factory com.google.gerrit.sshd.DispatchCommand$Factory will be slow because class com.google.gerrit.sshd.DispatchCommand has assisted Provider dependencies [27/90344]
 the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact offending deps are: [Key[type=com.g
oogle.inject.Injector, annotation=[none]]@com.google.gerrit.sshd.BaseCommand.injector]
[2024-03-13T21:53:53.658Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SshCommandStart' queue
[2024-03-13T21:53:53.978Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Stream-Worker' queue
[2024-03-13T21:53:53.980Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Interactive-Worker' queue
[2024-03-13T21:53:53.980Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'SSH-Batch-Worker' queue
[2024-03-13T21:53:53.981Z] [main] INFO  com.google.gerrit.server.git.WorkQueue : Adding metrics for 'MigrateExternalIdCase' queue
[2024-03-13T21:53:54.255Z] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @6833ms to org.eclipse.jetty.util.log.Slf4jLog
[2024-03-13T21:53:54.402Z] [main] INFO  com.google.gerrit.server.git.SystemReaderInstaller : Set JGit's SystemReader to read system config from /var/gerrit/etc/jgit.config
[2024-03-13T21:53:54.406Z] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 2047m
[2024-03-13T21:53:55.175Z] [main] WARN  com.google.gerrit.server.project.PeriodicProjectListCacheWarmer : project_list cache warmer is disabled
[2024-03-13T21:53:55.178Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /var/gerrit/plugins
[2024-03-13T21:53:55.272Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin codemirror-editor, version v3.9.2
Mar 13, 2024 9:53:55 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely igno
re this message. If this is NOT deliberate however, your application may not work as expected.
[2024-03-13T21:53:55.338Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin commit-message-length-validator, version v3.9.2
Mar 13, 2024 9:53:55 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely igno
re this message. If this is NOT deliberate however, your application may not work as expected.
[2024-03-13T21:53:55.458Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin delete-project, version v3.9.2
[2024-03-13T21:53:55.529Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v3.9.2
[2024-03-13T21:53:55.576Z] [main] INFO  com.google.gerrit.server.config.FileBasedGlobalPluginConfigProvider : No /var/gerrit/etc/gitiles.config; assuming defaults
Mar 13, 2024 9:53:55 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely igno
re this message. If this is NOT deliberate however, your application may not work as expected.
[2024-03-13T21:53:55.596Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin gitiles, version v3.9.2
[2024-03-13T21:53:55.682Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin hooks, version v3.9.2
Mar 13, 2024 9:53:55 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely igno
re this message. If this is NOT deliberate however, your application may not work as expected.
[2024-03-13T21:53:55.763Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin plugin-manager, version v3.9.2
[2024-03-13T21:53:55.771Z] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : Start-up: pre-loading list of plugins from registry
Mar 13, 2024 9:53:55 PM com.google.inject.servlet.GuiceFilter setPipeline
WARNING: Multiple Servlet injectors detected. This is a warning indicating that you have more than one GuiceFilter running in your web application. If this is deliberate, you may safely igno
re this message. If this is NOT deliberate however, your application may not work as expected.
[2024-03-13T21:53:55.899Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin readonly, version fd0d01b1f8
Mar 13, 2024 9:53:56 PM com.google.inject.assistedinject.FactoryProvider2 isValidForOptimizedAssistedInject
WARNING: AssistedInject factory com.googlesource.gerrit.plugins.replication.Destination$Factory will be slow because class com.googlesource.gerrit.plugins.replication.Destination has assiste
d Provider dependencies or injects the Injector. Stop injecting @Assisted Provider<T> (instead use @Assisted T) or Injector to speed things up. (It will be a ~6500% speed bump!)  The exact o
ffending deps are: [Key[type=com.google.inject.Injector, annotation=[none]]@com.googlesource.gerrit.plugins.replication.Destination.<init>()[0]]
[2024-03-13T21:53:56.076Z] [main] WARN  com.googlesource.gerrit.plugins.replication.DestinationConfigParser : Replication config does not exist or it's empty; not replicating
[2024-03-13T21:53:56.090Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v3.9.2
[2024-03-13T21:53:56.148Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v3.9.2
[2024-03-13T21:53:56.207Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin singleusergroup, version v3.9.2
[2024-03-13T21:53:56.383Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin uploadvalidator, version v3.0.0-rc1-316-gd83c4085af
[2024-03-13T21:53:56.457Z] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7a6c8dbc[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@214
af01f[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@635791b7]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expi
red git repositories...
[2024-03-13T21:53:56.466Z] [WorkQueue-1[java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7a6c8dbc[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@214
af01f[Wrapped task = com.google.gerrit.server.logging.LoggingContextAwareRunnable@635791b7]]]] INFO  com.googlesource.gerrit.plugins.deleteproject.fs.RepositoryCleanupTask : Cleaning up expi
red git repositories... Done
[2024-03-13T21:53:56.524Z] [main] INFO  com.google.gerrit.server.plugins.PluginLoader : Loaded plugin webhooks, version v3.9.2
[2024-03-13T21:53:56.554Z] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : No schedule configuration for "accountDeactivation".
[2024-03-13T21:53:56.589Z] [main] INFO  com.google.gerrit.sshd.SshDaemon : Started Gerrit APACHE-SSHD-2.12.0 on *:29418
[2024-03-13T21:53:56.593Z] [main] INFO  org.eclipse.jetty.server.Server : jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.10+
7
[2024-03-13T21:53:56.686Z] [main] INFO  org.eclipse.jetty.server.session : DefaultSessionIdManager workerName=node0
[2024-03-13T21:53:56.686Z] [main] INFO  org.eclipse.jetty.server.session : No SessionScavenger set, using defaults
[2024-03-13T21:53:56.700Z] [main] INFO  org.eclipse.jetty.server.session : node0 Scavenging every 600000ms
[2024-03-13T21:53:56.750Z] [main] INFO  com.google.gerrit.server.config.FileBasedGlobalPluginConfigProvider : No /var/gerrit/etc/readonly.config; assuming defaults
[2024-03-13T21:53:56.899Z] [main] INFO  com.googlesource.gerrit.plugins.gitiles.HttpModule : No /var/gerrit/etc/gitiles.config; assuming defaults
[2024-03-13T21:53:58.210Z] [main] INFO  org.eclipse.jetty.server.handler.ContextHandler : Started o.e.j.s.ServletContextHandler@18092b96{/,null,AVAILABLE}
[2024-03-13T21:53:58.231Z] [main] INFO  org.eclipse.jetty.server.AbstractConnector : Started ServerConnector@3ce34b92{HTTP/1.1, (http/1.1)}{0.0.0.0:8081}
[2024-03-13T21:53:58.245Z] [main] INFO  org.eclipse.jetty.server.Server : Started @10824ms
[2024-03-13T21:53:58.246Z] [main] INFO  com.google.gerrit.pgm.Daemon : Gerrit Code Review 3.9.2 ready
[2024-03-13T21:54:06.077Z] [plugin-manager-preloader] INFO  com.googlesource.gerrit.plugins.manager.OnStartStop : 41 plugins successfully pre-loaded


Martin Fick

unread,
Mar 13, 2024, 6:41:01 PM3/13/24
to Aaron Smith, Repo and Gerrit Discussion
On Wed, Mar 13, 2024 at 4:34 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

Sounds like an indexing issue. Are you sure the container volumes where the indexes are kept are persisted and visible to the running Gerrit process?

-Martin

Matthias Sohn

unread,
Mar 13, 2024, 6:41:49 PM3/13/24
to Aaron Smith, Repo and Gerrit Discussion
On Wed, Mar 13, 2024 at 11:34 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
I upgraded our production Gerrit server from 3.9.1 to 3.9.2 this morning and all is well. I used a somewhat manual process that I've successfully used in the past.

In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

I did the upgrade in the sandbox by running the container with the argument 'init' to invoke the upgrade, something I hadn't tried before. I also manually ran the commands from the entrypoint script. Both scenarios resulted in all changes disappearing from the web UI. Changes are also not found when using gerrit query over SSH.

I determined that the step that causes changes to disappear is the reindex after init.

if [ ! -d /var/gerrit/git/All-Projects.git ] || [ "$1" == "init" ]
then
  echo "Initializing Gerrit site ..."
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war init --batch --install-all-plugins -d /var/gerrit
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "-Djava.security.egd=file:/dev/./urandom"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.net=ALL-UNNAMED"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.lang.invoke=ALL-UNNAMED"
fi

The git repositories and the postgres database are still there and are about the same size as before the reindex, so the changes haven't been removed from disk. Gerrit just isn't finding them.

I would appreciate any insight into why this happened, whether it can be fixed in-place, if I did something incorrect, and if whether is a bug.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0cc39463-c37e-432b-aab1-30a8720a8bdan%40googlegroups.com.

Aaron Smith

unread,
Mar 13, 2024, 6:48:48 PM3/13/24
to Repo and Gerrit Discussion
Yes, this is a setup I've used since 2022 with no issues. Running the upgrade without the reindexing step works as expected. Fortunately, the upgrade didn't require a reindex on this system.

-Martin

Aaron Smith

unread,
Mar 13, 2024, 6:55:19 PM3/13/24
to Repo and Gerrit Discussion
On Wednesday, March 13, 2024 at 3:41:49 PM UTC-7 Matthias Sohn wrote:
On Wed, Mar 13, 2024 at 11:34 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
I upgraded our production Gerrit server from 3.9.1 to 3.9.2 this morning and all is well. I used a somewhat manual process that I've successfully used in the past.

In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

I did the upgrade in the sandbox by running the container with the argument 'init' to invoke the upgrade, something I hadn't tried before. I also manually ran the commands from the entrypoint script. Both scenarios resulted in all changes disappearing from the web UI. Changes are also not found when using gerrit query over SSH.

I determined that the step that causes changes to disappear is the reindex after init.

if [ ! -d /var/gerrit/git/All-Projects.git ] || [ "$1" == "init" ]
then
  echo "Initializing Gerrit site ..."
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war init --batch --install-all-plugins -d /var/gerrit
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "-Djava.security.egd=file:/dev/./urandom"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.net=ALL-UNNAMED"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.lang.invoke=ALL-UNNAMED"
fi

The git repositories and the postgres database are still there and are about the same size as before the reindex, so the changes haven't been removed from disk. Gerrit just isn't finding them.

I would appreciate any insight into why this happened, whether it can be fixed in-place, if I did something incorrect, and if whether is a bug.


You are correct. I re-ran the full reindex and (as noted in a bug comment "if you re-run the offline reindex they all appear again. The index is empty, then the reindex will re-create them.") the changes all reappeared.

It's too bad that the defective change wasn't reverted for the 3.9.2 release. I hope I'm the only one who runs into this -- when changes disappear after an upgrade, it definitely increases the admin's blood pressure!

Thank you for pointing me to the source of the issue and the fix. Here's hoping this is fixed in the next patch release.

Matthias Sohn

unread,
Mar 13, 2024, 7:13:31 PM3/13/24
to Aaron Smith, Repo and Gerrit Discussion
On Wed, Mar 13, 2024 at 11:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, March 13, 2024 at 3:41:49 PM UTC-7 Matthias Sohn wrote:
On Wed, Mar 13, 2024 at 11:34 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
I upgraded our production Gerrit server from 3.9.1 to 3.9.2 this morning and all is well. I used a somewhat manual process that I've successfully used in the past.

In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

I did the upgrade in the sandbox by running the container with the argument 'init' to invoke the upgrade, something I hadn't tried before. I also manually ran the commands from the entrypoint script. Both scenarios resulted in all changes disappearing from the web UI. Changes are also not found when using gerrit query over SSH.

I determined that the step that causes changes to disappear is the reindex after init.

if [ ! -d /var/gerrit/git/All-Projects.git ] || [ "$1" == "init" ]
then
  echo "Initializing Gerrit site ..."
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war init --batch --install-all-plugins -d /var/gerrit
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "-Djava.security.egd=file:/dev/./urandom"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.net=ALL-UNNAMED"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.lang.invoke=ALL-UNNAMED"
fi

The git repositories and the postgres database are still there and are about the same size as before the reindex, so the changes haven't been removed from disk. Gerrit just isn't finding them.

I would appreciate any insight into why this happened, whether it can be fixed in-place, if I did something incorrect, and if whether is a bug.


You are correct. I re-ran the full reindex and (as noted in a bug comment "if you re-run the offline reindex they all appear again. The index is empty, then the reindex will re-create them.") the changes all reappeared.

It's too bad that the defective change wasn't reverted for the 3.9.2 release. I hope I'm the only one who runs into this -- when changes disappear after an upgrade, it definitely increases the admin's blood pressure!

Thank you for pointing me to the source of the issue and the fix. Here's hoping this is fixed in the next patch release.

Nasser Grainawi

unread,
Apr 4, 2024, 12:01:41 PM4/4/24
to Matthias Sohn, Aaron Smith, Repo and Gerrit Discussion
On Wed, Mar 13, 2024 at 5:13 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Wed, Mar 13, 2024 at 11:55 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
On Wednesday, March 13, 2024 at 3:41:49 PM UTC-7 Matthias Sohn wrote:
On Wed, Mar 13, 2024 at 11:34 PM 'Aaron Smith' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:
I upgraded our production Gerrit server from 3.9.1 to 3.9.2 this morning and all is well. I used a somewhat manual process that I've successfully used in the past.

In preparation for this, I performed this upgrade in a sandbox environment. I tried using the Gerrit docker container's entrypoint script to perform the upgrade and ended up with no changes showing up in Gerrit.

I did the upgrade in the sandbox by running the container with the argument 'init' to invoke the upgrade, something I hadn't tried before. I also manually ran the commands from the entrypoint script. Both scenarios resulted in all changes disappearing from the web UI. Changes are also not found when using gerrit query over SSH.

I determined that the step that causes changes to disappear is the reindex after init.

if [ ! -d /var/gerrit/git/All-Projects.git ] || [ "$1" == "init" ]
then
  echo "Initializing Gerrit site ..."
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war init --batch --install-all-plugins -d /var/gerrit
  java $JAVA_OPTS -jar /var/gerrit/bin/gerrit.war reindex -d /var/gerrit
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "-Djava.security.egd=file:/dev/./urandom"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.net=ALL-UNNAMED"
  git config -f /var/gerrit/etc/gerrit.config --add container.javaOptions "--add-opens java.base/java.lang.invoke=ALL-UNNAMED"
fi

The git repositories and the postgres database are still there and are about the same size as before the reindex, so the changes haven't been removed from disk. Gerrit just isn't finding them.

I would appreciate any insight into why this happened, whether it can be fixed in-place, if I did something incorrect, and if whether is a bug.


You are correct. I re-ran the full reindex and (as noted in a bug comment "if you re-run the offline reindex they all appear again. The index is empty, then the reindex will re-create them.") the changes all reappeared.

It's too bad that the defective change wasn't reverted for the 3.9.2 release. I hope I'm the only one who runs into this -- when changes disappear after an upgrade, it definitely increases the admin's blood pressure!

It will be reverted for 3.9.3, sorry about that: 417857: Revert "During online reindexing of all changes skip changes already present" | https://gerrit-review.googlesource.com/c/gerrit/+/417857
 
Reply all
Reply to author
Forward
0 new messages