Gerrit won't do an online index (during upgrade to 2.14.3)

[Matthew Webber]

I just upgraded Gerrit from 2.13.9 to 2.14.3, and I thought I'd try doing an online index update, rather than the offline reindex as normal.

However, it didn't work. What did I do wrong?

# Upgrade, taking the default answers to every prompt

java -jar gerrit-2.14.3.war init -d ${site_path}/ --delete-caches --no-auto-start

That works ok, and the database schema is updated.

 

# I see this message, which tells me I need to do an offline reindex:

The index must be rebuilt before starting Gerrit:

  java -jar gerrit.war reindex -d site_path

# Let's ignore that, it must be an old message, online reindex is the modern way, right?

# Start Gerrit. It fails to start, and in error_log I see:

[2017-09-05 19:03:49,035] [Reindex v32-v39] INFO  com.google.gerrit.server.index.OnlineReindexer : Starting online reindex from schema version 32 to 39

[2017-09-05 19:03:49,042] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /mnt/gerrit/prod/gerrit/bin/gerrit.war reindex

1 error

at com.google.gerrit.server.index.AbstractVersionManager.initIndex(AbstractVersionManager.java:145)

at com.google.gerrit.server.index.AbstractVersionManager.start(AbstractVersionManager.java:76)

at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:92)

at com.google.gerrit.pgm.Daemon.start(Daemon.java:323)

at com.google.gerrit.pgm.Daemon.run(Daemon.java:232)

at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:204)

at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:108)

at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:63)

at Main.main(Main.java:24)

# So I do an offline reindex, and after that Gerrit starts ok

java -jar ${site_path}/bin/gerrit.war reindex --threads 20 -d ${site_path}

Matthew

[Saša Živkov]

On Wed, Sep 6, 2017 at 3:15 PM, Matthew Webber <mat...@unsolvable.org> wrote:

I just upgraded Gerrit from 2.13.9 to 2.14.3, and I thought I'd try doing an online index update, rather than the offline reindex as normal.

However, it didn't work. What did I do wrong?

# Upgrade, taking the default answers to every prompt

java -jar gerrit-2.14.3.war init -d ${site_path}/ --delete-caches --no-auto-start

That works ok, and the database schema is updated.

 

# I see this message, which tells me I need to do an offline reindex:

The index must be rebuilt before starting Gerrit:

  java -jar gerrit.war reindex -d site_path

# Let's ignore that, it must be an old message, online reindex is the modern way, right?

# Start Gerrit. It fails to start, and in error_log I see:

[2017-09-05 19:03:49,035] [Reindex v32-v39] INFO  com.google.gerrit.server.index.OnlineReindexer : Starting online reindex from schema version 32 to 39

[2017-09-05 19:03:49,042] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /mnt/gerrit/prod/gerrit/bin/gerrit.war reindex

My guess is that your previous upgrade (to 2.13.x) didn't successfully finish reindexing to the v32 index version.

Obviously, 2.14 requires at least v32 to be ready (to serve searches until online reindexing to v39 is in progress) but it doesn't find one.

You can verify my assumption by looking inside: $SITE/index/gerrit_index.config. 

What is the status of the changes index version 32? I guess it is "ready = false".

If my assumptions are true, offline reindexing is probably the only way to proceed.

1 error

at com.google.gerrit.server.index.AbstractVersionManager.initIndex(AbstractVersionManager.java:145)

at com.google.gerrit.server.index.AbstractVersionManager.start(AbstractVersionManager.java:76)

at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:92)

at com.google.gerrit.pgm.Daemon.start(Daemon.java:323)

at com.google.gerrit.pgm.Daemon.run(Daemon.java:232)

at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:204)

at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:108)

at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:63)

at Main.main(Main.java:24)

# So I do an offline reindex, and after that Gerrit starts ok

java -jar ${site_path}/bin/gerrit.war reindex --threads 20 -d ${site_path}

Matthew

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Borowitz]

On Wed, Sep 6, 2017 at 9:37 AM, Saša Živkov <ziv...@gmail.com> wrote:

On Wed, Sep 6, 2017 at 3:15 PM, Matthew Webber <mat...@unsolvable.org> wrote:

I just upgraded Gerrit from 2.13.9 to 2.14.3, and I thought I'd try doing an online index update, rather than the offline reindex as normal.

However, it didn't work. What did I do wrong?

# Upgrade, taking the default answers to every prompt

java -jar gerrit-2.14.3.war init -d ${site_path}/ --delete-caches --no-auto-start

That works ok, and the database schema is updated.

 

# I see this message, which tells me I need to do an offline reindex:

The index must be rebuilt before starting Gerrit:

  java -jar gerrit.war reindex -d site_path

# Let's ignore that, it must be an old message, online reindex is the modern way, right?

# Start Gerrit. It fails to start, and in error_log I see:

[2017-09-05 19:03:49,035] [Reindex v32-v39] INFO  com.google.gerrit.server.index.OnlineReindexer : Starting online reindex from schema version 32 to 39

[2017-09-05 19:03:49,042] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /mnt/gerrit/prod/gerrit/bin/gerrit.war reindex

My guess is that your previous upgrade (to 2.13.x) didn't successfully finish reindexing to the v32 index version.

Obviously, 2.14 requires at least v32 to be ready (to serve searches until online reindexing to v39 is in progress) but it doesn't find one.

You can verify my assumption by looking inside: $SITE/index/gerrit_index.config. 

What is the status of the changes index version 32? I guess it is "ready = false".

Unfortunately, this can't be checked after the fact, because the offline reindex (final step in OP) would have set all old versions to ready=false.

Other than that, your hypothesis is reasonable, I just don't know of a way to check one way or the other what happened unless Matthew has a full backup of the site dir prior to reindexing.

[Marcelo Ávila de Oliveira]

I got the same problem with my upgrade.

Before the upgrade my gerrit_index.config was:

[index "changes_0004"]

ready = false

[index "changes_0008"]

ready = false

[index "changes_0011"]

ready = false

[index "changes_0014"]

ready = false

[index "changes_0025"]

ready = true

[index "accounts_0003"]

ready = true

[index "changes_0032"]

ready = true

There were two changes_ with ready=true, 0025 and 0032, is this expected?

Now, after update + offline reindex I have:

[index "changes_0004"]

ready = false

[index "changes_0008"]

ready = false

[index "changes_0011"]

ready = false

[index "changes_0014"]

ready = false

[index "changes_0025"]

ready = false

[index "accounts_0003"]

ready = false

[index "changes_0032"]

ready = false

[index "accounts_0004"]

ready = true

[index "groups_0002"]

ready = true

[index "changes_0039"]

ready = true

Is it OK?

--

Marcelo Ávila de Oliveira

mav...@cpqd.com.br

[Matthew Webber]

>> Other than that, your hypothesis is reasonable, I just don't know of a way to check one way or the other what happened unless Matthew has a full backup of the site dir prior to reindexing

Luckily, I do! (Not luck actually, I'm just careful about that kind of thing).

Here's what it looked like before the upgrade

[index "25"]

[index "changes_0025"]

ready = true

[index "accounts_0003"]

ready = true

[index "changes_0032"]

ready = true

And here's what it looks like now, after upgrade and offline reindex

[index "25"]

[index "changes_0025"]

ready = false

[index "accounts_0003"]

ready = false

[index "changes_0032"]

ready = false

[index "accounts_0004"]

ready = true

[index "groups_0002"]

ready = true

[index "changes_0039"]

ready = true

Matthew

[Saša Živkov]

On Wed, Sep 6, 2017 at 4:26 PM, Matthew Webber <mat...@unsolvable.org> wrote:

>> Other than that, your hypothesis is reasonable, I just don't know of a way to check one way or the other what happened unless Matthew has a full backup of the site dir prior to reindexing

Luckily, I do! (Not luck actually, I'm just careful about that kind of thing).

Here's what it looked like before the upgrade

[index "25"]

[index "changes_0025"]

ready = true

[index "accounts_0003"]

ready = true

[index "changes_0032"]

ready = true

This proves that my hypothesis was wrong.

[Hugo Arès]

On Wednesday, September 6, 2017 at 9:15:43 AM UTC-4, Matthew Webber wrote:

I just upgraded Gerrit from 2.13.9 to 2.14.3, and I thought I'd try doing an online index update, rather than the offline reindex as normal.

However, it didn't work. What did I do wrong?

# Upgrade, taking the default answers to every prompt

java -jar gerrit-2.14.3.war init -d ${site_path}/ --delete-caches --no-auto-start

That works ok, and the database schema is updated.

 

# I see this message, which tells me I need to do an offline reindex:

The index must be rebuilt before starting Gerrit:

  java -jar gerrit.war reindex -d site_path

# Let's ignore that, it must be an old message, online reindex is the modern way, right?

Online reindex is only possible if there is a previous version of an index. In Gerrit 2.14, there are 3 indexes:
changes, accounts and groups. Groups index was introduced in 2.14 so that one must be reindexed offline.

The init message is not precise enough, it does not tell you which index absolutely need to be reindexed offline.

To upgrade from 2.13 to 2.14 and use the online reindex for accounts and changes:

-java -jar gerrit-war.war init -d /your/review-site
-java -jar gerrit-war.war reindex --index groups -d /your/review-site
-start gerrit and online reindex should be started for accounts and changes indexes
 

[Matthew Webber]

On Thursday, 7 September 2017 00:45:03 UTC+1, Hugo Arès wrote:

Online reindex is only possible if there is a previous version of an index. In Gerrit 2.14, there are 3 indexes:
changes, accounts and groups. Groups index was introduced in 2.14 so that one must be reindexed offline.

The init message is not precise enough, it does not tell you which index absolutely need to be reindexed offline.

To upgrade from 2.13 to 2.14 and use the online reindex for accounts and changes:

-java -jar gerrit-war.war init -d /your/review-site
-java -jar gerrit-war.war reindex --index groups -d /your/review-site
-start gerrit and online reindex should be started for accounts and changes indexes 

Perfect! That explains everything.

In additional to an unhelpful init message. there's also a documentation issue here.

I'll try and write up something to add to the documentation.

Matthew