"No index versions ready" error after update from 2.12.5 to 2.13.(already run reindex)

[Ben Wang]

After update gerrit server from 2.12.5 to 2.13, I can't start up the server with the error message below.

I have already run "reindex" and still got this error.

Any guys have ideas/tips/suggestions on this? Thanks a lot.

Error Log:

[2016-11-07 18:20:11,687] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @5996ms

[2016-11-07 18:20:11,933] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 2047m

[2016-11-07 18:20:11,961] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /home/gerrit/site_path/bin/gerrit.war reindex

1 error

        at com.google.gerrit.lucene.LuceneVersionManager.initIndex(LuceneVersionManager.java:146)

        at com.google.gerrit.lucene.LuceneVersionManager.start(LuceneVersionManager.java:119)

        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:89)

        at com.google.gerrit.pgm.Daemon.start(Daemon.java:311)

        at com.google.gerrit.pgm.Daemon.run(Daemon.java:212)

        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:64)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:161)

        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:102)

        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:59)

        at Main.main(Main.java:25)

[Saša Živkov]

On Mon, Nov 7, 2016 at 11:28 AM, Ben Wang <alber...@gmail.com> wrote:

After update gerrit server from 2.12.5 to 2.13, I can't start up the server with the error message below.

I have already run "reindex" and still got this error.

Any guys have ideas/tips/suggestions on this? Thanks a lot.

Error Log:

[2016-11-07 18:20:11,687] [main] INFO  org.eclipse.jetty.util.log : Logging initialized @5996ms

[2016-11-07 18:20:11,933] [main] INFO  com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 2047m

[2016-11-07 18:20:11,961] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon

com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /home/gerrit/site_path/bin/gerrit.war reindex

What is the content of your site_path/index/gerrit_index.config?

 

1 error

        at com.google.gerrit.lucene.LuceneVersionManager.initIndex(LuceneVersionManager.java:146)

        at com.google.gerrit.lucene.LuceneVersionManager.start(LuceneVersionManager.java:119)

        at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:89)

        at com.google.gerrit.pgm.Daemon.start(Daemon.java:311)

        at com.google.gerrit.pgm.Daemon.run(Daemon.java:212)

        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:64)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:161)

        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:102)

        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:59)

        at Main.main(Main.java:25)

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Wang]

Contents of site_path/index/gerrit_index.config are:

[index "accounts_0003"]

ready = true

[index "changes_0032"]

ready = false

And structure under path site_path/index/:

$ ls site_path/index/

accounts_0003  changes_0032  gerrit_index.config

Thanks.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

[Saša Živkov]

On Tue, Nov 8, 2016 at 6:44 AM, Ben Wang <alber...@gmail.com> wrote:

Contents of site_path/index/gerrit_index.config are:

[index "accounts_0003"]

ready = true

[index "changes_0032"]

ready = false

This is the problem. There is no ready "changes_xxxx" index.

When you run reindex did it report some errors/warnings?

 

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[Luca Milanesio]

Actually this happened to us also: the issue is that the reindex consider the index "not ready" when a significant portion of changes failed to be indexed successfully.

However, this may sometimes be highly debatable because there are known situations where we are expecting this to happen:

a) Project removed or archived

b) Project was migrated to/from another instance

We should IMHO request a confirmation at the end saying:

"N changes failed to be indexed (X % of the total): would you like to mark the index ready anyway? [y/N]"

Luca.

[Ben Wang]

Yes, zivkov, there are some errors and warnings, all of them are four types:

1. Missing project.

    such as, "Caused by: com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /home/gerrit/android/external"

2. Get unknown field.

     "ERROR com.google.gerrit.server.index.Schema : error getting field committer of ChangeData{Change{17026 (XXX), dest=/boards/bub,refs/heads/master, status=A}}"

     ....

     "Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown XXX"

3. PatchListKey computing error.

     " WARN  com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[BASE..0XXX IGNORE_NONE]"

     " Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 0XXX"

4. Mergeability checking failed.

    "ERROR com.google.gerrit.server.change.MergeabilityCacheImpl : Error checking mergeability of 0a... into d5... (MERGE_IF_NECESSARY)"

The result output:

Reindexing changes: projects: 100% (751/751), 95% (66225/68994), done    

Reindexed 66225 documents in changes index in 287.3s (230.5/s)

It seems errors and warnings above can be ignored.

Because I migrate my gerrit from 2.3 to 2.8 and to 2.11 and then to 2.12, all these upgrades look well. And for the /index/gerrit_index.config file versions before 2.13 the index is using "changes_xxx" not "accounts_003", I don't know the reason and internal details about gerrit. Am I missing some configuration steps before the upgrade from 2.12.5 to 2.13?

[Ben Wang]

Yes, some of my git projects in gerrit are removed, and I get the error like you mentioned. But after looking into the reindex output, I can't find any significant fails. That's quite weird.

[Saša Živkov]

On Tue, Nov 8, 2016 at 11:58 AM, Ben Wang <alber...@gmail.com> wrote:

Yes, zivkov, there are some errors and warnings, all of them are four types:

1. Missing project.

    such as, "Caused by: com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /home/gerrit/android/external"

2. Get unknown field.

     "ERROR com.google.gerrit.server.index.Schema : error getting field committer of ChangeData{Change{17026 (XXX), dest=/boards/bub,refs/heads/master, status=A}}"

     ....

     "Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown XXX"

3. PatchListKey computing error.

     " WARN  com.google.gerrit.server.patch.PatchListLoader : Error computing PatchListKey[BASE..0XXX IGNORE_NONE]"

     " Caused by: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 0XXX"

4. Mergeability checking failed.

    "ERROR com.google.gerrit.server.change.MergeabilityCacheImpl : Error checking mergeability of 0a... into d5... (MERGE_IF_NECESSARY)"

How did you delete project(s) in the past?

Do you have any changes (in the changes table in the database) for non-existing projects?

 

 

The result output:

Reindexing changes: projects: 100% (751/751), 95% (66225/68994), done    

Reindexed 66225 documents in changes index in 287.3s (230.5/s)

It seems errors and warnings above can be ignored.

Because I migrate my gerrit from 2.3 to 2.8 and to 2.11 and then to 2.12, all these upgrades look well. And for the /index/gerrit_index.config file versions before 2.13 the index is using "changes_xxx" not "accounts_003",

The accounts_003 index is new in 2.13.

The changes_xxx index exists in all release since the lucene index was introduced.

This means that in 2.13 you need both accounts_... and changes_... index.

 

I don't know the reason and internal details about gerrit. Am I missing some configuration steps before the upgrade from 2.12.5 to 2.13?

You can edit the index/gerrit_index.config and set the changes_0032.ready to true.

After that Gerrit will start and the 5% of the changes where (re)indexing failed will not be visible.

--
--
To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss+unsubscribe@googlegroups.com.

[zafar shamim]

Hi Zivkov,

           I am facing similar issue while updating from  2.12.3 to 2.14 .

And as recommended edited the index/gerrit_index.config and set the changes_0039.  to true.

Afterwards when i restarted gerrit through tomcat , gerrit site is not reachable.

 

if i try ssh i get message "ping: unknown host git.sand.com" and through browser git.sand.com’s server DNS address could not be found..

However jsvc service is listening on all ports 

 

tcp        0      0 :::38054                    :::*                        LISTEN      18582/jsvc.exec

tcp        0      0 :::8009                     :::*                        LISTEN      18582/jsvc.exec

tcp        0      0 :::29418                    :::*                        LISTEN      18582/jsvc.exec

tcp        0      0 :::8080                     :::*                        LISTEN      18582/jsvc.exec

tcp        0      0 :::9015                     :::*                        LISTEN      18582/jsvc.exec

tcp        0      0 :::35960                    :::*                        LISTEN      18582/jsvc.exec

 

Any pointer will be a great help

[Jason Schoon]

I have similar issues when upgrading from 2.13.8 to 2.14. Suddenly I get lots of extra failures on reindex (I always got failures about missing projects in the past).

Then, Gerrit will not start. I can then go update the gerrit_index.confg and set to true. However, we then get the following error on every push.

remote: error: internal error while processing changes

To ssh://

 ! [remote rejected] DF -> refs/drafts/develop (internal error)

error: failed to push some refs to 'ssh://

----------------------------------------------------------------------

[FAILED]                          (Upload failed)

The push still goes through, but obviously getting this error everytime is not ideal.

Anything I can do to cleanup the failures on reindex? I can provide any additional information that would be helpful in troubleshooting.

Thanks!

[Luca Milanesio]

The problem with off-line reindex is that is happening without the whole Gerrit infrastructure available, including full logging.

Best would be doing an on-line reindex because all the logs and failures are properly recorded.

Question: why did you opt for an off-line reindex? At GerritHub.io we gave up a long ago with the offline option because of three reasons:

1. It is *potentially*incorrect, especially if you have plugins influencing your prolog rules (e.g. owners, group backends, ...)

2. Takes a *very very long time* for us (several hours) and thus is never a good choice when upgrading.

3. May leave you in a "failed" state (as you've noticed)

We prefer one of the following two options:

Option 1: on-line reindex. Just run without reindexing, the on-line reindex kicks in and do the job for you.

Option 2: delta reindex. Just force the index as ready and reindex in background using the REST API.

Luca.

[Jason Schoon]

Thanks for the reply.

Largely - because the documentation used to recommend (or at least mention) doing an offline reindex so I have always done that. However, as noted, it does take a long time and did fail.

Until tracking this down and then reading your message, I didn't realize I could just skip the reindex and let it happen online. That's my plan from this point forward.

I will note that while I still have failure indexing offline, I did fix the error on upload. That was actually caused by an outdated reviewers plugin. Hopefully less of an issue now that I'm using the new plugin manager :-)

[Luca Milanesio]

On 11 May 2017, at 14:16, Jason Schoon <jason....@gmail.com> wrote:

Thanks for the reply.

Largely - because the documentation used to recommend (or at least mention) doing an offline reindex so I have always done that. However, as noted, it does take a long time and did fail.

Oh gosh, let me fix the documentation: online reindexing is definitely better.

Luca.

[hari]

Hi Luca,

Option 2 seems interesting. Could you let me know how to achieve this.? Do you mean to say after the upgrade, set gerrit_index.config to 'true', start the gerrit daemon and then reindex using rest api? Is my assumption correct? Also will be helpful if you can let me know how exactly this can be done with rest api. 

[Luca Milanesio]

Just use the index API:

https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#index-change

HTH

Luca.