OnlineReindexer & RepositoryNotFoundException

[Cyril Jaquier]

Hi all,

We upgraded from Gerrit 2.11.2 to 2.13.1 and are facing the following stacktrace:

[2016-09-29 14:32:36,086] [Reindex v25-v32] INFO com.google.gerrit.server.index.OnlineReindexer : Starting online reindex from schema version 25 to 32
[2016-09-29 14:32:36,346] [Reindex v25-v32] ERROR com.google.gerrit.server.index.change.AllChangesIndexer : Error collecting projects
org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository x86-km-src
  at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:202)
  at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:164)
  at com.google.gerrit.server.index.change.AllChangesIndexer.indexAll(AllChangesIndexer.java:145)
  at com.google.gerrit.server.index.change.AllChangesIndexer.indexAll(AllChangesIndexer.java:83)
  at com.google.gerrit.server.index.OnlineReindexer.reindex(OnlineReindexer.java:81)
  at com.google.gerrit.server.index.OnlineReindexer.access$000(OnlineReindexer.java:28)
  at com.google.gerrit.server.index.OnlineReindexer$1.run(OnlineReindexer.java:52)
Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /var/lib/git/repositories/esw_gitrepos/x86-km-src
  at org.eclipse.jgit.lib.RepositoryCache$FileKey.open(RepositoryCache.java:453)
  at org.eclipse.jgit.lib.RepositoryCache.openRepository(RepositoryCache.java:274)
  at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:117)
  at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:92)
  at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:199)
... 6 more
[2016-09-29 14:32:36,346] [Reindex v25-v32] ERROR com.google.gerrit.server.index.OnlineReindexer : Online reindex of schema version 32 failed. Successfully indexed 0 changes, failed to index 0 changes

The x86-km-src repository does not exist anymore and was likely a test repository we had some years ago. It seems however that some references to it are still present in the database. How can we fix this problem and have the indexer complete his work properly? Thank you

Regards,
Cyril

[Cyril Jaquier]

Any ideas where the repository information is coming from? I looked at the database and add expected to have a project table but that doesn't seem to be the case!? Any pointers are welcome. Thanks

Cyril

gerrit> \d
                     List of relations
 TABLE_SCHEM | TABLE_NAME                  | TABLE_TYPE
 ------------+-----------------------------+-----------
 public      | account_external_ids        | TABLE
 public      | account_group_by_id         | TABLE
 public      | account_group_by_id_aud     | TABLE
 public      | account_group_members       | TABLE
 public      | account_group_members_audit | TABLE
 public      | account_group_names         | TABLE
 public      | account_groups              | TABLE
 public      | account_project_watches     | TABLE
 public      | accounts                    | TABLE
 public      | change_messages             | TABLE
 public      | changes                     | TABLE
 public      | patch_comments              | TABLE
 public      | patch_set_approvals         | TABLE
 public      | patch_sets                  | TABLE
 public      | schema_version              | TABLE
 public      | system_config               | TABLE

[Luca Milanesio]

Hi Cyril,

you still have references to that repository from the changes table.

Luca.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luca Milanesio]

Btw ... I believe we should include an option to just skip those changes instead of failing the migration process :-(

Luca.

[Cyril Jaquier]

Thank you Luca.

Strange because I can find anything there:

gerrit> select * from changes where dest_project_name = 'x86-km-src';
 change_key | created_on | current_patch_set_id | dest_branch_name | dest_project_name | last_updated_on | owner_account_id | status | subject | topic | row_version | change_id | original_subject | submission_id | note_db_state
 -----------+------------+----------------------+------------------+-------------------+-----------------+------------------+--------+---------+-------+-------------+-----------+------------------+---------------+--------------
(0 rows; 2 ms)

Also looking at all the dest_project_name, I can find any potential candidate for this repository. Am I looking at the wrong place?

Cyril

[Luca Milanesio]

There was a bug a while ago where the reindex was listing the changes from the previous index.

Have you tried to get rid of all indexes before running the migration?

Luca.

[Cyril Jaquier]

Excuse my ignorance but how do you get rid of all indexes?

Cyril

[Luca Milanesio]

What about running from Gerrit site:

rm index/*

Luca.

[Cyril Jaquier]

Did not help, same error, and now Gerrit does not come up again since it needs the index:

[2016-10-06 09:30:28,575] [main] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) No index versions ready; run java -jar /opt/gerrit-code-review/bin/gerrit.war reindex

Running the reindex as proposed by Gerrit results in the same error:

Reindexing accounts: 100% (91/91)
Reindexed 91 documents in accounts index in 2.2s (41.6/s)
Collecting projects: 1[2016-10-06 09:28:54,654] [main] ERROR com.google.gerrit.server.index.change.AllChangesIndexer : Error collecting projects

org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository x86-km-src

Any help welcome...

[Luca Milanesio]

Hold on ... but now it fails in this part of the code:

for (Project.NameKey name : projectCache.all()) {
try (Repository repo = repoManager.openRepository(name)) {
int size = ChangeNotes.Factory.scan(repo).size();
changeCount += size;
projects.add(new ProjectHolder(name, size));
} catch (IOException e) {
log.error("Error collecting projects", e);
return new Result(sw, false, 0, 0);
}
pm.update(1);
}
pm.endTask();
setTotalWork(changeCount);

return indexAll(index, projects);
}

*ARE YOU SURE* that the x86-km-src directory doesn't exist anymore on the $GERRIT_SITE/git filesystem?

Sounds strange to me that even if the directory doesn't exist it still shows up in the project list cache.

Luca.

[Björn Pedersen]

As this probably was deleted manually (not using the delete-project plugin), the project cache maybe has not been flushed.
From [1]:

cache "projects"

Caches the project description records, from the projects table in the database. If a project record is updated or deleted, this cache should be flushed. Newly inserted projects do not require a cache flush, as they will be read upon first reference.

So try to delete <gerrit-dir>/cache as well.

Björn


[1] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#cache_names

[Luca Milanesio]

Hold on ... if you are shutting down Gerrit and run the off-line reindex, the project cache is gone anyway :-O

Unless they have configured a H2 persistent cache for that.

@Cyril can you share your gerrit.config?

Luca.

[Björn Pedersen]

Am Donnerstag, 6. Oktober 2016 10:24:47 UTC+2 schrieb lucamilanesio:

Hold on ... if you are shutting down Gerrit and run the off-line reindex, the project cache is gone anyway :-O

Unless they have configured a H2 persistent cache for that.

@Cyril can you share your gerrit.config?

Luca.

If this is the reason, the I guess running init should flush all persistent caches to be on the safe side. For  a normal restart the cache should bekept, of course.

Björn

[Luca Milanesio]

Yes, it should and I thought it did it already :-O

@Cryrill do you have persistent project cache?

Luca.

[Cyril Jaquier]

Sooo... Gerrit up and running smooth again :-)

I was not the guy behind the keyboard, but here is the story:

- Indeed we had a folder "x86-km-src.git" under /git. The access time and the ".git" ending confused the admin who was looking for "x86-km-src" (without ".git" at the end) and for an untouched folder since ages (but access time was updated by the reindexer itself).
- Removing this folder did not fix it completely. We got other exceptions, the main one being another project not accessible. That one was moved probably manually some months ago. Since it was a tryout repo, we deleted it. The other exceptions were like
com.google.gerrit.server.git.IntegrationException: Cannot merge d2f716c621452fa380d922c358a8b682a7515031. After removing the tryout project and rerunning the indexer, those exceptions also disappeared. Likely some commits from that tryout project.
- After that, the indexes could be generated and Gerrit started successfully.

We would have probably faced these problems in the future so good we cleaned up our mess a bit. Many thanks to all of you for helping us. That was very much appreciated.

Regards,
Cyril

[Björn Pedersen]

Am Donnerstag, 6. Oktober 2016 10:43:09 UTC+2 schrieb lucamilanesio:

Yes, it should and I thought it did it already :-O

@Cryrill do you have persistent project cache?

Luca.

I checked in the code (for 2.13), there initCache only  ensures the directory existence.
I will try to  do a patch for that.


Björn