Recently plagued by "Repository unknown" errors

[James Mills]

Hey all,

Recently, one of the gerrit servers I help maintain has started getting just a ton of errors like this:

[2012-07-24 00:14:37,052] ERROR com.google.gerrit.server.query.change.MessagePredicate : Repository myrepo unknown.

org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Invalid name: myrepo

        at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:100)

        at com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:95)

        at com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:45)

        at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:78)

        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:114)

...

From a cold start, gerrit runs fine for about 10 minutes, then gets a ton of these messages for multiple repositories.  Once that happens, the ssh path via gerrit for pushing changes is completely useless and gerrit needs to be restarted again.

All of the repositories that gerit is throwing exceptions about do in fact exist, and work just fine up until the first exception is thrown.

We have not recently made any changes to the gerrit server configuration or version.  The only real change I can think of is that we recently got an infusion of upstream code, causing many of the repositories to grow.

Any help would be appreciated!

~james

[Saša Živkov]

On Tue, Jul 24, 2012 at 4:41 PM, James Mills <james...@gmail.com> wrote:
> Hey all,
>
> Recently, one of the gerrit servers I help maintain has started getting just
> a ton of errors like this:
>
> [2012-07-24 00:14:37,052] ERROR
> com.google.gerrit.server.query.change.MessagePredicate : Repository myrepo
> unknown.
> org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found:
> Invalid name: myrepo
> at
> com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:100)
> at
> com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:95)
> at
> com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:45)
> at
> com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:78)
> at
> com.google.gerrit.server.query.change.AndSource.read(AndSource.java:114)
> ...

The stack trace is not complete. Is there any "Caused by..." further down?

> From a cold start, gerrit runs fine for about 10 minutes, then gets a ton of
> these messages for multiple repositories. Once that happens, the ssh path
> via gerrit for pushing changes is completely useless and gerrit needs to be
> restarted again.

Which version of Gerrit do you run?
Make a thread dump at the moment when ssh channel is not responding
and post it here.
$ jstack $(cat REVIEW_SITE/logs/gerrit.pid)

>
> All of the repositories that gerit is throwing exceptions about do in fact
> exist, and work just fine up until the first exception is thrown.
>
> We have not recently made any changes to the gerrit server configuration or
> version. The only real change I can think of is that we recently got an
> infusion of upstream code, causing many of the repositories to grow.
>
> Any help would be appreciated!
> ~james
>

> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en

[James Mills]

On Tuesday, July 24, 2012 10:55:03 AM UTC-5, zivkov wrote:

On Tue, Jul 24, 2012 at 4:41 PM, James Mills <james...@gmail.com> wrote:
> Hey all,
>
> Recently, one of the gerrit servers I help maintain has started getting just
> a ton of errors like this:
>
> [2012-07-24 00:14:37,052] ERROR
> com.google.gerrit.server.query.change.MessagePredicate : Repository myrepo
> unknown.
> org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found:
> Invalid name: myrepo
>         at
> com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:100)
>         at
> com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:95)
>         at
> com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:45)
>         at
> com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:78)
>         at
> com.google.gerrit.server.query.change.AndSource.read(AndSource.java:114)
> ...

The stack trace is not complete. Is there any "Caused by..." further down?

Here is a complete trace:

2012-07-24 00:14:37,068] ERROR com.google.gerrit.server.query.change.MessagePredicate : Repository "myrepo" unknown.

org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository myrepo

        at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:108)

        at com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:95)

        at com.google.gerrit.server.query.change.MessagePredicate.match(MessagePredicate.java:45)

        at com.google.gerrit.server.query.AndPredicate.match(AndPredicate.java:78)

        at com.google.gerrit.server.query.change.AndSource.read(AndSource.java:114)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.searchQuery(ChangeListServiceImpl.java:170)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.access$100(ChangeListServiceImpl.java:56)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$6.query(ChangeListServiceImpl.java:137)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$QueryNext.run(ChangeListServiceImpl.java:338)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl$QueryNext.run(ChangeListServiceImpl.java:320)

        at com.google.gerrit.httpd.rpc.BaseServiceImplementation.run(BaseServiceImplementation.java:63)

        at com.google.gerrit.httpd.rpc.ChangeListServiceImpl.allQueryNext(ChangeListServiceImpl.java:133)

        at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

        at java.lang.reflect.Method.invoke(Unknown Source)

        at com.google.gwtjsonrpc.server.MethodHandle.invoke(MethodHandle.java:91)

        at com.google.gwtjsonrpc.server.JsonServlet.doService(JsonServlet.java:382)

        at com.google.gwtjsonrpc.server.JsonServlet.service(JsonServlet.java:268)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)

        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:216)

        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:141)

        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:63)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:76)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:134)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:54)

        at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:129)

        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:59)

        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:122)

        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:110)

        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1322)

        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:473)

        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:921)

        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:403)

        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:856)

        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)

        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114)

        at org.eclipse.jetty.server.Server.handle(Server.java:352)

        at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596)

        at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1069)

        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805)

        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218)

        at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:510)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34)

        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40)

        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:450)

        at java.lang.Thread.run(Unknown Source)

Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /home/git/root/myrepo

        at org.eclipse.jgit.lib.RepositoryCache$FileKey.open(RepositoryCache.java:322)

        at org.eclipse.jgit.lib.RepositoryCache.openRepository(RepositoryCache.java:171)

        at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:106)

        at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:81)

        at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:105)

        ... 56 more

 

> From a cold start, gerrit runs fine for about 10 minutes, then gets a ton of
> these messages for multiple repositories.  Once that happens, the ssh path
> via gerrit for pushing changes is completely useless and gerrit needs to be
> restarted again.

Which version of Gerrit do you run?

Version 2.2.2

 

Make a thread dump at the moment when ssh channel is not responding
and post it here.
$ jstack $(cat REVIEW_SITE/logs/gerrit.pid)

I'll keep an eye on it and get that info when it happens.  Thanks!

 

>
> All of the repositories that gerit is throwing exceptions about do in fact
> exist, and work just fine up until the first exception is thrown.
>
> We have not recently made any changes to the gerrit server configuration or
> version.  The only real change I can think of is that we recently got an
> infusion of upstream code, causing many of the repositories to grow.
>
> Any help would be appreciated!
> ~james
>
> --

> To unsubscribe, email repo-discuss+unsubscribe@googlegroups.com

[Martin Fick]

If you search they ML you will find similar reports. We see the same thing every now and then, but not repeatedly. I have my suspicions that it has to do with exceptions being thrown when jgit first reads a pack file, perhaps because a user interrupts a clone, or perhaps a partially written pack file which jgit opens and then throws an exception on because it is an incomplete file (Git never creates such files, but mirrors may see them depending on how you are replicating the files, rsync, NFS snapmirrors...), or perhaps because of an NFS stale filehandle. These are just guesses, is it possible that any of these problems may be affecting you? One way to confirm that you are indeed experiencing a similar problem is to run a git repack on the repo while Gerrit is running to see if it helps resets the repo to a working state,

-Martin

>--
>To unsubscribe, email repo-discuss...@googlegroups.com

>More info at http://groups.google.com/group/repo-discuss?hl=en

Employee of Qualcomm Innovation Center,Inc. which is a member of Code Aurora Forum

[Luca Milanesio]

In our case we solved completely the problem with a fix to Gerrit code (but the problem was originated in jGit).

Can share the patch for review today.

I think happened because of a background Git pack.

Luca
---------
Sent from my iPhone
Luca Milanesio
+44-(0)7928 617383
Skype: lucamilanesio

[Saša Živkov]

Let's see what JGit does at exactly this place (RepositoryCache.java:322)

320 public Repository open(final boolean mustExist) throws
IOException {
321 if (mustExist && !isGitRepository(path, fs))
322 throw new RepositoryNotFoundException(path);
323 return new FileRepository(path);
324 }

So the condition at line 321 must be true, which means that
isGitRepository(path, fs)
returns false. Lets look inside the isGitRepository method (the same class):

356 public static boolean isGitRepository(final File dir, FS fs) {
357 return fs.resolve(dir, "objects").exists()
358 && fs.resolve(dir, "refs").exists()
359 && isValidHead(new File(dir, Constants.HEAD));
360 }
361
362 private static boolean isValidHead(final File head) {
363 final String ref = readFirstLine(head);
364 return ref != null
365 && (ref.startsWith("ref: refs/") || ObjectId.isId(ref));
366 }

so it checks existence of "objects" and "refs" folders directly under the
git repository root folder. It also checks if the content of the HEAD
is valid by checking if the first line in the HEAD file starts with "ref: refs/"
or if it is parseable as objectId (means 40 char long hexadecimal number).

Could you perform all these checks manually at the time when you see
this exception in the error_log and when Gerrit is not able to serve requests
for this repository?

Saša

>> > To unsubscribe, email repo-discuss...@googlegroups.com

>> > More info at http://groups.google.com/group/repo-discuss?hl=en
>

> --
> To unsubscribe, email repo-discuss...@googlegroups.com

[Saša Živkov]

found some interesting information on why File.exists() could return false
even though the file exists:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5003595
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6191254

[Martin Fick]

Luca,

Can you share this patch? :)

-Martin

On Wednesday, July 25, 2012 12:27:43 am Luca Milanesio
wrote:

> > Employee of Qualcomm Innovation Center,Inc. which is a
> > member of Code Aurora Forum

--

Employee of Qualcomm Innovation Center, Inc. which is a

[Luca Milanesio]

Hi Martin,
I looking for the patch in out support branches and submitting today for review :-)

Luca.

[Martin Fick]

On Wednesday, August 15, 2012 03:54:49 pm Luca Milanesio
wrote:

> Hi Martin,
> I looking for the patch in out support branches and
> submitting today for review :-)

Thanks!

[Luca Milanesio]

Hi Martin,

here you go:
https://gerrit-review.googlesource.com/#/c/37440/

This additional retry cycle has resolved our problem completely: our missing unknown were NOT actually repo corruptions but just temporary access problems to the pack files (possibly due to the concurrent 'git gc' operations we run periodically).

IMHO the retry mechanism in JGit itself need some "maintenance" as this patch is a workaround on Gerrit.

Luca.