repository not found...

[Nasser Grainawi]

Does anyone see "org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found:" errors in their log ever? We're running 2.1.8 and we see this occasionally (maybe a few times a week?). We'll also get missing object exceptions sometimes as well.

Usually (always?) the exceptions are from our kernel repo, but the host doesn't ever report any filesystem problems or anything else that would actually indicate trouble. If anyone has ideas here I'm very interested to hear them.

Nasser

--
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

[Edwin Kempin]

We are running 2.1.8 but we don't see these errors in our logs.

2012/3/26 Nasser Grainawi <nas...@codeaurora.org>

--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

[Shawn Pearce]

On Mon, Mar 26, 2012 at 06:27, Nasser Grainawi <nas...@codeaurora.org> wrote:
> Does anyone see "org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found:" errors in their log ever? We're running 2.1.8 and we see this occasionally (maybe a few times a week?).

Example stack traces would help narrow down where this happens.
RepositoryNotFoundException gets thrown internally when the user makes
a typo on a name, e.g. they ask to clone "kenel" rather than "kernel".
We try to avoid putting these into the log and just return an error to
the user.

> We'll also get missing object exceptions sometimes as well.

These aren't usually expected so some example traces might be good.
Are you running a concurrent git gc or git repack in the background?
Can you correlate a missing object with the gc/repack activity?

> Usually (always?) the exceptions are from our kernel repo, but the host doesn't ever report any filesystem problems or anything else that would actually indicate trouble. If anyone has ideas here I'm very interested to hear them.

Our cluster implementation used by gerrit-review also sometimes sees
both of these. I have been thus far assuming it is unique to our
cluster environment and been trying to solve them from that level.

[Nasser Grainawi]

On Mar 26, 2012, at 9:15 AM, Shawn Pearce wrote:

> On Mon, Mar 26, 2012 at 06:27, Nasser Grainawi <nas...@codeaurora.org> wrote:
>> Does anyone see "org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found:" errors in their log ever? We're running 2.1.8 and we see this occasionally (maybe a few times a week?).
>
> Example stack traces would help narrow down where this happens.
> RepositoryNotFoundException gets thrown internally when the user makes
> a typo on a name, e.g. they ask to clone "kenel" rather than "kernel".
> We try to avoid putting these into the log and just return an error to
> the user.

Here's one from earlier today:

[2012-03-26 10:24:44,120] ERROR com.google.gerrit.server.patch.PublishComments : Failed to obtain PatchSetInfo for patch set 113338,2
com.google.gerrit.server.patch.PatchSetInfoNotAvailableException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository kernel/msm
at com.google.gerrit.server.patch.PatchSetInfoFactory.get(PatchSetInfoFactory.java:97)
at com.google.gerrit.server.patch.PublishComments.email(PublishComments.java:284)
at com.google.gerrit.server.patch.PublishComments.call(PublishComments.java:127)
at com.google.gerrit.sshd.commands.ReviewCommand.approveOne(ReviewCommand.java:225)
at com.google.gerrit.sshd.commands.ReviewCommand.access$700(ReviewCommand.java:60)
at com.google.gerrit.sshd.commands.ReviewCommand$1.run(ReviewCommand.java:154)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:399)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository kernel/msm
at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:103)
at com.google.gerrit.server.patch.PatchSetInfoFactory.get(PatchSetInfoFactory.java:83)
... 15 more
Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /path/to/repos/kernel/msm
at org.eclipse.jgit.lib.RepositoryCache$FileKey.open(RepositoryCache.java:322)
at org.eclipse.jgit.lib.RepositoryCache.openRepository(RepositoryCache.java:171)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:106)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:81)
at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:100)
... 16 more

It's not always PublishComments that starts the stacktrace, sometimes ChangeHookRunner does and it's not always kernel/msm.

>
>> We'll also get missing object exceptions sometimes as well.
>
> These aren't usually expected so some example traces might be good.
> Are you running a concurrent git gc or git repack in the background?
> Can you correlate a missing object with the gc/repack activity?

We likely were running a git gc in the background. Looks like the cases I've seen recently are during upload.

[2012-03-26 07:23:21,123] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user <username> account 1001844) during git-upload-pack '/platform/frameworks/base'
org.eclipse.jgit.errors.MissingObjectException: Missing commit 9fceff03ec9518cbbacbf4fe6b736a522d6b1ff2
at org.eclipse.jgit.storage.file.WindowCursor.open(WindowCursor.java:126)
at org.eclipse.jgit.revwalk.RevWalk.getCachedBytes(RevWalk.java:855)
at org.eclipse.jgit.revwalk.RevCommit.parseHeaders(RevCommit.java:136)
at org.eclipse.jgit.revwalk.PendingGenerator.next(PendingGenerator.java:148)
at org.eclipse.jgit.revwalk.TopoSortGenerator.<init>(TopoSortGenerator.java:78)
at org.eclipse.jgit.revwalk.StartGenerator.next(StartGenerator.java:157)
at org.eclipse.jgit.revwalk.RevWalk.next(RevWalk.java:420)
at org.eclipse.jgit.revwalk.ObjectWalk.next(ObjectWalk.java:238)
at org.eclipse.jgit.storage.pack.PackWriter.findObjectsToPack(PackWriter.java:1369)
at org.eclipse.jgit.storage.pack.PackWriter.preparePack(PackWriter.java:519)
at org.eclipse.jgit.transport.UploadPack.sendPack(UploadPack.java:827)
at org.eclipse.jgit.transport.UploadPack.service(UploadPack.java:419)
at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:368)
at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:55)
at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:103)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:399)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

>
>> Usually (always?) the exceptions are from our kernel repo, but the host doesn't ever report any filesystem problems or anything else that would actually indicate trouble. If anyone has ideas here I'm very interested to hear them.
>
> Our cluster implementation used by gerrit-review also sometimes sees
> both of these. I have been thus far assuming it is unique to our
> cluster environment and been trying to solve them from that level.
>

> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en

--

[Nasser Grainawi]

This stack trace is appearing when one user is trying to upload a change. On their end they get the message "change state corrupt".

git push ssh://review.example.com:29418/kernel.git c7e93872fb04f2e62fb9ee92c0ae60e5210db32f:refs/for/master
Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 681 bytes, done.
Total 6 (delta 5), reused 0 (delta 0)
remote: Resolving deltas: 0% (0/5)
remote: (W) c7e9387: commit subject >65 characters; use shorter first paragraph
To ssh://review.example.com:29418/kernel.git
! [remote rejected] c7e93872fb04f2e62fb9ee92c0ae60e5210db32f -> refs/for/master (change state corrupt)

[2012-03-29 12:29:00,239] ERROR com.google.gerrit.server.git.ReceiveCommits : Change 112351 missing 6baa73ba6551bf768f4ce3e58d9220fac479bfa4
org.eclipse.jgit.errors.MissingObjectException: Missing unknown 6baa73ba6551bf768f4ce3e58d9220fac479bfa4
at org.eclipse.jgit.storage.file.WindowCursor.open(WindowCursor.java:125)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:228)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:811)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:724)
at com.google.gerrit.server.git.ReceiveCommits.doReplace(ReceiveCommits.java:1190)
at com.google.gerrit.server.git.ReceiveCommits.doReplaces(ReceiveCommits.java:1104)
at com.google.gerrit.server.git.ReceiveCommits.onPreReceive(ReceiveCommits.java:359)
at org.eclipse.jgit.transport.ReceivePack.executeCommands(ReceivePack.java:1027)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:680)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:600)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:90)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:103)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:69)
at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:399)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:324)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Any ideas?

[Nasser Grainawi]

On Mar 29, 2012, at 2:32 PM, Nasser Grainawi wrote:

>>>> Usually (always?) the exceptions are from our kernel repo, but the host doesn't ever report any filesystem problems or anything else that would actually indicate trouble. If anyone has ideas here I'm very interested to hear them.
>>>
>>> Our cluster implementation used by gerrit-review also sometimes sees
>>> both of these. I have been thus far assuming it is unique to our
>>> cluster environment and been trying to solve them from that level.

Ok, might have traced this back to a line in JGit.

[2012-04-03 13:32:19,234] WARN com.google.gerrit.common.ChangeHookRunner : Cannot open repository kernel/msm

org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository kernel/msm

at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:129)
at com.google.gerrit.common.ChangeHookRunner.openRepository(ChangeHookRunner.java:212)
at com.google.gerrit.common.ChangeHookRunner.openRepository(ChangeHookRunner.java:201)
at com.google.gerrit.common.ChangeHookRunner.doPatchsetCreatedHook(ChangeHookRunner.java:245)
at com.google.gerrit.server.git.ReceiveCommits.doReplace(ReceiveCommits.java:1334)
at com.google.gerrit.server.git.ReceiveCommits.doReplaces(ReceiveCommits.java:1026)
at com.google.gerrit.server.git.ReceiveCommits.onPreReceive(ReceiveCommits.java:325)
at org.eclipse.jgit.transport.ReceivePack.executeCommands(ReceivePack.java:1047)
at org.eclipse.jgit.transport.ReceivePack.service(ReceivePack.java:697)
at org.eclipse.jgit.transport.ReceivePack.receive(ReceivePack.java:617)
at com.google.gerrit.sshd.commands.Receive.runImpl(Receive.java:91)

at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:103)
at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:34)
at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:69)

at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:397)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:333)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Caused by: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: /path/to/kernel/msm

at org.eclipse.jgit.lib.RepositoryCache$FileKey.open(RepositoryCache.java:322)
at org.eclipse.jgit.lib.RepositoryCache.openRepository(RepositoryCache.java:171)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:106)
at org.eclipse.jgit.lib.RepositoryCache.open(RepositoryCache.java:81)

at com.google.gerrit.server.git.LocalDiskRepositoryManager.openRepository(LocalDiskRepositoryManager.java:126)
... 23 more

From jgit:lib/RepositoryCache.java

public static boolean isGitRepository(final File dir, FS fs) {
return fs.resolve(dir, "objects").exists()
&& fs.resolve(dir, "refs").exists()
&& isValidHead(new File(dir, Constants.HEAD));
}

Why isn't the File being passed to isValidHead going through fs.resolve?

Our server has git repos that end in '.git', but from the exception above, it looks like we're cutting that off.

Nasser