Missing blob still remains, now escalating

866 views
Skip to first unread message

Fredrik Luthander

unread,
Sep 2, 2010, 4:10:10 PM9/2/10
to Repo and Gerrit Discussion, s...@google.com, ulrik....@gmail.com
Hi there everyone!

Hate to bring this up again, but it seems we're still having problems
with the missing blob thing that we wrote about on this list a lot in
June. [1]

It's been a fairly constant thing so far, showing up every now and
then. Most of the occurences have been solved, users have upgraded
from git 1.5.xx to 1.7.0.3 and ran git gc in their projects. Usually,
this did the trick.

However, last week things have been escalating, because now not only
our own commits are affected, these days also the regular imports we
do from Google is hopeless. So now I'm involved in manually bypassing
gerrit and pushing directly to the git through my openssh-account.
This is not good, mainly because I'm not supposed to handle these
things myself, and I also have other things to work with.

We then discovered that this change was added where Shawn states that
the missing blob error was fixed (in JGit). [2] What a coincidence!

So, we spent quite a lot of time to build the latest 2.1.5+56 build
(It seems we had to download JGit from the eclipse-page rather than
the android site?)
Once we had a binary we of course tested it on our testserver before
rolling it out on our production server.

But you know what? The error remained. So, under the condidition that
we did build gerrit correctly, the missing blob error is (badly) not
gone. It's still there!

Can you give us some hints as to what can we do to help solve this
problem quicker?
It's really really painful for us as it stands.

BR,
Fredrik

[1] http://groups.google.com/group/repo-discuss/browse_thread/thread/3ad5047b234cdd7c/826584066c464b74?

[2] https://review.source.android.com/#change,publish,16886,1

Shawn Pearce

unread,
Sep 3, 2010, 2:42:30 PM9/3/10
to Fredrik Luthander, Repo and Gerrit Discussion, ulrik....@gmail.com
On Thu, Sep 2, 2010 at 13:10, Fredrik Luthander
<fredrik....@sonyericsson.com> wrote:
> Hate to bring this up again, but it seems we're still having problems
> with the missing blob thing that we wrote about on this list a lot in
> June. [1]

:-(

Just to refresh everyone's memory, the error manifests itself on the
client like this:

$ repo upload
...
Writing objects: 100% (178/178), 30.91 KiB, done.
Total 178 (delta 99), reused 171 (delta 92)
error: unpack failed: error Missing blob
8f440ca1c535e9e7633ec34314cb63fa8d3df2fb

After a lot of digging through code, I suspect this is caused by the
connectivity check that is performed when there are branch level read
access controls enabled. Unfortunately the stack trace is being
discarded and never makes it to the server log, so we don't know
exactly where in the connectivity check code this error gets detected
and thrown out.

Consequently I've patched JGit and Gerrit Code Review to try and log
this more verbosely. Can you rebuild and run [3]?

[3] https://review.source.android.com/16991

> (It seems we had to download JGit from the eclipse-page rather than
> the android site?)

It should be available through the Maven repository that I use to
track some of the packages that go into Gerrit. And yup, its there.
Our current JGit version is 0.8.4-328-ge6bd689, which is in [4]. So
I'm not sure why mvn didn't pick that up for you automatically. Its
2nd on our repository list in the top-level pom.

[4] http://gerrit.googlecode.com/svn/mavenrepo/org/eclipse/jgit/org.eclipse.jgit/0.8.4.328-ge6bd689/

Fredrik Luthander

unread,
Sep 3, 2010, 8:20:01 PM9/3/10
to Repo and Gerrit Discussion
Hi Shawn!
It seems you've put a lot of effort into this case, I'm very grateful
for that. Thanks for the awesome job!

On Sep 3, 8:42 pm, Shawn Pearce <s...@google.com> wrote:
> On Thu, Sep 2, 2010 at 13:10, Fredrik Luthander
>
> <fredrik.luthan...@sonyericsson.com> wrote:
> > Hate to bring this up again, but it seems we're still having problems
> > with the missing blob thing that we wrote about on this list a lot in
> > June. [1]
>
> :-(
>
> Just to refresh everyone's memory, the error manifests itself on the
> client like this:
>
>   $ repo upload
>   ...
>   Writing objects: 100% (178/178), 30.91 KiB, done.
>   Total 178 (delta 99), reused 171 (delta 92)
>   error: unpack failed: error Missing blob
> 8f440ca1c535e9e7633ec34314cb63fa8d3df2fb
>
> After a lot of digging through code, I suspect this is caused by the
> connectivity check that is performed when there are branch level read
> access controls enabled.  Unfortunately the stack trace is being
> discarded and never makes it to the server log, so we don't know
> exactly where in the connectivity check code this error gets detected
> and thrown out.

Ulrik has one case which is 100% repeatable for him, so he's been
debugging today. However, he didn't come down far enough to find the
true cause. It seem he agrees with you analysis though, it's got
something to do with the branch level read access, a rule case that
we've put our All Projects level.

:-(

>
> Consequently I've patched JGit and Gerrit Code Review to try and log
> this more verbosely.  Can you rebuild and run [3]?
>
> [3]https://review.source.android.com/16991
>
Thank you! I'll make Ulrik aware of this, and I'll make him build me a
new binary to try things out on. I'll get back you as soon as we have
some output of some kind!

> > (It seems we had to download JGit from the eclipse-page rather than
> > the android site?)
>
> It should be available through the Maven repository that I use to
> track some of the packages that go into Gerrit.  And yup, its there.
> Our current JGit version is 0.8.4-328-ge6bd689, which is in [4].  So
> I'm not sure why mvn didn't pick that up for you automatically.  Its
> 2nd on our repository list in the top-level pom.
>
> [4]http://gerrit.googlecode.com/svn/mavenrepo/org/eclipse/jgit/org.eclip...

I will leave this for Ulrik to reply, as he currently is the one
building gerrit binaries for us.

Have a nice weekend everyone!

BR,
Fredrik

Ulrik Sjölin

unread,
Sep 6, 2010, 9:11:13 AM9/6/10
to Repo and Gerrit Discussion
Ok, so here is a first printout of my 100% reproducible case which
makes it impossible for me to update our internal gerrit repo with new
code. So the command I invoke is this:

$ git push ssh://ulrik....@review-test.sonyericsson.net:29418/tools/gerrit.git
70a7ca07267b367461dee5a2739535a27baa2e9c:refs/heads/master

This is the output:

Counting objects: 1495, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (392/392), done.
Writing objects: 100% (1214/1214), 265.91 KiB, done.
Total 1214 (delta 510), reused 1169 (delta 468)
error: unpack failed: error Missing blob
bdddb98a04637387444b80139e1c758c72d8e1ab
To ssh://ulrik....@review-test.sonyericsson.net:29418/tools/gerrit.git
! [remote rejected] 70a7ca07267b367461dee5a2739535a27baa2e9c ->
master (n/a (unpacker error))
error: failed to push some refs to 'ssh://ulrik.sjolin@review-
test.sonyericsson.net:29418/tools/gerrit.git'

The missing blob is of tree-type and contains:
100644 blob 2a83ed14bb3cec40f758e439e8a5f88949061369
AbandonChangeDialog.java
100644 blob e122ce428f40c95ab114cb92b4b3abf78377b26e
AccountDashboardScreen.java
100644 blob f850a33eda7912a84a777cfba5fb2d33da072e60
ApprovalTable.java
100644 blob 9a08932687d9e2a53598819b2a12c972f28d3c2c
ChangeConstants.java
100644 blob 59d0bd52c1b04277630558355f9850be272c779e
ChangeConstants.properties
100644 blob 05323263fd9285b7a8d28ef16af13dd83054c97f
ChangeDescriptionBlock.java
100644 blob 0c7106eefa4909118d94cf31d22ddb8a7fd1087d
ChangeInfoBlock.java
100644 blob 51e2bc58960d4a03fef604b71f98fb7efbaefce5
ChangeMessages.java
100644 blob 5d2d3a5014d6c395f17b321f19d7f66dbd62c7b8
ChangeMessages.properties
100644 blob a5e9426b6d354a732362242d9d848349621778c8
ChangeMessages_en.properties
100644 blob 892143c157d4a4259124afbf8fbdac8fef5abadb
ChangeResources.java
100644 blob ae2dd119146fb5241ee50f02569903f2f6f84b1f ChangeScreen.java
100644 blob 76b9f49e10747e4dafd12a3fe77a769851bc3894 ChangeTable.java
100644 blob 48e257de34996fca74397a91e4274a49bbf99796
CommitMessageBlock.java
100644 blob 53771a974d3597c58579605b98f747a7e43e9678
DownloadCommandLink.java
100644 blob 0affcfb3bfbba5ff1ac306c6ec4492a2688bba72
DownloadCommandPanel.java
100644 blob 6e72e20685ee4bbade1e51baa78660344e6cd162
DownloadUrlLink.java
100644 blob f4260e2baa8aeda9f9a74a761b65dd5f47da4c51
DownloadUrlPanel.java
100644 blob 533999d326e7426c91addbbcb861cca9632a1c59
IncludedInTable.java
100644 blob 9016b21fdca8cc8de7bf1b8950e0e09f426e3b35
MineSingleListScreen.java
100644 blob 6ca102f33dea0790008d5157b2e37a8bfdc21882
PagedSingleListScreen.java
100644 blob ce0cb6ddc3fe9be3a3c189825fb9bb006f206ac7
PatchSetComplexDisclosurePanel.java
100644 blob b0f06a4e3ed9f311eec499ea329a7476398ee49b
PatchSetsBlock.java
100644 blob f448b67daf69bf389e8b10cfb6cf112368248bc3 PatchTable.java
100644 blob 0335178391ca505867f7aca9a7a45e701ce6c26d
PublishCommentScreen.java
100644 blob bcd2ed22ce48cd01b5acdf1e4f1128db35547e08 QueryScreen.java
100644 blob f9f6f99d3bcfdff13c0f94ea3d3de78fab86d879
SubmitFailureDialog.java
100644 blob a7d468e96f051dde96f5b0008f32840e7c4d9e94 Util.java
100644 blob 839e8ef15a7f5a87e7f6f3ef5ae10f58f9d3616d
removeReviewerNormal.png
100644 blob 2e509eccb9d48f2cb58da448b94b1ab5d48af098
removeReviewerPressed.png

My thought was to poke around here a bit more to find out which commit
this is and so on.... I have not had time for that today though ;-)

This is in the log:
[2010-09-06 15:00:10,652] ERROR com.google.gerrit.sshd.BaseCommand :
Internal server error (user ulrik.sjolin account 1000003) during git-
receive-pack '/tools/gerrit.git'
com.google.gerrit.sshd.BaseCommand$Failure: Unpack error on project
"tools/gerrit":
RefFilter:
com.google.gerrit.server.git.VisibleRefFilter@7e6ab533VisibleRefFilter
Visible references (284):
- c78f8975 refs/changes/03/21003/1
- 3778dd8b refs/changes/25/36325/14
- 7e9a2a98 refs/changes/12/26412/1
- 90519bc9 refs/changes/25/36325/13
- a0e8dbdf refs/changes/25/36325/12
- 687dc093 refs/changes/25/36325/11
- 4353b4b4 refs/changes/25/36325/10
- 7245966b refs/changes/98/27798/11
- 8c36688d refs/changes/98/27798/10
- c35012ce refs/changes/98/27798/14
- 47a71635 refs/changes/98/27798/13
- 89ccc042 refs/changes/98/27798/12
- d70ae9ef refs/changes/58/44158/1
- 7a7562d2 refs/tags/v2.0.14.1
- bcefac5b refs/heads/merge-strategy-per-branch
- b0cd858f refs/changes/53/19553/1
- 309de5fa refs/changes/26/36326/1
- d86eddf0 refs/changes/77/25677/1
- 4564563d refs/tags/v2.0.13.1
- c75eff68 refs/changes/65/28565/1
- 03810032 refs/changes/23/15523/1
- c3316956 refs/changes/65/28565/2
- f855f399 refs/heads/master
- c17303e5 refs/changes/30/26030/1
- 56bf9479 refs/changes/11/26411/1
- 41b90bce refs/changes/83/16383/1
- e2adb928 refs/changes/70/23970/1
- 48fe83cc refs/tags/v2.1.4-rc4
- c8f17a98 refs/tags/v2.1.4-rc3
- bcefac5b refs/changes/59/27159/1
- 16f809cd refs/tags/v2.1.4-rc2
- 8ef3b69f refs/heads/project-group-ui
- 8aa42c35 refs/tags/v2.0.24.1
- 8c0245f7 refs/tags/v2.0.24.2
- b1c5ac78 refs/changes/87/22587/1
- 68ecdf07 refs/changes/71/29171/1
- 307d2eb5 refs/changes/56/26956/2
- 181b0576 refs/changes/56/26956/3
- 1ed33cb5 refs/changes/56/26956/1
- a927a0e6 refs/heads/FBAC
- 4ab4e381 refs/changes/82/15182/1
- f61a880e refs/heads/FPUR
- 987631ea refs/changes/78/21378/2
- 17b75684 refs/changes/78/21378/1
- b66e0c13 refs/changes/01/22601/1
- 8a896384 refs/changes/78/21378/3
- 1bfb5e15 refs/changes/84/16284/1
- 834f40c4 refs/changes/98/27798/4
- d8ee1828 refs/changes/98/27798/3
- 5f843d2f refs/changes/98/27798/6
- 5743e82a refs/changes/98/27798/5
- 8c8a5ae2 refs/changes/98/27798/8
- 3a68e277 refs/changes/98/27798/7
- e5e5807e refs/tags/v2.0.9
- 54be5d26 refs/changes/98/27798/9
- ce74324c refs/tags/v2.0.8
- 9029365b refs/tags/v2.0.7
- 6360dbc0 refs/changes/49/29349/9
- 7ed593b4 refs/tags/v2.0.6
- 80908e45 refs/tags/v2.0.5
- e269eae7 refs/tags/v2.0.4
- 315ec1d2 refs/changes/49/29349/6
- 14b64ea5 refs/tags/v2.0.3
- 49fa9c78 refs/changes/49/29349/5
- 69badf31 refs/tags/v2.0.2
- ea0933fe refs/changes/49/29349/8
- 52027eab refs/tags/v2.0.1
- b20805bf refs/changes/49/29349/7
- 75a6ac88 refs/tags/v2.1.3
- 21e00ab4 refs/changes/22/23222/4
- 2323592d refs/changes/49/29349/2
- a67229a5 refs/tags/v2.1.2
- d662da53 refs/changes/49/29349/1
- 3f77b9f1 refs/changes/22/23222/5
- 49384781 refs/tags/v2.1.1
- e4a61586 refs/changes/22/23222/2
- db498fb9 refs/changes/49/29349/4
- 1a7a931b refs/changes/22/23222/3
- 55566e93 refs/changes/49/29349/3
- 94a4719b refs/tags/v2.1.4
- 8ff06898 refs/changes/49/29349/12
- 2ade6e1b refs/changes/49/29349/13
- 9229f90b refs/tags/v2.1.4-rc1
- 1a772931 refs/changes/49/29349/10
- 706b47f1 refs/tags/v2.1.4-rc0
- 1cf59765 refs/changes/49/29349/11
- af98d54f refs/changes/22/23222/1
- 74b40b2b refs/heads/hooks
- 4f631487 refs/changes/98/27798/1
- ed90006e refs/changes/98/27798/2
- 2cad0536 refs/heads/FBAC2
- 333ce99c refs/changes/49/29349/15
- 5e6112ab refs/changes/49/29349/14
- 0725deb3 refs/changes/82/23282/1
- 03c04e8e refs/changes/69/33769/1
- c2d5ce34 refs/changes/69/33769/2
- 0d38b0f8 refs/changes/20/15720/1
- 7c962612 refs/changes/54/15854/1
- 45f2e244 refs/changes/25/36325/9
- fdfa855e refs/changes/73/21373/1
- 09eb6905 refs/changes/14/16814/1
- 8a10e89b refs/changes/25/36325/5
- 0880b265 refs/changes/25/36325/6
- db602ad3 refs/changes/25/36325/7
- 25665221 refs/changes/25/36325/8
- 7ba17c4d refs/changes/10/31010/1
- 6f983370 refs/tags/v2.1.1.1
- 2ada1af2 refs/changes/25/36325/1
- 309de5fa refs/changes/25/36325/2
- 7abd9aa5 refs/changes/06/21606/1
- c1855b78 refs/changes/25/36325/3
- e4ee1794 refs/changes/06/21606/2
- 05205d9c refs/changes/25/36325/4
- 401715cf refs/changes/06/21606/3
- 92c01d92 refs/tags/v2.1.2-rc2
- 8ef3b69f refs/heads/access-tab-regexp
- 4d320c71 refs/changes/14/16814/3
- 1fe430c9 refs/tags/v2.1.2-rc3
- 30405d21 refs/changes/14/16814/2
- d605b990 refs/tags/v2.1.2-rc0
- 826293a4 refs/tags/v2.1.2-rc1
- 9d823b2e refs/heads/project-group
- 25bfbdfc refs/changes/53/26753/2
- 3290d411 refs/changes/53/26753/3
- 6aaeb203 refs/changes/71/25671/1
- ae922cf5 refs/changes/53/26753/1
- 9a4e1416 refs/changes/03/21503/1
- 7f615c30 refs/changes/10/27910/1
- 5bfa61db refs/changes/45/33245/1
- 9bfc2492 refs/changes/04/17304/1
- 6308ceaa refs/changes/45/33245/2
- 8bf2587f refs/changes/04/17304/5
- 178937d0 refs/changes/04/17304/4
- 6e406118 refs/changes/45/33245/3
- 0793eb70 refs/changes/04/17304/3
- 6197d831 refs/changes/04/17304/2
- 5c07a722 refs/changes/36/15936/1
- ec1894b3 refs/changes/21/23321/4
- 1730b485 refs/changes/89/40789/4
- 274922b0 refs/changes/21/23321/3
- a26e3d29 refs/changes/89/40789/5
- 14022113 refs/changes/89/40789/2
- 806ac3b0 refs/changes/89/40789/3
- 94e2d95b refs/changes/89/40789/1
- f4aaa1b9 refs/changes/87/33287/1
- e96a9319 refs/changes/89/40789/9
- 236f32e9 refs/changes/89/40789/8
- 372291ee refs/changes/89/40789/7
- 95f4955a refs/changes/89/40789/6
- 3330fd96 refs/changes/51/34651/1
- 6ba9fb9f refs/changes/69/33969/1
- 9b26111f refs/tags/v2.0-rc0
- b0275ba4 refs/changes/38/20038/3
- 01cb372a refs/changes/69/33969/3
- 175d553e refs/changes/69/33969/2
- 37177b9c refs/changes/51/34651/5
- d1e0459a refs/changes/51/34651/4
- 1f190c7f refs/changes/38/20038/2
- d7e2ac9b refs/changes/51/34651/3
- a232418e refs/changes/38/20038/1
- b3bbb964 refs/changes/51/34651/2
- 8283be78 refs/tags/v2.1.2.3
- 9bf0c6ef refs/tags/v2.1.2.2
- 85356c79 refs/tags/v2.1.2.1
- 78303b7a refs/changes/22/35822/4
- 42df10c3 refs/changes/22/35822/3
- 7a4f2c05 refs/changes/22/35822/2
- 08a2ad3a refs/changes/22/35822/1
- b1226676 refs/changes/22/35822/7
- b6b5a502 refs/changes/22/35822/6
- bfb7cdd7 refs/changes/22/35822/5
- a3172217 refs/changes/98/14998/1
- aad316ff refs/changes/45/28545/2
- 3578fc5d refs/changes/21/23321/1
- 34ca99ad refs/changes/21/23321/2
- fb04e709 refs/changes/74/25674/1
- 05ec98a7 refs/tags/v2.0.19.1
- 2ef98598 refs/tags/v2.0.19.2
- 464f42a8 refs/changes/01/15501/1
- 60d1049e refs/tags/v2.1.2.4
- 9d6c9f46 refs/tags/v2.1.2.5
- 591d65d4 refs/changes/78/25678/1
- 76b7025f refs/changes/45/28545/1
- b8602569 refs/heads/tracking-id
- a608bbc2 refs/heads/merge-strategy-per-branch-2
- e9225d37 refs/changes/84/16184/1
- f76312a1 refs/tags/v2.1.3.1
- 197b2eec refs/changes/98/21298/1
- 730e293d refs/changes/61/15561/1
- 3879fb85 refs/changes/76/25676/1
- 9221da10 refs/changes/96/18296/1
- 2cad0536 refs/changes/05/21305/2
- c23bfdfc refs/changes/05/21305/1
- a927a0e6 refs/changes/87/18187/3
- 25e42acf refs/changes/87/18187/1
- 34e726d3 refs/changes/87/18187/2
- d822a471 refs/changes/27/27727/1
- 951510a0 refs/tags/v2.0.14
- f3da46d1 refs/tags/v2.0.15
- 6dec2fdf refs/tags/v2.0.16
- 22074dc0 refs/tags/v2.0.17
- 2a5a0448 refs/tags/v2.0.10
- 17576fbc refs/changes/91/22591/1
- a8e16386 refs/tags/v2.0.11
- 9f142a39 refs/tags/v2.0.12
- 4cdf3a43 refs/tags/v2.0.13
- 1a135c93 refs/changes/50/19650/2
- 4f3c51c7 refs/changes/50/19650/1
- d0e200df refs/tags/v2.0
- 8565c3c3 refs/changes/50/19650/3
- 98f8dec5 refs/tags/v2.1
- 85b3e0e6 refs/changes/67/15967/1
- 87765911 refs/changes/03/17103/1
- d2994b6c refs/tags/v1.0
- b186adaa refs/tags/v2.0.19
- 1365c76c refs/tags/v2.0.18
- 81fafb5f refs/changes/19/27319/4
- 5a3c67c3 refs/changes/50/34650/2
- d80095f0 refs/changes/19/27319/3
- dc1b9e20 refs/changes/50/34650/3
- 8565c3c3 refs/heads/FBAC-Upload
- f1aec7b8 refs/changes/12/32912/1
- 7b2c85b4 refs/changes/19/27319/2
- b54e4ae3 refs/changes/19/27319/1
- 4e2331f4 refs/changes/50/34650/1
- a50275db refs/tags/v2.0.23
- 1f0a065d refs/changes/50/34650/6
- d7cdc0d9 refs/changes/76/21376/3
- f9c9a76e refs/tags/v2.0.24
- c8a38731 refs/changes/50/34650/7
- 3436cead refs/tags/v2.0.21
- dd97317f refs/changes/50/34650/4
- 5d284a84 refs/changes/76/21376/1
- 973b93fe refs/tags/v2.0.22
- cf9e5e0b refs/changes/50/34650/5
- d470cf7c refs/changes/76/21376/2
- 4a7b351c refs/tags/v2.0.20
- 31c6f4a2 refs/changes/02/21002/1
- 1486d156 refs/changes/03/17103/2
- 8793285c refs/changes/08/27608/1
- 84aaeb69 refs/changes/02/21002/2
- db158f44 refs/changes/73/18373/1
- a3116dce refs/changes/62/28962/6
- bc5548fc refs/changes/62/28962/7
- 94ac8a5e refs/changes/62/28962/4
- 795e9fb8 refs/changes/63/19363/1
- b246df62 refs/changes/62/28962/5
- 8ef3b69f refs/heads/non-admin-project-creation
- 7aec2c99 refs/changes/62/28962/2
- ef21ac52 refs/changes/62/28962/3
- de9f6aef refs/changes/65/25565/1
- 55a98f2f refs/changes/62/28962/1
- d93c6f55 refs/changes/19/27319/7
- f7056a4c refs/changes/19/27319/8
- aef9cc17 refs/changes/19/27319/5
- c579df2c refs/changes/19/27319/6
- 2a66eb38 refs/changes/07/22607/1
- 4ab5e064 refs/changes/07/22607/2
- e4b97faf refs/tags/v2.1-rc1
- 21802b70 refs/changes/07/22607/3
- 1eb7a207 refs/tags/v2.1-rc0
- 7c26d1f3 refs/changes/07/22607/4
- 42be0a0a refs/changes/23/24423/1
- 0d7f6aaf refs/changes/84/15984/1
- 45844057 refs/changes/15/26415/1
- b66b2c1c refs/changes/20/15020/1
- 216cf587 refs/changes/50/34650/9
- 631cd190 refs/changes/50/34650/8
- 0821c812 refs/changes/74/21374/1
- 30c20f47 refs/tags/v2.1.3-rc0
- f8b3fbb8 refs/changes/51/34651/7
- 530d638b refs/changes/51/34651/6
- b20e38ea refs/tags/v2.1.3-rc2
- ff333f3c refs/tags/v2.1.3-rc1
- 07398f79 refs/changes/49/20849/3
- eb4f09a4 refs/changes/49/20849/4
- 5a6687c6 refs/changes/54/23954/1
- 8bdcd701 refs/changes/49/20849/5
- bb50fe11 refs/changes/49/20849/6
- 7ff112d3 refs/changes/10/26410/1
- 5e3adc20 refs/changes/49/20849/1
- 192f8678 refs/changes/49/20849/2
- a9ce851a refs/changes/49/20849/7
- 7d7b051f refs/changes/49/20849/8
Hidden references (1):
- f855f399 HEAD

Ok so that is it.... if anyone has any smart thoughts here I would
really love to hear them ;-)

Ulrik


Shawn Pearce

unread,
Sep 6, 2010, 10:40:43 AM9/6/10
to Ulrik Sjölin, Repo and Gerrit Discussion
On Mon, Sep 6, 2010 at 06:11, Ulrik Sjölin <ulrik....@gmail.com> wrote:
> Ok, so here is a first printout of my 100% reproducible case which
> makes it impossible for me to update our internal gerrit repo with new
> code. So the command I invoke is this:
>
> $ git push ssh://ulrik....@review-test.sonyericsson.net:29418/tools/gerrit.git
> 70a7ca07267b367461dee5a2739535a27baa2e9c:refs/heads/master
>
> This is the output:
>
> Counting objects: 1495, done.
> Delta compression using up to 2 threads.
> Compressing objects: 100% (392/392), done.
> Writing objects: 100% (1214/1214), 265.91 KiB, done.
> Total 1214 (delta 510), reused 1169 (delta 468)
> error: unpack failed: error Missing blob
> bdddb98a04637387444b80139e1c758c72d8e1ab
> To ssh://ulrik....@review-test.sonyericsson.net:29418/tools/gerrit.git
>  ! [remote rejected] 70a7ca07267b367461dee5a2739535a27baa2e9c ->
> master (n/a (unpacker error))
> error: failed to push some refs to 'ssh://ulrik.sjolin@review-
> test.sonyericsson.net:29418/tools/gerrit.git'

Yay!

Sorry, I can't help myself. Usually we hear of problems in closed
repositories that I'm unable to reproduce locally. This time its in
gerrit itself, which should make things easier.

> The missing blob is of tree-type and contains:

Yikes. That is a problem. We are assuming this thing (bddb98a) is a
blob. But its actually a tree. I have it locally as well and just
confirmed, its actually a tree. As it turns out, its the
gerrit-gwtui/src/main/java/com/google/gerrit/client/changes directory
introduced by commit 32002457744e9e0c17b258fcfea5dd6d30732950. But
given how enumeration works, we should be encountering it first as
part of commit 5f11b2954cf149e001a5d34ea7d867674370acc6.

Was there a stack trace attached to this spew as well? I really could
use it, because it might help me to narrow down where we were inside
of the JGit code when this triggers. But I at least have a starting
point. Most of those refs above I probably have already, so I may be
able to setup a test case locally and reproduce the problem.

Shawn Pearce

unread,
Sep 6, 2010, 3:53:02 PM9/6/10
to Ulrik Sjölin, Repo and Gerrit Discussion
On Mon, Sep 6, 2010 at 07:40, Shawn Pearce <s...@google.com> wrote:
> On Mon, Sep 6, 2010 at 06:11, Ulrik Sjölin <ulrik....@gmail.com> wrote:
>> Ok, so here is a first printout of my 100% reproducible case which
>> makes it impossible for me to update our internal gerrit repo with new
>> code. So the command I invoke is this:
>>
>> $ git push ssh://ulrik....@review-test.sonyericsson.net:29418/tools/gerrit.git
>> 70a7ca07267b367461dee5a2739535a27baa2e9c:refs/heads/master
>>
>> This is the output:
>>
>> Counting objects: 1495, done.
>> Delta compression using up to 2 threads.
>> Compressing objects: 100% (392/392), done.
>> Writing objects: 100% (1214/1214), 265.91 KiB, done.
>> Total 1214 (delta 510), reused 1169 (delta 468)
>> error: unpack failed: error Missing blob
>> bdddb98a04637387444b80139e1c758c72d8e1ab

I was able to reproduce the bug locally, and I believe this to now be
fixed as of [1].

The problem was actually deep within JGit. When branch level access
controls are enabled, the server tries to prove that any object the
client mentions but did not provide on the wire is actually visible to
the client through one of the branches he has READ access to.
Unfortunately this is somewhat complex as the client can mention any
object as a delta base for almost anything else, and we err on the
side of caution by proving that the client can reach the delta base
before we accept it. The check works by building up a list of
potential delta bases and then validating that all delta bases
mentioned by the client actually appear on that list. Because the
list can be arbitrarily large (2+ million entries for the Linux
kernel), the server tries to prune the list the same way the client
builds the pack file. To save time and memory we only put an object
onto this list if the client would realistically consider the object
as a valid delta base for communication with this server.
Unfortunately JGit wasn't using the same parameters to build the list,
which meant we omitted a very large number of candidates and reported
spurious reachability failures.

[1] https://review.source.android.com/17027

Ulrik Sjölin

unread,
Sep 7, 2010, 2:23:40 PM9/7/10
to Repo and Gerrit Discussion
I have tried the fix today and it works!! (naturally ;-))

Thanks for killing this bug, it has haunted us for some time.

Ulrik
Reply all
Reply to author
Forward
0 new messages