Gerrit upload "missing blob"
Ulrik Sjölin
We have run into problems here with the branch-access stuff. A number
of users have run into this issue, we have tried to debug this, but
our “failsafe” workaround has proven... uhhh...un-failsafe ;-)
The symptom looks like this:
$repo upload .
Upload project xyz:
branch contact-branch-1 ( 1 commit, Wed Jun 2 17:00:00 2010 +0200):
72bbd242 Display details screen.
to review.sonyericsson.net (y/n)? y
Counting objects: 29, done.
Compressing objects: 100% (13/13), done.
Writing objects: 100% (16/16), 19.23 KiB, done.
Total 16 (delta 2), reused 13 (delta 0)
error: unpack failed: error Missing blob
8f57a524712b5183bd57ba84ed78b848e62971f9
To ssh://john...@review.sonyericsson.net:29418/xyz
! [remote rejected] contact-branch-1 -> refs/for/feature-branch (n/a
(unpacker error))
error: failed to push some refs to 'ssh://
john...@review.sonyericsson.net:29418/xyz'
There are no restrictions on any of the branches involved here for the
user in question... he does have access to the missing blob. The blob
reported missing is part of his own commit. It just doesn’t seem the
user is allowed to refer to/access it for some reason.
Many times it helps to start anew with a clean repository (either with
repo init + sync or git clone), but not always. Some users has had
success with "git gc" and then push, another user reported success by
doing "git rm" of the file that contained the blob, committed and did
a separate commit+upload of the file in question later on.
Thoughts, advice and/or help is greatly appreciated!!!
I get the feeling there is some kind of bug in the branch-access
code.... would you agree or are we just doing things the wrong way?
Nicolas Sallembien
I've also had a similar problem lately on a repository here where
there was a change on the Gerrit repository, i.e.
refs/changes/88/1199/1 that was pointing to a bad object. This was
causing errors for the users when they were trying to upload with a
similar message. You could try running git fsck --full on the
repository and see if there are any error messages.
It's also quite possible there is a bug in the branch access code, I
was not very familiar with the JGit code I modified and I might have
overlooked something.
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
Shawn Pearce
> Do you think it could be a manifestation of bug 390?
>
> I've also had a similar problem lately on a repository here where
> there was a change on the Gerrit repository, i.e.
> refs/changes/88/1199/1 that was pointing to a bad object. This was
> causing errors for the users when they were trying to upload with a
> similar message. You could try running git fsck --full on the
> repository and see if there are any error messages.
Sounds like that might be the case. If the problem seems to disappear
by the client creating a new clone, it might be issue 390 on the
server, or a reference that is pointing at something that really is
gone and that `git fsck --full` would complain about.
During a push the server tells the client things it has. If the
client has similar content, it can create deltas for the server that
assumes the server has what it told the client it has. If the server
doesn't actually have something (e.g. it lied in that initial message
to the client), then the server can't unpack that delta and will crash
out.
In a fresh clone, its possible the client doesn't have something now
that the older clone had before. Which means the client might not be
using the same delta, and is therefore able to create a different
delta which the server can actually recognize.
Issue 390 caused the server to temporarily lose access to an object on
disk. But it may still have claimed to the client that it has that
object. Nico recently found a different bug where the server might
advertise it has something, even though its legitimately missing from
disk. This latter case can at least be discovered with `git fsck
--full`.
> It's also quite possible there is a bug in the branch access code, I
> was not very familiar with the JGit code I modified and I might have
> overlooked something.
Also possible, but I know I went back and fixed some issues in there.
It seemed reasonably correct to me when I left it. :-)
Ulrik Sjölin
On Jun 3, 9:06 pm, Shawn Pearce <s...@google.com> wrote:
> > Do you think it could be a manifestation of bug 390?
>
> > I've also had a similar problem lately on a repository here where
> > there was a change on the Gerrit repository, i.e.
> > refs/changes/88/1199/1 that was pointing to a bad object. This was
> > causing errors for the users when they were trying to upload with a
> > similar message. You could try running git fsck --full on the
> > repository and see if there are any error messages.
>
this problem. git reports a bunch of dangling commits... would that be
something that could cause these symptoms? would git fsck --full --
lost-found would take care of this?
Shawn Pearce
Dangling commits are OK. Its odd to see them on the server, but its
not corruption, and wouldn't be causing this. A future `git gc` will
remove them from the filesystem when its safe to do so.
If you aren't yet on 2.1.2.5 (or a recent master snapshot that uses
the newer JGit), I have to say that I still suspect issue 390 here.
Fredrik Luthander
> On Thu, Jun 3, 2010 at 12:39, Ulrik Sjölin
After some error searching on the server on the topic of missing
blobs, I have some more information to add to Ulriks previous posts.
We're currently running gerrit on SEMC internal commit
e745c5e877b3be8513c2e6914b4a533c0231612e, which basically is commit
a608bbc2eab9f98ebe84b1b060d9d1e337a6c705, but with commit
fc53f7fd7a5244e628803c8b543483db70875ed0 from N. Sallembien reverted
due to it containing a db schema upgrade (that we couldn't upgrade to
at this point in time).
With this version of gerrit running on all our servers, the following
behaviour can be observed:
#1 User performs repo init; repo sync
#2 User creates new topic branch in one git project
#3 User adds and commits a very small change (two-three lines at most)
#4 User runs repo upload, to deliver this as a change in Gerrit. this
upload works without any visible errors.
#5 User abandons change in Gerrit.
#6 User runs repo start anew, creates a second topic branch.
#7 Makes a new commit with changes to the same files, intended as a
new change.
#8 repo upload gives message -> missing blob <alphabet soup>
We are at this point not sure that this is 100% repeatable for all
users.
The user in question runs git version 1.5.6.3.
Investigation shows that the blob reported missing actually exists on
the server. In fact, the entire commit makes it to the server, not
only this particular blob that's supposedly missing.
If we disable fsck on upload, it makes no difference, which kind of
makes sense since the commit with objects makes it into the server
side git.
diff --git a/etc/gerrit.config b/etc/gerrit.config
index ec7b05a..d0eb8af 100644
--- a/etc/gerrit.config
+++ b/etc/gerrit.config
@@ -45,6 +45,8 @@
packedGitLimit = 2g
packedGitOpenFiles = 4096
packedGitWindowSize = 16k
+[receive]
+ fsckObjects = false
[cache]
directory = /var/cache/git
Also the error seems to be "concentrated" to some users, but these
users currently have no known common denominator.
The problems seems to have been reported first time last friday,
possibly in coinciding in time with us enabling read access controls
on a per branch basis.
The latest news is that it seems that an upgrade to gerrit 1.7.x.x
makes this problem go away.
We will add more information to this thread as we learn. :)
Nicolas Sallembien
I don't think any version that starts with 1.5 is supported.
Thanks
Nico
On Friday, June 4, 2010, Fredrik Luthander
Fredrik Luthander
On Jun 4, 4:10 pm, Nicolas Sallembien <nsallemb...@gmail.com> wrote:
> Yes, I think your git version is likely to be the problem.
>
> I don't think any version that starts with 1.5 is supported.
>
> Thanks
> Nico
>
> On Friday, June 4, 2010, Fredrik Luthander
>
Where can I find info on what's supported and not? I guess we need to
push our IT department to roll a newer version of git to our clients,
but we need to know the span first then.
BR,
Fredrik
Nicolas Sallembien
later, although for a different issue: there is a security issue in
1.6.5 and below.
Shawn Pearce
<fredrik....@sonyericsson.com> wrote:
>
> Where can I find info on what's supported and not? I guess we need to
> push our IT department to roll a newer version of git to our clients,
> but we need to know the span first then.
We don't publish this, because in theory any Git client newer than the
earliest 0.99 can talk to Gerrit Code Review over SSH. I haven't
tested something that old, but the basic protocol is the same. Any
Git client newer than 1.5.0 can most certainly talk to Gerrit over SSH
as most protocol advances came between 0.99 and 1.5.
To use the http:// protocols you need 1.6.6 or later.
I always recommend the latest Git release, because its pretty well
tested, and the latest release always contains more bug fixes and
features than the prior releases. Sure there's a risk of breakage or
regression, but I think the folks who build Git do an amazing job at
trying to keep the system more stable with each new version. I wish
we were that good with Gerrit Code Review. Of course it helps testing
when there are literally hundreds of contributors per release. :-)
Shawn Pearce
<fredrik....@sonyericsson.com> wrote:
>
> We're currently running gerrit on SEMC internal commit
> e745c5e877b3be8513c2e6914b4a533c0231612e, which basically is commit
> a608bbc2eab9f98ebe84b1b060d9d1e337a6c705, but with commit
> fc53f7fd7a5244e628803c8b543483db70875ed0 from N. Sallembien reverted
> due to it containing a db schema upgrade (that we couldn't upgrade to
> at this point in time).
> With this version of gerrit running on all our servers, the following
> behaviour can be observed:
Can you humor me and cherry-pick
2366d781b35de7e2e5740560a5877a01e24e44b1 into your running version?
Its the commit that fixes issue 390. According to what you have told
me above, you don't have that in your build, which means this may be
just another symptom of the same fundamental bug in issue 390... JGit
loses access to objects on disk because a file was closed behind its
back.
Shawn Pearce
> users currently have no known common denominator.
> The problems seems to have been reported first time last friday,
> possibly in coinciding in time with us enabling read access controls
> on a per branch basis.
>
> The latest news is that it seems that an upgrade to gerrit 1.7.x.x
> makes this problem go away.
Also, I think these are both red-herrings. The client version (I
assume you meant git 1.7.x.x above?) shouldn't have an impact on
whether or not the server can properly scan the pack that was sent to
it. No known client version of Git has ever produced an invalid pack
on the network. Some versions (1.5.4?) have had bugs while scanning a
pack fetched over the network in particular circumstances, but its
never sent a bad pack.
The read access controls add an extra level of processing during the
receive code path, which causes more objects to be checked inside of
the server. Because the server is checking more objects, its
increasing the odds that issue 390 will cause an object to be looked
for and not get found.
If I'm right about the root cause of issue 390, its pretty random.
The failure is typically started when a client uses Ctrl-C to kill a
running command, and then that project might start to show failures if
the server was accessing Git data when the Ctrl-C interrupt was
received. If you have enough users, you can't really predict who will
Ctrl-C when/where, it just happens. Humans are these fallible
creatures who hit up-arrow-enter in the wrong terminal window and
restart a command they didn't mean to run.
So please give commit 2366d781b35de7e2e5740560a5877a01e24e44b1 a try
and let us know if its resolved the problem.
Luthander, Fredrik
We must have made some mistake when looking at the SHA1s then, Ulrik specifically have delivered a binary to us that have the 390-fix, but without the schema upgrade.
Ulrik, comments on this? Can you upload our commit to the local gerrit repo so I can run my second set of eyes on the commit-tree? :)
BR,
Fredrik
Fredrik Luthander
> On Fri, Jun 4, 2010 at 07:03, Fredrik Luthander
>
After examining all commits closer, it seems that Ulrik by mistake
have created a war-binary which hasn't issue 390 integrated! So, you
could be onto something here, but unfortunately we won't be able to
pick this up until Tuesday, since Monday is public holiday for most of
us in Sweden.
So, thanks for all the help so far, we will report back once we have
the chance to test this with our users!
BR,
Fredrik
Fredrik Luthander
On Jun 5, 12:15 am, Fredrik Luthander
We now have some bad news to bring you.
We've upgraded Gerrit to really include the bugfix for issue #390, and
we've also upgraded users to git 1.7.
But today we've seen reports of this error from different users that
have git 1.7.0.x and 1.7.1.x.. It doesn't make any difference as far
as we can see. It seems we've emptied all the possibilities when it
comes to software versions.
However, we've made the error go away a few times, by just fiddling
with things like caches and rights and stuff. However, we have so far
not been able to spot any pattern, so this is still a long shot from
our side.
We had one users where we added full rights on the project to the
user, plussing everything as much as possible. That didn't help, but
when we then consequently flushed all caches on the upload server, it
suddenly work. However, we have not been able to repeat that behaviour
a second time. Using the latest version of gerrit on master doesn't
seem to help either. Currently we are quite confused to be honest.
Any light you can shed, or suggestions you can give is helpful at this
point.
Shawn Pearce
<fredrik....@sonyericsson.com> wrote:
> We now have some bad news to bring you.
> We've upgraded Gerrit to really include the bugfix for issue #390, and
> we've also upgraded users to git 1.7.
> But today we've seen reports of this error from different users that
> have git 1.7.0.x and 1.7.1.x.. It doesn't make any difference as far
> as we can see. It seems we've emptied all the possibilities when it
> comes to software versions.
Arrrgh.
> However, we've made the error go away a few times, by just fiddling
> with things like caches and rights and stuff. However, we have so far
> not been able to spot any pattern, so this is still a long shot from
> our side.
The only thing that should have an impact here is READ +1 permissions.
And the only cache that matters is the "projects" cache, as that is
what holds a cached copy of the permissions records from the database.
> We had one users where we added full rights on the project to the
> user, plussing everything as much as possible. That didn't help, but
> when we then consequently flushed all caches on the upload server, it
> suddenly work. However, we have not been able to repeat that behaviour
> a second time. Using the latest version of gerrit on master doesn't
> seem to help either. Currently we are quite confused to be honest.
Yikes.
Can we get more information about a particular occurrence of this problem?
When the unpack fails with a missing object, I'd like to know:
- Message(s) on the client side.
- Full stack trace(s) from the server when the upload failed.
- Is the missing SHA-1 actually in the server repository? (`git
cat-file -t SHA1`)
- Are there branch level READ permissions in the project?
- Does the user have READ on all branches, or just a subset?
- If its a subset, is the following true? `git rev-list --objects
refs/heads/branch-user-can-see refs/heads/other-branch-user-can-see
... | grep SHA1`
Fredrik Luthander
On Jun 11, 6:05 pm, Shawn Pearce <s...@google.com> wrote:
> On Fri, Jun 11, 2010 at 07:01, Fredrik Luthander
>
> > We now have some bad news to bring you.
> > We've upgraded Gerrit to really include the bugfix for issue #390, and
> > we've also upgraded users to git 1.7.
> > But today we've seen reports of this error from different users that
> > have git 1.7.0.x and 1.7.1.x.. It doesn't make any difference as far
> > as we can see. It seems we've emptied all the possibilities when it
> > comes to software versions.
>
> Arrrgh.
> > However, we've made the error go away a few times, by just fiddling
> > with things like caches and rights and stuff. However, we have so far
> > not been able to spot any pattern, so this is still a long shot from
> > our side.
>
> The only thing that should have an impact here is READ +1 permissions.
> And the only cache that matters is the "projects" cache, as that is
> what holds a cached copy of the permissions records from the database.
Projects-- and on the project specifically, also with no result.
> > We had one users where we added full rights on the project to the
> > user, plussing everything as much as possible. That didn't help, but
> > when we then consequently flushed all caches on the upload server, it
> > suddenly work. However, we have not been able to repeat that behaviour
> > a second time. Using the latest version of gerrit on master doesn't
> > seem to help either. Currently we are quite confused to be honest.
>
> Yikes.
>
> Can we get more information about a particular occurrence of this problem?
particular client git where the missing blob-error wouldn't go away no
matter how many times we pushed.
We discovered the error on our master server which has a version which
is a hack (Schema 32 + bugfix390).
We moved to our test server with the same version, error stayed, we
upgraded to a version that run schema 34 and it still stayed. Ulrik
said he would setup the gits in his dev environment and step the
upload. Let's see if that gives something!
> When the unpack fails with a missing object, I'd like to know:
>
> - Message(s) on the client side.
> - Full stack trace(s) from the server when the upload failed.
> - Is the missing SHA-1 actually in the server repository? (`git
> cat-file -t SHA1`)
not. If I remember correctly we've seen at least one occurence of a
missing tree, but I couldn't swear by it.
> - Are there branch level READ permissions in the project?
1.7.1.x too!) and we don't use any branch specific rights at all.
either you see the git, or you don't.
However, some rules are in effect on --All Projects-- now that you
mention it...
> - Does the user have READ on all branches, or just a subset?
>
> - If its a subset, is the following true? `git rev-list --objects
> refs/heads/branch-user-can-see refs/heads/other-branch-user-can-see
> ... | grep SHA1`
project does have some branch specific rules that is applied on --All
Projects--.
This means that there are branches in this small project that users of
this project are not able to "access" (even though the branch don't
exist)..
Hmm. That could be a pattern we've not looked for so far! If the
branch doesn't exist in the project.. but the rule is there to protect
it anyway (something that doesn't exist..)
I just checked with Ulrik, and from the top of our heads we could
match this negating rule to all the affected gits so far. To really
see if this is true I would have to dig through my support mailbox
though to get better statistics..
Do you think this could be feasible? Or should we drop this thought
again and look for other stuff?
Cecilia
We are still having several users reporting the missing blob error.
The problem is only reported from users at one site. The only
difference in server configuration we can find is that we
use git protocol for replicating to the slave-server at this site.
Also, this happens to be the site at which the master server resides,
not sure if the relative lack of network latency
can affect this error?
I have collected info from 2 users having the problem, user 1 has
downloaded code from external source.
User 1
> - Message(s) on the client side. git version 1.7.0.3
Repo upload
...
...
Counting objects: 233, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (86/86), done.
Writing objects: 100% (178/178), 30.91 KiB, done.
Total 178 (delta 99), reused 171 (delta 92)
To ssh://john...@review.sonyericsson.net:29418/xxx
! [remote rejected] my/master -> xxx/master (n/a (unpacker error))
> - Full stack trace(s) from the server when the upload failed.
No visible stack trace in the logs.
git cat-file -t 8f440ca1c535e9e7633ec34314cb63fa8d3df2fb
tree
> - Does the user have READ on all branches, or just a subset?
set on the project for another branch(PHD),and
the user is a member of that group.
To check the repository we did git fsck --full on the server side, a
few dangling commits were found but no SHA1
referring to above error message.
User 2
- Message(s) on the client side. git version 1.7.0.3
$ git commit -a --amend
[detached HEAD 400305a] New script
1 files changed, 164 insertions(+), 0 deletions(-)
create mode 100644 aaa-xx.pl
$ git push -f ssh://john....@review.sonyericsson.net:29418/xx/yy
HEAD:refs/changes/34346
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.69 KiB, done.
Total 3 (delta 1), reused 0 (delta 0)
To ssh://john...@review.sonyericsson.net:29418/xx/yy
! [remote rejected] HEAD -> refs/changes/34346 (n/a (unpacker error))
> - Full stack trace(s) from the server when the upload failed.
No visible stack trace in the logs.
> cat-file -t SHA1`)
git cat-file -t 6b4d641e6f7c08b0ee083efad8fda27390b1bcb5
tree
To check the repository we did git fsck --full on the server side, a
few dangling commits were found but no SHA1
referring to above error message.
> - Does the user have READ on all branches, or just a subset?
On Jun 11, 6:55 pm, Fredrik Luthander
> We discovered the error on our master server which has a version which
> is a hack (Schema 32 + bugfix390).
> We moved to our test server with the same version, error stayed, we
> upgraded to a version that run schema 34 and it still stayed. Ulrik
> said he would setup the gits in his dev environment and step the
> upload. Let's see if that gives something!
>
>
>
> > - Message(s) on the client side.
>
> Message is identical to the one in the OP.
>
> > - Full stack trace(s) from the server when the upload failed.
>
> No visible stack trace in the logs.. ?
>
> > cat-file -t SHA1`)
>
> Yes it's there. It is unclear to us if theblobis in other commits or
> not. If I remember correctly we've seen at least one occurence of amissingtree, but I couldn't swear by it.
Ragesh For U :-)
//GITAM team SonyEricsson