Issue with pushing tags -failed to lock, remote rejected

[cecilia svensson]

We are experiencing issues regarding push of tags to 20-25 repositories (out of 7000+)

Developers are pushing deliveries with “--tags” option. That compares all tags, updates where necessary and adds new ones.

 This has been working fine with a 30 minute sync interval, until end of last week.

No changes done for Gerrit master server regarding configuration, neither at server OS level.

Gerrit version 3.3.2

 Output from user terminal pushing

 remote: error: internal error       

To ssh://gerrit.com:29418/project/yyy/yyy/xxxxxxxxxx

! [remote rejected]   refs/tags/C4a-xx-0_xx -> refs/tags/C4a-xx-0_ xx  (failed to lock)

! [remote rejected]   refs/tags/K2-xx-0_xx  -> refs/tags/K2-xx-0_xx  (transaction aborted)

! [remote rejected]   refs/tags/K3-xx-0_xx  -> refs/tags/K3-xx-0_xx  (transaction aborted)

! [remote rejected]   refs/tags/K4-xx-0_xx  -> refs/tags/K4-xx-0_xx  (transaction aborted)

! [remote rejected]   refs/tags/K5-xx-0_xxx  -> refs/tags/K5-xx-0_xxx  (transaction aborted)

error: failed to push some refs to 'ssh://gerrit.com:29418/project/yyy/yyy/xxxxxxxxxx

+ echo 'git push gerrit --tags --force refs/remotes/upstream/master:refs/heads/master failed'

 

error_log from Gerrit Master 

 [SSH git-receive-pack /project/yyy/yyy/xxxxxxxxxx (userid)] WARN  com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning

 [SSH git-receive-pack /project/yyy/yyy/xxxxxxxxxx (userid)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push

java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.update.UpdateException: com.google.gerrit.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[

  CREATE: 0000000000000000000000000000000000000000 f3yyy..          refs/tags/C4a-xx-0_xx (LOCK_FAILURE)

  CREATE: 0000000000000000000000000000000000000000 50yy..         refs/tags/K2-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 f8yy..         refs/tags/K3-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 98yyy..        refs/tags/K4-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 a4yyy…         refs/tags/K5-xx-0_xxx (REJECTED_OTHER_REASON: transaction aborted)

 

At the same time these pushes hold up queue at Gerrit master server and receive-packs are halted and queued until we restart service.

 

Checking the git at master server disk, 

• the tags exist in the git (these are tags they are updating according to above). 
•  they are not visible in Gerrit UI. 
• -“git show tag” etc. gives output as well in the failing git.

 Git gc and git fsck executed for affected gits, no issues

 Developer tried to delete them via force push, since they reside on disk

$ git push -f gerrit  :C4a-xx-0_xx

error: unable to delete 'C4a-xx-0_xx': remote ref does not exist

 If I delete the tags in the git at master server (git tag -d tag) and then ask user to retry push, then tags are recreated, and visible in Gerrit UI

developer can also delete same tags and recreate once more after that deletion at server side.

 Restart of Gerrit service does not solve the issue, we haven’t done any reboot of server yet

Have found some similar threads searching for error – but no solution or root cause (a reboot of server helped for some)

 Looks like some permissions issue or lock issue, but haven't found anything related to that at server.

Anyone that recognize this issue?

Thanks

Cecilia

 

[Matthias Sohn]

Did this happen after a server crash ?

Check on the Gerrit server in the respective repositories if there are any stale packed-refs.lock files.

Which kind of filesystem do you use on the server ?
Do you use NFS ?

 

-Matthias

At the same time these pushes hold up queue at Gerrit master server and receive-packs are halted and queued until we restart service.

 

Checking the git at master server disk, 

• the tags exist in the git (these are tags they are updating according to above). 
•  they are not visible in Gerrit UI. 
• -“git show tag” etc. gives output as well in the failing git.

 Git gc and git fsck executed for affected gits, no issues

 Developer tried to delete them via force push, since they reside on disk

$ git push -f gerrit  :C4a-xx-0_xx

error: unable to delete 'C4a-xx-0_xx': remote ref does not exist

 If I delete the tags in the git at master server (git tag -d tag) and then ask user to retry push, then tags are recreated, and visible in Gerrit UI

developer can also delete same tags and recreate once more after that deletion at server side.

 Restart of Gerrit service does not solve the issue, we haven’t done any reboot of server yet

Have found some similar threads searching for error – but no solution or root cause (a reboot of server helped for some)

 Looks like some permissions issue or lock issue, but haven't found anything related to that at server.

Anyone that recognize this issue?

Thanks

Cecilia

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/97125f78-528b-4426-b7c6-d0ae09b95a40n%40googlegroups.com.

[cecilia svensson]

Thanks Mattias,

No issue just came up one afternoon, no server crash

there are no stale packed-refs.lock files, just checked

Server OS is Red Hat 7, and filesystem is XFS
We do not use NFS

For Gerrit we use java-11-openjdk

Some more from the server log, some entry was missed earlier

 [SSH git-receive-pack /project/yyy/yyy/xxxxxxxxxx (userid)] WARN  com.google.gerrit.server.git.MultiProgressMonitor : MultiProgressMonitor worker did not call end() before returning

 [SSH git-receive-pack /project/yyy/yyy/xxxxxxxxxx (userid)] ERROR com.google.gerrit.server.git.receive.AsyncReceiveCommits : error while processing push

java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.update.UpdateException: com.google.gerrit.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[

  CREATE: 0000000000000000000000000000000000000000 f3yyy..          refs/tags/C4a-xx-0_xx (LOCK_FAILURE)

  CREATE: 0000000000000000000000000000000000000000 50yy..         refs/tags/K2-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 f8yy..         refs/tags/K3-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 98yyy..        refs/tags/K4-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 a4yyy…         refs/tags/K5-xx-0_xxx (REJECTED_OTHER_REASON: transaction aborted) at com.google.gerrit.server.update.BatchUpdate.wrapAndThrowException(BatchUpdate.java:229)

at com.google.gerrit.server.update.BatchUpdate.execute(BatchUpdate.java:172)

at com.google.gerrit.server.update.SubmissionExecutor.execute(SubmissionExecutor.java:67)

at com.google.gerrit.server.git.receive.ReceiveCommits.handleRegularCommands(ReceiveCommits.java:747)

... 18 more

Java.util.concurrent.ExecutionException: com.google.gerrit.exceptions.StorageException: com.google.gerrit.server.update.UpdateException: com.google.gerrit.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[

  CREATE: 0000000000000000000000000000000000000000 f3yyy..          refs/tags/C4a-xx-0_xx (LOCK_FAILURE)

  CREATE: 0000000000000000000000000000000000000000 50yy..         refs/tags/K2-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 f8yy..         refs/tags/K3-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 98yyy..        refs/tags/K4-xx-0_xx (REJECTED_OTHER_REASON: transaction aborted)

  CREATE: 0000000000000000000000000000000000000000 a4yyy…         refs/tags/K5-xx-0_xxx (REJECTED_OTHER_REASON: transaction aborted)

at com.google.gerrit.git.RefUpdateUtil.checkResults(RefUpdateUtil.java:100)

at com.google.gerrit.git.RefUpdateUtil.executeChecked(RefUpdateUtil.java:57)

at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:382)

at com.google.gerrit.server.notedb.NoteDbUpdateManager.execute(NoteDbUpdateManager.java:332)

[cecilia svensson]

Some update

Server rebooted - issue solved for all gits except one (contains 3000+ tags)

If we delete failing tags at server side, then it is throwing error for new ones in next push

after an update from Sven Selberg (post didn't reach community) We will try to increase git_tags cache values now as next step (increased yesterday but maybe not enough)

"It smells like an issue with population of the git_tags cache.

This cache is persistent so it survives restarts, and git operations uses it to advertise tags.
My guess is that the tags was created on disk but the git_tags cache was not updated properly"

[Matthias Sohn]

On Thu, Mar 17, 2022 at 11:20 AM 'cecilia svensson' via Repo and Gerrit Discussion <repo-d...@googlegroups.com> wrote:

Some update

please avoid top posting on this list since it makes it harder to follow conversations

 

Server rebooted - issue solved for all gits except one (contains 3000+ tags)

If we delete failing tags at server side, then it is throwing error for new ones in next push

after an update from Sven Selberg (post didn't reach community) We will try to increase git_tags cache values now as next step (increased yesterday but maybe not enough)

"It smells like an issue with population of the git_tags cache.

This cache is persistent so it survives restarts, and git operations uses it to advertise tags.
My guess is that the tags was created on disk but the git_tags cache was not updated properly"

Did you try to flush the git_tags cache ?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ba337eda-4c44-4ee2-81cc-a075965067bfn%40googlegroups.com.

[Matthias Sohn]

On Thu, Mar 17, 2022 at 3:54 PM Svensson Cecilia (RBSN/ESW23) <Cecilia....@se.bosch.com> wrote:

please avoid top posting on this list

 

Yes, we flushed the git_tags cache, still same issue

We have now increased the git_tags cache values, but pending a restart of service

 

Could it be possible that the git_tags cache is corrupt?

And if so, is there anyway to recreate/rebuild it from scratch without reinit the site.

flushing the cache should remove all cached data and it should be repopulated when accessing tags

[Matthias Sohn]

On Fri, Mar 18, 2022 at 4:10 PM Svensson Cecilia (RBSN/ESW23) <Cecilia....@se.bosch.com> wrote:

Please avoid top-posting on this list since it makes it harder to follow the conversation.

Instead use interleaver posting style [1]. I already asked several times but you still ignore this.

 

The increase of git_tags cache did not solve issue

 

failing tags in push and error_log ends up as dangling tags and commits in the git

If we remove these, they end up in same state next push.

 

After this error occurred first time, we have seen increased exec time in sshd_log for the affected git but might be related to the error itself.

[SSH git-receive-pack /project/yyy/yyy/xxxxxxxxxx (userid)] userid a/10xxxxx git-receive-pack./ project/yyy/yyy/xxxxxxxxxx  2ms 9393ms 0 git/2.17.1

 

Will recreate git_tags cache as next step, or might be that the git is corrupt

run "git fsck --full" on the repository on the server to check if the repository is corrupt

[1] https://en.wikipedia.org/wiki/Posting_style#Placement_of_replies

[motorhe...@gmail.com]

Just experienced similar issue and flushing the git_tags cache did the trick. I would like to thank this thread for pointing me in that direction.

Cheers.