Stale packed-refs.lock file

[DARSHIT THAKKAR]

Hi,

Often we run into issues where we get errors while uploading a new patchset

Caused by: com.google.gerrit.server.git.LockFailureException: Update aborted with one or more lock failures: PackedBatchRefUpdate[ CREATE: 0000000000000000000000000000000000000000 ....... (LOCK_FAILURE) ...... (REJECTED_OTHER_REASON: transaction aborted) ]

and the issue turns out to be stale packed-refs.lock file

We stop gerrit and then we delete the packed-refs.lock and start gerrit

I wanted to check, can we delete this file w/o stopping gerrit? and what would be the ways to be super sure that this lock file is indeed stale?

[Sven Selberg]

Stale lock files are left by processes that died before they were able to clean up the lock file.
The usual approach is to check if the file is "old", in this case if it's 1-2 hours old it's unlikely that the process that created the lock (git gc et.al) is still alive. And if the process that created the lock-file is dead you should be able to safely remove it.

You should be able to check creator and usage with tools like lsof but I don't know if that will give you any useful information in the case of lock-files.

 

[Martin Fick]

On Tue, 5 Sept 2023 at 00:30, DARSHIT THAKKAR <dnt...@gmail.com> wrote:

Often we run into issues where we get errors while uploading a new patchset

...

and the issue turns out to be stale packed-refs.lock file

I am curious how often this happens for you and for others? once a day, week, month, year?

Gerrit 3.x+ now directly modifies the packed-refs file on every new change upload. This makes this somewhat riskier than it used to be, and I wonder if this is now something that we need to worry about and potentially design solutions to avoid?

Any data from others experiencing this would be very useful.

Thanks,

-Martin

[Matthias Sohn]

We observed on busy large repositories that git gc sometimes fails since it cannot lock packed-refs

if some concurrent receive-pack request in Gerrit already holds this lock during PackedBatchRefUpdate

to process an incoming push.

We disabled packing refs in git gc since they are anyway repacked on those receive-packs.

 

Thanks,

-Martin

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKFHYViwkDyhDO7ewCwjXqGUyRKCxr4MFq0k%3DQ0MXkpEm7b4aA%40mail.gmail.com.

[Luca Milanesio]

On 6 Sep 2023, at 23:26, Matthias Sohn <matthi...@gmail.com> wrote:

On Wed, Sep 6, 2023 at 11:47 PM Martin Fick <marti...@linaro.org> wrote:

On Tue, 5 Sept 2023 at 00:30, DARSHIT THAKKAR <dnt...@gmail.com> wrote:

Often we run into issues where we get errors while uploading a new patchset

... 

and the issue turns out to be stale packed-refs.lock file

I am curious how often this happens for you and for others? once a day, week, month, year?

Gerrit 3.x+ now directly modifies the packed-refs file on every new change upload. This makes this somewhat riskier than it used to be, and I wonder if this is now something that we need to worry about and potentially design solutions to avoid?

Any data from others experiencing this would be very useful.

We observed on busy large repositories that git gc sometimes fails since it cannot lock packed-refs

if some concurrent receive-pack request in Gerrit already holds this lock during PackedBatchRefUpdate

to process an incoming push.

We disabled packing refs in git gc since they are anyway repacked on those receive-packs.

Thanks for sharing your experience, we also experience the same issue when the below two conditions happen at the same time:

1. The repository has *a lot of refs* (over 2M) and the packed-refs file is around 100MBytes

2. The update for packed-refs fails when there is a lot of workload and JVM GC activity, causing the thread updating the refs failing

So far, we have some crontab detecting the packed-refs.lock as stale (the utime of the .lock file is older than X minutes) and we remove it.

Luca.

 

Thanks,

-Martin

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKFHYViwkDyhDO7ewCwjXqGUyRKCxr4MFq0k%3DQ0MXkpEm7b4aA%40mail.gmail.com.

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3S%3DBN-M%3DnV8CQec2tGSW-D8iv--Odz7XguTor4j53F4Dw%40mail.gmail.com.

[Martin Fick]

On Thu, 7 Sept 2023 at 08:24, Luca Milanesio <luca.mi...@gmail.com> wrote:

On 6 Sep 2023, at 23:26, Matthias Sohn <matthi...@gmail.com> wrote:

On Wed, Sep 6, 2023 at 11:47 PM Martin Fick <marti...@linaro.org> wrote:

On Tue, 5 Sept 2023 at 00:30, DARSHIT THAKKAR <dnt...@gmail.com> wrote:

Often we run into issues where we get errors while uploading a new patchset

... 

and the issue turns out to be stale packed-refs.lock file

I am curious how often this happens for you and for others? once a day, week, month, year?

Gerrit 3.x+ now directly modifies the packed-refs file on every new change upload. This makes this somewhat riskier than it used to be, and I wonder if this is now something that we need to worry about and potentially design solutions to avoid?

Any data from others experiencing this would be very useful.

We observed on busy large repositories that git gc sometimes fails since it cannot lock packed-refs

if some concurrent receive-pack request in Gerrit already holds this lock during PackedBatchRefUpdate

to process an incoming push.

We disabled packing refs in git gc since they are anyway repacked on those receive-packs.

Thanks for sharing your experience, we also experience the same issue when the below two conditions happen at the same time:

1. The repository has *a lot of refs* (over 2M) and the packed-refs file is around 100MBytes

2. The update for packed-refs fails when there is a lot of workload and JVM GC activity, causing the thread updating the refs failing

So far, we have some crontab detecting the packed-refs.lock as stale (the utime of the .lock file is older than X minutes) and we remove it.

Thanks for this extra useful info!

I guess a related question is, do you find that when this happens there also tends to be one or more stale loose refs locks also? I ask because it seems like anytime the packedrefs lock is taken, it likely also requires at least one (and it gerri'ts case likely at least 2?) loose ref lock(s) to be taken also for the update? So if there is a process interruption which leads to the packedrefs lock becoming stale, I would expect the loose refs locks also to become stale, do you find that to be the case?

-Martin

 

[Matthias Sohn]

We don't see stale locks but lock contention which sometimes causes git gc to fail.

Here is a fix which should help if we end up with a stale lock due to a graceful shutdown of the JVM:
https://git.eclipse.org/r/c/jgit/jgit/+/204213

 

-Martin

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKFHYVhTH5e6pCL--QW%3Dn-rdMeLFDRxocM6KPfBjxNbQKh-6VQ%40mail.gmail.com.