org.eclipse.jgit.transport.WantNotValidException: want

[Rajesh Babhulkar]

Hi ,

We are using Gerrit version 3.9.1 . Found the following error in error log. Can someone please help to fix this? ow can we fix this error?

[2024-04-16T00:45:09.240-04:00] [SSH git-upload-pack /abc (xxxx)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user xxxx account 1004013) during git-upload-pack '/wnc'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:819)
        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:492)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:675)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.eclipse.jgit.transport.WantNotValidException: want 38ce168b1ba9b5dee575d84f58099266a472370d not valid
        at org.eclipse.jgit.transport.UploadPack$AdvertisedRequestValidator.checkWants(UploadPack.java:1989)

Thanks and Regards,

Rajesh

[Daniele Sassoli]

As far as I'm aware this is due to a security feature in Gerrit that will prevent you from cloning/fetching if the sha1 you're trying to fetch is not advertised anymore when you get to fetch it.
Couldn't find any better docs on this than [1]

Retrying should fix the issue. You can also set uploadpack.allowanysha1inwant to true(as per link), but this might have some security implications. Ensure you understand them before enabling this setting.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=573390
 

Thanks and Regards,

Rajesh

[Sven Selberg]

On Tuesday, April 16, 2024 at 10:03:37 PM UTC+2 Daniele Sassoli wrote:

On Tuesday, April 16, 2024 at 6:29:48 PM UTC+1 Rajesh Babhulkar wrote:

Hi ,

We are using Gerrit version 3.9.1 . Found the following error in error log. Can someone please help to fix this? ow can we fix this error?

[2024-04-16T00:45:09.240-04:00] [SSH git-upload-pack /abc (xxxx)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user xxxx account 1004013) during git-upload-pack '/wnc'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:819)
        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:492)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:675)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: org.eclipse.jgit.transport.WantNotValidException: want 38ce168b1ba9b5dee575d84f58099266a472370d not valid
        at org.eclipse.jgit.transport.UploadPack$AdvertisedRequestValidator.checkWants(UploadPack.java:1989)

This can be a simple race-condition (when using git protocol version1):

CLIENT: What refs do you have?
SERVER:

... dc7d9180725d1bd4ca2d6c23fc7d0217c26f5d87 refs/heads/volatile-dev-branch ...

REF-UPDATE: refs/heads/volatile-dev-branch is force pushed making dc79180... orphan
CLIENT:: I want dc7d9180725d1bd4ca2d6c23fc7d0217c26f5d87
SERVER: Internal Server Error "want dc7d9180725d1bd4ca2d6c23fc7d0217c26f5d87 not valid" (since it's no longer reachable from any branch).

[Thomas Dräbing]

There also seems to be a bug in jgit that might cause a race condition, when using protocol V2 in case a wanted ref was updated during the fetch. We have opened a bug for it: https://github.com/eclipse-jgit/jgit/issues/48. Matthias is looking into it.

For us it affects large and busy repositories where some workflows include mirror clones or fetches of refs/changes/*, since those include notedb refs (e.g. refs/changes/01/101/meta), which tend to be updated a lot (comments, votes, etc.).

 

As far as I'm aware this is due to a security feature in Gerrit that will prevent you from cloning/fetching if the sha1 you're trying to fetch is not advertised anymore when you get to fetch it.
Couldn't find any better docs on this than [1]

Retrying should fix the issue. You can also set uploadpack.allowanysha1inwant to true(as per link), but this might have some security implications. Ensure you understand them before enabling this setting.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=573390
 

Thanks and Regards,

Rajesh

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/43ab1e4a-5867-4f0e-8805-9fd01f217a45n%40googlegroups.com.

[Sven Selberg]

Just to check if I understand things correctly:
The bug in v2 is basically the same as the one in v1 but in v1 you are affected even if you fetch one specific ref since v1 checks all refs regardless (so it's more frequent in v1), v2 is affected mostly if you fetch a wide ref-spec since v2 only verifies the refs you actually fetch?

[Thomas Dräbing]

Matthias is still analyzing the issue, but yes. Reading the code, jgit first computes the wanted ObjectIds based on the wanted Refs and then the advertised IDs based on the wanted refs. In a large and busy repo, it might happen that one ref has been updated between those two steps, because collecting refs takes relatively long and updates are frequent, leading to differences between wanted and advertised IDs. This leads to the AdvertisedRequestValidator to fail. However, this validator is not called in every case, but only when the condition [1] returns false. I am not familiar enough with the jgit code yet to say when this is the case.

[1] https://github.com/eclipse-jgit/jgit/blob/master/org.eclipse.jgit/src/org/eclipse/jgit/transport/UploadPack.java#L1262

 

For us it affects large and busy repositories where some workflows include mirror clones or fetches of refs/changes/*, since those include notedb refs (e.g. refs/changes/01/101/meta), which tend to be updated a lot (comments, votes, etc.).

 

As far as I'm aware this is due to a security feature in Gerrit that will prevent you from cloning/fetching if the sha1 you're trying to fetch is not advertised anymore when you get to fetch it.
Couldn't find any better docs on this than [1]

Retrying should fix the issue. You can also set uploadpack.allowanysha1inwant to true(as per link), but this might have some security implications. Ensure you understand them before enabling this setting.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=573390
 

Thanks and Regards,

Rajesh

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/43ab1e4a-5867-4f0e-8805-9fd01f217a45n%40googlegroups.com.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/eb9562cf-f5c6-46f2-9124-9b96e6954c01n%40googlegroups.com.

[Daniele Sassoli]

@Thomas, @Luca can correct me if I'm wrong, but this is not a bug, it's a feature. Setting that allowallsha1inwants will fix your use case I believe. You'll also see your Gerrit server reduces it's usage a fair bit as it doesn't need to compute the visibility of the refs at ecah step.

[Thomas Dräbing]

In our case we don't fetch a specific SHA1. We fetch for example `+refs/changes*:refs/changes*` or do a mirror clone. We have fetching specific SHA1 sums disabled on purpose due to the security considerations you mentioned above. We rely on Gerrit permissions being evaluated properly, which only works on refs.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/5676c71f-d627-4855-9706-d32cb2b21841n%40googlegroups.com.

[Daniele Sassoli]

Yes exactly. The setting I mentioned comes into play in this scenario. It will disable the second check, after the advertising, so you'll be able to fetch whatever has been advertised in the first place.

[Luca Milanesio]

What do you mean exactly? The Gerrit permissions are always evaluated regardless of fetching all refs, a subset of them or an individual one.

I did the security testing of the checking of the wants being reachable by the advertised (I hacked the Git client as requested and simulated a malicious client wanting to fetch a non-advertised an non-reachable ref) and verified that:

- Before the JGit fix in Gerrit, the client was able to fetch a non-advertised ref

- After the JGit fix in Gerrit, the client received a “wants-not-valid” error

One thing should be mentioned though: the difference between Git/SSH and Git/HTTP.

Git/SSH:

- there is a unique session with JGit

- JGit “remembers” the list of advertised refs

- JGit validates the “wants” against the ones “remembered” in the session, even if then have changed on the repo in the meantime

Git/HTTP:

- each HTTP call is separate, there no unique session (or state) with JGit on the server

- JGit “computes” the visible refs that are advertised

- JGit validates the “wants” against the ones “visible at the time” of the HTTP call

I have verified that the wants-not-valid because of the repo being very active happens on Git/HTTP a lot more than on Git/SSH, because of the way the protocol works.

The workaround suggested by Dani makes sense to me, with the caveat that it would potentially allow malicious Git clients to access SHA1s they do not have requested or have visibility.

HTH

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAG7bb4BYUVPppdxN2SD5vC4_F-3R2TpS821wNJrcx0%3DsKube8A%40mail.gmail.com.

[Matthias Sohn]

I think this issue is caused by jgit UploadPack#fetchV2 not caching refs it read in the first pass, and re-reading them when determining the advertised IDs 

which may cause discrepancies causing that AdvertisedRequestValidator fails. I am thinking about fixing that by using SnapshottingRefDirectory in this

method. This should have the effect that each ref is read at most once and such races shouldn't happen anymore without compromising visibility checks in Gerrit.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/831af0f1-59fa-41a3-b60d-55fb1cca57fbn%40googlegroups.com.

[Luca Milanesio]

Yep, that’s the case I’ve mentioned and it only happens with Git/HTTP.

which may cause discrepancies causing that AdvertisedRequestValidator fails. I am thinking about fixing that by using SnapshottingRefDirectory in this

method.

How? The two calls are independently served by two threads at different times. I would be interested in the solution, which would be highly appreciated :-)

This should have the effect that each ref is read at most once and such races shouldn't happen anymore without compromising visibility checks in Gerrit.

Yeah, that would be nice, agreed.

Luca.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3QEr1x2otS737uyyuagknn%3D0QysrRrXhTbsx9qYP%2BAf9Q%40mail.gmail.com.

[Piotr Szlązak]

On Wednesday, April 17, 2024 at 10:44:34 AM UTC+2 Luca Milanesio wrote:

One thing should be mentioned though: the difference between Git/SSH and Git/HTTP.

Git/SSH:

- there is a unique session with JGit

- JGit “remembers” the list of advertised refs

- JGit validates the “wants” against the ones “remembered” in the session, even if then have changed on the repo in the meantime

Git/HTTP:

- each HTTP call is separate, there no unique session (or state) with JGit on the server

- JGit “computes” the visible refs that are advertised

- JGit validates the “wants” against the ones “visible at the time” of the HTTP call

I have verified that the wants-not-valid because of the repo being very active happens on Git/HTTP a lot more than on Git/SSH, because of the way the protocol works.

Hello!

Since which version of JGit mentioned difference  between SSH and HTTP is present?

I'm pretty sure I was able to notice this problem also for SSH session.

By this problem I mean:

- refs want sent by client

- sha1s advertised  by server (besides others refs/changes/12/345612/meta -> abc)

- in other session update of refs/changes/12/345612/meta - previously advertised sha1 no longer being a tip of this branch (we only have allowTipSHA1InWant enabled)

- want sent by client with abc

- server response: fatal: remote error: want abc not valid

 

-- 

Piotr

[Luca Milanesio]

Just do a "git log -i --grep=allowTipSHA1InWant” on the JGit source code:

commit adc73c4ba1c5a0ddfaee9537df438bfa14f38a62

Author: Jonathan Tan <jonath...@google.com>

Date:   Fri Feb 23 14:07:02 2018 -0800

    Teach UploadPack basic "fetch" command

    Add basic support for the "fetch" command in the fetch-pack/upload-pack

    protocol v2. This patch teaches "have" and "done".

    The protocol specification (Documentation/technical/protocol-v2.txt in

    the Git project) states:

        want <oid>

            Indicates to the server an object which the client wants to

            retrieve.  Wants can be anything and are not limited to

            advertised objects.

    It is unspecified whether the server should respect the

    uploadpack.allowtipsha1inwant option etc. when serving packfiles. This

    patch is conservative in that the server respects them.

    Change-Id: I3dbec172239712ef9286a15b8407e86b87ea7863

    Signed-off-by: Jonathan Tan <jonath...@google.com>

HTH

Luca.

 

-- 

Piotr

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/1969464b-8407-4fd9-a4d6-afabf7d74a1dn%40googlegroups.com.

[Piotr Szlązak]

As far as I can see mentioned commit adc73c4ba1 introduces Git protocol V2. I understand this can help, as in V2 client can fetch arbitrary OID and in my case V0/V1 is being used.

But this commit does not introduce different handling of SSH vs. HTTP session.

Especially that V2 should be stateless:

https://git-scm.com/docs/protocol-v2#_capabilities

<quote>

Protocol version 2 is stateless by default. This means that all commands must only last a single round and be stateless from the perspective of the server side, unless the client has requested a capability indicating that state should be maintained by the server. Clients MUST NOT require state management on the server side in order to function correctly. This permits simple round-robin load-balancing on the server side, without needing to worry about state management.

</quote>

-- 

Piotr

[Piotr Szlązak]

On Thursday, June 6, 2024 at 6:23:26 PM UTC+2 Piotr Szlązak wrote:

[...] I understand this can help, as in V2 client can fetch arbitrary OID and in my case V0/V1 is being used.

If V0/V1 is used, allowAnySHA1InWant can help, so JGit 6.5 / Gerrit 3.7.1.

-- 

Piotr

[Piotr Szlązak]

On Thursday, June 6, 2024 at 11:45:57 PM UTC+2 Piotr Szlązak wrote:

If V0/V1 is used, allowAnySHA1InWant can help, so JGit 6.5 / Gerrit 3.7.1.

Or Gerrit 3.7.3 as  3.7.1 ships with JGit 6.4. Or Gerrit 3.8.0.

Yet I was not able to enable uploadPack.allowAnySHA1InWant. I tried to add it to:

- $site_path/etc/gerrit.config

- $site_path/etc/jgit.config

- $site_path/git/project.git/config

all without success.

-- 

Piotr

[Piotr Szlązak]

On Wednesday, June 26, 2024 at 7:10:08 PM UTC+2 Piotr Szlązak wrote:

Yet I was not able to enable uploadPack.allowAnySHA1InWant. I tried to add it to:

- $site_path/etc/gerrit.config

- $site_path/etc/jgit.config

- $site_path/git/project.git/config

all without success.

To double-check it I added uploadpack.hideRefs to  $site_path/etc/jgit.config. It works.

But adding uploadpack.allowTipSHA1InWant[1] to allow fetching SHA1 of hidden ref does not work.

I'm confused how and if uploadpack.allowXXX options work in Gerrit.

[1] https://git-scm.com/docs/git-config#Documentation/git-config.txt-uploadpackallowTipSHA1InWant

-- 

Piotr