replication over ssh with https_proxy broken in 3.4

[Peter Bruin]

Hi,

We upgraded to 3.4 recently and notice the replication was broken. For our docker image we use the base 3.4.1 image and add some python package for the hook scripts.

Do be able to access the internet there are the usual https_proxy and no_proxy environment vars. However, using these breaks the replication as it looks that the replication over ssh also uses these variables, but does not respect the no_proxy.

In the end I fixed it by using adding the proxy to /etc/apt/apt.conf.d/01_proxy but I wanted to report it anyway as I don't think the ssh can use most proxies.

Thanks,

Peter

[luca.mi...@gmail.com]

Sent from my iPhone

On 28 Oct 2021, at 05:58, Peter Bruin <peterb...@gmail.com> wrote:

Hi,

We upgraded to 3.4 recently and notice the replication was broken. For our docker image we use the base 3.4.1 image and add some python package for the hook scripts.

Do be able to access the internet there are the usual https_proxy and no_proxy environment vars. However, using these breaks the replication as it looks that the replication over ssh also uses these variables, but does not respect the no_proxy.

In the end I fixed it by using adding the proxy to /etc/apt/apt.conf.d/01_proxy but I wanted to report it anyway as I don't think the ssh can use most proxies.

Does this happen with a vanilla Gerrit jar? With native packages or with a Docker distribution?

Luca

Thanks,

Peter

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/42fd1d10-fbe1-4a62-8661-c222791b72edn%40googlegroups.com.

[Peter Bruin]

Yes, this happens with a vanila Gerrit jar with Docker distribution.

This is my Dockerfile. Don't think there is anything special here.

FROM gerritcodereview/gerrit:3.4.1-ubuntu20

# DO NOT USE THE PROXY VARIABLE

# The replication fails when the proxy is being used

# Use the apt proxy setting instead

#ENV http_proxy=http://proxy.example.com:9480

#ENV https_proxy=http://proxy.example.com:9480

#ENV no_proxy=127.0.0.1,.example.com

COPY ./apt/01proxy /etc/apt/apt.conf.d/

COPY ./ssh/* /var/gerrit/.ssh/

COPY ./etc/* /var/gerrit/etc/

COPY ./plugins/* /var/gerrit/plugins/

COPY ./static/* /var/gerrit/static/

COPY ./hooks/* /var/gerrit/hooks/

USER root

RUN apt-get update && apt-get -y install python3 python3-pip

RUN ln -s /usr/bin/python3 /usr/bin/python

ADD https://gerrit-ci.gerritforge.com/view/Plugins-stable-3.4/job/plugin-lfs-bazel-master-stable-3.4/lastSuccessfulBuild/artifact/bazel-bin/plugins/lfs/lfs.jar /var/gerrit/plugins/

ADD https://bootstrap.pypa.io/get-pip.py /tmp/get-pip.py

RUN python /tmp/get-pip.py --index-url=https://pypi.example.com/api/pypi/pypi_virtual/simple --trusted-host=pypi.example.com && rm /tmp/get-pip.py

RUN python -m pip install rtcclient --index-url=https://pypi.example.com/api/pypi/pypi_virtual/simple --trusted-host=pypi.example.com

RUN usermod --uid 1001 gerrit \

&&  groupmod --gid 1001 gerrit \

&&  rmdir /var/gerrit/tmp \

&&  mkdir /var/gerrit/tmp \

&&  chown -R gerrit:gerrit /var/gerrit \

&&  chmod 600 /var/gerrit/.ssh/*

RUN chown -R gerrit:gerrit /var/gerrit

USER gerrit

[Sven Selberg]

On Thursday, October 28, 2021 at 8:37:26 AM UTC+2 lucamilanesio wrote:

Sent from my iPhone

On 28 Oct 2021, at 05:58, Peter Bruin <peterb...@gmail.com> wrote:

Hi,

We upgraded to 3.4 recently and notice the replication was broken. For our docker image we use the base 3.4.1 image and add some python package for the hook scripts.

Do be able to access the internet there are the usual https_proxy and no_proxy environment vars. However, using these breaks the replication as it looks that the replication over ssh also uses these variables, but does not respect the no_proxy.

This is the release where Apache Mina became the default SSH library, is it possible that Mina is (mis)using these env-vars?
What happens when you switch back to Jcraft Jsch [1]?

gerrit.config:

   [ssh]

     clientImplementation = JSCH

[1] https://gerrit-documentation.storage.googleapis.com/Documentation/3.4.1/config-gerrit.html#ssh

[Peter Bruin]

Hi Sven,

I have tried that but it still doesn't work.

This is the exceptions I see in the replication_log

[2021-10-28 15:50:57,281] Cannot replicate to g...@code.example.com:group/scripts [CONTEXT pushOneId="eeb061cb" ]

org.eclipse.jgit.errors.TransportException: g...@code.example.com:group/scripts: Session is being closed

at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:250)

at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:74)

at org.eclipse.jgit.transport.SshTransport.getSession(SshTransport.java:107)

at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:281)

at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:274)

at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:146)

at com.googlesource.gerrit.plugins.replication.PushOne.listRemote(PushOne.java:711)

at com.googlesource.gerrit.plugins.replication.PushOne.doPushAll(PushOne.java:647)

at com.googlesource.gerrit.plugins.replication.PushOne.generateUpdates(PushOne.java:637)

at com.googlesource.gerrit.plugins.replication.PushOne.pushVia(PushOne.java:547)

at com.googlesource.gerrit.plugins.replication.PushOne.runImpl(PushOne.java:538)

at com.googlesource.gerrit.plugins.replication.PushOne.doRunPushOperation(PushOne.java:421)

at com.googlesource.gerrit.plugins.replication.PushOne.runPushOperation(PushOne.java:389)

at com.googlesource.gerrit.plugins.replication.PushOne.lambda$run$2(PushOne.java:375)

at com.google.gerrit.server.util.RequestScopePropagator.lambda$cleanup$1(RequestScopePropagator.java:182)

at com.google.gerrit.server.util.RequestScopePropagator.lambda$context$0(RequestScopePropagator.java:170)

at com.google.gerrit.server.git.PerThreadRequestScope$Propagator.lambda$scope$0(PerThreadRequestScope.java:70)

at com.googlesource.gerrit.plugins.replication.PushOne.run(PushOne.java:378)

at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)

at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)

at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused by: org.apache.sshd.common.SshException: Session is being closed

at org.apache.sshd.common.future.AbstractSshFuture.verifyResult(AbstractSshFuture.java:126)

at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:39)

at org.apache.sshd.client.future.DefaultAuthFuture.verify(DefaultAuthFuture.java:32)

at org.apache.sshd.common.future.VerifiableFuture.verify(VerifiableFuture.java:68)

at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:164)

at org.eclipse.jgit.transport.sshd.SshdSession.connect(SshdSession.java:99)

at org.eclipse.jgit.transport.sshd.SshdSessionFactory.getSession(SshdSessionFactory.java:237)

... 25 more

Caused by: org.apache.sshd.common.SshException: Session is being closed

at org.apache.sshd.client.session.ClientSessionImpl.preClose(ClientSessionImpl.java:172)

at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:94)

at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.sessionClosed(AbstractSessionIoHandler.java:46)

at org.apache.sshd.common.io.nio2.Nio2Session.doCloseImmediately(Nio2Session.java:276)

at org.apache.sshd.common.util.closeable.AbstractCloseable.close(AbstractCloseable.java:95)

at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:373)

at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:335)

at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:332)

at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)

at java.base/java.security.AccessController.doPrivileged(Native Method)

at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)

at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)

at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)

at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)

... 3 more

[Sven Selberg]

To me it looks like apache is still being used, perhaps my assumptions about what the ssh.clientImplementation setting did was erroneous.

[Makson Lee]

we have same issue, our gerrit version is 3.6.0, if we remove system http_proxy & https_proxy setting, then problem gone.

[Thomas Wolf]

On Wednesday, June 1, 2022 at 5:58:39 AM UTC+2 cdle...@gmail.com wrote:

we have same issue, our gerrit version is 3.6.0, if we remove system http_proxy & https_proxy setting, then problem gone.

On Thursday, October 28, 2021 at 11:58:22 AM UTC+8 peterb...@gmail.com wrote:

Hi,

We upgraded to 3.4 recently and notice the replication was broken. For our docker image we use the base 3.4.1 image and add some python package for the hook scripts.

Do be able to access the internet there are the usual https_proxy and no_proxy environment vars. However, using these breaks the replication as it looks that the replication over ssh also uses these variables, but does not respect the no_proxy.

Gerrit configures its SshSessionFactory instance for Apache MINA sshd to use an org.eclipse.jgit.transport.sshd.DefaultProxyDataFactory, which is based on the standard java.net.ProxySelector.

The DefaultProxyDataFactory queries the java.net.ProxySelector first for a SOCKS proxy, but if there none, it queries again for an HTTP proxy. (An SSH connection can perfectly well be proxied via HTTP CONNECT.)

I have no idea why the no_proxy would not be respected; that bit is all done inside the java.net.ProxySelector implementation. (Typically sun.net.spi.DefaultProxySelector.)

[Makson Lee]

FYR, https://bugs.java.com/bugdatabase/view_bug.do?bug_id=JDK-8144008

[Makson Lee]

On Thursday, June 2, 2022 at 4:34:16 AM UTC+8 Thomas Wolf wrote:

On Wednesday, June 1, 2022 at 5:58:39 AM UTC+2 cdle...@gmail.com wrote:

we have same issue, our gerrit version is 3.6.0, if we remove system http_proxy & https_proxy setting, then problem gone.

On Thursday, October 28, 2021 at 11:58:22 AM UTC+8 peterb...@gmail.com wrote:

Hi,

We upgraded to 3.4 recently and notice the replication was broken. For our docker image we use the base 3.4.1 image and add some python package for the hook scripts.

Do be able to access the internet there are the usual https_proxy and no_proxy environment vars. However, using these breaks the replication as it looks that the replication over ssh also uses these variables, but does not respect the no_proxy.

Gerrit configures its SshSessionFactory instance for Apache MINA sshd to use an org.eclipse.jgit.transport.sshd.DefaultProxyDataFactory, which is based on the standard java.net.ProxySelector.

The DefaultProxyDataFactory queries the java.net.ProxySelector first for a SOCKS proxy, but if there none, it queries again for an HTTP proxy. (An SSH connection can perfectly well be proxied via HTTP CONNECT.)

we are using squid proxy server, ssh connection is not allowed by default, need to add following acl to make it working,

acl Safe_ports port 22

 

I have no idea why the no_proxy would not be respected; that bit is all done inside the java.net.ProxySelector implementation. (Typically sun.net.spi.DefaultProxySelector.)

still have this issue in gerrit version 3.8.1.