Replication - Pushing to Bitbucket has begun failing with "channel is not opened"

585 views
Skip to first unread message

Tom Kerswill

unread,
Jul 26, 2016, 7:47:35 AM7/26/16
to Repo and Gerrit Discussion
Hi

We have replication set up on Gerrit to replicate all our repositories over to Bitbucket. This has been working great for a couple of years. Around 11th July 2016, the repos stopped replicating. The gerrit replication logs show messages like:

Cannot replicate to g...@bitbucket.org:myteam/myrepo.git

org.eclipse.jgit.errors.TransportException: g...@bitbucket.org:myrepo.git: channel is not opened.


... It seems like BitBucket recently switched off SSH Multiplexing; perhaps this could be the issue? It's possible to turn off SSH multiplexing in .ssh/config --- but that doesn't seem to affect gerrit.


Gerrit's SSH options seem to be set in the config, but I couldn't see anything related to the "ControlMaster" setting which usually turns of multiplexing.


It looks like other people have hit a very similar issue using TeamCity:


http://stackoverflow.com/questions/38301318/channel-is-not-opened-errors-in-automatic-merge-on-teamcity-git

Is there any workaround on Gerrit currently?

Thanks in advance!

Tom

mick.b...@medialinksaustralia.com.au

unread,
Aug 24, 2016, 9:19:18 PM8/24/16
to Repo and Gerrit Discussion
Hi, we have a same issue. Are there any solutions for this?

Regards,
Mick

cla...@zzzcomputing.com

unread,
Oct 20, 2016, 2:52:11 AM10/20/16
to Repo and Gerrit Discussion
I'm having this problem as well and I can no longer replicate to Bitbucket.    the solution at https://answers.atlassian.com/questions/39449158/cant-repo-sync-anymore suggests setting "export GIT_SSH_COMMAND="ssh -o ControlPath=none"" however it's not clear if Gerrit makes use of this environment variable, setting this in /etc/default/gerritcodereview did not work for me.

Luca Milanesio

unread,
Oct 20, 2016, 4:43:48 AM10/20/16
to cla...@zzzcomputing.com, Repo and Gerrit Discussion
Why don't you just use Git/HTTPS for replication?
Relying on Git/SSH isn't a safe and scalable option anyway, even when it is working.

Luca.

On 19 Oct 2016, at 18:42, cla...@zzzcomputing.com wrote:

I'm having this problem as well and I can no longer replicate to Bitbucket.    the solution at https://answers.atlassian.com/questions/39449158/cant-repo-sync-anymore suggests setting "export GIT_SSH_COMMAND="ssh -o ControlPath=none"" however it's not clear if Gerrit makes use of this environment variable, setting this in /etc/default/gerritcodereview did not work for me.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

mike bayer

unread,
Oct 20, 2016, 8:27:11 AM10/20/16
to Luca Milanesio, Repo and Gerrit Discussion


On 10/20/2016 04:43 AM, Luca Milanesio wrote:
> Why don't you just use Git/HTTPS for replication?
> Relying on Git/SSH isn't a safe and scalable option anyway, even when it
> is working.

Well like others with the same problem I'm dropping in a shell script
(still using ssh to push though, works fine for my set of < 10 repos
that get less than a dozen commits / day and push model is much more
convenient than pull, if that's what you mean by using HTTPS, I'm not
about to put my atlassian password in plaintext on my server).

Also took the time to figure out that no, Jsch is a total rewrite of SSH
(write once, run anywhere, blah blah) so SSH_COMMAND is pointless and it
doesn't recognize the "ControlMaster" option inside of .ssh/config either.

However. We're all open source authors here, when our projects have
some feature that we tell users, "don't use that thing, it's buggy", do
we just leave it in to further confuse / mislead others? Or do we say,
"yeah, if it's broke, nobody uses it, let's deprecate it and document
that there's a better way". Why does gerrit have a replication feature
if it's not recommended ?





>
> Luca.
>
>> On 19 Oct 2016, at 18:42, cla...@zzzcomputing.com
>> <mailto:cla...@zzzcomputing.com> wrote:
>>
>> I'm having this problem as well and I can no longer replicate to
>> Bitbucket. the solution at
>> https://answers.atlassian.com/questions/39449158/cant-repo-sync-anymore suggests
>> setting "export GIT_SSH_COMMAND="ssh -o ControlPath=none"" however
>> it's not clear if Gerrit makes use of this environment variable,
>> setting this in /etc/default/gerritcodereview did not work for me.
>>
>>
>> --
>> --
>> To unsubscribe, email repo-discuss...@googlegroups.com
>> <mailto:repo-discuss...@googlegroups.com>
>> More info at http://groups.google.com/group/repo-discuss?hl=en
>>
>> ---
>> You received this message because you are subscribed to the Google
>> Groups "Repo and Gerrit Discussion" group.
>> To unsubscribe from this group and stop receiving emails from it, send
>> an email to repo-discuss...@googlegroups.com
>> <mailto:repo-discuss...@googlegroups.com>.

David Pursehouse

unread,
Oct 20, 2016, 8:40:34 AM10/20/16
to mike bayer, Luca Milanesio, Repo and Gerrit Discussion
On Thu, Oct 20, 2016 at 9:27 PM mike bayer <cla...@zzzcomputing.com> wrote:


On 10/20/2016 04:43 AM, Luca Milanesio wrote:
> Why don't you just use Git/HTTPS for replication?
> Relying on Git/SSH isn't a safe and scalable option anyway, even when it
> is working.

Well like others with the same problem I'm dropping in a shell script
(still using ssh to push though, works fine for my set of < 10 repos
that get less than a dozen commits / day and push model is much more
convenient than pull, if that's what you mean by using HTTPS, I'm not
about to put my atlassian password in plaintext on my server).

Also took the time to figure out that no, Jsch is a total rewrite of SSH
(write once, run anywhere, blah blah) so SSH_COMMAND is pointless and it
doesn't recognize the "ControlMaster" option inside of .ssh/config either.

However.   We're all open source authors here, when our projects have
some feature that we tell users, "don't use that thing, it's buggy", do
we just leave it in to further confuse / mislead others?   Or do we say,
"yeah, if it's broke, nobody uses it, let's deprecate it and document
that there's a better way".   Why does gerrit have a replication feature
if it's not recommended ?


I know it doesn't really help your situation, but actually the recommendation to use ssh for replication was removed recently:


 
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

mike bayer

unread,
Oct 20, 2016, 8:54:07 AM10/20/16
to David Pursehouse, Luca Milanesio, Repo and Gerrit Discussion
So can I just get an understanding of the replication engine, it only
does push over ssh or push to local filesystem, right? The
above-mentioned "https" recommendation basically means "roll your own",
is that right or is there some feature I'm not aware of?

If the replication engine is no longer recommended this is helpful,
sure. Means I need not be grumpy about rolling my own because everyone
else is too :)





>
>
>
>
>
>
>
> >
> > Luca.
> >
> >> On 19 Oct 2016, at 18:42, cla...@zzzcomputing.com
> <mailto:cla...@zzzcomputing.com>
> >> <mailto:cla...@zzzcomputing.com
> <mailto:cla...@zzzcomputing.com>> wrote:
> >>
> >> I'm having this problem as well and I can no longer replicate to
> >> Bitbucket. the solution at
> >>
> https://answers.atlassian.com/questions/39449158/cant-repo-sync-anymore
> suggests
> >> setting "export GIT_SSH_COMMAND="ssh -o ControlPath=none"" however
> >> it's not clear if Gerrit makes use of this environment variable,
> >> setting this in /etc/default/gerritcodereview did not work for me.
> >>
> >>
> >> --
> >> --
> >> To unsubscribe, email repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>
> >> <mailto:repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>>
> >> More info at http://groups.google.com/group/repo-discuss?hl=en
> >>
> >> ---
> >> You received this message because you are subscribed to the Google
> >> Groups "Repo and Gerrit Discussion" group.
> >> To unsubscribe from this group and stop receiving emails from it,
> send
> >> an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>
> >> <mailto:repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>>.
> >> For more options, visit https://groups.google.com/d/optout.
> >
>
> --
> --
> To unsubscribe, email repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>
> More info at http://groups.google.com/group/repo-discuss?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "Repo and Gerrit Discussion" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to repo-discuss...@googlegroups.com
> <mailto:repo-discuss%2Bunsu...@googlegroups.com>.

Luca Milanesio

unread,
Oct 20, 2016, 9:07:03 AM10/20/16
to mike bayer, Repo and Gerrit Discussion

> On 20 Oct 2016, at 13:27, mike bayer <cla...@zzzcomputing.com> wrote:
>
>
>
> On 10/20/2016 04:43 AM, Luca Milanesio wrote:
>> Why don't you just use Git/HTTPS for replication?
>> Relying on Git/SSH isn't a safe and scalable option anyway, even when it
>> is working.
>
> Well like others with the same problem I'm dropping in a shell script (still using ssh to push though, works fine for my set of < 10 repos that get less than a dozen commits / day and push model is much more convenient than pull, if that's what you mean by using HTTPS, I'm not about to put my atlassian password in plaintext on my server).

I believe Atlassian supports OAuth as well and you don't need to put any password in cleartext anywhere.
The fact that HTTPS is more scalable than SSH, well that is a fact :-)

But again, if you have a very small set of repos and commits, a script would just do the job, even with Git/SSH.

>
> Also took the time to figure out that no, Jsch is a total rewrite of SSH (write once, run anywhere, blah blah) so SSH_COMMAND is pointless and it doesn't recognize the "ControlMaster" option inside of .ssh/config either.

That's the reason why Git/SSH isn't recommended for Gerrit anymore. I would say as well that is "not the best rewrite of SSH" I've seen so far :-(

>
> However. We're all open source authors here, when our projects have some feature that we tell users, "don't use that thing, it's buggy", do we just leave it in to further confuse / mislead others?

Let me explain: Gerrit is OpenSource and JSch is OpenSource and as developers we don't just highlight the problem but we typically fix it.
In this case, the problem is in JSch and need to be sorted out at that level.

We do not deprecate it because:
- it may just work for smaller-scale setups, like yours
- It is needed anyway when creating new repos: how could you create a repo remotely over HTTPS?

> Or do we say, "yeah, if it's broke, nobody uses it, let's deprecate it and document that there's a better way". Why does gerrit have a replication feature if it's not recommended ?

Ad David pointed out, it is not recommended anymore. It is just simpler to configure, so it's a good way to start experimenting Gerrit replication.

Luca Milanesio

unread,
Oct 20, 2016, 9:08:03 AM10/20/16
to mike bayer, David Pursehouse, Repo and Gerrit Discussion
Nope, Git/HTTPS is managed through the Gerrit Replication plugin as well: just use a HTTPS URL for git pushes and set your credentials over .netrc or secure.config.

Luca.

mike bayer

unread,
Oct 20, 2016, 9:39:05 AM10/20/16
to Luca Milanesio, David Pursehouse, Repo and Gerrit Discussion


On 10/20/2016 09:07 AM, Luca Milanesio wrote:
>>
>> So can I just get an understanding of the replication engine, it only does push over ssh or push to local filesystem, right? The above-mentioned "https" recommendation basically means "roll your own", is that right or is there some feature I'm not aware of?
>>
>> If the replication engine is no longer recommended this is helpful, sure. Means I need not be grumpy about rolling my own because everyone else is too :)
>
> Nope, Git/HTTPS is managed through the Gerrit Replication plugin as well: just use a HTTPS URL for git pushes and set your credentials over .netrc or secure.config.

oh, oh now I was looking for that. the only docs I see are
https://gerrit.googlesource.com/plugins/replication/+doc/master/src/main/resources/Documentation/config.md
and
https://gerrit.googlesource.com/plugins/replication/+doc/master/src/main/resources/Documentation/about.md.
This paragraph:

Typically replication should be done over SSH, with a passwordless
public/private key pair. On a trusted network it is also possible to use
replication over the insecure (but much faster due to no authentication
overhead or encryption) git:// protocol, by enabling the receive-pack
service on the receiving system, but this configuration is not
recommended. It is also possible to specify a local path as replication
target. This makes e.g. sense if a network share is mounted to which the
repositories should be replicated.


as well as the lack of any examples w/ https (and considering the
transport has to be reinvented inside of the java libraries) strongly
indicated to me that there is no https option for this.

We can put https URLs into the replication config? When you say set
credientials in secure.config, how does that interact with bitbucket
Oauth ?

Luca Milanesio

unread,
Oct 20, 2016, 9:44:19 AM10/20/16
to mike bayer, David Pursehouse, Repo and Gerrit Discussion
I believe you are right, there is lack of documentation of Git/HTTPS usage.

It is "implicitly said" here:

The optional file $site_path/secure.config is a Git-style config file that provides secure values that should not be world-readable, such as passwords. Passwords for HTTP remotes can be obtained from this file.

remote.NAME.username : Username to use for HTTP authentication on this remote, if not given in the URL.

remote.NAME.password : Password to use for HTTP authentication on this remote.

From the fact that you have place where to put HTTP authentication info, gives you the clue that you could use Git/HTTPS :-)

With regards to documentation on how a Git/HTTPS URL look like, have a read on:

Hope this helps.

Luca.

cros...@gmail.com

unread,
Apr 21, 2017, 2:46:28 AM4/21/17
to Repo and Gerrit Discussion, cla...@zzzcomputing.com, david.pu...@gmail.com
Hi, I have the same issue when replicating over SSH. So I decided to investigate this problem, but unfortunately haven't found any solution even force disabling SSH multiplexing doesn't work. As a result I tried to configure it over HTTPS and it works. I have written a blog post about that, hope this helps someone. Here is the link https://crosp.net/blog/administration/devops/setting-gerrit-bitbucket-replication-https/

Luca.







  <mailto:...@zzzcomputing.com>> wrote:
  >>
  >> I'm having this problem as well and I can no longer replicate to
  >> Bitbucket.    the solution at
  >>
  https://answers.atlassian.com/questions/39449158/cant-repo-sync-anymore
  suggests
  >> setting "export GIT_SSH_COMMAND="ssh -o ControlPath=none"" however
  >> it's not clear if Gerrit makes use of this environment variable,
  >> setting this in /etc/default/gerritcodereview did not work for me.
  >>
  >>
  >> --
  >> --
  >> To unsubscribe, email repo-discuss...@googlegroups.com

  >> More info at http://groups.google.com/group/repo-discuss?hl=en
  >>
  >> ---
  >> You received this message because you are subscribed to the Google
  >> Groups "Repo and Gerrit Discussion" group.
  >> To unsubscribe from this group and stop receiving emails from it,
  send
  >> an email to repo-discuss...@googlegroups.com

  >> For more options, visit https://groups.google.com/d/optout.
  >

  --
  --
  To unsubscribe, email repo-discuss...@googlegroups.com

  More info at http://groups.google.com/group/repo-discuss?hl=en

  ---
  You received this message because you are subscribed to the Google
  Groups "Repo and Gerrit Discussion" group.
  To unsubscribe from this group and stop receiving emails from it,
  send an email to repo-discuss...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages