[JIRA] (JENKINS-62195) ec2-1.50.2 doesn't work with SSH <7.5

[multani (JIRA)]

Jonathan Ballet created an issue

Jenkins / JENKINS-62195 ec2-1.50.2 doesn't work with SSH <7.5 Issue Type: Bug Assignee: FABRIZIO MANFREDI Components: ec2-plugin Created: 2020-05-07 06:12 Priority: Critical Reporter: Jonathan Ballet Version 1.50.2 introduces security mitigations by proposing new options for SSH. 2 of the 3 options have been introduced by [SSH version 7.6](https://www.openssh.com/txt/release-7.6): ssh(1): expand the StrictHostKeyChecking option with two new settings. The first "accept-new" will automatically accept hitherto-unseen keys but will refuse connections for changed or invalid hostkeys. This is a safer subset of the current behaviour of StrictHostKeyChecking=no. The second setting "off", is a synonym for the current behaviour of StrictHostKeyChecking=no: accept new host keys, and continue connection for hosts with incorrect hostkeys. A future release will change the meaning of StrictHostKeyChecking=no to the behaviour of "accept-new". bz#2400 Although it was released almost 3 years ago, this seriously breaks compatibility with non-recent Jenkins installations. For instance, the current default Docker image for Jenkins is currently based off Debian Stretch which provides SSH 7.4 and doesn't support these new options: {{ $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=off command-line line 0: unsupported option "off". $ docker run --rm -ti jenkins/jenkins:2.235 ssh -o StrictHostKeyChecking=accept-new command-line line 0: unsupported option "accept-new". $ docker run --rm -ti jenkins/jenkins:lts ssh -o StrictHostKeyChecking=accept-new command-line line 0: unsupported option "accept-new". }} Add Comment

This message was sent by Atlassian Jira (v7.13.12#713012-sha1:6e07c38)

[dtroup@hotmail.com (JIRA)]

David Troup commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5 You can change the strategy in the config   Host Key Verification Strategy in cloud config

Add Comment

[dtroup@hotmail.com (JIRA)]

David Troup updated an issue

Jenkins / JENKINS-62195

ec2-1.50.2 doesn't work with SSH <7.5

Change By: David Troup Comment:

You can change the strategy in the config   Host Key Verification Strategy in cloud config

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5 Oleg Nenashev FYA (platform SIG)

Add Comment

[john@jeffers.cc (JIRA)]

John Jeffers commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Confirmed, happening here as well. We are using the latest LTS image, jenkins/jenkins:2.222.3 root@jenkins-master-fb7584fbb-s6nnl:/# ssh -V OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u 20 Dec 2019 Also worth noting that when I attempted to downgrade the plugin, it did not downgrade properly and instead seemed to uninstall the plugin, taking all of its config with it. I had to manually downgrade and restore config.xml from a backup. I believe this has something to do with the ec2.xml file it drops into $JENKINS_HOME, because I could not get 1.50.1 working again until I removed that file.

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev assigned an issue to Unassigned

Jenkins / JENKINS-62195

ec2-1.50.2 doesn't work with SSH <7.5

Change By: Oleg Nenashev Assignee: FABRIZIO MANFREDI

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev updated an issue

Jenkins / JENKINS-62195 ec2-1.50.2 doesn't work with SSH <7.5

Change By: Oleg Nenashev Labels: regression

Add Comment

[o.v.nenashev@gmail.com (JIRA)]

Oleg Nenashev commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5 We also hit the issues after upgrading the plugin on ci.jenkins.io which currently uses the plugin to provision agents in AWS. https://groups.google.com/forum/#!topic/jenkinsci-dev/2_WmJWSjtuc for a general discussion about agents stability, CC Mark Waite. Daniel Beck FYI this plugin is not really within the scope of the platform SIG. I am working to get the issue reviewed by the maintainers, but it is unlikely to happen immediately due to bank holidays, etc.

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon assigned an issue to Ramon Leon

Jenkins / JENKINS-62195

ec2-1.50.2 doesn't work with SSH <7.5

Change By: Ramon Leon Assignee: Ramon Leon

Add Comment

[rcampbell@cloudbees.com (JIRA)]

Ryan Campbell commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5 Noting in case it isn't clear, that a valid workaround is to update the ssh client to a more recent version which supports these more secure options.

Add Comment

[dbeck@cloudbees.com (JIRA)]

Daniel Beck commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Oleg Nenashev As this problem seems to occur because of very outdated base images, it's reasonable to inform the SIG about the consequences of that.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Yes, Debian Stretch is the current Debian "oldstable" release and is delivering OpenSSH 7.4p1. It is a distribution which the Debian project continues to patch and will continue to patch until the release of Debian next-stable ("Bullseye"). No release date has been set for Bullseye. After Bullseye releases, Debian Stretch will stop receiving patches. Red Hat 7 and CentOS 7 are also still actively being patched by their maintainers and are delivering OpenSSH 7.4p1. Red Hat Enterprise Linux 7 is slated to be supported through 2024.

Add Comment

[mark.earl.waite@gmail.com (JIRA)]

Mark Waite edited a comment on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Yes, Debian Stretch is the current Debian "[oldstable|https://wiki.debian.org/DebianOldStable]" release and is delivering OpenSSH 7.4p1.  It is a distribution which the Debian project continues to patch and will continue to patch until the release of [Debian next-stable|https://wiki.debian.org/DebianReleases] ("Bullseye").  No release date has been set for Bullseye.  After Bullseye releases, Debian Stretch will stop receiving patches. [~danielbeck] is correct that we'll need to update the Docker base images.  That seems like a good topic for the Platform SIG and a good place to reach consensus on labeling patterns and practices. Red Hat 7 and CentOS 7 are also still actively being patched by their maintainers and are delivering OpenSSH 7.4p1.  Red Hat Enterprise Linux 7 is slated to be [supported through 2024|https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#Product_life_cycle].

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Another workaround is to set the strategy to Check New Hard which will set the option to yes which is supported by all versions. This strategy requires the key is added to the known_hosts file Another workaround is to use avoid using the ssh command and the plugin will use a pure-java ssh command to do that.

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon edited a comment on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Another workaround is to set the strategy to Check New Hard which will set the option to yes which is supported by all versions. This strategy requires the key is added to the known_hosts file

Another workaround is to avoid using the ssh command and the plugin will use a pure-java ssh command client to do that the connection .

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon edited a comment on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Another workaround is to set the strategy to Check New Hard which will set the option to yes which is supported by all versions. This strategy requires the key is added to the known_hosts file

Another workaround is to use avoid using the ssh command and the plugin will use a pure-java ssh command to do that.

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

A PR to use the no option instead of off which still these days is a synonym, although it was advertised they may differ in the future (from 2017): https://github.com/jenkinsci/ec2-plugin/pull/460

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon updated JENKINS-62195

Jenkins / JENKINS-62195

ec2-1.50.2 doesn't work with SSH <7.5

Change By: Ramon Leon Status: In Progress Review

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon started work on JENKINS-62195

Change By: Ramon Leon Status: Open In Progress

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5 Another PR with the documentation updated: https://github.com/jenkinsci/ec2-plugin/pull/455 , best to use this one.

Add Comment

[manuelramonleonjimenez@gmail.com (JIRA)]

Ramon Leon commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

John Jeffers regarding the downgrade: https://issues.jenkins-ci.org/browse/JENKINS-62231

Add Comment

[john@jeffers.cc (JIRA)]

John Jeffers commented on JENKINS-62195

Re: ec2-1.50.2 doesn't work with SSH <7.5

Thank you. I was lucky that I had a backup to recover from. Would have been very bad otherwise.

Add Comment