Jira (BOLT-920) Bolt unable to connect with ecdsa key

6 views
Skip to first unread message

Michael Smith (JIRA)

unread,
Oct 10, 2018, 5:40:04 PM10/10/18
to puppe...@googlegroups.com
Michael Smith created an issue
 
Puppet Task Runner / Bug BOLT-920
Bolt unable to connect with ecdsa key
Issue Type: Bug Bug
Affects Versions: BOLT 1.0.0
Assignee: Unassigned
Created: 2018/10/10 2:39 PM
Priority: Normal Normal
Reporter: Michael Smith

Using a ecdsa-sha2-nistp256 key generated by ssh-keygen -t ecdsa, Bolt quietly fails to connect. See https://github.com/puppetlabs/bolt/issues/700 for original issue and debugging.

This doesn't appear to be resolved yet in net-ssh#master, or by including ed25519 and bcrypt_pbkdf (with net-ssh 5.0.2).

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

Michael Smith (JIRA)

unread,
Oct 11, 2018, 12:46:03 PM10/11/18
to puppe...@googlegroups.com
Michael Smith updated an issue
Change By: Michael Smith
Sprint: Bolt Ready for Grooming

Michael Smith (JIRA)

unread,
Oct 11, 2018, 2:42:03 PM10/11/18
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

SSH 7.8 switched ssh-keygen to outputting keys in the OpenSSH format (rather than OpenSSL's PEM format). This breaks net-ssh for everything except ed25519 keys due to https://github.com/net-ssh/net-ssh/blob/v5.0.2/lib/net/ssh/key_factory.rb#L112. Issue at https://github.com/net-ssh/net-ssh/issues/633.

Michael Smith (JIRA)

unread,
Dec 4, 2018, 5:06:02 PM12/4/18
to puppe...@googlegroups.com
Michael Smith updated an issue
Change By: Michael Smith
Sprint: Bolt Ready for Grooming

Duncan X Simpson (JIRA)

unread,
Jan 21, 2019, 6:22:02 PM1/21/19
to puppe...@googlegroups.com

Cas Donoghue (JIRA)

unread,
Jan 22, 2019, 12:15:04 PM1/22/19
to puppe...@googlegroups.com

Michael Smith (JIRA)

unread,
Jan 22, 2019, 12:15:04 PM1/22/19
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

Looks like we can address this with net-ssh 5.1.0.

Cas Donoghue (JIRA)

unread,
Feb 1, 2019, 6:00:04 PM2/1/19
to puppe...@googlegroups.com

Cas Donoghue (JIRA)

unread,
Feb 1, 2019, 6:01:05 PM2/1/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

Verified ecdsa key works with 5.1.0

Cas Donoghue (JIRA)

unread,
Feb 6, 2019, 10:25:04 AM2/6/19
to puppe...@googlegroups.com
Cas Donoghue updated an issue
Change By: Cas Donoghue
Fix Version/s: BOLT Next

Cas Donoghue (JIRA)

unread,
Feb 6, 2019, 10:30:04 AM2/6/19
to puppe...@googlegroups.com
Cas Donoghue updated an issue
Change By: Cas Donoghue
Release Notes Summary: Bolt can now use an {{ecdsa}} key to authenticate {{ssh}} connections.
Release Notes: Bug Fix

Cas Donoghue (JIRA)

unread,
Feb 6, 2019, 12:06:03 PM2/6/19
to puppe...@googlegroups.com
Cas Donoghue updated an issue
Change By: Cas Donoghue
Release Notes Summary: Bolt can now use keys in an {{ecdsa}} key updated OpenSSH format to authenticate {{ssh}} connections.

Melissa Amos (JIRA)

unread,
Feb 7, 2019, 12:04:03 PM2/7/19
to puppe...@googlegroups.com
Melissa Amos commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

To clarify, the known issue documented in pre-docs for this ticket is being FIXED in 1.11.0?

Cas Donoghue (JIRA)

unread,
Feb 7, 2019, 2:20:13 PM2/7/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-920

When I tested this originally I did not have my test case quite right. I went to do a final verification on a clean install and found that net-ssh 5.1.0 does not resolve the issue.

Replication:

[root@ebo9k1qf1fpcs67 gems]# ssh -V
OpenSSH_7.8p1, OpenSSL 1.1.1 FIPS  11 Sep 2018
[root@ebo9k1qf1fpcs67 ~]# ssh-keygen -t ecdsa
[root@ebo9k1qf1fpcs67 ~]# ssh -i id_ecdsa root@localhost
[root@ebo9k1qf1fpcs67 Boltdir]# bolt command run whoami -n ecdsa
Started on localhost...
Failed on localhost:
  Failed to connect to localhost: the given identity is known, but the private key could not be loaded: Net::SSH::Exception (Cannot decode private key of type ecdsa-sha2-nistp256)
Failed on 1 node: localhost
Ran on 1 node in 0.16 seconds

inventory.yaml

nodes:
  - name: localhost
    alias: ecdsa
    config:
      transport: ssh
      ssh:
        host-key-check: false
        private-key: /root/id_ecdsa
        user: root

Cas Donoghue (JIRA)

unread,
Feb 7, 2019, 3:25:03 PM2/7/19
to puppe...@googlegroups.com
Cas Donoghue updated an issue
Change By: Cas Donoghue
Fix Version/s: BOLT Next

Melissa Amos (JIRA)

unread,
Feb 7, 2019, 5:07:03 PM2/7/19
to puppe...@googlegroups.com
Melissa Amos commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

Here's the known issue. I tweaked the wording on the workaround (last paragraph) so please check my understanding.

http://docs-internal.puppet.com/docs/bolt/dev/bolt_known_issues.html#ssh-keys-generated-with-ssh-keygen-from-openssh-7-8-fail

Melissa Amos (JIRA)

unread,
Feb 7, 2019, 5:10:03 PM2/7/19
to puppe...@googlegroups.com
Melissa Amos updated an issue
 
Change By: Melissa Amos
Labels: known-issue-added

Cas Donoghue (JIRA)

unread,
Feb 8, 2019, 5:21:02 PM2/8/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-920
 
Re: Bolt unable to connect with ecdsa key

Update:

The title of this ticket indicates that ecdsa keys do not work with bolt. If the ecdsa key is generated with a version of Openssh that is pre 7.8 (or with the non-openssh format) then the key is compatible with the version of net-ssh (5.0.2) which shipped with bolt (pre 1.11.0).

As noted in the comments the root cause is the Openssh formatted key. So keys (including keypairs that use the popular rsa algorithm) that use the Openssh format were incompatible with net-ssh (pre 5.0.2) and therefor with bolt.

The bolt 1.11.0 release updated to the latest net-ssh version (5.1.0) which added support for rsa keys with the openssh format. However ecdsa keys with the Openssh format are still incompatible with the latest version (see comment above showing Net::SSH::Exception). So with bolt 1.11.0 an rsa key formatted in the Openssh format will work where previously it did not but ecdsa keys with Openssh format are still incompatible.

Previously an Openssh formatted rsa key with bolt < 1.11.0 would produce the following error:

[root@ebo9k1qf1fpcs67 bolt]# bolt command run whoami -n rsa-openssh
Started on jlmfz1sost45dup.delivery.puppetlabs.net...
Failed on jlmfz1sost45dup.delivery.puppetlabs.net:
  Failed to connect to jlmfz1sost45dup.delivery.puppetlabs.net: expected 64-byte String, got 3
Failed on 1 node: jlmfz1sost45dup.delivery.puppetlabs.net
Ran on 1 node in 0.19 seconds

Ecdsa keys stored in the Openssh format still produce the following error:

[root@ebo9k1qf1fpcs67 Boltdir]# bolt command run whoami -n ecdsa-openssh
Started on localhost...
Failed on localhost:
  Failed to connect to localhost: the given identity is known, but the private key could not be loaded: Net::SSH::Exception (Cannot decode private key of type ecdsa-sha2-nistp256)
Failed on 1 node: localhost
Ran on 1 node in 0.16 seconds

Cas Donoghue (JIRA)

unread,
Feb 11, 2019, 3:58:03 PM2/11/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-920

An issue was filed against the net-ssh project: https://github.com/net-ssh/net-ssh/issues/657

A ticket describing compatibility with rsa keys in the OpenSSH format was created in order to track that work separately. 

David Kramer (JIRA)

unread,
Feb 13, 2019, 12:41:04 PM2/13/19
to puppe...@googlegroups.com
David Kramer assigned an issue to Unassigned
 
Change By: David Kramer
Assignee: Cas Donoghue

David Kramer (JIRA)

unread,
Feb 13, 2019, 12:42:04 PM2/13/19
to puppe...@googlegroups.com
David Kramer updated an issue
Change By: David Kramer
Sprint: Bolt Kanban Ready for Grooming

David Kramer (JIRA)

unread,
Apr 16, 2019, 12:40:03 PM4/16/19
to puppe...@googlegroups.com
David Kramer updated an issue
Change By: David Kramer
Sprint: Bolt Ready for Grooming

Yasmin Rajabi (JIRA)

unread,
Aug 15, 2019, 6:49:03 PM8/15/19
to puppe...@googlegroups.com
Yasmin Rajabi updated an issue
Change By: Yasmin Rajabi
Labels: ghm known-issue-added

Cas Donoghue (JIRA)

unread,
Aug 20, 2019, 10:47:03 AM8/20/19
to puppe...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages