Jira (BOLT-939) Bolt from Homebrew throws errors on command run when using ed25519

14 views
Skip to first unread message

David Kramer (JIRA)

unread,
Nov 27, 2018, 12:43:02 PM11/27/18
to puppe...@googlegroups.com
David Kramer updated an issue
 
Puppet Task Runner / Bug BOLT-939
Bolt from Homebrew throws errors on command run when using ed25519
Change By: David Kramer
Summary: Bolt from Homebrew throws errors on command run when using ed25519
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)
Atlassian logo

David Kramer (JIRA)

unread,
Nov 27, 2018, 12:46:03 PM11/27/18
to puppe...@googlegroups.com

Cas Donoghue (JIRA)

unread,
Nov 27, 2018, 2:45:02 PM11/27/18
to puppe...@googlegroups.com

Cas Donoghue (JIRA)

unread,
Nov 27, 2018, 8:28:01 PM11/27/18
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-939
 
Re: Bolt from Homebrew throws errors on command run when using ed25519

I am trying to reproduce this on an available 10.13 image: 
1. Install puppet-bolt 1.1.0

2. Generate ed25519 key

ssh-keygen -o -a 100 -t ed25519 -f ~/.ssh/id_ed25519 -C "cas...@example.com"

3. Configure SSH on bolt controller

r49ndj4tcq21vsv:.ssh osx$ cat ~/.ssh/config 
Host *
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519
 
r49ndj4tcq21vsv:.ssh osx$ ssh-add -K ~/.ssh/id_ed25519

4. Push key to target node

r49ndj4tcq21vsv:.ssh osx$ ssh-copy-id ro...@yez2m1udxvdah4k.delivery.puppetlabs.net
 
root@yez2m1udxvdah4k:~# cat ~/.ssh/authorized_keys | grep cas...@example.com
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIbQ8a6PM8T0NXfSYLmMS1ITjCf214GlBtle5bqdzrzv cas...@example.com

4. Execute using new key for auth

r49ndj4tcq21vsv:.ssh osx$ bolt command run "uname -a" -n yez2m1udxvdah4k.delivery.puppetlabs.net
Started on yez2m1udxvdah4k.delivery.puppetlabs.net...
Finished on yez2m1udxvdah4k.delivery.puppetlabs.net:
  STDOUT:
    Linux yez2m1udxvdah4k 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Successful on 1 node: yez2m1udxvdah4k.delivery.puppetlabs.net
Ran on 1 node in 0.42 seconds

I am unable to reproduce on OSX 10.13. I am starting to dig in to what openssl library is loaded.

otool -L /opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bcrypt_pbkdf-1.0.0/lib/bcrypt_pbkdf_ext.bundle
/opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bcrypt_pbkdf-1.0.0/lib/bcrypt_pbkdf_ext.bundle:
	/opt/puppetlabs/bolt/lib/libruby.2.5.dylib (compatibility version 2.5.0, current version 2.5.1)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.0.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)

r49ndj4tcq21vsv:lib osx$ nm -gu libssl.dylib | grep SHA
_SHA1_Init
_SHA1_Transform
_SHA224_Init
_SHA256_Init
_SHA256_Transform
_SHA384_Init
_SHA512_Init
_SHA512_Transform

I am wondering if it is possible that we need to deal with another environment variable in https://github.com/puppetlabs/bolt-vanagon/blob/8fa10095f4eaf219883d02caa431d48f746879be/resources/files/posix/bolt_env_wrapper#L4

Nick Le Mouton (JIRA)

unread,
Nov 28, 2018, 5:15:02 PM11/28/18
to puppe...@googlegroups.com

I tried it with a new ed25519 key and it seemed to work as expected. My old key seems to be causing issues, but works fine for ssh/scp etc etc.

Michael Smith (JIRA)

unread,
Nov 28, 2018, 5:21:03 PM11/28/18
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-939

Are there differences in the format of your private keys? Different headers for example.

Nick Le Mouton (JIRA)

unread,
Nov 28, 2018, 5:45:03 PM11/28/18
to puppe...@googlegroups.com

Not sure how to see the headers of those keys, but if I run `ssh-keygen -lf <key>` the output of the two keys looks similar

Michael Smith (JIRA)

unread,
Nov 29, 2018, 12:27:02 PM11/29/18
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-939

I was wondering whether the file itself starts with

-----BEGIN OPENSSH PRIVATE KEY-----

or

-----BEGIN EC PRIVATE KEY-----

or something else.

Nick Le Mouton (JIRA)

unread,
Nov 29, 2018, 2:46:02 PM11/29/18
to puppe...@googlegroups.com

Both are:

```
----BEGIN OPENSSH PRIVATE KEY----
```

Michael Smith (JIRA)

unread,
Nov 30, 2018, 12:56:02 PM11/30/18
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-939

I'm not sure what's going on. Some things you could try that might give us more info are:

  • print any RUBY* environment variables
  • install Bolt into the system Ruby or your own Ruby, and additionally install ed25519 and bcrypt_pbkdf gems. Try using that.

A possible explanation relates to the use of inline - https://stackoverflow.com/questions/19068705/undefined-reference-when-calling-inline-function. We may be compiling with C99 semantics, where inline expects an external definition in a separate translation unit.

I'm not sure how openssh-keygen decides to use bcrypt. It may be the new keys we're generating don't.

Michael Smith (JIRA)

unread,
Nov 30, 2018, 1:03:02 PM11/30/18
to puppe...@googlegroups.com
Michael Smith commented on Bug BOLT-939

If you'd like to test out my inline theory, pull down https://github.com/MikaelSmith/bcrypt_pbkdf-ruby/tree/static-inline, do gem build bcrypt_pbkdf.gemspec, then sudo /opt/puppetlabs/bolt/bin/gem install bcrypt_pbkdf-1.0.0.gem.

Nick Lewis (JIRA)

unread,
Mar 6, 2019, 4:11:02 PM3/6/19
to puppe...@googlegroups.com
Nick Lewis commented on Bug BOLT-939

It appears that {{ssh-add}}ing the ssh key may work around this issue. We're still not clear what's actually going on here though...

Rosalind Benoit (JIRA)

unread,
May 21, 2019, 1:16:03 AM5/21/19
to puppe...@googlegroups.com

Hi, can I get troubleshooting assistance with an error that related to me? I'm working on a demo and am new to Bolt and plans.

When I run this plan: https://github.com/puppetlabs/brownbag-demo  on my OSX machine with target nodes hosted on Platform9, with a recently generated rsa keypair that is associated with the target hostnames in my ~/.ssh/config file, I get:

```

rosalind:brownbag-demo rosalind.benoit$ bolt plan run brownbag

Starting: plan brownbag

Starting: install puppet and gather facts on centos-7-large-0.forrester-fun.puppet.vm, centos-7-0.forrester-fun.puppet.vm, centos-7-1.forrester-fun.puppet.vm

dyld: lazy symbol binding failed: Symbol not found: _SHA512Init

  Referenced from: /opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bcrypt_pbkdf-1.0.0/lib/bcrypt_pbkdf_ext.bundle

  Expected in: flat namespace

 

dyld: Symbol not found: _SHA512Init

  Referenced from: /opt/puppetlabs/bolt/lib/ruby/gems/2.5.0/gems/bcrypt_pbkdf-1.0.0/lib/bcrypt_pbkdf_ext.bundle

  Expected in: flat namespace

 

/opt/puppetlabs/bin/bolt: line 4: 69955 Abort trap: 6           env -u GEM_HOME -u GEM_PATH -u DLN_LIBRARY_PATH -u RUBYLIB -u RUBYLIB_PREFIX -u RUBYOPT -u RUBYPATH -u RUBYSHELL -u LD_LIBRARY_PATH -u LD_PRELOAD SHELL=/bin/sh /opt/puppetlabs/bolt/bin/bolt "$@"

```

Michael Smith David Kramer Nick Le Mouton Nick Lewis  is there something I can do to just get the plan to...work? : ) I tried adding a password for the centos user and then using -u and -p flags but I'm getting the same result. lmk if there is a better contact for this issue. thanks!!

Cas Donoghue (JIRA)

unread,
May 21, 2019, 10:25:05 AM5/21/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-939

Rosalind Benoit Did you try adding the key to your ssh agent? ssh-add /path/to/key.

I am curious some of the other details.
1. Do you know the key type? ssh-keygen -l -f /path/to/key
2. Is the key password protected?
3. What version of macOS?
4. What version of Bolt?

Rosalind Benoit (JIRA)

unread,
May 21, 2019, 1:27:03 PM5/21/19
to puppe...@googlegroups.com

Cas Donoghue NO like an idiot I did not. Added it and it worked...thanks for that. Not that the plan itself worked but that is another realm to troubleshoot altogether. In case it's still helpful, here are my answers:

  1. It's RSA. Output: 2048 SHA256:Ez8Psu8ZHBY2mkcZ7/e61NDFyiYm92epT7HZB23HFoY rosalin...@rosalind.benoit-C02LJQ7SFD57 (RSA)
  2. Yes I'm using a passphrase on the key. I was wondering if that can create problems for Bolt?
  3. 10.14.4
  4. 1.18.0

Cas Donoghue (JIRA)

unread,
May 21, 2019, 1:33:04 PM5/21/19
to puppe...@googlegroups.com
Cas Donoghue commented on Bug BOLT-939

Unfortunately the only way to use a password protected key with bolt is by adding it to the ssh-agent. The error that is surfaced is horrible. Will see if we can figure out a better message to surface. Thanks so much for the details. Feel free ask questions in #bolt in either the internal puppet slack instance or the public one. 

Rosalind Benoit (JIRA)

unread,
May 21, 2019, 1:35:04 PM5/21/19
to puppe...@googlegroups.com

Cool, will do. Thanks so much for your quick response on this and for directing me on where to go in future!!

Reply all
Reply to author
Forward
0 new messages