Getting SSL error connecting to NATS

526 views
Skip to first unread message

Ishtiyak Siddiqui

unread,
Oct 21, 2019, 12:40:44 PM10/21/19
to Choria Users
Hi,

I have successfully installed Choria and it was working until we had to update puppet certificate.
After puppet certificate update, Choria is giving SSL error:
---------------------------------------------------------------------------------------------
mco ping
error 2019/10/21 12:15:54: natswrapper.rb:145:in `block in start' Error in NATS connection: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
error 2019/10/21 12:15:56: natswrapper.rb:145:in `block in start' Error in NATS connection: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
error 2019/10/21 12:15:56: client.rb:39:in `rescue in initialize' Timeout occured while trying to connect to middleware

The ping application failed to run, use -v for full error backtrace details: execution expired
warn 2019/10/21 12:15:56: natswrapper.rb:138:in `block in start' Disconnected from NATS: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
--------------------------------------------------------------------------------------------

My understanding is that Choria uses puppet certificates, looks like NATs certificates are not same as puppet cert anymore after the puppet cert update.
Wondering where do I need to update this cert to make choria working again?

I have looked at "mco choria show_config", it says SSL setup is valid.
----------------------------------------------------------------------------------------------------------
 mco choria show_config
Active Choria configuration:

The active configuration used in Choria comes from using Puppet AIO defaults, querying SRV
records and reading configuration files.  The below information shows the completely resolved
configuration that will be used when running MCollective commands

MCollective related:

    MCollective Version: 2.12.4
         Choria Version: 0.16.1
     Client Config File: /etc/puppetlabs/mcollective/client.cfg
     Active Config File: /etc/puppetlabs/mcollective/client.cfg
      Plugin Config Dir: /etc/puppetlabs/mcollective/plugin.d
      Using SRV Records: true
              Federated: false
             SRV Domain: <removed>
     Middleware Servers: puppet:4222

Puppet related:

       Puppet Server: puppet:8140
     PuppetCA Server: puppet:8140
     PuppetDB Server: puppet:8081
     Discovery Proxy: not using a proxy
      Facter Command: /opt/puppetlabs/bin/facter
       Facter Domain: <removed>

SSL setup:

     Valid SSL Setup: yes
   Security Provider: puppet
            Certname: appadmin.mcollective
       SSL Directory: /home/appadmin/.puppetlabs/etc/puppet/ssl (found)
  Client Public Cert: /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/appadmin.mcollective.pem (found)
  Client Private Key: /home/appadmin/.puppetlabs/etc/puppet/ssl/private_keys/appadmin.mcollective.pem (found)
             CA Path: /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/ca.pem (found)
            CSR Path: /home/appadmin/.puppetlabs/etc/puppet/ssl/certificate_requests/appadmin.mcollective.pem (found)
      Public Cert CN: appadmin.mcollective (match)

Active Choria configuration settings as found in configuration files:

  No custom Choria settings found in your configuration files
--------------------------------------------------------------------------------------------------------------------------------------------------------------

Vincent Janelle

unread,
Oct 21, 2019, 1:07:35 PM10/21/19
to choria...@googlegroups.com
Is the certificate on the choria server valid?

--
You received this message because you are subscribed to the Google Groups "Choria Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/ac2f046e-5a6e-4bd0-9922-54676636bb4b%40googlegroups.com.

Ishtiyak Siddiqui

unread,
Oct 21, 2019, 1:59:59 PM10/21/19
to Choria Users
"mco choris show_config" shows SSL setup as valid.
How do we troubleshoot further? What else to check and where?

Choria agent is installed on Puppet server and lot of puppet agents are already connected....I can't touch Puppet puppet cert.
Have to find out where in Chroia or NATS, certs are mismatched and fix that.
To unsubscribe from this group and stop receiving emails from it, send an email to choria...@googlegroups.com.

Vincent Janelle

unread,
Oct 21, 2019, 3:37:17 PM10/21/19
to choria...@googlegroups.com
It would be the certificates on the choria brokers that you would need to investigate - the mco choria show_config command would not necessarily give you the right results.

You can also try restarting the choria-broker process and seeing if that helps.

To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/2d925e5d-8091-4df4-bdb6-c51541fa63c9%40googlegroups.com.

Ishtiyak Siddiqui

unread,
Oct 21, 2019, 3:58:00 PM10/21/19
to Choria Users
I have tried restarting several times, it didn't help!
I very well understand that issue is at chroa broker certificate because it doesn't match with puppet cert anymore.
Question is: how do we fix it?

Let's say tomorrow my puppet cert expires and I need to renew my puppet cert, how do I update that new puppet cert to Choria Broker for it to continue working?
Is there any documentation how to update broker cert?

R.I.Pienaar

unread,
Oct 22, 2019, 4:54:58 AM10/22/19
to choria-users
hello,

show_config tests the local validity, it does not attempt to make remote connections to very those for you.

In what way does the choria broker cert not match the Puppet one? Show config says its trying to connect to `puppet:4222` which if things are setup correctly would mean its the puppet server, that would have a name `puppet` by default - and if you renamed that, you also have to tell choria via configuration what new name you gave it to.

Should the certs expire - and given that it appears the broker is on the same node - you would just need to restart it to pick up the new cert.  By default it uses the exact same file as puppet does for certs so they would match.
To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.

--
R.I.Pienaar / www.devco.net / @ripienaar

Ishtiyak

unread,
Oct 22, 2019, 7:08:13 AM10/22/19
to choria...@googlegroups.com
Thanks for clarification that NATS broker is using exactly same certs as puppet cert from same location.
Yes, Nats broker is installed on same server as puppet and host entry (/etc/hosts) has been added to add “puppet” as hostname.

I have tried restarting the broker as well, still I have same SSL error, anything else that I should be checking? Any other tricks to run some troubleshooting or validation to find where is the problem?

Thanks,
Istiyak

On Oct 22, 2019, at 4:54 AM, R.I.Pienaar <r...@devco.net> wrote:



R.I.Pienaar

unread,
Oct 22, 2019, 7:55:24 AM10/22/19
to choria-users
Unfortunately because NATS have their weird protocol elevation thing its a bit hard, easiest to figure out what your NATS broker thinks is this:

Ensure you have plugin.choria.network.peer_port = 5222 in your broker config and restarted with that there, then:

$ openssl s_client \
     -CApath /home/rip/.puppetlabs/etc/puppet/ssl/certs/ca.pem \
     -cert /home/rip/.puppetlabs/etc/puppet/ssl/certs/rip.mcollective.pem \
     -key /home/rip/.puppetlabs/etc/puppet/ssl/private_keys/rip.mcollective.pem \
     -connect c1.example.net:5222

this will connect to the cluster port of the broker that speaks native TLS and will use your certs and so forth (obtained from show_config) to try and verify the TLS.  It'll print errors etc on fail, if it works your last line of output will look similar to:

INFO {"server_id":"NBX2HOILKUOXP4J5BUUUBBDHRUBPWKB7VB6OENQPAXXXE4BFSN2UL3CA","version":"2.0.4","proto":2,"go":"go1.13","host":"::","port":5222,"tls_required":true,"tls_verify":true,
"max_payload":1048576,"nonce":"tmfY3M5LLDZY9vM"}

If not, we'll have more information to maybe try figure it out

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 4:38:50 PM10/22/19
to Choria Users
Here is the output (last few lines):
-------------------------------------------------------------------------------------------
    Start Time: 1571776300
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
INFO {"server_id":"9tzCsm25fQFUMSJT7Gsu8K","version":"1.4.1","proto":1,"go":"go1.10.7","host":"server.domain.com (server name changed)","port":4223,"tls_required":true,"tls_verify":true,"max_payload":1048576} 
--------------------------------------------------------------------------------------------

first few lines of output:
---------------------------------------------------------------------------------------------
CONNECTED(00000003)
depth=0 CN = server.domain.com (server name changed)
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.domain.com (server name changed)
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=server.domain.com (server name changed)
   i:/CN=Puppet CA: server.domain.com (server name changed)
---
Server certificate
-----BEGIN CERTIFICATE-----
-----------------------------------------------------------------------------------------

Thanks,

--
R.I.Pienaar / www.devco.net / @ripienaar


--
You received this message because you are subscribed to the Google Groups "Choria Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to choria...@googlegroups.com.


--
You received this message because you are subscribed to the Google Groups "Choria Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to choria...@googlegroups.com.

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 4:39:40 PM10/22/19
to Choria Users
Command executed was:

openssl s_client \
>      -CApath /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/ca.pem \
>      -cert /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/appadmin.mcollective.pem \
>      -key /home/appadmin/.puppetlabs/etc/puppet/ssl/private_keys/appadmin.mcollective.pem \
>      -connect puppet:4223

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 4:50:48 PM10/22/19
to Choria Users
When I look inside the certs, I see:

/home/appadmin/.puppetlabs/etc/puppet/ssl/certs/ca.pem
Owner: CN=Puppet CA: server.domain.com (server name changed)
Issuer: CN=Puppet CA: server.domain.com (server name changed)

/home/appadmin/.puppetlabs/etc/puppet/ssl/certs/appadmin.mcollective.pem
Owner: CN=appadmin.mcollective
Issuer: CN=Puppet CA: server.domain.com (server name changed)

/home/appadmin/.puppetlabs/etc/puppet/ssl/private_keys
keytool -printcert -file appadmin.mcollective.pem 
keytool error: java.lang.Exception: Failed to parse input



R.I.Pienaar

unread,
Oct 22, 2019, 4:51:42 PM10/22/19
to choria-users
So I guess if you do -connect server.domain.com:5222 it would work? Can you perform the same test against 8140 (puppet port)
To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 5:01:04 PM10/22/19
to Choria Users
openssl s_client \
>      -CApath /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/ca.pem \
>      -cert /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/appadmin.mcollective.pem \
>      -key /home/appadmin/.puppetlabs/etc/puppet/ssl/private_keys/appadmin.mcollective.pem \
>      -connect puppet:8140
CONNECTED(00000003)
depth=0 CN = server.domain.com 
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = server.domain.com 
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=server.domain.com 
   i:/CN=Puppet CA: server.domain.com 
---
Server certificate
-----BEGIN CERTIFICATE-----
Content removed
-----END CERTIFICATE-----
subject=/CN=server.domain.com 
issuer=/CN=Puppet CA: server.domain.com 
---
Acceptable client certificate CA names
/CN=Puppet CA: server.domain.com 
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2277 bytes and written 2410 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA256
    Session-ID: 5DAF6D5833B7F8EE3E84D560E91A2416FE5D30980972EA774EC9E06A3032B01A
    Session-ID-ctx: 
    Master-Key: 9788467F1FBFAA5E5D147E0ADE3E1F770C93A4FF6E878D46138A8AB8D60CF8ECD90D4123A8967694A510BCFFC5B14AD5
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1571777880
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
closed


Thanks!

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 5:04:26 PM10/22/19
to Choria Users
openssl s_client \
     -CApath /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/ca.pem \
     -cert /home/appadmin/.puppetlabs/etc/puppet/ssl/certs/appadmin.mcollective.pem \
     -key /home/appadmin/.puppetlabs/etc/puppet/ssl/private_keys/appadmin.mcollective.pem \
     -connect server.domain.com:4223

Above command returns exactly same result as puppet:4223.

Ishtiyak Siddiqui

unread,
Oct 22, 2019, 5:43:47 PM10/22/19
to Choria Users
so, if certs are good and we can connect using these commands, why Choria is giving SSL error?
I see error when I run "mco ping"

R.I. Pienaar

unread,
Oct 23, 2019, 2:33:24 PM10/23/19
to choria...@googlegroups.com
Because choria is trying to connect to “puppet” and you carts do not have that as a valid name in them. 

Configure choria to connect to the name in the cert. by default puppet server certs have puppet as an alias, seems yours don’t. 

---
R.I.Pienaar

On 22 Oct 2019, at 23:43, Ishtiyak Siddiqui <ishtiyak...@gmail.com> wrote:


To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/c61fdf90-8a6d-4eb6-ad72-c72b90550453%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages