Latest Choria version and Ubuntu Xenial (16.04)

Vladimir Stafievsky

unread,

Jan 26, 2021, 9:08:25 AM1/26/21

to Choria Users

Hi all!

I'm new with choria and making some tests on local vagrant setup:

- puppetmaster (Ubuntu 20) puppet.localdomain

- 3 puppet nodes with choria-broker installed (Ubuntu 16, cause a lot of servers in my current company run on this version) puppetnode-(01-03).localdomain

- 1 puppet node (ubuntu 20) puppetnode-04.localdomain

I faced with problems when try to run any mco commands ( mco service status puppet, for example ) :

mco service status puppet

* [ ============================================================> ] 5 / 5

puppetnode-02.localdomain: Choria MCollective Compatibility Layer failed to invoke the action: cannot load such file -- mcollective/connector/nats.rb

puppetnode-03.localdomain: Choria MCollective Compatibility Layer failed to invoke the action: cannot load such file -- mcollective/connector/nats.rb

puppetnode-01.localdomain: Choria MCollective Compatibility Layer failed to invoke the action: cannot load such file -- mcollective/connector/nats.rb

puppet.localdomain: stopped

puppetnode-04.localdomain: stopped

Summary of Service Status:

stopped = 2

Finished processing 5 / 5 hosts in 1071.10 ms

===================

From the last choria version there's no any lib inside /opt/puppetlabs/mcollective/plugins as I saw. I found that there's no choria-mcorpc-support gem installed by /opt/puppetlabs/puppet/bin/gem at Xenial nodes. I've tried to install it then, but nothing changed from mco side, same error about nats.dll. I've enabled debug logs at Xenial nodes and tried to run choria show_config. Here's the output:

=============================

root@puppetnode-01:/etc/choria# mco choria show_config

debug 2021/01/26 12:44:20: pluginmanager.rb:162:in `loadclass' Loading Mcollective::Facts::Yaml_facts from mcollective/facts/yaml_facts.rb

debug 2021/01/26 12:44:20: pluginmanager.rb:43:in `<<' Registering plugin facts_plugin with class MCollective::Facts::Yaml_facts single_instance: true

debug 2021/01/26 12:44:20: pluginmanager.rb:162:in `loadclass' Loading Mcollective::Connector::Nats from mcollective/connector/nats.rb

debug 2021/01/26 12:44:20: cache.rb:117:in `block in ttl' Cache miss on 'ddl' key 'connector/nats'

debug 2021/01/26 12:44:20: base.rb:100:in `block in findddlfile' Found nats ddl at /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/choria-mcorpc-support-2.23.3/lib/mcollective/connector/nats.ddl

debug 2021/01/26 12:44:20: pluginmanager.rb:43:in `<<' Registering plugin connector_plugin with class MCollective::Connector::Nats single_instance: true

debug 2021/01/26 12:44:20: pluginmanager.rb:162:in `loadclass' Loading Mcollective::Security::Choria from mcollective/security/choria.rb

debug 2021/01/26 12:44:20: pluginmanager.rb:43:in `<<' Registering plugin security_plugin with class MCollective::Security::Choria single_instance: true

info 2021/01/26 12:44:20: config.rb:151:in `loadconfig' The Marionette Collective version 2.23.3 started by /usr/bin/mco using config file /etc/choria/client.conf

debug 2021/01/26 12:44:20: pluginmanager.rb:162:in `loadclass' Loading MCollective::Application::Choria from mcollective/application/choria.rb

debug 2021/01/26 12:44:20: pluginmanager.rb:43:in `<<' Registering plugin choria_application with class MCollective::Application::Choria single_instance: true

debug 2021/01/26 12:44:20: pluginmanager.rb:79:in `[]' Returning new plugin choria_application with class MCollective::Application::Choria

debug 2021/01/26 12:44:20: pluginmanager.rb:162:in `loadclass' Loading MCollective::Util::Choria from mcollective/util/choria.rb

debug 2021/01/26 12:44:20: pluginmanager.rb:79:in `[]' Returning new plugin connector_plugin with class MCollective::Connector::Nats

info 2021/01/26 12:44:20: nats.rb:15:in `initialize' Choria NATS.io connector using pure ruby nats/io/client 0.6.2 with protocol version 1

Active Choria configuration:

The active configuration used in Choria comes from using Puppet AIO defaults, querying SRV

records and reading configuration files. The below information shows the completely resolved

configuration that will be used when running MCollective commands

MCollective related:

MCollective Version: 2.23.3

Choria Version: 0.19.0

Client Config File: /etc/choria/client.conf

Active Config File: /etc/choria/client.conf

Plugin Config Dir: /etc/choria/plugin.d

Using SRV Records: false

Federated: false

SRV Domain: localdomain

NATS NGS: false

Middleware Servers: puppetnode-01.localdomain:4222, puppetnode-02.localdomain:4222, puppetnode-03.localdomain:4222

info 2021/01/26 12:44:20: choria.rb:179:in `query_srv_records' Skipping SRV record queries due to choria.query_srv_records setting

Puppet related:

Puppet Server: puppet:8140

PuppetCA Server: puppet:8140

PuppetDB Server: puppet:8081

Discovery Proxy: not using a proxy

Facter Command: /opt/puppetlabs/bin/facter

Facter Domain: localdomain

Security setup:

debug 2021/01/26 12:44:20: pluginmanager.rb:79:in `[]' Returning new plugin security_plugin with class MCollective::Security::Choria

debug 2021/01/26 12:44:21: choria.rb:356:in `block in have_ssl_files?' Checking for SSL file /etc/puppetlabs/puppet/ssl/certs/puppetnode-01.localdomain.pem

debug 2021/01/26 12:44:21: choria.rb:356:in `block in have_ssl_files?' Checking for SSL file /etc/puppetlabs/puppet/ssl/private_keys/puppetnode-01.localdomain.pem

debug 2021/01/26 12:44:21: choria.rb:356:in `block in have_ssl_files?' Checking for SSL file /etc/puppetlabs/puppet/ssl/certs/ca.pem

debug 2021/01/26 12:44:21: choria.rb:401:in `valid_certificate?' Verified certificate /CN=puppetnode-01.localdomain against CA /CN=Puppet CA: puppet.localdomain

Valid SSL Setup: yes

Security Provider: puppet

Certname: puppetnode-01.localdomain

SSL Directory: /etc/puppetlabs/puppet/ssl (found)

Client Public Cert: /etc/puppetlabs/puppet/ssl/certs/puppetnode-01.localdomain.pem (found)

Client Private Key: /etc/puppetlabs/puppet/ssl/private_keys/puppetnode-01.localdomain.pem (found)

CA Path: /etc/puppetlabs/puppet/ssl/certs/ca.pem (found)

CSR Path: /etc/puppetlabs/puppet/ssl/certificate_requests/puppetnode-01.localdomain.pem (found)

Public Cert CN: puppetnode-01.localdomain (match)

Active Choria configuration settings as found in configuration files:

choria.middleware_hosts: puppetnode-01.localdomain:4222,puppetnode-02.localdomain:4222,puppetnode-03.localdomain:4222

choria.use_srv_records: false

debug 2021/01/26 12:44:21: pluginmanager.rb:82:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Nats

=====================================

As you can see, mco at Xenial node found nats.dll from the manually installed gem.

Why mco from the client node can't execute commands on that node then? :) I can collect any additional logs if needed.

Thank you for your answers in advance.

V.

R.I.Pienaar

unread,

Jan 26, 2021, 9:13:03 AM1/26/21

to choria-users

Hmm, can you ensure that the "mcollective" class is included? It should set all this up, and I know there was some cases where this didnt happen (fixed in next)

$ grep mcollective /opt/puppetlabs/puppet/cache/state/classes.txt

--
You received this message because you are subscribed to the Google Groups "Choria Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to choria-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/d0f8af16-ca70-47ea-92b2-566312272aa0n%40googlegroups.com.

--

R.I.Pienaar / www.devco.net / @ripienaar

Vladimir Stafievsky

unread,

Jan 26, 2021, 9:23:04 AM1/26/21

to Choria Users

Here's my common.yaml hiera config for all nodes:

---

classes:

- mcollective

choria::server_config:

plugin.choria.use_srv: false

plugin.choria.middleware_hosts: "puppetnode-01.localdomain:4222,puppetnode-02.localdomain:4222,puppetnode-03.localdomain:4222"

mcollective::plugin_classes:

- mcollective_agent_shell

mcollective_choria::config:

use_srv_records: false

middleware_hosts: "puppetnode-01.localdomain:4222,puppetnode-02.localdomain:4222,puppetnode-03.localdomain:4222"

#mcollective_choria::package_dependencies:

# "choria-mcorpc-support": "2.23.3"

mcollective::site_policies:

- action: "allow"

callers: "choria=vagrant.mcollective"

actions: "*"

facts: "*"

classes: "*"

Here's grep from xenial node:

root@puppetnode-01:/etc/puppetlabs/mcollective# grep mcollective /opt/puppetlabs/puppet/cache/state/classes.txt

mcollective

mcollective::plugin_dirs

mcollective::config

mcollective::facts

mcollective_agent_shell

mcollective_choria

mcollective_agent_puppet

mcollective_agent_service

mcollective_agent_package

mcollective_agent_filemgr

mcollective_util_actionpolicy

вторник, 26 января 2021 г. в 17:13:03 UTC+3, r...@devco.net:

R.I.Pienaar

unread,

Jan 26, 2021, 9:26:23 AM1/26/21

to choria-users

ok, the mcollective_choria installs it - if you are using Puppet All In One versions 6 or 7 (we do not support 5 anymore)

To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/8f45f6a2-4736-4e28-b299-e552e1af4a17n%40googlegroups.com.

Vladimir Stafievsky

unread,

Jan 26, 2021, 9:30:23 AM1/26/21

to choria...@googlegroups.com

At this setup I'm using 6.20

вт, 26 янв. 2021 г., 17:26 R.I.Pienaar <r...@devco.net>:

You received this message because you are subscribed to a topic in the Google Groups "Choria Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/choria-users/S2Nx0ibXIvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to choria-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/bfaa3976-82ce-4da7-a0a7-702cf544fd88%40www.fastmail.com.

R.I.Pienaar

unread,

Jan 26, 2021, 9:32:45 AM1/26/21

to choria-users

You're on 0.20.0 of the mcollective_choria module?

To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/CAADsC--MAQVfmvWLtkGf%3DU%2BAnjYqmeq5qKdk2wVn_d%3DKWPkcVw%40mail.gmail.com.

Vladimir Stafievsky

unread,

Jan 26, 2021, 1:43:17 PM1/26/21

to Choria Users

Yep,

etc/puppetlabs/code/environments/choria_tests/modules

├── choria-choria (v0.21.0)

├── choria-mcollective (v0.13.0)

├── choria-mcollective_agent_filemgr (v2.0.1)

├── choria-mcollective_agent_package (v5.3.0)

├── choria-mcollective_agent_puppet (v2.4.0)

├── choria-mcollective_agent_service (v4.0.1)

├── choria-mcollective_agent_shell (v1.0.4)

├── choria-mcollective_choria (v0.20.0)

├── choria-mcollective_util_actionpolicy (v3.2.0)

├── puppetlabs-apt (v7.7.0)

├── puppetlabs-stdlib (v6.5.0)

└── puppetlabs-translate (v2.2.0)

/etc/puppetlabs/code/environments/common (no modules installed)

/etc/puppetlabs/code/modules (no modules installed)

/opt/puppetlabs/puppet/modules (no modules installed)

/usr/share/puppet/modules (no modules installed)

вторник, 26 января 2021 г. в 17:32:45 UTC+3, r...@devco.net:

R.I.Pienaar

unread,

Jan 26, 2021, 2:53:47 PM1/26/21

to choria-users

$ grep mcorpc /opt/puppetlabs/puppet/cache/state/resources.txt ?

To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/bd4c6634-65ef-4acc-9560-dc0e4225a9dbn%40googlegroups.com.

Vladimir Stafievsky

unread,

Jan 27, 2021, 12:41:37 AM1/27/21

to Choria Users

root@puppetnode-03:/etc# grep mcorpc /opt/puppetlabs/puppet/cache/state/resources.txt

root@puppetnode-03:/etc# /opt/puppetlabs/puppet/bin/gem list

*** LOCAL GEMS ***

bigdecimal (1.2.4)

choria-mcorpc-support (2.23.3)

deep_merge (1.0.1)

facter (3.6.10)

fast_gettext (1.1.0)

gettext (3.2.2)

gettext-setup (0.30)

hiera (3.3.3)

hocon (1.2.5)

io-console (0.4.3)

json (1.8.1)

locale (2.1.2)

minitest (4.7.5)

nats-pure (0.6.2)

net-ssh (4.1.0)

psych (2.0.5)

puppet (4.10.12)

rake (10.1.0)

rdoc (4.1.0)

semantic_puppet (0.1.2)

stomp (1.3.3)

systemu (2.6.5)

test-unit (2.1.9.0)

text (1.3.1)

choria-mcorpc-support (2.23.3) installed manually here for tests, as I said in a first message

вторник, 26 января 2021 г. в 22:53:47 UTC+3, r...@devco.net:

R.I. Pienaar

unread,

Jan 27, 2021, 1:40:09 AM1/27/21

to choria...@googlegroups.com

Ok, I am not sure. As it’s a vagrant setup are you able to share the vagrant file and puppet code? Probably easiest for all our time if I just look at it that way

---

R.I.Pienaar

On 27 Jan 2021, at 06:41, Vladimir Stafievsky <just...@gmail.com> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/79dd2cd5-e748-4269-999a-6ffe3b1ea7a9n%40googlegroups.com.

Vladimir Stafievsky

unread,

Jan 27, 2021, 2:00:50 AM1/27/21

to Choria Users

Ok, thank you! I have some flights today, but, I think, I'll be able to share this test code with you by the end of this day. I'll send the link directly to your email

среда, 27 января 2021 г. в 09:40:09 UTC+3, r...@devco.net:

Vladimir Stafievsky

unread,

Jan 27, 2021, 3:07:42 PM1/27/21

to Choria Users

I've found the reason :) as usual, it's because of a stupid mistake. There's a wrong puppet version inside xenial vms.

if it's possible, just delete this topic, please

среда, 27 января 2021 г. в 10:00:50 UTC+3, Vladimir Stafievsky:

R.I. Pienaar

unread,

Jan 27, 2021, 5:02:26 PM1/27/21

to choria...@googlegroups.com

That’s great. Glad you found it :)

---

R.I.Pienaar

On 27 Jan 2021, at 21:07, Vladimir Stafievsky <just...@gmail.com> wrote:

I've found the reason :) as usual, it's because of a stupid mistake. There's a wrong puppet version inside xenial vms.

To view this discussion on the web visit https://groups.google.com/d/msgid/choria-users/ab7737fe-b97e-492c-a531-1b686f3c2b5dn%40googlegroups.com.

Reply all

Reply to author

Forward