Bootstrap Issues - Method 'enable_cfengine_agents' failed in some repairs - cfengine-community 3.6.4

[Jean-Samuel Christophe]

Hi List,

I'm testing cfengine-community 3.6.4 on a linux-vserver (base debian wheezy) setup where host is hub and vserver are clients.

It get the following error on bootstrap and can't seem to nail it down.

2015-02-26T18:02:34+0000    error: /default/cfe_internal_update_processes/methods/'ENABLING CFE AGENTS'/default/enable_cfengine_agents/commands/'"/var/cfengine/bin/cf-serverd"'[0]: Command related to promiser '"/var/cfengine/bin/cf-serverd"' returned code defined as promise failed 1

2015-02-26T18:02:34+0000   notice: /default/cfe_internal_update_processes/methods/'ENABLING CFE AGENTS'/default/enable_cfengine_agents/commands/'"/var/cfengine/bin/cf-serverd"'[0]: Q: "...f-serverd"": 2015-02-26T18:02:34+0000    error: Unable to start server

2015-02-26T18:02:34+0000    error: /default/cfe_internal_update_processes/methods/'ENABLING CFE AGENTS'[1]: Method 'enable_cfengine_agents' failed in some repairs

Files are being copied from hub, cf-monitord and cf-execd are running on the vserver instance, cf-agent seems to be running properly when invoked

self-diagnostics for agent using workdir '/var/cfengine'

self-diagnostics for agent using inputdir '/var/cfengine/inputs'

self-diagnostics for agent using logdir '/var/cfengine'

[ YES ] Check that agent is bootstrapped: 192.168.1.137

[ NO  ] Check if agent is acting as a policy server: Not acting as a policy server

[ YES ] Check private key: OK at '/var/cfengine/ppkeys/localhost.priv'

[ YES ] Check public key: OK at '/var/cfengine/ppkeys/localhost.pub'

[ NO  ] Check persistent classes DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/state/cf_state.lmdb'

[ NO  ] Check checksums DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/checksum_digests.lmdb'

[ NO  ] Check classes DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/cf_classes.lmdb'

[ NO  ] Check observations DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/state/cf_observations.lmdb'

[ NO  ] Check file stats DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/stats.lmdb'

[ NO  ] Check locks DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/state/cf_lock.lmdb'

[ NO  ] Check performance DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/performance.lmdb'

[ NO  ] Check lastseen DB: Unable to diagnose LMDB file (not implemented) for '/var/cfengine/cf_lastseen.lmdb'

But I keep on receiving mail alerts with the same error as the on generated on bootstrap.

Any hints would be greatly appreciated.

JS

[Neil Watson]

On Thu, Feb 26, 2015 at 10:41:24AM -0800, Jean-Samuel Christophe wrote:
> It get the following error on bootstrap and can't seem to nail it down.
> 2015-02-26T18:02:34+0000    error:
> /default/cfe_internal_update_processes/methods/'ENABLING CFE
> AGENTS'/default/enable_cfengine_agents/commands/'"/var/cfengine/bin/cf-serverd"'[0]:
> Command related to promiser '"/var/cfengine/bin/cf-serverd"' returned
> code defined as promise failed 1

Here ^^^^

cf-serverd fails to start. Start it by and to find the error.

--
Neil H Watson
Sr. Partner, Architecture and Infrastructure
CFEngine reporting: https://github.com/evolvethinking/delta_reporting
CFEngine policy: https://github.com/evolvethinking/evolve_cfengine_freelib
CFEngine and vim: https://github.com/neilhwatson/vim_cf3
CFEngine support: http://evolvethinking.com

[Jean-Samuel Christophe]

Hi Neil,

Thanks for the quick update.. Here is the output of cf-serverd -I

2015-02-26T19:04:58+0000     info: /default/access_rules/access/'/var/cfengine/masterfiles'[0]: Found hostname admit/deny in access_rules, turning on reverse DNS lookups for every connection

2015-02-26T19:04:58+0000     info: /default/access_rules/access/'/var/cfengine/master_software_updates'[0]: Failed to canonicalise filename '/var/cfengine/master_software_updates' (realpath: No such file or directory)

2015-02-26T19:04:58+0000     info: /default/access_rules/access/'/var/cfengine/master_software_updates'[0]: Path does not exist, it's added as-is in access rules: /var/cfengine/master_software_updates

2015-02-26T19:04:58+0000     info: /default/access_rules/access/'/var/cfengine/master_software_updates'[0]: WARNING: this means that (not) having a trailing slash defines if it's (not) a directory!

2015-02-26T19:04:58+0000     info: Could not bind server address. (bind: Address already in use)

2015-02-26T19:04:58+0000    error: Unable to start server

Also why would I want cf-serverd running on the client?

Many thanks again,

JS

[Neil Watson]

On Thu, Feb 26, 2015 at 11:10:05AM -0800, Jean-Samuel Christophe wrote:
WARNING: this means that (not) having a trailing slash defines if it's
> (not) a directory!
> 2015-02-26T19:04:58+0000     info: Could not bind server address. (bind:
> Address already in use)

^^^^

It's common to run the server on agent hosts to a) collect files from
each host, and b) to use cf-runagent.

[Jean-Samuel Christophe]

Ok.. Thanks for the info.. Just starting with cfengine and still not familiar with its bits & pieces..

So cf-serverd can't start because it cant bind to an address. Are we talking the vserver's ip address or the host's ip address (192.168.1.137) which is also the hub's address?

My understanding

1 - install cfe on host

2 - bootstrap host to itself to make it a policy server

3 - install cfe on vserver

4 - bootstrap vserver to host's address

Did I skip something?

Thanks again,

JS

[Neil Watson]

Usually that error means that cf-serverd is already running. And by
address they usually mean port which defaults to 5308.

CFEngine's architecture is pretty simple, but poorly explained.

cf-serverd is a file server. It's how agents get files. Typically agent
hosts requests new inputs from the hub server's cf-serverd process.

cf-agent is the agent that keeps promises. It can talk to cf-serverd to
request files like inputs, templates, or others.

cf-execd is as scheduler. It sleeps, waking up every five minutes to
determine if cf-agent should run.

There are other programs that come with CFEngine, but the above are the
important ones.

So, all agents, including the hub, are bootstrapped to the hub. Get the
hub bootstrapped to itself and then proceed to other agent hosts.

[Jean-Samuel Christophe]

Thanks a bunch for the insight.. What I still don't understand is why it's not binding to the port..

root@hermes[/]#:ps aux | grep cf-*           

root      4766  0.0  0.0  39880  4092 ?        Ss   20:05   0:00 /var/cfengine/bin/cf-execd

root      5075  0.0  0.1  41296  6760 ?        Ss   20:10   0:00 /var/cfengine/bin/cf-monitord

root      5457  0.0  0.0   7832   852 pts/9    S+   20:16   0:00 grep --color=auto cf-*

root@hermes[/]#:netstat -o -n -a  

Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State       Timer

tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      off (0.00/0/0)

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      off (0.00/0/0)

tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      off (0.00/0/0)

tcp        0      0 192.168.1.126:326       0.0.0.0:*               LISTEN      off (0.00/0/0)

tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      off (0.00/0/0)

tcp        0      0 127.0.0.1:36093         127.0.0.1:25            TIME_WAIT   timewait (7.86/0/0)

udp        0      0 127.0.0.1:36968         127.0.0.1:36968         ESTABLISHED off (0.00/0/0)

I am pretty sure that no other instance of cf-serverd is running on the vserver and that port 5308 is not being used.

Looks a bit like a zabbix issue I had where server (the physical host) had an agent listening on a specific port and agents in the vservers had to use another port. Must be a some kind of side effect to using linux-vserver.

Is it possible to change the port cf-serverd is listening on for my vserver instances while keeping the default port on host machine, and what would I need to do to have them all working properly together?

JS

[Brian Bennett]

Are you running RHEL or CentOS? Have you given appropriate SELinux permissions to cf-serverd and cf-agent?

You say you're using "vservers". What kind? KVM? Xen? VMware? Hyper-V? OpenVZ?

What kind of network adapter has been set up? NAT? Bridge? Host only?

-- 
Brian Bennett
Looking for CFEngine training?
http://www.verticalsysadmin.com/

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Jean-Samuel Christophe]

Am on debian wheezy running linux-vserver (linux-vserver.org) and I now clearly see where the issue is

root@inmtxx-local-02[/v/c/masterfiles]#:netstat -plnt       

Active Internet connections (only servers)

Proto Recv-Q Send-Q Local Address           Foreign Address         State

tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN

tcp        0      0 0.0.0.0:5308            0.0.0.0:*               LISTEN

tcp        0      0 192.168.1.137:10050     0.0.0.0:*               LISTEN

tcp        0      0 192.168.1.137:326       0.0.0.0:*               LISTEN

On the host cf-serverd is listening on all ips on port 5308, hence on all vserver ips.. So I don't need to change the port but  would really need to force it to listen to it's own ip instead.

[Jean-Samuel Christophe]

UPDATE

================================================

Many thanks Neil for your explanations and for pointing me in the right direction.

Here's a summary and fix for anyone encountering the issue.

Env - Running Linux-Vserver (linux-vserver.org) on Debian Wheezy

Issue - cf-serverd in vserver refused to start as it could not bind to it's interface's ip

Reason - cf-serverd on the host machine was listening on all of the interface's ips (0.0.0.0)

Solution - added the following to /var/cfengine/masterfiles/controls/cf-serverd.cf in the "body server control" section

bindtointerface => "$(sys.ipv4)";