Cfengine bootstrap failure TLS key error

[Tom De Sloovere]

Hi,

we are working quiet some time with cfengine and everything is working ok.

But there is one issue from time to time that makes me pull my hair out. It mostly happens when I start playing around with the removal of cf-keys with the command cf-keys.

I cannot figure out how the authentication works when you start bootstrapping a client to the policy hub. I cannot connect the dots between the key files in /var/cfengine/ppkeys (localhost and rootMD5 files on both server and client), lastseen database, ...

The error I receive is:

root@laptop:~# cf-agent --bootstrap cfengine0102
  notice: Bootstrap mode: implicitly trusting server, use --trust-server=no if server trust is already established
   error: Failed to establish TLS connection: underlying network error (Connection reset by peer)
   error: No suitable server found
   error: Failed to establish TLS connection: underlying network error (Connection reset by peer)
   error: No suitable server found
R: This autonomous node assumes the role of voluntary client
R: Failed to copy policy from policy server at 10.1.16.45:/var/cfengine/masterfiles
       Please check
       * cf-serverd is running on 10.1.16.45
       * CFEngine version on the policy hub is 3.6.0 or latest - otherwise you need to tweak the protocol_version setting
       * network connectivity to 10.1.16.45 on port 5308
       * masterfiles 'body server control' - in particular allowconnects, trustkeysfrom and skipverify
       * masterfiles 'bundle server' -> access: -> masterfiles -> admit/deny
       It is often useful to restart cf-serverd in verbose mode (cf-serverd -v) on 10.1.16.45 to diagnose connection issues.
       When updating masterfiles, wait (usually 5 minutes) for files to propagate to inputs on 10.1.16.45 before retrying.
R: Did not start the scheduler

Running on the server side cf-serverd -d or cf-serverd -v only reveils only the following output:

Nov 18 11:52:44 cfengine0102 cf-serverd[7752]: CFEngine(server)  Remote host '10.3.130.107' not in allowconnects, denying connection

Which makes no sense since we allow the entire 10.x.x.x IP range in the allowconnects (both client and server are in the same network), and in the trustkeysfrom slist I even allow all IP addresses. But it looks like the cf-serverd process does not pick up this configuration. Which is even weirder because because it previously worked (before fiddling with the cf-key command).

Def.cf:

   "acl"     slist => {
                           # Allow everything in my own domain.

                           # Note that this:
                           # 1. requires def.domain to be correctly set
                           # 2. will cause a DNS lookup for every access
                           # ".*$(def.domain)",

                           # Assume /16 LAN clients to start with
                           "$(sys.policy_hub)/8",

                           #  "2001:700:700:3.*",
                           #  "217.77.34.18",
                           #  "217.77.34.19",
},
...
  "trustkeysfrom" slist => {
                                 # COMMENT THE NEXT LINE OUT AFTER ALL MACHINES HAVE BEEN BOOTSTRAPPED.
                                 "0.0.0.0/0", # allow any IP
      },

As far as I can understand what is happening, is that the key trust between policy hub and client does not work anymore. It looks like there is still a reference to this particular host. Removing cf_lastseen.lmdb, all keys in /var/cfengine/ppkeys, regenerate keys, rebootstrap policy server to itself (which works) restarting services, there is still a reference of this client host.

I have no clue how to "reset" the policy server regarding trust keys for a host or even multiple hosts. For example: do I need to remove the lasseen database, generated keys, all root-MD5 keys, ... stop services, rebootstrap policy server to itself, ...?
Or if someone can explain in simple words, what happens regarding the trusting of keys during bootstrap?

Thanks,

Tom

[Dimitrios Apostolou]

Hi Tom,

what about the "allowconnects" settings in body server control? That's
the *first* thing checked by the server, if the IP is in
allowconnects. After that it goes on to the "trustkeysfrom" and
finally to the paths ACLs, which are checked on each and every file
requested.


Hope this helps,
Dimitris

[Nick Anderson]

On 11/19/2015 03:23 PM, Dimitrios Apostolou wrote:
> what about the "allowconnects" settings in body server control? That's
> the *first* thing checked by the server, if the IP is in
> allowconnects. After that it goes on to the "trustkeysfrom" and
> finally to the paths ACLs, which are checked on each and every file
> requested.

If its using the stock cf_serverd.cf policy, allowconnects defaults to
include def.acl.

https://github.com/cfengine/masterfiles/blob/3.7.x/controls/3.7/cf_serverd.cf#L17

[Tom De Sloovere]

Hi,

thanks for the response. And yes, my first thought was the acl slist in def.acl as well. And after I added an extra line (the IP address of the client), I could re-bootstrap the client again.
Still, it confuses me. Why in the first place could I bootstrap a client initially when this line was not added?

So,

1. Default setting (allowconnect acl has only $(sys.policy_hub/8) in def.cf), bootstrapping a client works perfectly
   
2. Fiddling with keys, ... => result: re-bootstrapping failed
3. Adding the IP address to the acl slist in def.cf, re-bootstrap works again

As far as I can understand, the first bootstrap should have never worked. But, it has. So can this be considered a bug? If so, I will post a bug request on cfengine.

greetings,

Tom


Op woensdag 18 november 2015 12:29:59 UTC+1 schreef Tom De Sloovere:

[Dimitrios Apostolou]

Hi, sorry I saw the response late but I was not CC'd.


On Fri, Nov 20, 2015 at 8:39 PM, Tom De Sloovere
<tomdes...@gmail.com> wrote:
> Default settings (allowconnect acls in def.cf), bootstrapping a client works

> perfectly
> Fiddling with keys, ... => result: re-bootstrapping failed
> Adding the IP address to the acl slist in def.cf, re-bootstrap works again
>
> As far as I can understand, the first bootstrap should have never worked.
> But, it has. So can this be considered a bug? If so, I will post a bug
> request on cfengine.

So you are saying that cf-serverd's ACL excluded the client's IP
address from the beginning, but despite that it successfully
bootstrapped the first time? If so I would be very surprised, and yes,
I believe it would be a bug.

Dimitris

[Tom De Sloovere]

Hi,

it does seem weird but maybe I just do not understand the bootstrap and rebootstrap procedure.

As soon as I have some time on my hands, I will set up a test environment and try to reproduce the issue.
IF I can reproduce it, I will report it as a bug and keep you informed.

greetings

Tom

Op dinsdag 24 november 2015 00:45:09 UTC+1 schreef Dimitrios Apostolou:

[Louis Gillette]

Did this issue get any traction later on? I've been running into this for a couple of weeks now, and any new information would be appreciated.

What information should I be provide from my own mis-adventures with bootstrapping to further this discussion?

[Dimitrios Apostolou]

Just an idea: by default (check def.cf) the hub allows the connections from its own IP address' /16 subnet. Could it be that you are bootstrapping from all over the internet, and the default subnet is not enough?

Regards,

Dimitris

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengine+unsubscribe@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.

Visit this group at https://groups.google.com/group/help-cfengine.

For more options, visit https://groups.google.com/d/optout.

[Dimitrios Apostolou]

On Mon, Nov 7, 2016 at 12:37 PM, Dimitrios Apostolou <ji...@cfengine.com> wrote:

Just an idea: by default (check def.cf) the hub allows the connections from its own IP address' /16 subnet. Could it be that you are bootstrapping from all over the internet, and the default subnet is not enough?

To be more specific, there is a small guide on the topic of bootstrapping CFEngine over exposed networks:

https://cfengine.com/company/blog-detail/securely-deploying-cfengine-on-untrusted-networks/
https://cfengine.com/wp-content/uploads/2015/02/cfgcamp-2015.pdf

In general you can check "def.cf" to see and change the /16 default setting, and kill cf-serverd and re-run it in verbose foreground mode ("cf-served -vF") to see the "summary of access rules" at the end of the wall of text.

In short, "allowconnects" can be completely open if you wish (something like 0.0.0.0/0), but you should be really careful with "trustkeysfrom" setting.

We're open to contributions for improving our standard documentation, just check the https://github.com/cfengine/documentation repo.

Regards,

Dimitris