Checking variables holding IP Addresses....

Martin Simons

unread,

Jun 19, 2019, 8:41:36 AM6/19/19

to help-cfengine

Dear CFEngineer,

Too long time, no see.

After pulling all my hair out.

I am trying to configure ssh to use only specific IP-Addresses, depending on the role. I am using a regular expression, which seems to work. It validates an IP if it is valid and it sets a class if you feed it an invalid value., but it does not set the class in case I feed it a variable that is empty or undefind.

bundle agent ip_list {

vars:

"ip_1"                        string => "10.68.71.5";
"ip_2"                        string => "10.68.171.5";
"ip_3"                        string => "192.168.123.123";

"ips"                          slist => { "$(ip_1)", "$(ip_2)", "$(ip_3)" };

methods:

"check_ip list"            usebundle => check_ip( "$(ips)" );

reports:

"$(this.bundle) ip's: *$(ips)*";

}

bundle agent empty_var {

vars:

"ip_1"                        string => "10.68.71.5";
"ip_2"                        string => "10.68.171.5";
"ip_3"                        string => "$(nic.nic_admin)";

"ips"                          slist => { "$(ip_1)", "$(ip_2)", "$(ip_3)" };

methods:

"check_ip empty var"       usebundle => check_ip( "$(ips)" );

reports:

"$(this.bundle) ip's: *$(ips)*";

}

bundle agent bogus {

vars:

"ip_1"                        string => "10.68.71.5";
"ip_2"                        string => "10.68.171.5";
"ip_3"                        string => "bogus";

"ips"                          slist => { "$(ip_1)", "$(ip_2)", "$(ip_3)" };

methods:

"check_ip bogus"           usebundle => check_ip( "$(ips)" );

reports:

"$(this.bundle) ip's: *$(ips)*";

}
bundle agent empty_string {

vars:

"ip_1"                        string => "10.68.71.5";
"ip_2"                        string => "10.68.171.5";
"ip_3"                        string => "";

"ips"                          slist => { "$(ip_1)", "$(ip_2)", "$(ip_3)" };

methods:

"check_ip empty string"    usebundle => check_ip( "$(ips)" );

reports:

"$(this.bundle) ip's: *$(ips)*";

}

bundle agent check_ip(ips) {

classes:
"match_$(ips)"
   comment    => "Oh, the horror!",
   expression => regcmp(
#"^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
"^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$",
            $(ips)
            );

methods:

"!match_$(ips)"::
"Bell!!"            usebundle => bell($(ips));

reports:

"$(this.bundle): match_$(ips)*";

"!num_$(ips)"::
"$(this.bundle) iprange: $(ips)*";

"num_$(ips)"::
"$(this.bundle) iprange: $(ips)*";

}

bundle agent bell(ips) {

classes:

"bad_ip"        expression => regcmp( "bad_ip", $(bad_ip) );

vars:

"bad_ip"            string => "bad_ip";

methods:

bad_ip::

"The show stops here!" usebundle => stop_the_show;

reports:

"$(this.bundle): Yell Bell! $(ips)";

bad_ip::

"$(this.bundle): $(bad_ip)";

}

bundle agent stop_the_show {

reports:

"$(this.bundle): We stop the show!";

}

body common control {

        bundlesequence => { "ip_list", "empty_var", "bogus", "empty_string" };

}

It stops when it encottners the first error, but lets the empty variable pass. It does report the variable.

What do I miss?

Regards,

Martin.

Nick Anderson

unread,

Jun 19, 2019, 10:32:40 AM6/19/19

to Martin Simons, help-cfengine

Hi Martin,

Can you be a bit more specific about the issue? What output do you get, what do you expect? What do you mean it stops when it encounters the first error but lets the empty variable pass

This is the output that I get when I run the policy.

R: check_ip: match_10.68.71.5*
R: check_ip: match_10.68.171.5*
R: check_ip: match_192.168.123.123*
R: ip_list ip's: *10.68.71.5*
R: ip_list ip's: *10.68.171.5*
R: ip_list ip's: *192.168.123.123*
R: check_ip: match_$(nic.nic_admin)*
R: empty_var ip's: *10.68.71.5*
R: empty_var ip's: *10.68.171.5*
R: empty_var ip's: *$(nic.nic_admin)*
R: stop_the_show: We stop the show!
R: bell: Yell Bell! bogus
R: bell: bad_ip
R: check_ip: match_bogus*
R: check_ip iprange: bogus*
R: bogus ip's: *10.68.71.5*
R: bogus ip's: *10.68.171.5*
R: bogus ip's: *bogus*
R: bell: Yell Bell!
R: check_ip: match_*
R: check_ip iprange: *
R: empty_string ip's: *10.68.71.5*
R: empty_string ip's: *10.68.171.5*
R: empty_string ip's: **

One thing that does jump out at me is this glass guard you are trying to use inside bundle agent check_ip().

"!match_$(ips)"::

$(ips) does not expand to a valid class string. That is going to expand to something like !match_10.68.71.5:: which is probably not what you are actually trying to test.

You can either canonify $(ips) and use that, or you can move your condition to the promise itself and canonify on the fly.

Perhaps something like this:

"Bell!!"
  usebundle => bell( $(ips) ),
  if => not( canonify( "match_$(ips)" ) );

– Nick Anderson| Doer of Things | (+1) 785-550-1767 | https://northern.tech

Martin Simons

unread,

Jun 20, 2019, 6:21:22 PM6/20/19

to help-cfengine

Hi Nick,

Thank you for your swift answer.

> Can you be a bit more specific about the issue? What output do you get, what do you expect?

This is data driven thingy.

The servers have a role based configuration and the sshd configuration depends on the role, with regards to the interface it is listening on.

So I feed the listen addresses to the role as variables. I want the policy to check the IP's on those variables and only execute the sshd policy is the IP addresses are valid.

> What do you mean it stops when it encounters the first error but lets the empty variable pass

It does not catch an empty variable, but it catches bogus content.

One thing that does jump out at me is this glass guard you are trying to use inside bundle agent check_ip().
"!match_$(ips)"::
$(ips) does not expand to a valid class string. That is going to expand to something like !match_10.68.71.5:: which is probably not what you are actually trying to test.

You can either canonify $(ips) and use that, or you can move your condition to the promise itself and canonify on the fly.

Perhaps something like this:
"Bell!!"
  usebundle => bell( $(ips) ),
  if => not( canonify( "match_$(ips)" ) );

I changed the script accordingly. The filter works for bogus IP 's, but it still does not work for the empty variable:

bundle agent empty_var {
vars:
"ip_1"                        string => "10.68.71.5";
"ip_2"                        string => "10.68.171.5";
"ip_3"                        string => "$(nic.nic_admin)";
"ips"                          slist => { "$(ip_1)", "$(ip_2)", "$(ip_3)" };
methods:
"check_ip empty var"       usebundle => check_ip( "$(ips)" );
reports:
"$(this.bundle) ip's: *$(ips)*";
}

bundle agent check_ip(ips) {
classes:
"match_$(ips_canonified)"

comment => "Oh, the horror!",
expression => regcmp(

"^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
#"^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\_){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$",
$(ips_canonified)
);

"match_$(ips)"
comment => "Oh, the horror!",
expression => regcmp(

"^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
            $(ips)
            );
vars:
"ips_canonified"       string => canonify($(ips));
methods:
"!match_$(ips_canonified)"::
"Bell!!"            usebundle => bell($(ips));

"Bell"              usebundle => bell($(ips)),
                             if => not(canonify("match_$(ips)" ) );
reports:
"!match_$(ips_canonified)"::
"$(this.bundle) IP does not match: $(ips_canonified)*";
}

bundle agent bell(ips) {
classes:
"bad_ip" expression => regcmp( "bad_ip", $(bad_ip) );
vars:
"bad_ip" string => "bad_ip";
methods:
bad_ip::
"The show stops here!" usebundle => stop_the_show;
reports:
"$(this.bundle): Yell Bell! $(ips)";
bad_ip::
"$(this.bundle): $(bad_ip)";
}

bundle agent stop_the_show {
reports:
"$(this.bundle): We stop the show!";
}

body common control {
bundlesequence => { "empty_var" };
}

I tried functions like string_length too, but no result so far.

Regards,

Martin.

Nick Anderson

unread,

Jun 21, 2019, 2:07:59 PM6/21/19

to Martin Simons, help-cfengine

Martin Simons writes:

It does not catch an empty variable, but it catches bogus content.

OK, so you have a list of things that should be IPv4 addresses and you want to iterate over them if they are actually ipv4 addresses.

The regular expression from your example seems to be effective distilled down to a more simple example.

bundle agent example_filter_ipv4
{
  vars:
      "variable_value_from_varible" string => "$(nosuch.variable)";
      "candidates"
        slist => {
                   "10.68.71.5", "10.168.171.5",
                   "not-an-ip-address", "",
                   "$(missing.variable)", $(another_missing.variable),
                   "$(variable_value_from_varible)"
                 };

  reports:

      "'$(candidates)' is a valid IPv4 address"
        if => regcmp(
                      #"^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
                      "^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$",
                      $(candidates)
        );

      "'$(candidates)' is *NOT* a valid IPv4 address"
        if => not( regcmp(
                      #"^(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
                      "^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$",
                      $(candidates)
        ));


}
bundle agent __main__
{
  methods:
      "example_filter_ipv4";

  reports:
      "CFEngine: $(sys.cf_version)"; 
}

R: '10.68.71.5' is a valid IPv4 address
R: '10.168.171.5' is a valid IPv4 address
R: 'not-an-ip-address' is *NOT* a valid IPv4 address
R: '' is *NOT* a valid IPv4 address
R: CFEngine: 3.14.0a.4e12fcf75

What is the specific output that you expected to see from the policy?

Martin Simons

unread,

Jun 22, 2019, 7:26:29 AM6/22/19

to help-cfengine

Dear Nick,

> What is the specific output that you expected to see from the policy?

The policy indeed reports there is an unspecifed invalid or missing IP address. I have been dwelling on the topic for quite some time and I now feel I have been struggling with the topic because I did not have the right approach. I was also hampered by the fact that de Debian network on the hosts did not come up after a reboot. https://www.opencloudblog.com/?p=240

I only resolved the issue very recently with a (dirty) work around.

When looking at the ISO 7 layer model, with regards to the missing IP addresses, my conclusion is that I tried to resolve a layer two (the virtual bridge) or a layer one (the virtual ethernet device) problem with a layer three (the IP layer) technique. Of course, an IP address can be invalid, that is a layer three problem and that issue can be resolved using a regular expression. The issue with a missing virtual bridge or even a missing virtual ethernet device, however, cannot truly be resolved with this technique. My experience is that virtual bridges do collapse and that it thus is very useful to have a policy, that resurrects a bridge if collapsed, in place.

I provide a short description of the VLAN / LAN setup in the landscape:

I use vlan enabled switches and I use openswitch as a virtual switch on my kvm hosts. On the kvm host, vconfig defines virtual ethernet devices on which ovs defines tagged virtual bridges for each and every vlan.

I do have the following vlans:

1001 the service front end

123, support

22, admin

213, wlan

254, gateways

1111, vpn

42, cloud management

So the kvm hosts operate seven bridges, all but one being layer two. All guests are present on the support and admin vlans and the service providing guests have a service IP address. The other four vlans are special to very specifc roles in the network, 213 for the Internet facing services like a http proxy, mail or name server. Most of this setup can be found here: https://github.com/Webhuis/CFEngine-Roadshow/tree/opennebula-1

I am grateful for your contribution, because it made clear it is at least possible to raise an alarm bell on an IP address, before a policy destroys an IP address depending configuration.

I will shift my focus now to developing a complete policy set for layers one, two and three. Policies with the defaults policy type and mac address finding like this may be useful:

"$(kvm_host.$(bridge_prefix))_$(vlan)" expression => regextract(
                                     ".*HWaddr ([^\s]+).*(\n.*)*",
                                     "$(interface_breth0_$(vlan))",
                                     "mac_breth0_$(vlan)");

I will also look for policies on layer one and two that are available already and I am open to suggestions on those policies or other tips.

Beste regards,

Martin.

Op vrijdag 21 juni 2019 20:07:59 UTC+2 schreef Nick Anderson:

Martin Simons

unread,

Jun 25, 2019, 3:06:08 PM6/25/19

to help-cfengine

Dear CFEngineer,

This is the first part of the sultion, it deals with creating the virtual interface on the kvm host, if it is not available.

bundle agent check_network {

vars:

"vlan[front]"                       string => "11";
"vlan[support]"                     string => "12";
"vlan[admin]"                       string => "22";
"vlan[wan]"                         string => "23";
"vlan[gw]"                          string => "25";
"vlan[life]"                        string => "48";
"vlan[roadshow]"                    string => "89";

"vlan_index"                         slist => getindices("vlan");

methods:

"Check interfaces and bridges are up"    usebundle => check_vswitch( "$(vlan[$(vlan_index)])" ),
                                           comment   => "This is a layer I check on the virtual switch";

reports:

"$(this.bundle): $(vlan_index)";

}

bundle agent check_vswitch(vlan) {

vars:

"interface"                         string => "eth0";
"var_interface_$(vlan)"             string => execresult("/sbin/ip link show dev $(interface).$(vlan)","noshell");

methods:

"Set interface class"            usebundle => set_class( "$(vlan)", "$(var_interface_$(vlan))" );

reports:

"$(this.bundle): $(interface)_$(vlan)";

}

bundle agent set_class(vlan, var_interface_vlan) {

classes:

"no_device_$(vlan)"             expression => regcmp(
                                     ".*does not exist.*",
                                     "$(var_interface_vlan)");

vars:

"interface"                         string => "eth0";

methods:

"no_device_$(vlan)"::
"Bring up interface $(vlan)"     usebundle => bring_device_up( "$(interface)", "$(vlan)" );

reports:

"no_device_$(vlan)"::
"$(this.bundle): $(var_interface_vlan) Bingo!";

}

bundle agent bring_device_up(interface, vlan) {

commands:

"/sbin/ip link add link eth0 name $(interface).$(vlan) type vlan id $(vlan)";
"/sbin/ip link set dev $(interface).$(vlan) up";
"/sbin/vconfig add $(interface) $(vlan)";

reports:

"$(this.bundle): interface $(interface).$(vlan) created!";

}

body common control {

bundlesequence => { check_network };

inputs => { "/var/cfengine/masterfiles/lib/3.7/stdlib.cf", };

}

Regards,

Martin.

Mike Weilgart

unread,

Jun 25, 2019, 3:39:11 PM6/25/19

to Martin Simons, help-cfengine

I will admit I haven't studied my way through ALL of this code, but have you tried simply using the "grep" function?

It filters an slist and returns an slist containing only the elements which match the given regex. Seems like it exactly fits the bill here.

If you need to abort if there is a bad IP (rather than just using all of the valid IPs passed and ignoring the bad ones), you could probably do it most easily by checking the length of the slist result from grep compared to the slist passed into grep.

Is that something you've already considered?

Best,

—Mike Weilgart

Vertical Sysadmin, Inc.

--

Need training on CFEngine, Git, Bash or Vim? Email trai...@verticalsysadmin.com.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/01136aae-47b7-4631-8af4-ffd2ad4cf4ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Martin Simons

unread,

Jun 25, 2019, 3:50:25 PM6/25/19

to help-cfengine

Dear Mike,

To grep through the slist certainly is an idea.

The very point was the variable that should contain the IP address was ambiguous. Neither the true class nor the not true class would come up.

Thank you!

Regards,

Martin.

Op dinsdag 25 juni 2019 21:39:11 UTC+2 schreef Mike Weilgart:

To unsubscribe from this group and stop receiving emails from it, send an email to help-c...@googlegroups.com.

Martin Simons

unread,

Jul 10, 2019, 3:24:06 PM7/10/19

to help-cfengine

Dear CFEngineer,

It has been a while since my latest post on the matter of validating IP Adresses. The problem was that invalid IP Addresses ruined my configurations, because faulty or missing variables mentioned in templates just ended up a like a variable in a configuration file like this: $(role.ssh_address). The ssh daemon would subsequently die after changing the configuration file.

I struggled and my first attempts did not succeed, because my focus was on the very point where the problem emerged: the OSI layer III IP address. It became clear to me, however, that an OSI layer I or II problem cannot be resolved with a layer III strategy or tooling available.

I am using KVM hosts throughout my network to host libvirt / Qemu machines. The KVM hosts offer a openvswitch service, in order to facilitate VLANs. One of the nasty problems I encountered in operations was the collapse of a bridge or an unavailable virtual device. I simply makes no sense to check for IP Addresses in case a bridge or a device is unavailable.

So I had to split things up along the lines of the OSI model. I made my work available on Github. https://github.com/Webhuis/CFEngine-Roadshow/tree/master

I did my work in the opennebula-1 branch and I merged the branch with master moments ago. I am now continuing work on branch opennebula-2, but I consider the IP stuff done for know, although it will be extended. I will provide some snippets here, please consult the Github repo for context and more detail.

IP stuff and addresses come from various directions:

- from the role a machine has, different roles listen with different services on different vlans and thus interfaces

- data provided in domain related policies

- data from external sources

An IP addresses provided through a data policy or from an external source are simple. The complexity comes in when there are more IP Addresses from multiple sources. IP Addresses that rely on a device or a bridge are not that simple, but they are generally provided from a role.

This the snippet of the kvm role data bundle:

bundle agent kvmo_data {

vars:

"vlan[service]" string => "10";

"vlan[support]" string => "12";

"vlan[admin]" string => "14";
"vlan[wan]" string => "21";

"vlan[gw]" string => "25";

"vlan[life]"                        string => "26";
"vlan[roadshow]"                    string => "100";

"vlan_index"                        slist => getindices("kvmo_data.vlan");

"etc_hosts_nic"                     slist => { "$(sys.ipv4[breth0_$(vlan[support])])" };
"admin_nics"                        slist => { "$(sys.ipv4[breth0_$(vlan[admin])])", "$(sys.ipv4[breth0_$(vlan[life])])" };

methods:

"Check interfaces and bridges are up" usebundle => check_vswitch( "$(vlan[$(vlan_index)])" ),

comment => "This is a layer I and II check on the virtual switch";

bundle agent check_vswitch(vlan) {

vars:

"interface" string => "eth0";

"bridge" string => "breth0";

"var_interface_$(vlan)" string => execresult("/sbin/ip link show dev $(interface).$(vlan)","noshell");

"var_bridge_$(vlan)"                string => execresult("/sbin/ip link show dev $(bridge).$(vlan)","noshell");

methods:

"Set interface class"            usebundle => set_interface_class( "$(vlan)", "$(var_interface_$(vlan))" );
"Set bridge class"               usebundle => set_bridge_class( "$(vlan)", "$(var_bridge_$(vlan))" );

bundle agent set_interface_class(vlan, var_interface_vlan) {

classes:

"no_device_$(vlan)" expression => regcmp( ".*does not exist.*",
"$(var_interface_vlan)");

"if_down $(vlan)" expression => regcmp( ".*DOWN.*",

"$(var_interface_vlan)");

vars:

"interface" string => "eth0";

methods:

"no_device_$(vlan)"::

"Set status bad"                 usebundle => status_device("bad");
"Add device $(vlan)"             usebundle => add_device( "$(interface)", "$(vlan)" );
"Bring up interface $(vlan)"     usebundle => bring_device_up( "$(interface)", "$(vlan)" ),
                                     comment => "layer one";

"if_down_$(vlan)"::
"Set status bad"                 usebundle => status_device("bad");
"Bring up interface $(vlan)"     usebundle => bring_device_up( "$(interface)", "$(vlan)" ),
       bundle agent status_device(device_status) {

classes:

"stop_network_editing"     expression => regcmp( "bad", $(device_status) );

defaults:

"device_status"                string => "good", if_match_regex => "";

methods:

"Stop IP processing"        usebundle => "stop_ip";
                              comment => "layer one";

bundle agent status_device(device_status) {

classes:

"stop_network_editing"     expression => regcmp( "bad", $(device_status) );

defaults:

"device_status"                string => "good", if_match_regex => "";

methods:

"Stop IP processing"        usebundle => "stop_ip";

bundle common stop_ip {

vars:

"bad_device" string => $(status_device.device_status);

classes:

"stop_ip" expression => regcmp( "bad", "$(bad_device)" );

The stop condition has to be generated in case an interface or a bridge is bad. The policies will try to get the virtual interface and bridge up again and wait until the next iteration of cf-agent to process network information.

The defaults: promise type had to be applied in order to have a meaningful class set.

The IP check_vnic policy is analogue to the policy above, so I leave it out. The IP check policy is more simple, but it has an interesting regular expression:

bundle agent check_ip(ips) {

classes:

"match_$(can_ips)"
   comment    => "Oh, the horror! Process the list of IP Addresses.",
   expression => regcmp(
"(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",
            $(can_ips)
            );

vars:

"can_ips"              string => canonify( $(ips) );

methods:

'!match_$(can_ips)'::
"Set status bad"                 usebundle => status_ip("bad");

bundle agent status_ip(ip_status) {

classes:

"stop_ip_editing" expression => regcmp( "bad", $(ip_status) );

defaults:

"ip_status" string => "good", if_match_regex => "";

methods:

"Stop IP processing" usebundle => "bad_ip";

bundle common bad_ip {

vars:

"bad_ip" string => "$(status_ip.ip_status)";

classes:

"bad_ip" expression => regcmp( "bad_ip", $(bad_ip) );

reports:

"$(this.bundle) bad ip: $(bad_ip)";

stop_ip::
"$(this.bundle): prevent configuring bad ip addresses";

}

The defaults: promise type in stop_ip and bad_ip does the trick, because it provides a meaningful common class in either case.

I have been working on this for quite a while, but I may not have found the best solution. I am thus open to and beforehand grateful about any contributions and suggestions.

Op woensdag 19 juni 2019 16:32:40 UTC+2 schreef Nick Anderson:

Hi Martin,

Can you be a bit more specific about the issue? What output do you get, what do you expect? What do you mean it stops when it encounters the first error but lets the empty variable pass

Best regards,

MArtin.

Martin Simons

unread,

Aug 28, 2020, 8:09:49 AM8/28/20

to help-cfengine

Dear CFEngineer,

Left the topic for a while for what it was, until it started to bite me again.

Problem

Ok, I was able to filter out some invalid ip-addresses, but the policy still would allow errant variables to mess things up and break /etc/hosts, /etc/resolv.conf or ip-address based configuration files like /etc/ssh/sshd_config.

Like this:

bundle agent common {
vars:

"ip" slist => { "10.10.117.10", "$(bar)" };

methods:
"Check IP Variables"                      usebundle => check_ip($(ip));
bad_ip::
"Do nothing with variables"               usebundle => do_nothing("$(ip)");
!bad_ip::
"Do things with variables"                usebundle => do_things("$(ip)");
reports:
"$(this.bundle) ip input: $(ip)";
}
bundle common check_ip(ip) {
classes:
"is_variable_$(can_ip)"                  expression => isvariable("$(can_ip)");
"match_ip_$(can_$(ip))"                  expression => regcmp(

"(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",

            "$(can_$(ip))"
            );
"!match_ip_$(can_$(ip))"::
"bad_ip"                                 expression => "any";

vars:
"can_$(ip)" string => canonify( "$(ip)" );
reports:
"$(this.bundle) ip input: $(ip)";
"$(this.bundle) can ip input: $(can_$(ip))";
"!match_ip_$(can_$(ip))"::
"$(this.bundle) bad ip : $(can_$(ip))";
}

bundle agent do_things(ip) {
reports:
"$(this.bundle) do things with variable: $(ip)";
}
bundle agent do_nothing(ip) {
reports:
"$(this.bundle) do nothing with variable: $(ip)";
}

Output:

R: check_ip ip input: 10.10.117.10
R: check_ip can ip input: 10_10_117_10
R: check_ip ip input: $(bar)
R: check_ip can ip input: $(can_$(bar))
R: do_things do things with variable: 10.10.117.10
R: do_things do things with variable: $(bar)
R: common ip input: 10.10.117.10
R: common ip input: $(bar)

So this policy effectively messed everything up.

Solution

Enter the defaults: promise type and it changed it all.

bundle agent common {
vars:
"foo"                                        string => "foo";
#"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10" };
#"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10", "$(foo)" };
#"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10", "foo" };
#"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10", "" };
#"ip"                                          slist => { "10510.17.10", "10.110.17.10", "10.10.117.10" };
"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10", "$(bar[$(index)])" };
#"ip"                                          slist => { "10.10.17.10", "10.110.17.10", "10.10.117.10", "$(bar)" };
#"ip"                                          slist => { $(bar) };
methods:
"Check IP Variables"                      usebundle => check_ip($(ip));
bad_ip::
"Do nothing with variables"               usebundle => do_nothing("$(ip)");
!bad_ip::
"Do things with variables"                usebundle => do_things("$(ip)");
reports:
"$(this.bundle) ip input: $(ip)";
}
bundle common check_ip(ip) {
classes:
"match_ip_$(can_ip)"                     expression => regcmp(

"(25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[_](25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})",

            $(can_ip)
            );
"!match_ip_$(can_ip)"::
"bad_ip"                                 expression => "any";
defaults:
"ip"                                         string => "bar", if_match_regex => ".*$[\$\[\]\($a-zA-Z0-9_.]+\)";
vars:
"can_ip"                                     string => canonify( "$(ip)" );
reports:
"$(this.bundle) ip input: $(ip)";
"$(this.bundle) can ip input: $(can_ip)";
"!match_ip_$(can_ip)"::
"$(this.bundle) bad ip : $(can_ip)";
}
bundle agent do_things(ip) {
reports:
"$(this.bundle) do things with variable: $(ip)";
}
bundle agent do_nothing(ip) {
reports:
"$(this.bundle) do nothing with variable: $(ip)";
}
body common control {
bundlesequence => { "common", };
inputs         => { "/var/cfengine/inputs/lib/3.7/stdlib.cf", };
}

Output:

R: check_ip ip input: 10.10.17.10
R: check_ip can ip input: 10_10_17_10
R: check_ip ip input: 10.110.17.10
R: check_ip can ip input: 10_110_17_10
R: check_ip ip input: 10.10.117.10
R: check_ip can ip input: 10_10_117_10
R: check_ip ip input: bar
R: check_ip can ip input: bar
R: check_ip bad ip : bar
R: do_nothing do nothing with variable: 10.10.17.10
R: do_nothing do nothing with variable: 10.110.17.10
R: do_nothing do nothing with variable: 10.10.117.10
R: do_nothing do nothing with variable: $(bar[$(index)])
R: common ip input: 10.10.17.10
R: common ip input: 10.110.17.10
R: common ip input: 10.10.117.10
R: common ip input: $(bar[$(index)])

Problem solved. It even deals with complex erroneous variables stemming from a multidimensional array.

I wish I had this before.

Regards,

Martin.

Op woensdag 10 juli 2019 om 21:24:06 UTC+2 schreef Martin Simons:

Ted Zlatanov

unread,

Oct 27, 2020, 10:21:24 AM10/27/20

to help-c...@googlegroups.com

Here's a neat way to validate IPs in CFEngine, but it requires 3.11 and
only does IPv4. It works in my testing - see what you think.

I couldn't use a /0 subnet, the C code shortcuts it to make everything
valid. That's why I extract the first octet with the format() call both
to check the first octet and to make the subnet.

(It would be nice if the internal IP parsing functions were exposed as
either standalone functions or something like `mapdata(validate_ip,
$(this), @(candidates))`.)

Ted

```cfengine3
bundle agent main
{
vars:
"candidates" slist => { "10510.17.10", "1.2.3.4", "", "100", "127.0.0.1", "5.9.10.22", "233.444.23.1" };

classes:
"valid_$(candidates)" and => { isgreaterthan(format("%d", $(candidates)), 0),
isipinsubnet(format("%d.0.0.0/8", $(candidates)), $(candidates)) };

reports:
"$(this.bundle): candidates '$(candidates)'";
"$(this.bundle): VALID IP '$(candidates)'"
if => canonify("valid_$(candidates)");
"$(this.bundle): BAD IP '$(candidates)'"
unless => canonify("valid_$(candidates)");
"$(this.bundle): mask of $(candidates) is $(with)" with => format("%d.0.0.0/8", $(candidates));
}
```

Martin Simons

unread,

Aug 26, 2024, 6:20:28 AM8/26/24

to help-cfengine

Dear CFEngineer,

Issue CFE-3446 seems to be solved, so in preparation of the move to 3.24 I am fixing some policies dealing with variables.

One of thos is IP checking.

Good old Neil has this solution:

https://watson-wilson.ca/blog/2015/08/20/build-better-regular-expressions-in-cfengine/

It allows an address starting with '0.', however.

So I tweaked it a bit:

classes:

"valid_hosts_ip" expression => regcmp("(1[0-9]{0,2}|[3-9][0-9]{0,1}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])",
"$(ip)");

It now checks IP addresses starting with a '1[0-9]{0,2}' first, that's how all private address ranges start anyway. I have a test bundle available, if needed.

There also is this funny observation:

https://en.wikipedia.org/wiki/Benford%27s_law

Regards,

Martin.

Vratislav Podzimek

unread,

Aug 26, 2024, 11:10:30 AM8/26/24

to help-c...@googlegroups.com

Hi Martin!

On Mon, 2024-08-26 at 03:20 -0700, Martin Simons wrote:
> Dear CFEngineer,
>

> Issue CFE-3446 seems to be solved, so in preparation of the move to 3.24 I am fixing some policies dealing with variables.
> One of thos is IP checking.
>
> Good old Neil has this solution:
> https://watson-wilson.ca/blog/2015/08/20/build-better-regular-expressions-in-cfengine/
>
> It allows an address starting with '0.', however.
> So I tweaked it a bit:
> classes:
>
> "valid_hosts_ip" expression => regcmp("(1[0-9]{0,2}|[3-9][0-9]{0,1}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[\.]([0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])",
> "$(ip)");
>
> It now checks IP addresses starting with a '1[0-9]{0,2}' first, that's how all private address ranges start anyway. I have a test bundle available, if needed.

👍️

I was just wondering if `isipinsubnet()` [1] could be an answer here. If not,
what is the reason? Maybe we could extend it?

[1] https://docs.cfengine.com/docs/master/reference-functions-isipinsubnet.html

Thanks!

--
Vratislav

> --
> You received this message because you are subscribed to the Google Groups "help-cfengine" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/4021c7ba-e455-47b0-b2f6-62c4d10542ccn%40googlegroups.com.

signature.asc

Message has been deleted

Martin Simons

unread,

Aug 27, 2024, 6:07:12 PM8/27/24

to help-cfengine

Hi Vratislav,

Thank you for working on CFE-3446!

Because of the issue, I am still trapped in CFEngine 3.7.5.

I just checked, isipinsubnet() is not available for me.

I am cleaning up my plocies right now and really look forward to jumping into 3.24 LTS!

Martin.

Vratislav Podzimek

unread,

Aug 28, 2024, 4:12:38 AM8/28/24

to help-c...@googlegroups.com

On Mon, 2024-08-26 at 12:59 -0700, Martin Simons wrote:
> Hi Vratislav,
> Thank you for working on CFE-3446!

I wish that was justified. But thanks for the reminder! Actually, Lars made some
changes in the evaluation engine recently that may have fixed the bad behavior
in 3.24.0. Or it should at least be strongly related.

> Because of the issue, I am still trapped in CFEngine 3.7.5.
> I just checked, isipinsubnet() is not available for me.
> I am cleaning up my plocies right now and really look forward to jumping into 3.24 LTS!