JSON optional keys - difference behaviour 3.7.3 - 3.10.0

[S]

Hi,

We were considering upgrading from 3.7.3 to 3.10.0 but ran into an issue when taking input from json files

We have the following policy to create users based on data stored in a json file.

---

bundle agent _create_all_users(conf) {
  classes:
    "users_conf_exists" expression => fileexists("$(conf)");

  vars:
    users_conf_exists::
      "data" data => readjson("$(conf)", 1m);
      "i" slist => getindices("data");

  methods:

    users_conf_exists::
      "" usebundle => _create_user(
        "$(data[$(i)][user])",
        "$(data[$(i)][uid])",
        "$(data[$(i)][home_dir])",
        "$(data[$(i)][shell])",
        "$(data[$(i)][description])",
        "$(data[$(i)][gid])",
        "$(data[$(i)][password])",
       
        "$(data[$(i)][last_change])",
        "$(data[$(i)][min_age])",
        "$(data[$(i)][max_age])",
        "$(data[$(i)][warn_age])",

        "$(data[$(i)][inactive_age])",
        "$(data[$(i)][expire_age])"
      ),
     ifvarclass => "$(data[$(i)][class])";

  reports:
    "DEBUG $(data[$(i)][user]) $(data[$(i)][last_change]) $(data[$(i)][min_age]) $(data[$(i)][max_age]) $(data[$(i)][warn_age])";
}

bundle agent _create_user(user, uid, home_dir, shell, description, gid, password, last_change, min_age, max_age, warn_age, inactive_age, expire_age)
{
  defaults:
    any::
      "last_change"  string => "",         if_match_regex => "$(empty_or_undef)";
      "min_age"      string => "",         if_match_regex => "$(empty_or_undef)";
      "max_age"      string => "",         if_match_regex => "$(empty_or_undef)";
      "warn_age"     string => "",         if_match_regex => "$(empty_or_undef)";
      "inactive_age" string => "",         if_match_regex => "$(empty_or_undef)";
      "expire_age"   string => "",         if_match_regex => "$(empty_or_undef)";
    solaris::
      "password"  string => "NP",       if_match_regex => "$(empty_or_undef)";
    linux::
      "password"  string => "!!",       if_match_regex => "$(empty_or_undef)";
 
  vars:
    "empty_or_undef"  string => "^$|\$\(.*|\@\(.*"; # empty string or $... or @... undef variable

    "passwd" slist => { "$(user)", "x", "$(uid)", "$(gid)", "$(description)", "$(home_dir)", "$(shell)" };
    "shadow" slist => { "$(user)", "$(password)", "$(last_change)", "$(min_age)", "$(max_age)", "$(warn_age)", "$(inactive_age)", "$(expire_age)", "" };

    "passwd_str" string => join(":", "passwd");
    "shadow_str" string => join(":", "shadow");
   
  files:
    # do magic here; not relevant to inquiry
}

---

And here is a snippet of the JSON file which it takes as input


---
    {
        "comment": "SERVDESK",
        "class": "linux",
        "user": "servdesk",
        "uid": "3536",
        "gid": "2413",
        "description": "SERVICE DESK",
        "home_dir": "/home/user/servdesk",
        "shell": "/bin/ksh",
        "password": "xxx",
        "last_change": "14776"
    },

---

As you can see, not all keys are necessarily defined in the JSON file (eg. inactive_age).

In 3.7.3, the behaviour would be that for a missing JSON key, the content of the variable would be set to the variable name.

R: DEBUG servdesk 14776 $(data[15][min_age]) $(data[15][max_age]) $(data[15][warn_age])

We worked around that by using the defaults section in _create_user and reset the var to a "" string

In 3.10 however, any JSON entry with missing keys is just skipped; _create_user will not be called

How can we deal with optional keys when taking input from JSON data files?

Gregory

[Ted Zlatanov]

On Wed, 15 Mar 2017 05:05:30 -0700 (PDT) S <saw...@gmail.com> wrote:

S> How can we deal with missing or optional keyswhen taking input from JSON
S> data files?

For your specific example (user creation) where sensible defaults are
known, I would simply merge the input on top of a defaults data
container. Note you can have multiple arguments to mergedata() so it can
merge more than 2 containers.

Otherwise, I would use isvariable() to build more complex logic.

Ted

#+begin_src cfengine3
bundle agent main
{
methods:
"test";

vars:
"test_state" data => bundlestate(test);
"test_string" string => storejson(test_state);

reports:
"$(this.bundle): state of things = $(test_string)";
}

bundle agent test
{
vars:
"defaults" data => '{ "office location": "unknown" }';
"me" data => mergedata(defaults, '{ "name": "Ted" }');
"carol" data => mergedata(defaults, '{ "office location": "Main office", "name": "Carol" }');
}

#+end_src

Output:

#+begin_src text
% cf-agent -KI ./test_mergedata_defaults.cf
R: main: state of things = {
"carol": {
"name": "Carol",
"office location": "Main office"
},
"defaults": {
"office location": "unknown"
},
"me": {
"name": "Ted",
"office location": "unknown"
}
}
#+end_src

[S]

On Wednesday, March 15, 2017 at 1:53:01 PM UTC+1, Ted Zlatanov wrote:

        For your specific example (user creation) where sensible defaults are
        known, I would simply merge the input on top of a defaults data
        container. Note you can have multiple arguments to mergedata() so it can
        merge more than 2 containers.

Hi Ted,

Thanks for the reponse. mergedata looks interesting - I have some trouble seeing how we could
apply it to JSON array.

So we would have these two data structures:


{
    "office location" : "unknown",
}

and we'd have to apply it to any element in this array

[
    {
        "name": "Carold",

        "office location: "Main office"
    },

    {
        "name": "Ted",
    }

]

To end up with this:

[
    {
        "name": "Carold",

        "office location: "Main office"
    },

    {

        "name": "Ted",
        "office location: "unknown"
    }

]

which we then pass along to another bundle

[Ted Zlatanov]

1) Read your data from the file. Make list of indices.

"data" data => readjson("$(conf)", 1m);
"i" slist => getindices("data");

2) Pass your data and the index to the bundle

users_conf_exists::
"" usebundle => _create_user(@(data), $(i));

3) inside the _create_user bundle, merge the defaults (wherever they
come from) with the data from the caller

bundle agent _create_user(udata, index)
...
"user" data => mergedata(defaults, "udata[$(index)]");

4) now in _create_user you can use elements of udata, like `$(udata[uid])`

Ted

[S]

That worked nicely - thank you, Ted. :)