Reading, parsing, and using data from a JSON file

[Kermit Short]

Greetings!

   I've got a JSON file in the following format:
{
  "groupa": {
              "gid": "1000",
              "members": [ "usera", "userb" ]
            }
}

I'm using the following bundles to try and parse this file and add the groups to the group file if they don't already exist:

bundle agent legacygroups
{
  vars:
    "jsondat"   string => "$(sys.workdir)/files/legacy_tidal_groups.json";
    "datagrp"   data => readjson("$(legacygroups.jsondat)", 10000);
    "grpkeys"    slist => getindices("datagrp");
    "lstgroups" slist => sort(grpkeys, "lex");

  methods:
    "addgroup"
      usebundle => addlclgrp(@(legacygroups.datagrp), $(legacygroups.lstgroups));
}

bundle agent addlclgrp(datagrp, strgrp)
{
  vars:
    "members"     slist => getvalues("datagrp[$(strgrp)][members]");
#   Miserable attempt at typecasting the slist to a string to avoid problems
#   in the subsequent format() concatenation.
    "strgid"      slist => getvalues("datagrp[$(strgrp)][gid]");
    "gid"         string => $(strgid);

    "strmembers"  string => join(",", $(members));
    "arygrp[$(strgrp)]" string => format("%s:x:%s:%s", "$(strgrp)", "$(gid)",
                                   "$(strmembers)");
 
  files:
    "/etc/group"
      edit_line => append_groups_starting ($(addlclgrp.arygrp));
}

There are two items of note.  Some of the members sections in the JSON file will be simply "".   Is there a better way to specify a null value?

I'm getting an error from the join function:
Function 'join', argument 'members' did not resolve to a variable

There are likely other syntax or logic errors that I haven't yet discovered.  Any suggestions are greatly appreciated!  I've been pounding on this policy file for 3 days.  It's winning so far, and I'm fairly numb to the code at this point.

Thanks!
-Kermit Short, RHCE, RHCVA
Sr. Solutions Architect
Los Alamos National Laboratory

[bishwa SHRESTHA]

Hi,

"strgid"      slist => getvalues("datagrp[$(strgrp)][gid]");

 

should be:

"strgid"      string => "$(datagrp[$(strgrp)][gid])");

Can you also share the: "append_groups_starting" bundle? I am guessing it should be:

edit_line => append_groups_starting ($(addlclgrp.arygrp[$(strgrp)]));

-bishwa

On Fri, Nov 21, 2014 at 12:08 AM, Kermit Short <heral...@gmail.com> wrote:

Greetings!

   I've got a JSON file in the following format:
{
  "groupa": {
              "gid": "1000",
              "members": [ "usera", "userb" ]
            }
}

I'm using the following bundles to try and parse this file and add the groups to the group file if they don't already exist:

bundle agent legacygroups
{
  vars:
    "jsondat"   string => "$(sys.workdir)/files/legacy_tidal_groups.json";
    "datagrp"   data => readjson("$(legacygroups.jsondat)", 10000);
    "grpkeys"    slist => getindices("datagrp");
    "lstgroups" slist => sort(grpkeys, "lex");

  methods:
    "addgroup"
      usebundle => addlclgrp(@(legacygroups.datagrp), $(legacygroups.lstgroups));
}

bundle agent addlclgrp(datagrp, strgrp)
{

  varFlailings:

    "members"     slist => getvalues("datagrp[$(strgrp)][members]");
#   Miserable attempt at typecasting the slist to a string to avoid problems
#   in the subsequent format() concatenation.
    "strgid"      slist => getvalues("datagrp[$(strgrp)][gid]");
    "gid"         string => $(strgid);

    "strmembers"  string => join(",", $(members));
    "arygrp[$(strgrp)]" string => format("%s:x:%s:%s", "$(strgrp)", "$(gid)",
                                   "$(strmembers)");
 
  files:
    "/etc/group"
      edit_line => append_groups_starting ($(addlclgrp.arygrp));
}

There are two items of note.  Some of the members sections in the JSON file will be simply "".   Is there a better way to specify a null value?

I'm getting an error from the join function:
Function 'join', argument 'members' did not resolve to a variable

There are likely other syntax or logic errors that I haven't yet discovered.  Any suggestions are greatly appreciated!  I've been pounding on this policy file for 3 days.  It's winning so far, and I'm fairly numb to the code at this point.

Thanks!
-Kermit Short, RHCE, RHCVA
Sr. Solutions Architect
Los Alamos National Laboratory

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Ted Zlatanov]

OK, below is a working example (the JSON is inline, otherwise same idea
as yours, with longer bundle and variable names).

With the latest master, arrays seem to be cleared on bundle entry, so
you can't just use the "ary" array (this also seems to break the
standard library's `cmerge()` utility bundle). It works in 3.6.2 though,
so I think it's due to the resolution of https://dev.cfengine.com/issues/6369

Instead you have to use `variablesmatching()` which is not as clean.
Perhaps that issue will be addressed before the next release. I updated
https://dev.cfengine.com/issues/6369 with the example and my notes.

Ted

#+begin_src cfengine3
body common control
{
bundlesequence => { "legacy" };
inputs => { "$(sys.libdir)/stdlib.cf" };
}

bundle agent legacy
{
vars:
"groups" data => parsejson('{

"groupa": {
"gid": "1000",
"members": [ "usera", "userb" ]

},
"groupb": {
"gid": "1001",
"members": [ "userc", "userd" ]
}
}');
"unsorted_keys" slist => getindices("groups");
"keys" slist => sort(unsorted_keys, "lex");

methods:
"addgroup"
usebundle => add_legacy_group(@(groups), $(keys));

"report"
usebundle => report_legacy_groups();
}

bundle agent add_legacy_group(dg, group)
{
vars:
"members" slist => getvalues("dg[$(group)][members]");
"gid" string => "$(dg[$(group)][gid])";

"strmembers" string => join(",", members);

"created_$(group)"
meta => { "groupcreation" },
string => format("%s:x:%s:%s", $(group), $(gid), $(strmembers));

"ary[$(group)]" string => "$(created_$(group))";

reports:
"$(this.bundle): collecting data for $(group): members $(strmembers), GID $(gid)";
"$(this.bundle): collected data for $(group): $(created_$(group))";
}

bundle agent report_legacy_groups()
{
vars:
"created" slist => variablesmatching(".*", "groupcreation");
"collected[$(created)]" string => "$($(created))";
"todo" data => mergedata(collected);
"todo_s" string => format("%S", todo);

"ary_import" data => mergedata("add_legacy_group.ary");
"ary_import_s" string => format("%S", ary_import);

reports:
"$(this.bundle): $(created) = $(collected[$(created)])";
"$(this.bundle): as JSON: $(todo_s)";
"$(this.bundle): broken ary import as JSON: $(ary_import_s)";
}
#+end_src

Output with latest master:

#+begin_src text
R: add_legacy_group: collecting data for groupa: members usera,userb, GID 1000
R: add_legacy_group: collected data for groupa: groupa:x:1000:usera,userb
R: add_legacy_group: collecting data for groupb: members userc,userd, GID 1001
R: add_legacy_group: collected data for groupb: groupb:x:1001:userc,userd
R: report_legacy_groups: default:add_legacy_group.created_groupb = groupb:x:1001:userc,userd
R: report_legacy_groups: default:add_legacy_group.created_groupa = groupa:x:1000:usera,userb
R: report_legacy_groups: as JSON: {"default:add_legacy_group.created_groupa":"groupa:x:1000:usera,userb","default:add_legacy_group.created_groupb":"groupb:x:1001:userc,userd"}
R: report_legacy_groups: broken ary import as JSON: {"groupb":"groupb:x:1001:userc,userd"}
#+end_src

[Kermit Short]

Thanks to Bishra and Ted!

Verbose output from cf-agent indicates that the agent is still insulted by my policy code:
014-11-24T15:15:24-0700  verbose: /default/addlclgrp/vars: Function 'join', argument 'members' did not resolve to a variable.

Is this due to the array clearing that Ted was talking about?  I'm on Cfengine Community version 3.6.2.

The other Issue I have now is that when I look in my group file, instead of seeing a bunch of groups from my JSON file, I see:
$(grpstring)

Here is the updated code for the bundles:

bundle agent legacygroups
{
  vars:
    "jsondat"   string => "$(sys.workdir)/files/legacy_tidal_groups.json";
    "datagrp"   data => readjson("$(legacygroups.jsondat)", 10000);

    "lstgroups"    slist => getindices("datagrp");
    "grpkeys" slist => sort(lstgroups, "lex");

  methods:
    "addgroup"
      usebundle => addlclgrp(@(datagrp), $(grpkeys));
}

bundle agent addlclgrp(datagrp, strgrp)
{
  vars:
    "members"     slist => getvalues("datagrp[$(strgrp)][members]");
    "gid"         string => "$(datagrp[$(strgrp)][gid])";

    "strmembers"  string => join(",", members);

    "grpstring"   string => format("%s:x:%s:%s", $(strgroup), $(gid), $(strmembers));
    "arygrp[$(strgrp)]" string => "$(grpstring)";
 
  files:
    "/etc/group"
      edit_line => append_groups_starting ("addlclgrp.arygrp");

It looks like the $(grpstring) entry is coming from where I try to add it to the arygrp in the vars: section off the addlclgrp bundle.  Is there a way I can make sure that the grpstring variable gets expanded when I set the arygrp[$(strgrp)] element?

I can't tell if the bundles are iterating, because seemingly the promise is to add $(grpstring) to the file /etc/group, and once that has been done once, the promise has been fulfilled and no further entries will be added.

Just as an example, here's what my group file looks like (real group identities have been removed to protect the innocent):
<snip>
groupa:x:1000:usera
[...]
groupb:x:1001:usera,userb
$(grpstring)
</snip>

[Kermit Short]

Is there a way to add a bundle namespace to this reference to members in the join function?  I've tried using addlclgrp.members instead of members but cf-promises complains about the '.' so obviously that's not the correct syntax.

On Monday, November 24, 2014 3:20:45 PM UTC-7, Kermit Short wrote:

Verbose output from cf-agent indicates that the agent is still insulted by my policy code:
014-11-24T15:15:24-0700  verbose: /default/addlclgrp/vars: Function 'join', argument 'members' did not resolve to a variable.

bundle agent addlclgrp(datagrp, strgrp)
[...]

  vars:
    "members"     slist => getvalues("datagrp[$(strgrp)][members]");
    "gid"         string => "$(datagrp[$(strgrp)][gid])";
    "strmembers"  string => join(",", members);
    "grpstring"   string => format("%s:x:%s:%s", $(strgroup), $(gid), $(strmembers));
    "arygrp[$(strgrp)]" string => "$(grpstring)";

 

[...] 

[Kermit Short]

Well, I need to get this policy in production.  I can't figure out how to get the data added to the group file using JSON, which is apparently preferred instead of using arrays of strings (deprecated).  I will simply have to revert to the brute force method of appending a text file to the group file.  Thanks to Ted and Bishwa for their suggestions.  I'd like to revisit this, but for now I simply don't have the time to experiment.

[Kermit Short]

So once I got this working using the brute force method, I was fairly bothered that I couldn't figure out why this wasn't working.

Many days of testing and googling, I've figured it out.  Many thanks to Diego Zamboni, his book Learning Cfengine 3, and most specifically, this blog post:

How to Pass Arguments to Bundles Using Arrays

Turns out I was actually working with strings (names and names+indexes of variables to be precise) instead of values within expanded variables.  Diego refers to "dereferencing" in his blog, and that was exactly what I needed to do.  If it helps anyone as confused as I was, you handle data containers much like you handle arrays, and you have to use this "dereferencing" trick to be able to do that if you're passing these containers around from bundle to bundle.

On Friday, November 21, 2014 6:56:18 AM UTC-7, Ted Zlatanov wrote: