Converting associative array to list(data)/json list

[Christoffer Frost]

Hi Guys 

I seek some guidance or advice in my policy writing.

So I'm trying to create a json object from a two dimensional associative array that i can use in a mustache template. Problem is that converting this associative array using mergadata creates a nested json object that can not be iterated over within the mustache syntax. So this is what i get:

{
 "val1":{"name":"val1","key1":"val1_1","key2":"val1_2"..},
 "val2":{"name":"val2","key1":"val2_1","key2":"val2_2"..}
 ...
}

And what I want instead is a list of json objects: 

[
 {"name":"val1","key1":"val1_1","key2":"val1_2"..},
 {"name":"val2","key1":"val2_1","key2":"val2_2..}
 ...
]

Ideally Mustache should be able to handle iterating over nested json. But since it's not we gotta' try make CFEngine fix it. 

One way to achieve this could be to convert the two dimensional associative array to an list of single dimensioned associative arrays that i then can convert to json.

So hey! getvalues exists, so we can try that on the array:

bundle agent main {

  vars:

      "list" slist => {"1 - one", "2 - two", "3 - three"};

      "asso_array[$(list)][name]" string => "$(list)";

      "asso_array[$(list)][key_1]" string => "val_1 $(list)";

      "asso_array[$(list)][key_2]" string => "val_2 $(list)";

      "values" slist => getvalues("asso_array");

      "indices" slist => getindices("asso_array");

      "data" data => mergedata(asso_array);

      "values_str" string => format("%S", 'values');

      "indices_str" string => format("%S", 'indices');

      "data_str" string => format("%S", 'data');

  reports:

      "Values: $(values_str)";

      "Indices: $(indices_str)";

      "Data: $(data_str)";

}

# cf-agent -KIf ./test_help.cf

R: Values: { --empty-list-- }

R: Indices: { "3 - three", "1 - one", "2 - two" }

R: Data: {"1 - one":{"key_1":"val_1 1 - one","key_2":"val_2 1 - one","name":"1 - one"},"2 - two":{"key_1":"val_1 2 - two","key_2":"val_2 2 - two","name":"2 - two"},"3 - three":{"key_1":"val_1 3 - three","key_2":"val_2 3 - three","name":"3 - three"}}

As you see here this isn't really the case. getvalues just returns an empty list since the return value isn't a slist. This may have worked if getvalues allowed data as a return value, (Feature request?). 

I did come up with a solutions. Create the associative array as a json string, Collect all the entry strings in a slist. Join them with a comma(", "). Parse the json. But all this feels hacky and no way near the correct way to do it.:

bundle agent main {

  vars:

      "list" slist => {"1 - one", "2 - two", "3 - three"};

      "json[$(list)]" string => '{"name": "$(list)", "key_1":"val_1 $(list)", "key_2":"val_2 $(list)"}';

      "values" slist => getvalues("json");

      "json_joined" string => join(", ", values);

      "json_data" data => parsejson('{"packs": [ $(json_joined) ]}');

      "json_data_str" string => format("%S", 'json_data');

  reports:

      "Json Data: $(json_data_str)";

}

# cf-agent -KIf ./test_help.cf

R: Json Data: {"packs":[{"key_1":"val_1 2 - two","key_2":"val_2 2 - two","name":"2 - two"},{"key_1":"val_1 1 - one","key_2":"val_2 1 - one","name":"1 - one"},{"key_1":"val_1 3 - three","key_2":"val_2 3 - three","name":"3 - three"}]}

Do any of you guys have a better solution to this problem ?

/Christoffer

[Ted Zlatanov]

Hi Christoffer,

you can indeed iterate over a top map in Mustache in CFEngine. I
included that in the example below. That's very similar to the example I
posted for Robin just yesterday. Here I used the custom CFEngine
extensions to Mustache: `-top-` pointing to the top object, and `@`
pointing to the current iteration key.

I also included an example of using mapdata("json_pipe") calling `jq`
externally to convert a map to an array of arrays. It's easy to adapt
the example to whatever you need specifically, but maybe the Mustache
iteration is all you need.

HTH
Ted

#+begin_src cfengine3
bundle agent main
{
vars:
"map" data => '
{
"val1":{"name":"val1","key1":"val1_1","key2":"val1_2"},

"val2":{"name":"val2","key1":"val2_1","key2":"val2_2"}

}';
"output" string => string_mustache('
{{#-top-}}
key is {{{@}}}

name is {{{name}}}
{{/-top-}}
', map);

"map_as_array" data => mapdata("json_pipe", '$(def.jq) "to_entries|.[]|[.key, .value]"', map);

"map_as_array_str" string => format("%S", map_as_array);
reports:
"$(this.bundle): output = $(output)";
"$(this.bundle): map as array = $(map_as_array_str)";

}
#+end_src

#+begin_src text
% cf-agent -KI -f ./test_mustache_iteration2.cf
R: main: output =
key is val1

name is val1
key is val2

name is val2

R: main: map as array = [["val1",{"key1":"val1_1","key2":"val1_2","name":"val1"}],["val2",{"key1":"val2_1","key2":"val2_2","name":"val2"}]]
#+end_src

[Christoffer Frost]

Hi Ted

Thanks for the quick answer. 

I can see that the json_pipe is only available in 3.9, and as i am running 3.7. So this is unfortunately unavailable to, though i will upgrade to 3.10 as soon as it is launched. But great to see this feature, as also tried to find a way to filter json data.

Regarding the "#-top-" tag in mustache, I can't seem to find that one in the documentation. Do you know if all the modifications to mustache are documented anywhere?

/Christoffer

[Ted Zlatanov]

On Wed, 16 Nov 2016 07:25:25 -0800 (PST) Christoffer Frost <christof...@gmail.com> wrote:

CF> I can see that the json_pipe is only available in 3.9, and as i am running
CF> 3.7. So this is unfortunately unavailable to, though i will upgrade to 3.10
CF> as soon as it is launched. But great to see this feature, as also tried to
CF> find a way to filter json data.

Great!

CF> Regarding the "#-top-" tag in mustache, I can't seem to find that one in
CF> the documentation. Do you know if all the modifications to mustache are
CF> documented anywhere?

-top- was added in 3.9.
@ was added in 3.7.

They are documented here:
https://docs.cfengine.com/docs/3.10/reference-promise-types-files.html#edit_template

It may make sense to move the Mustache and JSON/data container documents
to the top level of the reference tree.

Ted

[Nick Anderson]

Christoffer Frost writes:

> 3.7. So this is unfortunately unavailable to, though i will upgrade to 3.10
> as soon as it is launched. But great to see this feature, as also tried to

Hey Christoffer,

Note we released the 3.10 beta recently.

https://cfengine.com/company/blog-detail/cfengine-3-10-0-lts-beta-ready-for-testing/

It would be great if you can test it with your policies and give us
feedback. We expect to release 3.10 final before end of December.


--
Nick Anderson
Doer of things @ CFEngine
https://cfengine.com