Merge data containers with arrays

[Matthias Rieber]

Hi,

I'd like to merge data containers where

{"zoneA": [ "dnsRecord1", ... ], ...} merged with
{"zoneA"; [ "dnsRecord2", ... ], ...} produces

{"zoneA": [ "dnsRecord1", "dnsRecord2", ... ], ...}


Using mergedata() does work probably like expected, but not as desired :)
It merges the data containers, but the last key 'wins' and overwrites the
previous array:

{"zoneA": [ "dnsRecord2", ...], ...} is the result.


Then I've tried to merge the zones individually, which basically works:

body common control
{
bundlesequence => { start };
}

bundle agent start
{
vars:
any::
"dns_vars" slist => variablesmatching(".*", "dns");

"zones" data => parsejson('[]');
"zones" data => mergedata(zones, getindices("$(dns_vars)"));
"zones" slist => unique("zones");

"z_$(zones)" data => parsejson('[]');
"z_$(zones)" data => mergedata("z_$(zones)", "$(dns_vars)[$(zones)]");
"z_str_$(zones)" string => format("%S", "z_$(zones)");
"dns" data => parsejson('{}');
"dns" data => mergedata("dns", '{ "$(zones)": z_$(zones) }');
"dns_str" string => format("%S", "dns");

reports:
any::
"Zones: $(zones)";
"Test: $(zones) => $(z_str_$(zones))";
"DNS Records: $(dns_str)";
}

bundle agent app_foo
{
vars:
"dns" data => parsejson('{
"zoneA": [
"entry1A",
"entry2A"
],
"zoneB": [
"entry1B",
"entry2B"
]}'),
meta => {"dns"};
}

bundle agent app_bar
{
vars:
"dns" data => parsejson('{
"zoneB": [
"entry3B",
"entry4B"
],
"zoneC": [
"entry1C",
"entry2C"
]}'),
meta => {"dns"};
}

However zoneX should be in my case domain names, which contains characters
which are not valid as variable names.

Can this solution be fixed to work with arbitrary zone names or is there
another solution.

Best regards,

Matthias

[Ted Zlatanov]

On Thu, 4 May 2017 21:35:52 +0200 (CEST) Matthias Rieber <matt...@zu-con.org> wrote:

MR> I'd like to merge data containers where

MR> {"zoneA": [ "dnsRecord1", ... ], ...} merged with
MR> {"zoneA"; [ "dnsRecord2", ... ], ...} produces

MR> {"zoneA": [ "dnsRecord1", "dnsRecord2", ... ], ...}


MR> Using mergedata() does work probably like expected, but not as desired :)
MR> It merges the data containers, but the last key 'wins' and overwrites the
MR> previous array:

MR> {"zoneA": [ "dnsRecord2", ...], ...} is the result.

Yes, this is as designed, mergedata() is simple and doesn't surprise the
user. I think you want the mapdata() "jq" functionality, which requires
jq itself to be installed.

Here's an example showing a one-level merge of arrays under the same
key. The only tricky thing from the CFEngine side is that we build a
temporary array with a and b in positions 0 and 1, and then use that in
the jq invocation.

HTH
Ted

#+begin_src cfengine3
bundle agent main
{
methods:
"test";

vars:
"test_state" data => bundlestate(test);
"test_string" string => storejson(test_state);

reports:
"$(this.bundle): state of things = $(test_string)";
}

bundle agent test
{
vars:
"a" data => '{ "key_array": [ 1, 2, 3 ] }';
"b" data => '{ "key_array": [ 4, 5, 6 ] }';

"jq_merge_arrays" data => mapdata("json_pipe", "$(def.jq) 'map(to_entries | add)
| group_by(.key)
| map({ (.[0].key): map(.value) | add })'", '[a, b]');
}
#+end_src

#+begin_src text
% cf-agent -KI -f ./test_mergedata_jq.cf
R: main: state of things = {
"a": {
"key_array": [
1,
2,
3
]
},
"b": {
"key_array": [
4,
5,
6
]
},
"jq_merge_arrays": [
[
{
"key_array": [
1,
2,
3,
4,
5,
6
]
}
]
]
}
#+end_src