List values from data containers

[Marco Marongiu]

Hello there!

I have been trying to brush some rust away from my CFEngine knowledge lately, and I've stumbled on a subject I had not become friends with: data containers. It looks to me that lists values behave like standard slists, but not quite, and I need your help.

I have a JSON file like this:

{
    "azure-cli" : {
        "sources" : ["deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ ${inventory_lsb.codename} main"],
        "key" : "https://packages.microsoft.com/keys/microsoft.asc",
        "packages" : [
            "ca-certificates",
            "curl",
            "apt-transport-https",
            "lsb-release",
            "gnupg",
            "azure-cli"
        ],
        "trigger" : "install_azure_cli"
    },
...

Basically, to each key I associate an object where the keys "sources" and "packages" are lists.

Now, if I iterate over those lists value in a string context, thus triggering implicit iteration, it works just fine, e.g. something like this works as expected:

  vars:

      "sw" data => readjson("software.json") ;

      "swlist" slist => getindices("sw") ;

  reports:

      "${swname} packages: ${sw[${swname}][packages]}" ;

However, if I try to use the whole list as a parameter in a methods promise, I get a giant middle finger back. Something like this in bundle agent main:

      # Ensure apt keys and sources
      "ensure_${swname}_repo"
        usebundle => add_apt_repo(
                                   "${swname}",
                                   "@{main.sw[${swname}][sources]}",
                                   "${sw[${swname}][key]}",
                                   "${henc.apt_keys_cachedir}",
                                   "${refresh_apt_keys}",
                                   "refresh_apt_cache"
        ),
        if => "${sw[${swname}][trigger]}",
        classes => classes_generic("${sw[${swname}][trigger]}_repo") ;

      "install_${swname}"
        usebundle => package_present("@{main.sw[$swname][packages]}"),
        if => "${sw[${swname}][trigger]}",
        classes => classes_generic("${sw[${swname}][trigger]}") ;

triggers this error:

    R: [add_local_apt_key/files KEPT] Check apt key /var/cfengine/cache/apt-keys/azure-cli.gpg for changes
       error: List or container parameter 'sources' not found while constructing scope 'add_apt_sourcefile' - use @(scope.variable) in calling reference
       error: List or container parameter 'add_apt_sourcefile.repos_list' not found while constructing scope 'insert_lines' - use @(scope.variable) in calling reference
    R: [add_apt_sourcefile/files KEPT] Ensure apt source azure-cli in /etc/apt/sources.list.d/azure-cli.list
       error: List or container parameter 'main.sw[$swname][packages]' not found while constructing scope 'package_present' - use @(scope.variable) in calling reference
        info: Installing @{main.sw[$swname][packages]}...
        info: Q:env DEBIAN_FRONTEND= ...:E: Malformed line 1 in source list /etc/apt/sources.list.d/azure-cli.list (type)
        info: Q:env DEBIAN_FRONTEND= ...:E: The list of sources could not be read.
        info: Q:env DEBIAN_FRONTEND= ...:E: Malformed line 1 in source list /etc/apt/sources.list.d/azure-cli.list (type)
        info: Q:env DEBIAN_FRONTEND= ...:E: The list of sources could not be read.
       error: Finished command related to promiser '@{main.sw[$swname][packages]}' -- an error occurred, returned 100
       error: Bulk package schedule execution failed somewhere - unknown outcome for '@{main.sw[$swname][packages]}'
    R: [package_present/methods REPD] Ensure azure-cli is installed

The list is not dereferenced, the apt source gets corrupted and everything breaks.

I can see this is about dereferencing the list from the data container, but I am not sure how I should fix it to make it work. Using CFEngine 3.15.

Any help is appreciated!

Thanks, ciao!

-- bronto

[Mike Weilgart]

I believe what you're looking for is one of the functions "maparray()" or "getvalues()".  Mind you, I haven't tested revising your code, but one of those should be what you need to convert a data container (or sub-container) into a bona-fide CFEngine slist.

Let me know if that gets you unstuck?

Documentation links:

- https://docs.cfengine.com/docs/3.15/reference-functions-getvalues.html

- https://docs.cfengine.com/docs/3.15/reference-functions-maparray.html

Best,
—Mike Weilgart
Vertical Sysadmin, Inc.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/CAMG599cSZ5QwCJ4jbXPHQ0j1YBGwn3%3D4URufHmDTas3dbNogAg%40mail.gmail.com.

[Marco Marongiu]

Thanks Mike

So, with getvalues() used like this:

      "sw"
        comment => "Read information about regular software",
        data => data_expand(readjson("${sys.libdir}/local/software.json")) ;

      "swname"
        comment => "List of managed software names",

        slist => getindices("sw") ;

      "packages[${swname}]"
        slist => getvalues("sw[$swname][packages]") ;

I get this error when running in verbose:

 verbose: getvalues: argument 'default:main.sw[$swname][packages]' does not resolve to a container or a list or a CFEngine array

So either I am using getvalues wrong, or it doesn't work.

As for maparray, I am not sure how you are suggesting that I should use it to extract those lists out of the data container. Can you please elaborate?

Thanks

-- bronto

[Nick Anderson]

Can you post a runnable standalone example?

To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/CAMG599d61Lw%3D0Disq97rd3NPvFhkdk-cmTu1%2BXiGn7wDavS71A%40mail.gmail.com.

[Marco Marongiu]

Hi Nick

Sure, here you go. Feel free to place the json file wherever you see fit, but update the readjson call in test.cf.

When I run this on a test machine I get:

# cf-agent -KI -f /var/cfengine/inputs/services/test.cf -b test -C
    info: Using command line specified bundlesequence
   error: List or container parameter 'sw[${swname}][packages]' not found while constructing scope 'report' - use @(scope.variable) in calling reference
R: report: list value: @{sw[${swname}][packages]}
R: test: vivaldi package: vivaldi-stable
R: test: azure-cli package: ca-certificates
R: test: azure-cli package: curl
R: test: azure-cli package: apt-transport-https
R: test: azure-cli package: lsb-release
R: test: azure-cli package: gnupg
R: test: azure-cli package: azure-cli
R: test: dotnet package: apt-transport-https
R: test: dotnet package: dotnet-sdk-3.1
R: test: kubectl package: apt-transport-https
R: test: kubectl package: kubectl
# 

The error remains even if I use the fully qualified name (test.sw in this case).

As you can see, I can use the scalar values in the packages list (all but the first report), but as soon as I try to use the entire list for anything, and in particular to pass it to a bundle for processing, the list is not recognised as such.

Looking forward to your findings!

Ciao
-- bronto

[Bas van der Vlies]

Marcoe

`````:
bundle agent main
{
vars:
any::

"sw"
comment => "Read information about regular software",

data => data_expand(readjson("/root/bas/software.json")) ;

"swname"
comment => "List of managed software names",
slist => getindices("sw") ;

methods:
any::
"test" usebundle => report("main.sw[${swname}][packages]") ;

reports:
any::
"${this.bundle}: ${swname} package: ${sw[${swname}][packages]}" ;
}

bundle agent report(name)
{
reports:
"${this.bundle}: $(name) value: $($(name))" ;
}
```

Outpur:
````
R: report: main.sw[vivaldi][packages] value: vivaldi-stable
R: report: main.sw[azure-cli][packages] value: ca-certificates
R: report: main.sw[azure-cli][packages] value: curl
R: report: main.sw[azure-cli][packages] value: apt-transport-https
R: report: main.sw[azure-cli][packages] value: lsb-release
R: report: main.sw[azure-cli][packages] value: gnupg
R: report: main.sw[azure-cli][packages] value: azure-cli
R: report: main.sw[dotnet][packages] value: apt-transport-https
R: report: main.sw[dotnet][packages] value: dotnet-sdk-3.1
R: report: main.sw[kubectl][packages] value: apt-transport-https
R: report: main.sw[kubectl][packages] value: kubectl
R: main: vivaldi package: vivaldi-stable
R: main: azure-cli package: azure-cli
R: main: dotnet package: dotnet-sdk-3.1
R: main: kubectl package: kubectl
```

--
Bas van der Vlies
| Operations, Support & Development | SURFsara | Science Park 140 | 1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surfsara.nl | www.surfsara.nl |

> "sources" : ["deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/${inventory_lsb.codename} main"],

> To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/CAMG599cy18kscTXqarU2yD-v9_0H6M6z84eMQ5fV_VfHSyT8DA%40mail.gmail.com.
> <test.cf><software.json>

[Marco Marongiu]

Hi Bas

Thanks. The problem is, you fixed the example but you didn't solve the problem ;-)

Your fix works because you can decide what to do with the value in bundle agent report. But still you are not addressing the list as a whole and you are not using it as a value. How would you pass the packages list to the package_present bundle from the standard library? Because

usebundle => package_present("main.sw[${swname}][packages]")

that just won't work. How would you fix that?

Thanks, ciao!
-- bronto

[Bas van der Vlies]

Marco,

First of all if you work with data files you know the data type and
what to do with it. Second the with the "@" operator you can not use
$(var) syntax in usebundle. I have hit that wall several times.

regards

On 18/03/2020 21:34, Marco Marongiu wrote:
> Hi Bas
>
> Thanks. The problem is, you fixed the example but you didn't solve
> the problem ;-)
>
> Your fix works because you can decide what to do with the value in
> bundle agent report. But still you are not addressing the list as a
> whole and you are not using it as a value. How would you pass the
> packages list to the package_present bundle from the standard library?
> Because
>
> usebundle => package_present("main.sw[${swname}][packages]")
>
> that just won't work. How would you fix that?
>
> Thanks, ciao!
> -- bronto
>
>
> Il giorno mer 18 mar 2020 alle ore 20:27 Bas van der Vlies

> <bas.van...@surfsara.nl <mailto:bas.van...@surfsara.nl>> ha

> <mailto:bas.van...@surfsara.nl> | www.surfsara.nl
> <http://www.surfsara.nl> |

>
>
>
> > On 18 Mar 2020, at 17:19, Marco Marongiu <bront...@gmail.com

> <mailto:bront...@gmail.com>> wrote:
> >
> > Hi Nick
> > Sure, here you go. Feel free to place the json file wherever you

> see fit, but update the readjson call in test.cf <http://test.cf>.

> > When I run this on a test machine I get:
> >
> > # cf-agent -KI -f /var/cfengine/inputs/services/test.cf

> <http://test.cf> -b test -C

> <mailto:mike.w...@verticalsysadmin.com>> ha scritto:

> > I believe what you're looking for is one of the functions
> "maparray()" or "getvalues()".  Mind you, I haven't tested revising
> your code, but one of those should be what you need to convert a
> data container (or sub-container) into a bona-fide CFEngine slist.
> >
> > Let me know if that gets you unstuck?
> >
> > Documentation links:
> >
> > -
> https://docs.cfengine.com/docs/3.15/reference-functions-getvalues.html
> >
> > -
> https://docs.cfengine.com/docs/3.15/reference-functions-maparray.html
> >
> > Best,
> > —Mike Weilgart
> > Vertical Sysadmin, Inc.
> >
> > On Mar 18, 2020, at 12:47 AM, Marco Marongiu

> <bront...@gmail.com <mailto:bront...@gmail.com>> wrote:
> >
> > Hello there!
> >
> > I have been trying to brush some rust away from my CFEngine
> knowledge lately, and I've stumbled on a subject I had not become
> friends with: data containers. It looks to me that lists values
> behave like standard slists, but not quite, and I need your help.
> >
> > I have a JSON file like this:
> >
> > {
> >     "azure-cli" : {
> >         "sources" : ["deb [arch=amd64]
> https://packages.microsoft.com/repos/azure-cli/${inventory_lsb.codename}

> <https://packages.microsoft.com/repos/azure-cli/$%7Binventory_lsb.codename%7D>

> <mailto:help-cfengine%2Bunsu...@googlegroups.com>.

> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/help-cfengine/CAMG599cSZ5QwCJ4jbXPHQ0j1YBGwn3%3D4URufHmDTas3dbNogAg%40mail.gmail.com.
> >
> >
> > --
> > You received this message because you are subscribed to the
> Google Groups "help-cfengine" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to help-cfengin...@googlegroups.com

> <mailto:help-cfengine%2Bunsu...@googlegroups.com>.

> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/help-cfengine/CAMG599d61Lw%3D0Disq97rd3NPvFhkdk-cmTu1%2BXiGn7wDavS71A%40mail.gmail.com.
> >
> > --
> > You received this message because you are subscribed to the
> Google Groups "help-cfengine" group.
> > To unsubscribe from this group and stop receiving emails from it,
> send an email to help-cfengin...@googlegroups.com

> <mailto:help-cfengine%2Bunsu...@googlegroups.com>.

> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/help-cfengine/CAMG599cy18kscTXqarU2yD-v9_0H6M6z84eMQ5fV_VfHSyT8DA%40mail.gmail.com.

> > <test.cf <http://test.cf>><software.json>
>

[Marco Marongiu]

Il giorno gio 19 mar 2020 alle ore 09:14 Bas van der Vlies <bas.van...@surfsara.nl> ha scritto:

  First of all if you work with data files you know the data type and
what to do with it. Second the with the "@" operator you can not use
$(var) syntax in usebundle. I have hit that wall several times.

 

Yeah, actually that's the point: there must be a way to extract a list value out of a data container and into a normal slist, am I right? I am trying to understand what that way is, but I ran out of options :(

Anyone?

Ciao
-- bronto

[Vratislav Podzimek]

My attempt was this:

bundle agent __main__ {
vars:
"sw" data => readjson("$(this.promise_dirname)/data.json");

"swlist" slist => getindices("sw");

"pkgs[${swlist}]" slist => maparray("$(this)", "${sw[${swlist}][packages]}");

"pkgs_all" slist => getvalues(@(pkgs));

reports:
"${swlist} packages: ${sw[${swlist}][packages]}";
"${pkgs_all}";
}

But it also doesn't work. And I believe that is actually a bug because
in the first reports promise, the each '${sw[${swlist}][packages]}' is
an slist which is properly expanded. Feel free to create a bug ticket
for this at https://tracker.mender.io (under the 'CFE' project).

Thanks,

--
Vratislav

[Marco Marongiu]

Hi Vratislav

Thanks for trying. After your suggestions, I did a number of attempts with maplist or getvalues, and the result is always the same: the function doesn't do what it is supposed to do because it fails to recognise the list from the container as a valid list ("While setting variable 'packages[azure-cli]' in bundle 'test', function 'getvalues' failed - skipping").

I can create a bug report, yes. However, is there any workaround that you or anyone would suggest for this case? As I am very sure that this issue is not going to be addressed any time soon...

I didn't have good memories about data containers from my experience with CFEngine 3.7. This was the type of problems (or bugs, if we want to call them with the right name) that discouraged us from using data containers more than marginally when I worked in Opera.

It is a bit disappointing that some three and a half years have passed since I left that company, and the same problems are there in CFEngine 3.15... One one side, it is said that associative arrays are deprecated and they will be eventually replaced, since data containers and supporting function make them superfluous and redundant. On the other hand, here we go with a clear case where it's pretty clear that they don't actually work, no matter what.

Anyway, any suggestions for workarounds are greatly appreciated!

Ciao
-- bronto

[Marco Marongiu]

Hello all again

I don't know why I am insisting so much on this thing, I guess it's stubbornness or just stupidity :) Anyway, I think I understood where the problem lies. I would appreciate if someone in Northern Tech can have a look and confirm or amend my findings.

I ran a number of different tests, trying to make the thing work. When I felt the battle was lost with data containers, I tried to approach the problem with using a good old multi-level array with the same structure as the data container. E.g., using the module protocol:

@repo[azure-cli][sources]={"deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ ${inventory_lsb.codename} main"}
=repo[azure-cli][key]=https://packages.microsoft.com/keys/microsoft.asc
@repo[azure-cli][packages]={"ca-certificates", "curl", "apt-transport-https", "lsb-release", "gnupg", "azure-cli"}
=repo[azure-cli][trigger]=install_azure_cli

Running cf-agent with different levels of verbosity (-I and -v) and with the help of --show-evaluated-vars I could see that the the values in the repo array were correct. Yet, trying to loop over the keys of repo and calling a bundle in a methods promise with the values in the key didn't work: the list were there, but CFEngine failed to expand them when building the scope of the called bundle, and everything broke. The errors I was seeing were something like:

   error: List or container parameter 'sources' not found while constructing scope 'add_apt_sourcefile' - use @(scope.variable) in calling reference

and, more importantly:

   error: List or container parameter 'henc_software.packages[${swname}]' not found while constructing scope 'package_present' - use @(scope.variable) in calling reference

It took me several changes in the test code and several runs in verbose mode to understand that the ${swname} I was seeing there was not simply referring to the literal parameter that I passed to the bundle: it was actually what got passed into the bundle! I was expecting that when I pass a parameter to a bundle like, say, "@{mybundle.packages[${swname}]}", that CFEngine would expand that ${swname}, build the full name of the element in the packages array (say: @{mybundle.packages[azure-cli]}) and pass it to the bundle. But no, it's not the case.

So, if I wanted this to work, I needed to find a way to pass a plain list to the called bundle, without any reference to variables. This is, in a way, inspired by the suggestion given by Bas: it didn't solve the problem, but it stayed in my mind as something that works.

But, I designed the data structure in the software.json file I posted earlier in order to iterate over the repository information for a number of packages and configure and install only when the trigger class is active. How could I retain the iteration while, at the same time, pass a plain list to the bundle I wanted to run?

That's where the suggestions given by Bas come into play once again. I needed one stage of indirection. Instead of having a main bundle where the data is manipulated and passed to an executor bundle, I needed a "launcher" bundle in the middle: the launcher would receive the name of the data structure (the slist) to expand, then expand it in a plain list and, finally, call the executor bundle.

I put a test together, hit a few more walls, and I finally got something that works. And it works both with arrays and data containers (where I use getvalues to extract a slist, as suggested in this thread -- thanks!)

I will try now to implement it in my "real" policies. Meanwhile, I would appreciate if someone in Northern Tech can check my assumptions and deductions and confirm if they are correct. See also the attached test.cf, which contains some important comments regarding the use of a promise handle.

Ciao
-- bronto

[Vratislav Podzimek]

On Sun, 2020-03-22 at 19:39 +0100, Marco Marongiu wrote:
> Hello all again
>
> I don't know why I am insisting so much on this thing, I guess it's
> stubbornness or just stupidity :) Anyway, I think I understood where
> the problem lies. I would appreciate if someone in Northern Tech can
> have a look and confirm or amend my findings.

There is definitely something buggy in the handling these arrays/lists
as variables. So let me ask you again to create a ticket at
https://tracker.mender.io so that this can be handled as a bug report.
Emails are not a great platform for this. If you can provide a small
isolated reproducer, I believe a fix could come soon. Much sooner in
JIRA than here on the mailing list.

Thanks.

--
Vratislav

[Marco Marongiu]

Here you go: https://tracker.mender.io/browse/CFE-3299

Is it good enough?

I managed to get the issue type wrong though :(

Ciao
-- bronto

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/7abaa57ff20861adfc03b16b73335204cb4aec7e.camel%40northern.tech.

[Vratislav Podzimek]

On Mon, 2020-03-23 at 22:41 +0100, Marco Marongiu wrote:
> Here you go: https://tracker.mender.io/browse/CFE-3299
> Is it good enough?
>
> I managed to get the issue type wrong though :(

Issue type fixed up. :)

Perfect! Thanks!

--
Vratislav

[Vratislav Podzimek]

On Fri, 2020-03-20 at 11:18 +0100, 'Vratislav Podzimek' via help-

cfengine wrote:
> On Thu, 2020-03-19 at 18:22 +0100, Marco Marongiu wrote:
> >
> > Il giorno gio 19 mar 2020 alle ore 09:14 Bas van der Vlies <
> > bas.van...@surfsara.nl> ha scritto:
> > > First of all if you work with data files you know the data type
> > > and
> > > what to do with it. Second the with the "@" operator you can not
> > > use
> > > $(var) syntax in usebundle. I have hit that wall several times.
> > >
> >
> >
> > Yeah, actually that's the point: there must be a way to extract a
> > list value out of a data container and into a normal slist, am I
> > right? I am trying to understand what that way is, but I ran out of
> > options :(
> >
> > Anyone?
> My attempt was this:

OK, my second attempt, that seems to work:

bundle agent __main__ {
vars:
"sw" data => parsejson('{
"sw1": {
"packages": [ "pkg1", "pkg2" ]
},
"sw2": {
"packages": [ "pkg3", "pkg4" ]
},
"sw3": {
"packages": []
}
}');

"swlist" slist => getindices("sw");

# "pkgs[$(swlist)]" slist => @(sw[$(swlist)][packages]);
"pkgs[$(swlist)]" slist => maparray("$(this.v)", "sw[$(swlist)][packages]");

"pkgs_all" slist => getvalues(@(pkgs));

reports:

"$(swlist) packages: $(sw[$(swlist)][packages])";
"$(pkgs_all)";
}

Hope that might help you, Marco.

I believe the commented-out version should also work, but it fails to
parse. So that's another bug on top of the list expansion for 'methods'
promises.

--
Vratislav

[Marco Marongiu]

Mike: I noticed, and in one of my many tests I also fixed it: it still didn't work.

Vratislav: you're still not trying to pass the list to a bundle, which is where things break. Or, maybe I am missing something?

Ciao
-- bronto

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/ec1f8a20795f29d080e7e64a6324d8c2110b3888.camel%40northern.tech.

[Vratislav Podzimek]

On Tue, 2020-03-24 at 21:36 +0100, Marco Marongiu wrote:
> Mike: I noticed, and in one of my many tests I also fixed it: it
> still didn't work.
>
> Vratislav: you're still not trying to pass the list to a bundle,
> which is where things break. Or, maybe I am missing something?

Yes, that's why I replied to my earlier message and backtracked in the
thread. My reply here really focuses on this part of the problem:

> > Yeah, actually that's the point: there must be a way to extract a
> > list value out of a data container and into a normal slist, am I
> > right? I am trying to understand what that way is, but I ran out of
> > options :(

--
Vratislav

[vratislav...@northern.tech]

Just for the record -- PRs fixing this (CFE-3299) were merged today.

--

Vratislav

[Marco Marongiu]

On 08/09/2020 09:55, 'vratislav...@northern.tech' via help-cfengine wrote:

Just for the record -- PRs fixing this (CFE-3299) were merged today.

Great stuff, good job everyone! It's always nice and motivating when a bug you report got fixed!!!

Ciao
-- bronto