Does CFEngine pass a list variable to a bundle by reference -- or by value?

[Aleksey Tsalolikhin]

https://docs.cfengine.com/docs/3.12/reference-promise-types-methods.html says:

a list can be passed as a implicitly iterated scalar and as a reference, while a data variable (a data container) can only be passed by reference

Passing by reference means the called bundle gets the address of the list variable, so the list variable isn't duplicated.

So then why does this CFEngine code:

bundle agent main {

vars:
"my_list"
slist => { "one", "two", "three" };

methods:
"any"
usebundle => worker(@(my_list));

reports:
"$(this.bundle): $(my_list)";

}

bundle agent worker(some_list) {

vars:
"some_list"
slist => { "four", "five" };

reports:
"$(this.bundle): $(some_list)";
}

Why doesn't this result in "four" and "five" being reported by both bundles?

[:~/git/cf3-tutorial/source] master* ± sudo cf-agent -KIC -f ./methods-pass-by-reference.cf
R: worker: four
R: worker: five
R: main: one
R: main: two
R: main: three
[:~/git/cf3-tutorial/source] master* ±

Is the documentation in error?  Or is my understanding off?

It looks like we're passing by value, not by reference.  (https://www.educative.io/edpresso/pass-by-value-vs-pass-by-reference has a pretty explanation of the two.)

Best,

Aleksey

-- 

Founder

Vertical Sysadmin, Inc.

Achieve real learning.

[Mike Weilgart]

I'm pretty sure the vars promise inside of the worker bundle is *overwriting* the value being passed in.  You can't use a parameter name as a local variable (vars promise) *also*; you have to do one or the other: that's why you're only getting "four, five" from the worker bundle.

As for value vs. reference, I believe that's in reference (sorry) to how some CFEngine functions can take the NAME of a variable as an argument.  This is passing by reference.  When you use ${varname} or @(varname), that's passing by value.  There's no way to pass a data container by value; the only way is to pass the name of the data container.

Best,
—Mike Weilgart

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/CANNWuVV3fyYLyysfzvKWN8H4G9YwmOWm%2BBQJZ1G%2BN_XbvyjDnQ%40mail.gmail.com.

[Ted Zlatanov]

On Tue, 9 Feb 2021 20:33:03 -0800 Mike Weilgart <mike.w...@verticalsysadmin.com> wrote:

MW> There's no way to pass a data container by value; the only way is to pass the name of the data container.

I can suggest one way to do it, though it's slow and limited by string
sizes of course. I hope this is useful.

Ted

#+BEGIN_SRC cfengine3
bundle agent main
{
vars:
"fromabove" data => '{ "this data": "will be passed by value" }';

methods:
"" usebundle => byvalue(storejson(fromabove));
}

bundle agent byvalue(contents)
{
vars:
"passed" data => parsejson($(contents));

reports:
"$(this.bundle): got passed contents $(with)" with => storejson(passed);
}
#+END_SRC

[Nick Anderson]

Ted Zlatanov writes:

MW> There's no way to pass a data container by value; the only way is to pass the name of the data container.

I can suggest one way to do it, though it's slow and limited by string sizes of course. I hope this is useful. Ted

More example:

bundle agent main
{
  vars:
    "my_data" data => '{ "this data": "will be passed by value or name" }';
    "my_string" string => "My string will be passed by value or name";
    "my_list" slist => { "My", "list", "will", "be", "passed", "by", "value", "or", "name" };

  methods:
    "" usebundle => data_byvalue(storejson(my_data));
    "" usebundle => data_byname("$(this.namespace):$(this.bundle).my_data");
    "" usebundle => string_byvalue( $(my_string) );
    "" usebundle => string_byname("$(this.namespace):$(this.bundle).my_string");
    "" usebundle => list_byvalue( @(my_list) );
    "" usebundle => list_byname("$(this.namespace):$(this.bundle).my_list");
}
bundle agent data_byvalue(value)
{
  vars:
    "passed" data => parsejson($(value));

  reports:
    "$(this.bundle): value is not defined"
      unless => isvariable( value );

    "$(this.bundle): got passed 'value' containing '$(with)' (BUG?)"
      with => storejson(value),
      if => isvariable( value ),
      comment => "This seems like a bug that you can not use the variable defined by the parameter directly";

    "$(this.bundle): got passed 'value', copied into 'passed' containing '$(with)'"
      with => storejson(passed),
      if => isvariable( passed );
}

bundle agent data_byname(reference)
{
  reports:
      "$(this.bundle): got passed reference '$(reference)' containing '$(with)'" with => storejson( $(reference) );
}
bundle agent string_byvalue(value)
{
  reports:
      "$(this.bundle): got passed value '$(with)'" with => "$(value)";
}
bundle agent string_byname(reference)
{
  reports:
      "$(this.bundle): got passed reference '$(reference)' containing '$(with)'" with => "$($(reference))";
}
bundle agent list_byvalue(value)
{
  reports:
      "$(this.bundle): got passed 'value' containing '$(with)'" with => "$(value)";
}
bundle agent list_byname(reference)
{
  reports:
      "$(this.bundle): got passed reference '$(reference)' containing '$(with)' via with (BUG?)" with => "$($(reference))",
        comment => "Seems like a bug that you can not iterate on a foreign list. I expected this promise to be actuated for each value of the list.";

      "$(this.bundle): got passed reference '$(reference)' containing '$((reference))' directly (BUG?)"
        comment => "Seems like a bug that you can not iterate directly on a foreign list";

      "$(this.bundle): got passed reference '$(reference)' containing '$(with)'" with => join( ", ", "$(reference)" );
}

R: data_byvalue: got passed 'value' containing '$(with)' (BUG?)
R: data_byvalue: got passed 'value', copied into 'passed' containing '{
  "this data": "will be passed by value or name"
}'
R: data_byname: got passed reference 'default:main.my_data' containing '{
  "this data": "will be passed by value or name"
}'
R: string_byvalue: got passed value 'My string will be passed by value or name'
R: string_byname: got passed reference 'default:main.my_string' containing 'My string will be passed by value or name'
R: list_byvalue: got passed 'value' containing 'My'
R: list_byvalue: got passed 'value' containing 'list'
R: list_byvalue: got passed 'value' containing 'will'
R: list_byvalue: got passed 'value' containing 'be'
R: list_byvalue: got passed 'value' containing 'passed'
R: list_byvalue: got passed 'value' containing 'by'
R: list_byvalue: got passed 'value' containing 'value'
R: list_byvalue: got passed 'value' containing 'or'
R: list_byvalue: got passed 'value' containing 'name'
R: list_byname: got passed reference 'default:main.my_list' containing '$(with)' via with (BUG?)
R: list_byname: got passed reference 'default:main.my_list' containing '$((reference))' directly (BUG?)
R: list_byname: got passed reference 'default:main.my_list' containing 'My, list, will, be, passed, by, value, or, name'

[Nick Anderson]

Mike Weilgart writes:

I'm pretty sure the vars promise inside of the worker bundle is overwriting the value being passed in. You can't use a parameter name as a local variable (vars promise) also; you have to do one or the other: that's why you're only getting "four, five" from the worker bundle.

Yep, that is my thought exactly.

bundle agent __main__

{
  vars:
    "my_list" slist => { "one", "two", "three" };

  methods

:
    "" usebundle => example( @(my_list) );
}
bundle agent example( my_param_var )
{
  vars:
    "my_param_var" string => "It's a string!", handle => "set_var_inside_bundle";

}

Let's checkout the debug log with all the log modules:

#+begin_src sh cf-agent -Kdf /tmp/locally_defined_vars_win.cf –log-modules=all | grep my_param_var #end_src

$ cf-agent -Kdf /tmp/locally_defined_vars_win.cf --log-modules=all | grep -P "my_param_var|BEGIN bundle|set_var_inside_bundle"                                                                                       


   debug: P:bundle:agent:example  arg id: my_param_var
   debug:       P:bundle:agent:example:vars:any promiser = my_param_var
   debug:       P:bundle:agent:example:vars:any:my_param_var attribute = string
   debug:       P:bundle:agent:example:vars:any:my_param_var attribute = handle
   debug:       P:bundle:agent:example:any qstring rval, handle = set_var_inside_bundle
   debug: DeRefCopyPromise(): promiser:'my_param_var'
   debug: PromiseIteratorPrepare("my_param_var")
   debug: PromiseIteratorPrepare("set_var_inside_bundle")
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: VariableTablePut(handle): string  => set_var_inside_bundle
   debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
   debug: VariableTablePut(promiser): string  => my_param_var
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: Evaluating vars promise: my_param_var
   debug: VariableTableGet(default:example.my_param_var): NOT FOUND
   debug: VariableTablePut(my_param_var): string  => It's a string!
   debug: DeRefCopyPromise(): promiser:'my_param_var'
   debug: PromiseIteratorPrepare("my_param_var")
   debug: PromiseIteratorPrepare("set_var_inside_bundle")
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: VariableTablePut(handle): string  => set_var_inside_bundle
   debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
   debug: VariableTablePut(promiser): string  => my_param_var
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: Evaluating vars promise: my_param_var
   debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
   debug: VariableTablePut(my_param_var): string  => It's a string!
   debug: DeRefCopyPromise(): promiser:'my_param_var'
   debug: PromiseIteratorPrepare("my_param_var")
   debug: PromiseIteratorPrepare("set_var_inside_bundle")
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: VariableTablePut(handle): string  => set_var_inside_bundle
   debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
   debug: VariableTablePut(promiser): string  => my_param_var
   debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
   debug: Evaluating vars promise: my_param_var
   debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
   debug: VariableTablePut(my_param_var): string  => It's a string!

That output at the beginning is from pre-eval where cf-agent eagerly resolves variables so that they are ready for use when the bundlesequence starts.

Here the main bundlesequence starts.

verbose: B: BEGIN bundle main
verbose: B: BEGIN bundle example( {"@(my_list)"})
verbose: V:     +  Private parameter: 'my_param_var' in scope 'example' (type: s) in pass 1
  debug: VariableTablePut(my_param_var): slist  =>  {"one","two","three"}
  debug: DeRefCopyPromise(): promiser:'my_param_var'

There we can see that bundle example is called and my_param_var is set to the value of the referenced variable.

debug: PromiseIteratorPrepare("my_param_var")
debug: PromiseIteratorPrepare("set_var_inside_bundle")
debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
debug: VariableTablePut(handle): string  => set_var_inside_bundle
debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
debug: VariableTablePut(promiser): string  => my_param_var

From here we are inside the bundle, and we can see the variable passed in is getting overwritten by the variable defined inside the bundle.

  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): slist  =>  {"one","two","three"}
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: DeRefCopyPromise(): promiser:'my_param_var'
  debug: PromiseIteratorPrepare("my_param_var")
  debug: PromiseIteratorPrepare("set_var_inside_bundle")
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
  debug: VariableTablePut(handle): string  => set_var_inside_bundle
  debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
  debug: VariableTablePut(promiser): string  => my_param_var
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
verbose: V:     Computing value of 'my_param_var'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: V: 'my_param_var' => 'It's a string!'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: DeRefCopyPromise(): promiser:'my_param_var'
  debug: PromiseIteratorPrepare("my_param_var")
  debug: PromiseIteratorPrepare("set_var_inside_bundle")
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
  debug: VariableTablePut(handle): string  => set_var_inside_bundle
  debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
  debug: VariableTablePut(promiser): string  => my_param_var
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
verbose: V:     Computing value of 'my_param_var'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: V: 'my_param_var' => 'It's a string!'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: DeRefCopyPromise(): promiser:'my_param_var'
  debug: PromiseIteratorPrepare("my_param_var")
  debug: PromiseIteratorPrepare("set_var_inside_bundle")
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
  debug: VariableTablePut(handle): string  => set_var_inside_bundle
  debug: ExpandScalar( (null) : this . my_param_var )  =>  my_param_var
  debug: VariableTablePut(promiser): string  => my_param_var
  debug: ExpandScalar( (null) : this . set_var_inside_bundle )  =>  set_var_inside_bundle
verbose: V:     Computing value of 'my_param_var'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: V: 'my_param_var' => 'It's a string!'
  debug: Evaluating vars promise: my_param_var
  debug: VariableTableGet(default:example.my_param_var): string  => It's a string!
  debug: VariableTablePut(my_param_var): string  => It's a string!

[Aleksey Tsalolikhin]

Nice, Ted.

Thanks Mike and Nick for the replies.   I appreciate all the debug logs, Nick.

So, I'm still a little confused, why does the documentation for methods talk about passing lists by reference?

a list can be passed as a implicitly iterated scalar and as a reference, while a data variable (a data container) can only be passed by reference

Is this applicable to methods promises?  If so, how?  I didn't find the inline example enlightening.  However the example you sent, Nick, were great.

BTW, I am absolutely delighted to see variables getting set shows up in the debug log, complete with value. That's great!

Nick, I tried this and it didn't work:

[:~] $ cf-agent -V
CFEngine Core 3.12.6
[:~] $ cat example.cf
bundle agent main
{
vars:

"my_list" slist => { "My", "list", "will", "be", "passed", "by",
"value", "or", "name" };

methods:

"" usebundle => list_byname("$(this.namespace):$(this.bundle).my_list");
}

bundle agent list_byname(reference)
{
reports:
"$($(reference))";
}

[:~] $ cf-agent -KIC -f ./example.cf
R: $(default:main.my_list)
[:~] $

Is that what you meant by not being able to iterate on a foreign list?  And that's a bug?

My client is on 3.10, so I can't use with which was introduced in 3.11.

Best,

-at

-- 

Founder

Vertical Sysadmin, Inc.

Achieve real learning.

--
Nick Anderson | Doer of Things | (+1) 785-550-1767 | https://northern.tech

[Nick Anderson]

Aleksey Tsalolikhin writes:

Nick, I tried this and it didn't work:

[:~] $ cf-agent -V CFEngine Core 3.12.6

Right, seems like iterating over a foreign list is buggy.

bundle agent main
{
  vars:
      "my_list" slist => { "My", "list", "will", "be", "passed", "by",
                           "value", "or", "name" };

  methods:
      "" usebundle => list_byname("$(this.namespace):$(this.bundle).my_list")

;
      

"" usebundle => list_byvalue( @(my_list) );
}

bundle agent list_byname(reference)
{
  vars:
      # Copy the list locally
      "local" slist => { "@{$(reference)}" };

  reports:
      "CFEngine Version: $(sys.cf_version)";
      "Seems like a BUG that can't Iterate over foreign list '$(reference)' $($(reference))";
      "Iterate over local list 'local' $(local)";
}
bundle agent list_byvalue(value)
{
  vars:
      "local" slist => { @(value) };

  reports:
      "Iterate over list defined as parameter 'value' $(value)";
      "Iterate over local list 'local' $(local)";
}

R: CFEngine Version: 3.17.0
R: Seems like a BUG that can't Iterate over foreign list 'default:main.my_list' $(default:main.my_list)
R: Iterate over local list 'local' My
R: Iterate over local list 'local' list
R: Iterate over local list 'local' will
R: Iterate over local list 'local' be
R: Iterate over local list 'local' passed
R: Iterate over local list 'local' by
R: Iterate over local list 'local' value
R: Iterate over local list 'local' or
R: Iterate over local list 'local' name
R: Iterate over list defined as parameter 'value' My
R: Iterate over list defined as parameter 'value' list
R: Iterate over list defined as parameter 'value' will
R: Iterate over list defined as parameter 'value' be
R: Iterate over list defined as parameter 'value' passed
R: Iterate over list defined as parameter 'value' by
R: Iterate over list defined as parameter 'value' value
R: Iterate over list defined as parameter 'value' or
R: Iterate over list defined as parameter 'value' name
R: Iterate over local list 'local' My
R: Iterate over local list 'local' list
R: Iterate over local list 'local' will
R: Iterate over local list 'local' be
R: Iterate over local list 'local' passed
R: Iterate over local list 'local' by
R: Iterate over local list 'local' value
R: Iterate over local list 'local' or
R: Iterate over local list 'local' name

Is that what you meant by not being able to iterate on a foreign list? And that's a bug?

Yes.

[Nick Anderson]

Aleksey Tsalolikhin writes:

So, I'm still a little confused, why does the documentation for methods talk about passing lists by reference?

a list can be passed as a implicitly iterated scalar and as a reference, while a data https://docs.cfengine.com/docs/3.12/reference-promise-types-vars.html#data variable (a data container) can only be passed by reference

Is this applicable to methods promises? If so, how? I didn't find the inline example enlightening. However the example you sent, Nick, were great.

data containers can be passed by value (not just a stringified version):

bundle agent __main__
{
  vars:
      "d" data => '{ "something": [ "in", "data" ] }';

  methods:
      "" usebundle => data_by_value( @(d) );
      "" usebundle => show_vars_in( "$(this.namespace):data_by_value\..*" );
}

bundle agent show_vars_in( regex_matching_variables )
{
   vars:
    "v" data => variablesmatching_as_data( "$(regex_matching_variables)" );

   reports:
      "Variables in $(regex_matching_variables) $(with)$(const.n)"
        with => storejson( @(v) );

       "Note: The contents of both the parameter (data_param) and the content of a variable defined inside the bundle (local) report their contents in bundle data_by_value.";
       "  variablesmatching() didn't find the variable defined as a parameter(default:data_by_value.data_param) ¯\_(ツ)_/¯";
       # https://github.com/cfengine/core/discussions/4510 
} 
bundle agent data_by_value( data_param )
{
  vars:
      "local" data => @(data_param);

  reports:
      "data_param: $(with)" with => storejson( data_param );
      "local: $(with)" with => storejson( local );

}

R: data_param: {
  "something": [
    "in",
    "data"
  ]
}
R: local: {
  "something": [
    "in",
    "data"
  ]
}
R: Variables in default:data_by_value\..* {
  "default:data_by_value.local": {
    "something": [
      "in",
      "data"
    ]
  }
}

R: Note: The contents of both the parameter (data_param) and the content of a variable defined inside the bundle (local) report their contents in bundle data_by_value.
R:   variablesmatching() didn't find the variable defined as a parameter(default:data_by_value.data_param) ¯\_(ツ)_/¯

As far as I know, anything can be passed by reference, all you do is pass the name you will use to reference the variable as a string. A variable is created for the parameter, that variable holds a string, the name of the thing you really want to access.