cf-agent and memory usage

[Stefan Skoglund]

I have a error in my policy such that cf-agent on the hub eats up all
memory (a virtual machine with 5G of RAM allocated) and when crashes
with xrealloc:
---
notice: Q: "...ne/bin/git pull": Already up to date.
xreallocCRITICAL: Unable to allocate memory
---

This is enterprise 18.1.

How to debug ?

Do cf-agent pulls down data from the clients or is it cf-hub who does
that ?

[Stefan Skoglund]

answering to myself, i have a policy which load data_state() into an
container in pass 1 and then it is redone in pass 2.

I wanted the classes part from data_state which is then used in a
custom template_data container.

It isn't the first time i have tried this.....

[craig.c...@northern.tech]

Here are the docs for the datastate() function for reference: https://docs.cfengine.com/docs/3.18/reference-functions-datastate.html

I would assume that the return from datastate() is quite large due to your policy defining many vars and classes.

Can you give a minimal reproducer of some sort or some more information about your policy that does something like

```

"state" data => datastate();

```

From what you said I assume you have something like that and using cf-agent --verbose or --debug you can see that it is "redone in pass 2"?

Could you use classesmatching to find just the classes you want and send those to your template? https://docs.cfengine.com/docs/3.18/reference-functions-classesmatching.html

-Craig

[craig.c...@northern.tech]

I did find this old ticket that I think is related: https://tracker.mender.io/browse/CFE-1785

The reproducer there seems to be similar to what you are describing and testing it myself seems to be a good reproducer of what is happening for you and have nothing to do with size of policy and number of classes/vars involved. I'll bump that ticket up in priority and we'll try and get into it soon.

-Craig

[Stefan Skoglund]

tis 2022-01-04 klockan 09:00 -0800 skrev 'craig.c...@northern.tech' via
help-cfengine:

> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to help-cfengin...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/help-cfengine/d00fbec3-42f8-4e3d-8cfd-53c48e009a6bn%40googlegroups.com
> .

Hmm, anyone which has used classesmatching(".*", "hardclass") to get
something which can be checked for in mustache templates using '#' ie
equal to:
---
{{#classes.debian_11}}
.....
{{/classes.debian_11}}
---

I would need the classes from the environment or those defined in
def.json.

[Bas van der Vlies]

>
> Hmm, anyone which has used classesmatching(".*", "hardclass") to get
> something which can be checked for in mustache templates using '#' ie
> equal to:
> ---
> {{#classes.debian_11}}
> .....
> {{/classes.debian_11}}
> ---
>
> I would need the classes from the environment or those defined in
> def.json.
>

I do not know what ypu exactly mean with this, but I use this in the SCL library a lot if I understand you correctly:. Classes defined in def.json are globally known and can be used in the mustache template. I do not use the datastate() functiion at all.
eg:
```

{{#classes.SSH_PUBKEY_AUTHENTICATION}}
AuthorizedKeysCommand {{vars.scl.ssh.authorized_keys_command}}
{{#classes.SLES}}
AuthorizedKeysCommandUser {{vars.scl.ssh.authorized_keys_commanduser}}
{{/classes.SLES}}
{{#classes.debian}}
AuthorizedKeysCommandUser {{vars.scl.ssh.authorized_keys_commanduser}}
{{/classes.debian}}
{{#classes.centos}}
AuthorizedKeysCommandRunAs {{vars.scl.ssh.authorized_keys_commanduser}}
{{/classes.centos}}
{{/classes.SSH_PUBKEY_AUTHENTICATION}}
```

the `SSH_PUBKEY_AUTHENTICATION` is defined in a `def.json` and converted by the SCL library to the `SSH_PUBKEY_AUTHENTICATION`
```
classes: {
BATCHSERVER_NODE: [ “debian_10” ]
},
ssh: {
classes: {
PUBKEY_AUTHENTICATION: any
},
```

Maybe this answer your question


--
Bas van der Vlies
| High Performance Computing Visualisation | SURF| Science Park 140 | 1098 XG Amsterdam
| T +31 (0) 20 800 1300 | bas.van...@surf.nl | www.surf.nl |

[Stefan Skoglund]

no i would want to create a custom template_data container
which below one key has the classes defined in def.json (and by cf-
agent from the environment)
and via other keys a number of variables (variables defined by the
current agent.)

This is basically compared with the normal built in template data
getting rid of having to nominate containing namespace for a class in
the mustache template.

I would like to have something like this:
---
vars:
"klassamling"
slist => classes_matching( ".*", "source=augment", "WiFi");

"templatedata"
data => mergedata( '{ "klasser": klassamling,
"testvar": "22"
}');
files:
"test"
edit_template= "test.mustache",
template_data => templatedata,
template_method => "mustache";
--

test.mustache:
---
{{#klasser.is_kdc_hub}}
variabel = {{testvar}}
{{/klasser.is_kdc_hub}}
---

For this to work if i understand mustache correctly would really mean
having some method of going from a string to a key in "klasser" where
the key's value is true ?

ONE another method would be to call datastate early on in my policy,
pull out the classes tree only and when if some agent wants access,
they can do so using the container's path - ie:

services/main.cf:
---
"datastate"
data => datastate();
"klasser"
data => mergedata( datastate[classes]);
---

And from other agent refer to klasser :
---
"template_data"
data => mergedata( '{ "klasser": default:main.klasser,
"var2": 3,
"var3": 4
}');
---

That agent is in another namespace, for example "WiFi".

My idea is to being able to tell a developer which wants to control
some type of text config file : refer to classes and vars using a
short name (and not having to tell/educate them about the concept of
namespaces in cfengine.)

A lot of complexity .... to be able to say :
----
test = {{test}}

{{#iftest}}
test2 = {{test2}}
{{/iftest}}
---

Instead of :
---
test = {{vars.DNS:agentName.test}}
{{#classes.DNS:iftest}}
test2 = {{vars.DNS:agentName.test2}}
{{/classes.DNS:iftest}}
----

[Bas van der Vlies]

Your setup is different then ours. I get your point of make it easy for developers. the scl library function to genenerate the file from a template can handle an optional data section arg to support the short names in templates. Do not know if is supports the namespaces. Maybe it can help you to solve your issue:
```
bundle agent scl_mustache_cf_data_2_file(bundle_name, template_file, destination, data_section)
{

vars:
any::
"template_dir" string => "$(def.node_template_dir)/$(bundle_name)";

"data_section_str" string => format("%S", "$(data_section)"),
if => "DEBUG_$(bundle_name)|DEBUG|DEBUG_MUSTACHE";

classes:
#"data_section" expression => isvariable("data_section"),
"data_section" expression => regcmp("[a-zA-Z].*", "$(data_section)"),
comment => "data_section defined do use it",
scope => "bundle";

"destination_set" expression => regcmp("^/.*", "$(destination)");


files:
data_section.destination_set::
"$(destination)"
comment => "Mustache expand with defined data_section variable ",
create => "true",
edit_template => "$(template_dir)/$(template_file)",
template_method => "mustache",
#template_data => mergedata("scl.$(bundle_name)[$(data_section)]"),
template_data => mergedata("$(data_section)"),
classes => if_repaired("scl$(destination)");

!data_section.destination_set::
"$(destination)"
comment => "Mustache expand with cfengine json data",
create => "true",
edit_template => "$(template_dir)/$(template_file)",
template_method => "mustache",
classes => if_repaired("scl$(destination)”);
}
```

--
Bas van der Vlies

| High Performance Computing & Visualization | SURF| Science Park 140 | 1098 XG Amsterdam