Could not load requested further augments from file

[Beto]

As we upgrade from RHEL7 to RHEL8 I'm seeing this in /var/log/messages:

Aug 16 12:54:06 zombie cf-serverd[1921]: CFEngine(server)  Rereading policy file '/var/cfengine/inputs/promises.cf'
Aug 16 12:54:06 zombie cf-serverd[1921]:   error: Could not load requested further augments from file '/srv/sysadmin/etc/
cfengine/augments.json'
Aug 16 12:54:06 zombie cf-serverd[1921]:   error: Could not load requested further augments from file '$(sys.inputdir)/au
gments/augments_$(def.network).json'
Aug 16 12:54:06 zombie cf-serverd[1921]: CFEngine(server)  Could not load requested further augments from file '/srv/sysa
dmin/etc/cfengine/augments.json'
Aug 16 12:54:06 zombie cf-serverd[1921]: CFEngine(server)  Could not load requested further augments from file '$(sys.inp
utdir)/augments/augments_$(def.network).json'

Not seeing this on RHEL7.

CFEngine Core 3.24.0
CFEngine Enterprise 3.24.0

Any ideas?

[Nick Anderson]

I dont think you can use non sys variables in augments like that becuase augmwnts is loaded before policy is read at all. Check the augments docs (i can't atm, on the move). 

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengin...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/857f33e7-4ac3-45f1-8b90-234de4c98bc8n%40googlegroups.com.

[craig.c...@northern.tech]

The docs might be a bit imprecise. https://docs.cfengine.com/docs/3.24/reference-language-concepts-augments.html says in two separate notes:

• CFEngine variables are not expanded unless otherwise noted.

• sys variables are expanded in def.json and all subsequently loaded augments as specified by the augments key.

---

I did a little test and it does seem your def.network and sys variables should both be expanded so I wonder if it is an access or selinux issue?

Maybe check /var/log/audit/audit.log for cf-serverd related entries?

---

My test:

$ cat def.json 

{

  "vars": {

    "foo": "${def.jq}",

    "bar": "baz"

  }

}

$ cat t.cf 

bundle agent main

{

  reports:

 "foo is ${default:def.foo}";

 "bar is ${default:def.bar}";

 "def.jq is ${def.jq}";

}

result:

R: foo is jq --compact-output --monochrome-output --ascii-output --unbuffered --sort-keys

R: bar is baz

R: def.jq is jq --compact-output --monochrome-output --ascii-output --unbuffered --sort-keys

[Nick Anderson]

Def.jq might itself be special what about def.foo? 

To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/3e1de5ef-ee2a-4db3-a8e7-5c76fa68ae02n%40googlegroups.com.

[Beto]

There are no selinux AVCs logged.

[Beto]

Weird thing is this occurs ONLY on RHEL8, not on RHEL7 and  this same augments configuration has been in use for years.

I thought maybe I could use host_specific.json to get around this issue but I can't tell that host_specific.json is being seen at all (as you say the doc is "imprecise").

[Vratislav Podzimek]

Hello,

On Fri, 2024-08-16 at 14:06 -0700, Beto wrote:
> Weird thing is this occurs ONLY on RHEL8, not on RHEL7 and  this same augments configuration has been in use for years.

That suggests the difference is most likely SELinux. Does it happen even if you
do a manual agent run from an interactive shell with `cf-agent -KI`? If not,
please double-check `ausearch -m AVC` on an affected system.

Another alternative is to run `strace -ff -- cf-agent -KI 2> cf-agent.strace`,
look for the particular file name in the trace and see what happens when the
file is attempted to be opened and read.

>
> I thought maybe I could use host_specific.json to get around this issue but I can't tell that host_specific.json is being seen at all (as you say the doc is "imprecise").

What do you think is imprecise in the docs? I think it's quite clear:

There are two canonical augments files, host_specific.json, and def.json
which may load additional Augments as specified by the augments key.

Notes:

CFEngine variables are not expanded unless otherwise noted.

and then there is a note for the def.json file:

sys variables are expanded in def.json and all subsequently loaded augments
as specified by the augments key.

You can do a verbose agent run using `--verbose` to see details of what it is doing
and which files it is loading.

One thing is quite clear -- the error message that further augments could not be
loaded should be enhanced to explain why.

--
Vratislav

>
> On Friday, August 16, 2024 at 4:00:51 PM UTC-5 Beto wrote:
> > There are no selinux AVCs logged.
> >
> > On Friday, August 16, 2024 at 3:42:41 PM UTC-5 nick.a...@northern.tech wrote:
> > > Def.jq might itself be special what about def.foo? 
> > >
> > > On Fri, Aug 16, 2024, 3:10 PM 'craig.c...@northern.tech' via help-cfengine <help-c...@googlegroups.com> wrote:
> > > > The docs might be a bit imprecise. https://docs.cfengine.com/docs/3.24/reference-language-concepts-augments.html says in two separate notes:
> > > >

> > > >  * CFEngine variables are not expanded unless otherwise noted.
> > > >  * sys variables are expanded in def.json and all subsequently loaded augments as specified by the augments key.

[craig.c...@northern.tech]

Yes, I introduced the idea that the docs are "imprecise" because I got lucky enough to choose the ONE non-sys variable which is available in augments for expansion: def.jq which is the only non-sys non-const variable which is defined in the agent aka C-code as can be seen with this command to show all variables and filter out sys and const variables:

$ cf-promises --show-vars | grep "source=agent" | grep -v sys | grep -v const default:def.jq jq --compact-output --monochrome-output --ascii-output --unbuffered --sort-keys invocation,source=agent,command_name=jq

So as Vrata mentioned, you are likely running into an SELinux issue.

-Craig

[Beto]

So, there were two issues:

1. AVCs on an augments file in /srv that selinux was blocking access to.
2. Typo in host_specific.json.  Turns out cfe doesn't recognize host_specfic.json.

Rather than fiddling with selinux contexts  I moved the augments file in /srv to host_specific.json and updated def.json to use the variables from host_specific.json.

Thank you all for your help.

[Nick Anderson]

Ah great, glad you got it sorted.

> Typo in host_specific.json.  Turns out cfe doesn't recognize host_specfic.json.

Classic kind of typo I would make. 

This should be visible in verbose outputs (if you don't miss it with all the other logging).

 verbose: No host-specific JSON data available at '/var/cfengine/data/host_specific.json'

To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/4035b131-5b0f-438b-97bb-6a01b58cf6ean%40googlegroups.com.

[Vratislav Podzimek]

On Mon, 2024-08-19 at 13:46 -0700, Beto wrote:
> So, there were two issues:
>

>    1. AVCs on an augments file in /srv that selinux was blocking access to.
>    2. Typo in host_specific.json.  Turns out cfe doesn't recognize host_specfic.json.

>
> Rather than fiddling with selinux contexts  I moved the augments file in /srv to host_specific.json and updated def.json to use the variables from host_specific.json.

Glad you found a solution. In case you wanted to use /srv for CFEngine
policy/JSON files anyway, it should be easy to add and apply a new SELinux file
context rule with something like this:

semanage fcontext -a -t cfengine_var_lib_t '/srv/whatever/specific/dir(/.*)?'
restorecon -Rv /srv/whatever/specific/dir

Of course, it would be better to do this from policy, but that would have to
check if the rule is already present first. Too bad that
https://build.cfengine.com/modules/?query=selinux gives 0 results, maybe someone
could add a small CFEngine module, perhaps with a new custom promise type, for
at least the fcontext rules...? 😉️ Modules welcome!

--
Vratislav

> To view this discussion on the web visit https://groups.google.com/d/msgid/help-cfengine/4035b131-5b0f-438b-97bb-6a01b58cf6ean%40googlegroups.com.

[Beto]

Thanks, Vrata!  I'm still trying to figure out the best way forward.