Problems while upgrading to latest cfengine-community LTS

[Berthold Cogel]

Hello,

we have a script that updates all our modules after the master files
have been updated. This script is designed to transfer changes to the
modules, add new modules to a host, and delete those that are no longer
needed. The script compares the installed modules with the expected
modules for the host.

Problem:

A change in the behavior of the latest version now causes all modules to
be deleted.

The script works up to and including version 3.23.0. The problem occurs
when upgrading to 3.24.0 and it also exists in 3.24.2.

The relevant part of the script appears to be this one:

------:<----------------

bundle agent controller {
# Delete everything not in module list.
vars:
"dirs_found" slist => findfiles("$(constants.conf_dir)/*/");
"dirs_expected" slist => {
maplist("$(constants.conf_dir)/$(this)/", @(constants.module_list)),
"$(constants.conf_dir)/host_config/" };
"dirs_to_delete" slist => difference(dirs_found, dirs_expected);
methods:
"any" usebundle => rm_rf("$(dirs_to_delete)");

methods:
# sync the modules
!debugflag_mode::
"any" usebundle => host_config;
"any" usebundle => sync_module($(constants.module_list));

reports:
inform_mode::
"Deleted directory '$(dirs_to_delete)' because it is not in
constants.module_list";
inform_mode.!debugflag_mode::
"controller was activated";
inform_mode.debugflag_mode::
"controller is deactivated";
}

------:<----------------

Debug output of working version:

verbose: V: Computing value of 'dirs_found'
debug: Evaluating vars promise: dirs_found
debug: V: 'dirs_found' => '/var/cfengine/config/host_config/, ......'
debug: Evaluating vars promise: dirs_found
debug: DeRefCopyPromise(): promiser:'dirs_expected'
debug: DeRefCopyPromise(): copying constraint: 'slist'
debug: Evaluating function:
maplist("/var/cfengine/config/$(this)/","@(constants.module_list)")
verbose: V: Computing value of 'dirs_expected'
debug: Evaluating vars promise: dirs_expected
debug: V: 'dirs_expected' => '/var/cfengine/config/afs/, ......'
debug: Evaluating vars promise: dirs_expected
debug: DeRefCopyPromise(): promiser:'dirs_to_delete'
debug: DeRefCopyPromise(): copying constraint: 'slist'
debug: Evaluating function: difference("dirs_found","dirs_expected")
verbose: V: Computing value of 'dirs_to_delete'
debug: Evaluating vars promise: dirs_to_delete
debug: V: 'dirs_to_delete' => ''
debug: Evaluating vars promise: dirs_to_delete
debug: DeRefCopyPromise(): promiser:'any'
debug: DeRefCopyPromise(): copying constraint: 'usebundle'
debug: Starting iteration engine with 1 wheels --- ENTERING WARP
SPEED
verbose: Skipping iteration since variable 'dirs_to_delete' resolves
to an empty list
debug: Iteration engine finished --- WARPING OUT

.....

Debug output of failing version:

verbose: V: Computing value of 'dirs_found'
debug: Evaluating vars promise: dirs_found
debug: V: 'dirs_found' => '/var/cfengine/config/krb, ......'
debug: Evaluating vars promise: dirs_found
debug: DeRefCopyPromise(): promiser:'dirs_expected'
debug: DeRefCopyPromise(): copying constraint: 'slist'
debug: Evaluating function:
maplist("/var/cfengine/config/$(this)/","@(constants.module_list)")
verbose: V: Computing value of 'dirs_expected'
debug: Evaluating vars promise: dirs_expected
debug: V: 'dirs_expected' => '/var/cfengine/config/afs/, ......'
debug: Evaluating vars promise: dirs_expected
debug: DeRefCopyPromise(): promiser:'dirs_to_delete'
debug: DeRefCopyPromise(): copying constraint: 'slist'
debug: Evaluating function: difference("dirs_found","dirs_expected")
verbose: V: Computing value of 'dirs_to_delete'
debug: Evaluating vars promise: dirs_to_delete
debug: V: 'dirs_to_delete' => '/var/cfengine/config/krb, ......'
debug: Evaluating vars promise: dirs_to_delete
debug: DeRefCopyPromise(): promiser:'any'
debug: DeRefCopyPromise(): copying constraint: 'usebundle'
debug: Starting iteration engine with 1 wheels --- ENTERING WARP
SPEED
verbose: P: .........................................................
verbose: P: BEGIN promise 'promise_user_update_lib_cf_8' of type
"methods" (pass 1)
verbose: P: Promiser/affected object: 'any'
verbose: P: Part of bundle: controller
verbose: P: Base context class: any
verbose: P: Stack path: /default/controller/methods/'any'[1]
verbose: B:
*****************************************************************
verbose: B: BEGIN bundle rm_rf( {"/var/cfengine/config/krb"})
verbose: B:
*****************************************************************

......



The problem seems to be the difference function.


Regards
Berthold Cogel

[Nick Anderson]

Hi Berthold,

Thanks for letting us know.

Can you please make a self contained policy that reproduces the behavior? I don't recall any changes to the difference() function in the recent past. If there is an unintended behavior change here there is little time to address it before the next release.

[Berthold Cogel]

Am 21.10.25 um 16:35 schrieb Nick Anderson:

Hello Nick,

until now I wasn't able to reproduce the problem with a separate script
in an isolated environment. I think I'm still missing something. Perhaps
I have to rebuild the complete directory structure of our setup.

Regards
Berthold Cogel

[Nick Anderson]

until now I wasn't able to reproduce the problem with a separate script in an isolated environment. I think I'm still missing something. Perhaps I have to rebuild the complete directory structure of our setup.

Well, that's unfortunate.

Looking at it again the output from the working version said …

verbose: Skipping iteration since variable 'dirs_to_delete' resolves
to an empty list

That doesn't have far to trace. It comes from dirs_found (derived from findfiles) and dirs_expected (derived from maplist)

There was this fix to findfiles() in 3.24, but it only affected windows as far as I know:

(ENT-11923)

• Fixed bug related to failing backwards directory traversial when using forward slashes in path argument of the findfiles_up() policy function on Windows.

[Berthold Cogel]

Am 28.10.25 um 20:56 schrieb 'Nick Anderson' via help-cfengine:

> Well, that's unfortunate.
>
> Looking at it again the output from the working version said …
>
> verbose: Skipping iteration since variable 'dirs_to_delete' resolves
> to an empty list
>
> That doesn't have far to trace. It comes from dirs_found (derived from
> findfiles) and dirs_expected (derived from maplist)
>
> There was this fix to findfiles() in 3.24, but it only affected windows
> as far as I know:
>
> (ENT-11923)
>

> * Fixed bug related to failing backwards directory traversial when

> using forward slashes in path argument of the findfiles_up() policy
> function on Windows.
>

Hello Nick,

I've got a test case now. I had to rebuild the directory structure of
the modules directory of my test system and copy our default module
list. You find the script and the directory attached to this email.

Extract the archive somewhere on one of your systems, change the
variable test_dir to the location of the files. I tested it against
3.23.0 and 3.24.0 and it shows the problem:

----------:<----------------
R: cf-version: 3.23.0
R: dir: /var/local/diff_test
R: modlist: afs

...

R: modlist: yum
R: found: /var/local/diff_test/idm_users/

...

R: found: /var/local/diff_test/net/
R: expected: /var/local/diff_test/afs/

...


R: expected: /var/local/diff_test/host_config/
----------:<----------------
R: cf-version: 3.24.0
R: dir: /var/local/diff_test
R: modlist: afs

...

R: modlist: yum
R: found: /var/local/diff_test/cmd

...

R: found: /var/local/diff_test/dnf
R: expected: /var/local/diff_test/afs/

...


R: expected: /var/local/diff_test/host_config/
R: diff: /var/local/diff_test/cmd

...

R: diff: /var/local/diff_test/dnf
----------:<----------------

Regards
Berthold Cogel

[Berthold Cogel]

Am 28.10.25 um 20:56 schrieb 'Nick Anderson' via help-cfengine:

> There was this fix to findfiles() in 3.24, but it only affected windows
> as far as I know:
>
> (ENT-11923)
>

> * Fixed bug related to failing backwards directory traversial when

> using forward slashes in path argument of the findfiles_up() policy
> function on Windows.
>

Hallo Nick,

I made a second attempt and it looks as if findfiles causes the trouble.

In our script we use this lines:

"dirs_found" slist => findfiles("$(constants.conf_dir)/*/");
"dirs_expected" slist => { maplist("$(constants.conf_dir)/$(this)/",

@(file_list) ), "$(constants.conf_dir)/host_config/"};

I modfied my test script:

------------:<-------------
body common control
{
bundlesequence => { "test" };
}

bundle agent test {
vars:

"file_list" slist => { "a", "b", "c", "d", "e", "f", "g", "x", "z" };

"dirs_found" slist => findfiles("/tmp/diff_test/*/");
"dirs2_found" slist => findfiles("/tmp/diff_test/*");
"dirs_expected" slist => { maplist("/tmp/diff_test/$(this)/",
@(file_list) ), "/tmp/diff_test/host_config/"};
"dirs_diff" slist => difference(dirs_found, dirs_expected);
"dirs2_diff" slist => difference(dirs2_found, dirs_expected);

reports:
"version: $(sys.cf_version)";
"found: $(dirs_found)";
"found2: $(dirs2_found)";
"expected: $(dirs_expected)";
"diff: $(dirs_diff)";
"diff2: $(dirs2_diff)";
}
------------:<-------------

Create /tmp/diff_test/ with the subdirectories from $(file_list) and run
the script in both cfengine versions:

------------:<-------------
R: version: 3.23.0
R: found: /tmp/diff_test/a/
R: found: /tmp/diff_test/b/
R: found: /tmp/diff_test/c/
R: found: /tmp/diff_test/d/
R: found: /tmp/diff_test/e/
R: found: /tmp/diff_test/f/
R: found: /tmp/diff_test/g/
R: found: /tmp/diff_test/x/
R: found: /tmp/diff_test/z/
R: found2: /tmp/diff_test/a
R: found2: /tmp/diff_test/b
R: found2: /tmp/diff_test/c
R: found2: /tmp/diff_test/d
R: found2: /tmp/diff_test/e
R: found2: /tmp/diff_test/f
R: found2: /tmp/diff_test/g
R: found2: /tmp/diff_test/x
R: found2: /tmp/diff_test/z
R: expected: /tmp/diff_test/a/
R: expected: /tmp/diff_test/b/
R: expected: /tmp/diff_test/c/
R: expected: /tmp/diff_test/d/
R: expected: /tmp/diff_test/e/
R: expected: /tmp/diff_test/f/
R: expected: /tmp/diff_test/g/
R: expected: /tmp/diff_test/x/
R: expected: /tmp/diff_test/z/
R: expected: /tmp/diff_test/host_config/
R: diff2: /tmp/diff_test/a
R: diff2: /tmp/diff_test/b
R: diff2: /tmp/diff_test/c
R: diff2: /tmp/diff_test/d
R: diff2: /tmp/diff_test/e
R: diff2: /tmp/diff_test/f
R: diff2: /tmp/diff_test/g
R: diff2: /tmp/diff_test/x
R: diff2: /tmp/diff_test/z
------------:<-------------
R: version: 3.24.0
R: found: /tmp/diff_test/a
R: found: /tmp/diff_test/b
R: found: /tmp/diff_test/c
R: found: /tmp/diff_test/d
R: found: /tmp/diff_test/e
R: found: /tmp/diff_test/f
R: found: /tmp/diff_test/g
R: found: /tmp/diff_test/x
R: found: /tmp/diff_test/z
R: found2: /tmp/diff_test/a
R: found2: /tmp/diff_test/b
R: found2: /tmp/diff_test/c
R: found2: /tmp/diff_test/d
R: found2: /tmp/diff_test/e
R: found2: /tmp/diff_test/f
R: found2: /tmp/diff_test/g
R: found2: /tmp/diff_test/x
R: found2: /tmp/diff_test/z
R: expected: /tmp/diff_test/a/
R: expected: /tmp/diff_test/b/
R: expected: /tmp/diff_test/c/
R: expected: /tmp/diff_test/d/
R: expected: /tmp/diff_test/e/
R: expected: /tmp/diff_test/f/
R: expected: /tmp/diff_test/g/
R: expected: /tmp/diff_test/x/
R: expected: /tmp/diff_test/z/
R: expected: /tmp/diff_test/host_config/
R: diff: /tmp/diff_test/a
R: diff: /tmp/diff_test/b
R: diff: /tmp/diff_test/c
R: diff: /tmp/diff_test/d
R: diff: /tmp/diff_test/e
R: diff: /tmp/diff_test/f
R: diff: /tmp/diff_test/g
R: diff: /tmp/diff_test/x
R: diff: /tmp/diff_test/z
R: diff2: /tmp/diff_test/a
R: diff2: /tmp/diff_test/b
R: diff2: /tmp/diff_test/c
R: diff2: /tmp/diff_test/d
R: diff2: /tmp/diff_test/e
R: diff2: /tmp/diff_test/f
R: diff2: /tmp/diff_test/g
R: diff2: /tmp/diff_test/x
R: diff2: /tmp/diff_test/z
------------:<-------------



Regards
Berthold Cogel

[Berthold Cogel]

Am 28.10.25 um 12:33 schrieb Berthold Cogel:

Hello Nick,

I was wondering why my last post remained unanswered and have now tested
with cfengine 3.24.3. In doing so, I realized that I accidentally posted
the wrong version of the script last time.

Sorry for the noise!

Here is the correct version with the results for versions 3.21.8 and
3.24.3:

In our script we use this lines:

"dirs_found" slist => findfiles("$(constants.conf_dir)/*/");
"dirs_expected" slist => { maplist("$(constants.conf_dir)/$(this)/",

@(file_list) ), "$(constants.conf_dir)host_config"};

------------:<-------------
body common control
{
bundlesequence => { "test" };
}

bundle agent test {
vars:

"test_dir" string => "/tmp/diff_test";

"file_list" slist => { "a", "b", "c", "d", "e", "f", "g", "x", "z" };

"dirs_found" slist => findfiles("$(test_dir)/*/");
"dirs2_found" slist => findfiles("$(test_dir)/*");
"dirs_expected" slist => { maplist("$(test_dir)/$(this)/",
@(file_list) ), "$(test_dir)/host_config"};

"dirs_diff" slist => difference(dirs_found, dirs_expected);
"dirs2_diff" slist => difference(dirs2_found, dirs_expected);

reports:
"version: $(sys.cf_version)";
"found: $(dirs_found)";
"found2: $(dirs2_found)";
"expected: $(dirs_expected)";
"diff: $(dirs_diff)";
"diff2: $(dirs2_diff)";
}

------------:<-------------

R: version: 3.21.8

R: diff2: /tmp/diff_test/a
R: diff2: /tmp/diff_test/b
R: diff2: /tmp/diff_test/c
R: diff2: /tmp/diff_test/d
R: diff2: /tmp/diff_test/e
R: diff2: /tmp/diff_test/f
R: diff2: /tmp/diff_test/g
R: diff2: /tmp/diff_test/x
R: diff2: /tmp/diff_test/z

------------:<-------------

R: version: 3.24.3

R: diff: /tmp/diff_test/a
R: diff: /tmp/diff_test/b
R: diff: /tmp/diff_test/c
R: diff: /tmp/diff_test/d
R: diff: /tmp/diff_test/e
R: diff: /tmp/diff_test/f
R: diff: /tmp/diff_test/g
R: diff: /tmp/diff_test/x
R: diff: /tmp/diff_test/z
R: diff2: /tmp/diff_test/a
R: diff2: /tmp/diff_test/b
R: diff2: /tmp/diff_test/c
R: diff2: /tmp/diff_test/d
R: diff2: /tmp/diff_test/e
R: diff2: /tmp/diff_test/f
R: diff2: /tmp/diff_test/g
R: diff2: /tmp/diff_test/x
R: diff2: /tmp/diff_test/z

------------:<-------------

If the new behavior is desired, I would be grateful for any information
regarding the necessary changes to our scripts.


Regards
Berthold Cogel

[Nick Anderson]

Hey Berthold,

That seems like it's related to this blog post about change in behavior for findfiles() (https://cfengine.com/blog/2025/change-in-behavior-findfiles/).

But, that post doesn't mention any intent to alter the trailing slash that signals directory. I am not sure if it was intentional or not.

• Old Behavior (<= 3.23.0): When findfiles("path/*/") is used, the returned file list included the trailing slash (e.g., path/to/dir).
• New Behavior (>= 3.24.0): The trailing slash is stripped from the returned file list (e.g., /path/to/dir), causing mismatches when compared against lists that expect the trailing slash.

This policy demonstrates the difference:

bundle agent main
{
  methods:
      "init";
      "test";
      "check";
  reports:
    inform_mode::
      "CFEngine: $(sys.cf_version)";
}

bundle agent init
{
  vars:
      "testdir" string => "/tmp/testdir";
      "subdirs" slist => { "a", "b", "c" };

  files:
      "$(testdir)/$(subdirs)/." create => "true";
}

bundle agent test
{
  vars:
      "found"
        slist => { findfiles( "$(init.testdir)/*/" ) };

}

bundle agent check
{
  vars:
      "foundtype[$(test.found)]"
        string => filestat( $(test.found), "type");

      "found_types"
        slist => getvalues( foundtype );

      "founddir[$(test.found)]"
        string => "$(foundtype[$(test.found)])",
        if => strcmp( "$(foundtype[$(test.found)])", "directory" );

      "found_dirs"
        slist => getindices( founddir );

  classes:
      "every_finding_has_trailing_slash"
        expression => every( ".*\/$", @(test.found) );

      "every_finding_is_directory"
        expression => every( "directory", @(found_types) );

      "every_found_directory_ends_trailing_slash"
        expression => every( ".*\/$", @(found_dirs) );

  reports:
      "Every findfiles() result has trailing slash"
        if => "every_finding_has_trailing_slash";

      "Found: $(test.found)";

      "Every findfiles() result is a directory"
        if => "every_finding_is_directory";

      "Type $(test.found) = $(foundtype[$(test.found)])";

      "Every directory returned by findfiles() has trailing slash"
        if => "every_found_directory_ends_trailing_slash";
}

# cf-agent -Kf ./t.cf --inform
R: Every findfiles() result has trailing slash
R: Found: /tmp/testdir/a/
R: Found: /tmp/testdir/b/
R: Found: /tmp/testdir/c/
R: Every findfiles() result is a directory
R: Type /tmp/testdir/a/ = directory
R: Type /tmp/testdir/b/ = directory
R: Type /tmp/testdir/c/ = directory
R: Every directory returned by findfiles() has trailing slash
R: CFEngine: 3.21.8

# cf-agent -KIf ./t.cf
    info: Created directory '/tmp/testdir/a/.', mode 0700
    info: Created directory '/tmp/testdir/b/.', mode 0700
    info: Created directory '/tmp/testdir/c/.', mode 0700
R: Found: /tmp/testdir/a
R: Found: /tmp/testdir/b
R: Found: /tmp/testdir/c
R: Every findfiles() result is a directory
R: Type /tmp/testdir/a = directory
R: Type /tmp/testdir/b = directory
R: Type /tmp/testdir/c = directory
R: CFEngine: 3.24.0