directory size

[Christian Linden]

Hi,

any w/o shell execresult way to determine a directory's size?

Thanks!

Chris

[Aleksey Tsalolikhin]

Have you checked the lists of functions in https://docs.cfengine.com/lts/reference-functions.html ? That's where all the functions are listed.

Hint: check the "files" column in the "Functions by Category" table (The first column in the first table in that reference.)

Best,

-at

ᐧ

-- 

Need training on CFEngine, Git or Time Management?  Email trai...@verticalsysadmin.com.

--
You received this message because you are subscribed to the Google Groups "help-cfengine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to help-cfengine+unsubscribe@googlegroups.com.
To post to this group, send email to help-c...@googlegroups.com.
Visit this group at https://groups.google.com/group/help-cfengine.
For more options, visit https://groups.google.com/d/optout.

[Christian Linden]

I found filesize and filestat but just dirname.

Are filesize and filestat useable for directories (using the /.) as well?

c

[Shane McEwan]

On 06/09/16 19:43, Christian Linden wrote:
> I found filesize and filestat but just dirname.
> Are filesize and filestat useable for directories (using the /.) as well?

filesize and filestat simply use the stat (or fstat or lstat) system
call (see 'man 2 stat'). There is no equivalent system call to determine
the total amount of disk space the files in a directory are using. The
only way to figure that out is to descend into the directory tree
'stat'ing every file you find. This is what the 'du' command does.

It would be a lot of work to implement du-like behaviour in CFEngine and
it wouldn't be any more efficient than calling du using execresult.

Shane.

[Christian Linden]

OK, thank you, Shane.

--
You received this message because you are subscribed to a topic in the Google Groups "help-cfengine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/help-cfengine/GCuCy32pJ0g/unsubscribe.
To unsubscribe from this group and all its topics, send an email to help-cfengine+unsubscribe@googlegroups.com.

[Ted Zlatanov]

On Tue, 6 Sep 2016 20:39:45 +0100 Shane McEwan <sh...@mcewan.id.au> wrote:

SM> It would be a lot of work to implement du-like behaviour in CFEngine and it
SM> wouldn't be any more efficient than calling du using execresult.

I can see the utility of a mapping function, though. Imagine filestat()
applied to a list of files. It would be useful in many other cases but
here especially:

vars:
# note ** matches up to 6 levels of subdirectories
# (which is actually a good limitation to cap resource usage)
# (and you can do **/** etc. for more levels)
"sizes" slist => maplist(filestat($(this), "size"), findfiles("/etc/**/*.conf"));
"total_size" real => sum(sizes);

Unfortunately maplist(function($(this)), ...) does not currently work
this way. It's been on my wishlist since at least 3.5 and there are
feature request tickets for it :)

For 3.9, this will work:

#+begin_src cfengine3
bundle agent main
{
vars:
"files" slist => findfiles("/etc/**/*.conf");
"cfiles[$(files)]" string => canonifyuniquely($(files));
"sizes[$(cfiles[$(files)])]" string => filestat($(files), "size");
"total_size" real => sum(getvalues(sizes));
reports:
"Total sizes = $(total_size)";
}
#+end_src

#+begin_src text
% cf-agent -KI ./test_filestats.cf
R: Total sizes = 741607.000000
#+end_src

HTH
Ted

[Christian Linden]

very cool, Ted, thanks a lot!

Chris

[Christian Linden]

Ted,

why do I get this:

cfetst1:/var/cfengine/masterfiles/R4Promises # cf-promises ./size.cf

 {'32','17299','436860','45','60','27352'})

  arg[0] range [a-zA-Z0-9_$(){}\[\].:]+ 32

   error: Fatal CFEngine error: Bad arguments

cfetst1:/var/cfengine/masterfiles/R4Promises # cf-agent -f ./size.cf

 {'32','17299','436860','45','60','27352'})

  arg[0] range [a-zA-Z0-9_$(){}\[\].:]+ 32

   error: Fatal CFEngine error: Bad arguments

   error: Policy failed validation with command '"/var/cfengine/bin/cf-promises" -c "./size.cf"'

   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

   error: Error reading CFEngine policy. Exiting...

cfetst1:/var/cfengine/masterfiles/R4Promises # cf-agent -KI ./size.cf

 {'32','17299','436860','45','60','27352'})

  arg[0] range [a-zA-Z0-9_$(){}\[\].:]+ 32

   error: Fatal CFEngine error: Bad arguments

   error: Policy failed validation with command '"/var/cfengine/bin/cf-promises" -c "./size.cf"'

   error: Failsafe condition triggered. Interactive session detected, skipping failsafe.cf execution.

   error: Error reading CFEngine policy. Exiting...

=(

cfetst1:/var/cfengine/masterfiles/R4Promises # cat size.cf

body common control

    {

        bundlesequence => { "size" };

    }

bundle agent size

{

  vars:

      "files" slist => findfiles("/tmp/*.txt");

      "cfiles[$(files)]" string => canonifyuniquely($(files));

      "sizes[$(cfiles[$(files)])]" string => filestat($(files), "size");

      "total_size" real => sum(getvalues(sizes));

  reports:

      "Total sizes = $(total_size)";

}

Thank you!

Chris

[Ted Zlatanov]

On Mon, 12 Sep 2016 04:45:07 -0700 (PDT) Christian Linden <lindo...@gmail.com> wrote:

CL> Ted,
CL> why do I get this:
CL> cfetst1:/var/cfengine/masterfiles/R4Promises # cf-promises ./size.cf
CL> {'32','17299','436860','45','60','27352'})
CL> arg[0] range [a-zA-Z0-9_$(){}\[\].:]+ 32
CL> error: Fatal CFEngine error: Bad arguments

I don't know.

CL> "files" slist => findfiles("/tmp/*.txt");

Can you show that file list on your system?

What version of CFEngine are you running? My code targets the latest release.

Ted

[Christian Linden]

3.8.2 Core

You are right, it's cause by the files, on another box it shows 0.00000 but there're 150k used:

HSLin2AddDsks:/var/cfengine/inputs/R4Promises # ls -altr /tmp/*.txt

-rw------- 1 root root 150 Jul 19 11:44 /tmp/execresult_testfile.txt

HSLin2AddDsks:/var/cfengine/inputs/R4Promises # cf-agent -f ./size.cf

R: Total sizes = 0.000000

Here's the file list from where "Bad arguments" is returned:

cfetst1:/var/cfengine/masterfiles/R4Promises # ls -al /tmp/*.txt

-rw-r--r-- 1 root root  17299 Feb  4  2014 /tmp/Readme.txt

-rw-r--r-- 1 root root     32 Jun  2 17:41 /tmp/append_and_repair.txt

-rw-r--r-- 1 root root     45 Jun  2 18:37 /tmp/append_and_repair_cfetst1.testlab1.it-fabrik.local.txt

-rw------- 1 root root 436860 Aug 24 11:02 /tmp/execresult_testfile.txt

-r--r--r-- 1 root root  27352 Oct 27  2013 /tmp/license.txt

-rw-r--r-- 1 root root     60 Mar 23 14:16 /tmp/oo-data-cfetst1.testlab1.it-fabrik.local.txt

Nothing bad.. I even renamed the files with several "." in the name.. still the same, odd to me =)

Thanks,
Chris

[Christian Linden]

With this file list I get bad args, too:

-rw------- 1 root root 2645 Sep 13 12:38 cfengine-conns.2016-09-13-12-38-50

-rw------- 1 root root 1916 Sep 13 12:44 cfengine-conns.2016-09-13-12-44-30

-rw------- 1 root root 2240 Sep 13 12:49 cfengine-conns.2016-09-13-12-49-10

-rw------- 1 root root 1754 Sep 13 12:53 cfengine-conns.2016-09-13-12-53-50

-rw------- 1 root root 1596 Sep 13 12:59 cfengine-conns.2016-09-13-12-59-30

-rw------- 1 root root 1997 Sep 13 13:04 cfengine-conns.2016-09-13-13-04-10

-rw------- 1 root root 1673 Sep 13 13:08 cfengine-conns.2016-09-13-13-08-50

-rw------- 1 root root 1754 Sep 13 13:14 cfengine-conns.2016-09-13-13-14-30

-rw------- 1 root root 2240 Sep 13 13:19 cfengine-conns.2016-09-13-13-19-10

-rw-r--r-- 1 root root 2159 Sep 13 13:22 cfengine-conns.2016-09-13-13-22-21

-rw-r--r-- 1 root root 1835 Sep 13 13:23 cfengine-conns.2016-09-13-13-23-36

-rw------- 1 root root 1916 Sep 13 13:23 cfengine-conns.2016-09-13-13-23-53

-rw-r--r-- 1 root root 1596 Sep 13 13:25 cfengine-conns.2016-09-13-13-25-05

Chris

[Ted Zlatanov]

On Tue, 13 Sep 2016 04:11:32 -0700 (PDT) Christian Linden <lindo...@gmail.com> wrote:

CL> 3.8.2 Core

As I said in my earlier post, my example was for 3.9 or newer only
because it uses function wrapping: `sum(getvalues(...))`

It just won't work on 3.8.x or older. You have to rewrite it with
intermediate variables to hold the getvalues() call:

"size_values" slist => getvalues(sizes);
"total_size" real => sum(size_values);

Ted

[Christian Linden]

Sorry, Ted, read that a wrong way.. got it, and it works! =)

[root@hub masterfiles]# cf-agent -f ./size.cf 

R: Total sizes = 25381.000000

Thanks again!

Chris

[Christian Linden]

But what kind of calc is it showing 26178 where it's 92K?

[root@hub size]# du -ha

4.0K ./host.conf

4.0K ./sudo-ldap.conf

0 ./gai.conf

4.0K ./yum.conf

4.0K ./gssapi_mech.conf

4.0K ./sysctl.conf

4.0K ./resolv.conf

4.0K ./request-key.conf

4.0K ./mke2fs.conf

4.0K ./dracut.conf

4.0K ./nfsmount.conf

4.0K ./mySizeTest.tst

4.0K ./ld.so.conf

4.0K ./libaudit.conf

4.0K ./idmapd.conf

4.0K ./sudo.conf

4.0K ./krb5.conf

4.0K ./sestatus.conf

4.0K ./rsyslog.conf

4.0K ./logrotate.conf

4.0K ./nsswitch.conf

4.0K ./libuser.conf

4.0K ./grub.conf

92K .

[root@hub size]# du -hs

92K .

[root@hub size]# vim size.cf

[root@hub size]# cd -

/var/cfengine/masterfiles

[root@hub masterfiles]# vim size.cf

[root@hub masterfiles]# cf-agent -f ./size.cf 

R: Total sizes = 26178.000000

[root@hub masterfiles]# 

Is it cylinders, sectors or what?

Chris

[Shane McEwan]

'du' reports on physical disk usage so all your small files are showing
as 4K as that must be the default block size on your filesystem. Files
will use up to the nearest block boundary even if they're not actually
that big.

Try running 'du' with the '--apparent-size' option and see if that
matches your CFEngine output.

Shane.

> [root@hub masterfiles]# cf-agent -f ./size.cf <http://size.cf>

>
> R: Total sizes = 25381.000000
>
>
> Thanks again!
>
> Chris
>

> --
> You received this message because you are subscribed to the Google
> Groups "help-cfengine" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to help-cfengin...@googlegroups.com
> <mailto:help-cfengin...@googlegroups.com>.

> To post to this group, send email to help-c...@googlegroups.com

> <mailto:help-c...@googlegroups.com>.

[Christian Linden]

[root@hub masterfiles]# du --apparent-size -b /size/

31084 /size/

[root@hub masterfiles]# cf-agent -f ./size.cf 

R: Total sizes = 26178.000000

[root@hub masterfiles]# 

Seems to be closed to bytes..

c

> <mailto:help-cfengine+unsub...@googlegroups.com>.