Checking for free disk space on mountpoint?

[Victor Hooi]

Hi,

I'm currently looking at migrating from some home-rolled Python scripts to using Puppet =).

One of the things we need to check for is that there is sufficient free disk space on a mountpoint (for logging etc.) this is currently quite easy in Python using statvfs().

Are there any resources in Puppet that might be able to do this? I checked the mount type (http://docs.puppetlabs.com/references/stable/type.html#mount), but it didn't seem to have anything to do this.

Cheers,

Victor

[Ken Barber]

Let me understand this correctly - you want to use the Puppet RAL
layer to interrogate data such as free space on a mount?

At the moment you can't use RAL data elsewhere in Puppet content ...
but you can use the RAL layer in your own code and in things like
MCollective using the ralsh plugin ...

Perhaps you're after facts?

http://docs.puppetlabs.com/guides/custom_facts.html

Excuse me if my understanding of your needs is completely out of whack
:-). Maybe you want to describe what you want to do with this data
once you have gathered it.

ken.

> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Users" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/puppet-users/-/lE1ahe-Z5KoJ.
> To post to this group, send email to puppet...@googlegroups.com.
> To unsubscribe from this group, send email to
> puppet-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/puppet-users?hl=en.
>

[Kent]

HI Victor,

You can write a simple custom Fact (leveraging the Facter library that
Puppet uses) that will run something like 'df' and parse the output.
Your puppet agents will report their disk usage, along with all the
other Facts, every time they make a Puppet run.

-Kent

[Doug Warner]

Here's my hack-y fact that adds facts for mounted partitions on linux w/ disk
used, available, and total as well as a fact for all available mounts.

-Doug

--- begin ---
mounts = []
mntpoints=`mount -t ext2,ext3,ext4,reiserfs,xfs`
mntpoints.split(/\n/).each do |m|
mount = m.split(/ /)[2]
mounts << mount
end

Facter.add("mounts") do
confine :kernel => :linux

setcode do
mounts.join(',')
end
end

mounts.each do |mount|
output = %x{df #{mount}}
output.each do |str|
dsk_size = nil
dsk_used = nil
dsk_avail = nil
if str =~ /^\S+\s+([0-9]+)\s+([0-9]+)\s+([0-9]+)\s+/
dsk_size = $1
dsk_used = $2
dsk_avail = $3
Facter.add("mount_#{mount}_size") do
setcode do
dsk_size
end
end
Facter.add("mount_#{mount}_used") do
setcode do
dsk_used
end
end
Facter.add("mount_#{mount}_avail") do
setcode do
dsk_avail
end
end
end
end
end
--- end ---

[Victor Hooi]

heya,

Excuse me if my understanding of your needs is completely out of whack :-). Maybe you want to describe what you want to do with this data once you have gathered it.

The reason we are doign this is to make sure that we have sufficient free disk space on various mountpoints to actually run the application on the server.

So this is something I'd like to enforce (or "ensure") as part of the configuration. Which would best point to integrate this Facter-based check into a Puppet configuration for a node?

@Doug: Thanks heaps for your Facter script =). Hmm, the only thing is, ideally I'd like it to be cross-platform (Linux, Solaris and Windows) - it seems there isn't really a abstracted way of checking this in-built into Facter/Puppet, which is a shame.

We also need to check the file permissions on certain files, to make sure the application will run correctly. I searched and found this:

http://projects.puppetlabs.com/projects/1/wiki/File_Permission_Check_Patterns

which seems to do the job.

I would have thought these two things (checking free disk space on various network mountpoints, and checking file/directory permissions) would be two fairly common tasks when configuring out boxes. How do people normally tackle this with Puppet?

Cheers,

Victor

[Ken Barber]

> The reason we are doign this is to make sure that we have sufficient free
> disk space on various mountpoints to actually run the application on the
> server.

Oh - and your monitoring doesn't do this already?

> So this is something I'd like to enforce (or "ensure") as part of the
> configuration. Which would best point to integrate this Facter-based check
> into a Puppet configuration for a node?

I guess so ... facter can be used as variables to decide if a
configuration can be applied. You could use conditionals to decide if
and service has ensure => running/stopped based on something from
facter using conditional. But this is during Puppet runtime only ...

> @Doug: Thanks heaps for your Facter script =). Hmm, the only thing is,
> ideally I'd like it to be cross-platform (Linux, Solaris and Windows) - it
> seems there isn't really a abstracted way of checking this in-built into
> Facter/Puppet, which is a shame.

Not necessarily. You can make decisions in Facter based on other
facts. 'operatingsystem' can be one of those facts. For example:

case Facter.value(:operatingsystem)
when "Solaris"
# ... Solaris related lookup code
when "centos", "redhat"
# ... centos/redhat related df lookup code
else
# ... some default behaviour
end

> We also need to check the file permissions on certain files, to make sure
> the application will run correctly. I searched and found this:
> http://projects.puppetlabs.com/projects/1/wiki/File_Permission_Check_Patterns
>
> which seems to do the job.
> I would have thought these two things (checking free disk space on various
> network mountpoints, and checking file/directory permissions) would be two
> fairly common tasks when configuring out boxes. How do people normally
> tackle this with Puppet?

I don't personally for application stop/start scripts. If I wanted to
I would put it in the system V startup script or whatever mechanism is
kicking around. That way it works for Puppet and human users who just
jump on a box and want to use the systems standard stop/start
mechanism.

Problem is - you do it all in Puppet ... only Puppet can stop and
start it. You could write start.pp and a stop.pp script if you wanted
and write all the scripting logic in Puppet I guess. Not normally done
though.

I can't say I have tested mount space before application start. I
usually rely on monitoring to tell me there is a problem there. Is
there a specific reason you need this, ie. avoid some odd corruption
during disk fulls or something? I would imagine the chances of
corruption during a "disk full" are much higher _after_ the
application has started and been running for a while though :-). Hence
why I guess I rely on monitoring more often ... monitoring will alert
me to a disk full even when I'm not trying to start the application.

ken.

[Victor Hooi]

Ken,

Currently, the existing processes we have are quite ad-hoc, and developers are often responsible for deployment, which is done by hand. I'm hoping Puppet will automate the process, provide better standardisation, and segregate development away from deployment/production.

All the things I'm attempting to check for (e.g. disk space, file/directory permissions) are things that somebody currently does manually from the command-line (e.g. using "df", "touch x" in a directory).

I need to to automate it all into Puppet to show that it's a viable option and can replace the manual processes, and then in phase two look at getting things like Nagios (for disk monitoring, or whatever else we have used elsehwere in the organisation for monitoring) up and running in tandem as well.

The disk space needs to be monitored because parts of the application don't handle full disks well and will crash nastily. This is only an issue intra-day. Basically, we spin up the application in the morning, and we know with good certainty the maximum space they'll consume, then the application is killed off at the end of the day. As long as we have more the <x> bytes available when we install things, we know we'll be fine.

The permissions is something I'd like to ensure when we setup a box - the directory/files are setup for us by sys-admins, and we just want to run some sanity checks to make sure they've set it up properly for us. Currently there's a lot of to-ing and fro-ing to check everythign is setup by the sys-admins correctly, I'd like to simply run-up Puppet, and have it ensure for these things.

So the best way is to write custom facts for all the things we're sanity checking, then just do a ensure-> on them, right?

I couldn't find anything on http://forge.puppetlabs.com/ about permissions or disk space...lol, guess they're too small/trivial to be there. There aren't other sources of these things, are there?

Cheers,

Victor

[vagn scott]

On 07/11/2011 10:37 PM, Victor Hooi wrote:
> As long as we have more the <x> bytes available when we install
> things, we know we'll be fine.

Here is a proof of concept.
With a little tweaking you should be able to test for
an absolute amount of free space on any mounted volume.

Hope that helps,

--vagn

#! /usr/bin/puppet apply

Exec {
path =>
"/u0/home/vagn/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
}

$script = inline_template("#! /bin/bash

d=\$1
p=\$2

t=\$( df -h \$d | tail -n +2 | sed 's/%//g' | while read a b c d e f ;
do echo \$e ; done )

if test \$t -gt \$p
then
echo \$t greater than \$p PASSED
exit 0
else
echo \$t not greater than \$p FAILED
exit 1
fi

)")

define diskspace($dev, $percent) {

exec { "diskspace $title $dev $percent":
command => "/usr/local/bin/diskspace-script ${dev}
${percent}",
logoutput => true,
require => File [ "/usr/local/bin/diskspace-script" ],
}

}

node default {

file { "/usr/local/bin/diskspace-script":
content => $script,
mode => 775,
}

diskspace { "check 1": dev => "/dev/sda1", percent => "30", }
diskspace { "check 2": dev => "/dev/sda1", percent => "40", }

}

[Patrick]

On Jul 11, 2011, at 7:37 PM, Victor Hooi wrote:

> Ken,
>
> Currently, the existing processes we have are quite ad-hoc, and developers are often responsible for deployment, which is done by hand. I'm hoping Puppet will automate the process, provide better standardisation, and segregate development away from deployment/production.
>
> All the things I'm attempting to check for (e.g. disk space, file/directory permissions) are things that somebody currently does manually from the command-line (e.g. using "df", "touch x" in a directory).
>
> I need to to automate it all into Puppet to show that it's a viable option and can replace the manual processes, and then in phase two look at getting things like Nagios (for disk monitoring, or whatever else we have used elsehwere in the organisation for monitoring) up and running in tandem as well.
>
> The disk space needs to be monitored because parts of the application don't handle full disks well and will crash nastily. This is only an issue intra-day. Basically, we spin up the application in the morning, and we know with good certainty the maximum space they'll consume, then the application is killed off at the end of the day. As long as we have more the <x> bytes available when we install things, we know we'll be fine.
>
> The permissions is something I'd like to ensure when we setup a box - the directory/files are setup for us by sys-admins, and we just want to run some sanity checks to make sure they've set it up properly for us. Currently there's a lot of to-ing and fro-ing to check everythign is setup by the sys-admins correctly, I'd like to simply run-up Puppet, and have it ensure for these things.
>
> So the best way is to write custom facts for all the things we're sanity checking, then just do a ensure-> on them, right?

So, here's what I see you doing. Tell me if I'm confused.

1) You want to do an "ensure" on permissions to make sure "stuff" is set right. In general this isn't done with facts. You just declare it, and puppet makes it so.

2) You have resources you don't want to run if there's not enough space available. This can be done by creating a "free_disk_space" fact and using that in an if statement that wraps resources. Keep in mind that, this being unix, there's no "drives" so if you will probably be checking the amount of space on root if you don't think about it. This might be what you want. If not you can create facts like free_space_var or whatever.

[vagn scott]

On 07/12/2011 01:05 AM, Patrick wrote:
> Keep in mind that, this being unix, there's no "drives" so if you will probably be checking the amount of space on root if you don't think about it.

There most certainly are drives in unix. Not drive letters, though.
The drives are normally denoted either by their device special file,
such as /dev/md127, or by their mount points, such as /u0.

Here is a list of everything mounted on one particular system.

vagn@nika:~$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 28G 9.3G 17G 36% /
tmpfs 7.9G 0 7.9G 0% /lib/init/rw
udev 7.9G 324K 7.9G 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/md127 2.7T 846G 1.8T 33% /u0
/dev/mapper/udisks-luks-uuid-3bb2bdd5-1b84-49b4-abe8-4a107f174702-uid1000
15G 7.9G 6.2G 56% /u0/home/vagn/s5
/dev/mapper/udisks-luks-uuid-d5ff5dbd-053b-48a7-b3ee-4f71a2e5c2fb-uid1000
2.7T 855G 1.8T 33% /bu02
/dev/sr0 6.8G 6.8G 0 100% /media/cdrom0

And here is the listing of one drive by device:

vagn@nika:~$ df -h /dev/md127
Filesystem Size Used Avail Use% Mounted on
/dev/md127 2.7T 846G 1.8T 33% /u0
vagn@nika:~$

And the same by mount point:

vagn@nika:~$ df -h /u0
Filesystem Size Used Avail Use% Mounted on
/dev/md127 2.7T 846G 1.8T 33% /u0

See man pages for stat, statvfs or statfs for details.

Also, the stat command is interesting:

vagn@nika:~$ stat -f -c "freeblocks on /u0 = %a" /u0
freeblocks on /u0 = 462905880

vagn@nika:~$ bc <<.
462905880 * 4
.
1851623520

vagn@nika:~$ df /u0
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/md127 2884294872 886157536 1851623520 33% /u0

--
vagn

[Patrick]

On Jul 11, 2011, at 11:10 PM, vagn scott wrote:

> On 07/12/2011 01:05 AM, Patrick wrote:
>> Keep in mind that, this being unix, there's no "drives" so if you will probably be checking the amount of space on root if you don't think about it.

> There most certainly are drives in unix. Not drive letters, though.

Ah. I misspoke. I meant that the path to where you put a file on the filesystem doesn't explicitly mention a drive name. For instance, you can't tell looking at the path if / and /var are on the same drive, where as you (usually) can tell with Windows.