Jira (FACT-1814) Mountpoints Fact available space does not accurately reflect disk usage

[Martin Ewings (JIRA)]

Martin Ewings created an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage Issue Type: Bug Affects Versions: FACT 3.6.5 Assignee: Unassigned Attachments: output.txt Created: 2018/02/02 6:56 AM Priority: Normal Reporter: Martin Ewings PE 2017.2.2 ( facter 3.6.5) RHEL 6.8 The customer is using mountpoints.(parition).available_bytes in conditional logging to determine if software should be installed on a system during Puppet Runs. In testing it appears that there is around a 400mb discrepancy between what is reported by facter as available, and what is actually available, which is backed up by df -h. In a test where by continual facter outputs and df outputs where collected during a large file transfer to the target partition, both facter and df -h tracked upwards in consumed disk space, but never agreed with each other, full attachments below, however a snippet can be seen here: ----facter---- 2568753152 2.39 GiB 58.62% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 3.4G 2.1G 62% /var ----facter---- 2568753152 2.39 GiB 58.62% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 3.4G 2.1G 62% /var ----facter---- 846790656 773.65 MiB 87.52% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.1G 431M 93% /var ----facter---- 415592448 396.34 MiB 93.30% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.4G 90M 99% /var ----facter---- 415592448 396.34 MiB 93.30% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.4G 90M 99% /var ----facter---- 415592448 396.34 MiB 93.30% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.4G 90M 99% /var ----facter---- 421269504 401.75 MiB 93.21% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.4G 95M 99% /var ----facter---- 421269504 401.75 MiB 93.21% ----system---- Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4 5.8G 5.4G 95M 99% /var It could be that facter is failing to take into consideration the blocksize for the partition, in which case i have provided the output below: blockdev --getbsz /dev/mapper/vg_root-lv_var 4096 Add Comment

This message was sent by Atlassian JIRA (v7.0.2#70111-sha1:88534db)

[Martin Ewings (JIRA)]

Martin Ewings updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage

Change By: Martin Ewings

PE 2017.2.2 ( facter 3.6.5) RHEL  6.8

The customer is using mountpoints.(parition).available_bytes in conditional  logging  logic  to determine if software should be installed on a system during Puppet Runs.

In testing it appears that there is around a 400mb discrepancy between what is reported by facter as available, and what is actually available, which is backed up by df -h. In a test where by continual facter outputs and df outputs where collected during a large file transfer to the target partition,  both facter and df -h tracked upwards in consumed disk space, but never agreed with each other, full attachments below, however a snippet can be seen here:

{code:java}

{code}

It could be that facter is failing to take into consideration the blocksize for the partition, in which case i have provided the output below:  blockdev --getbsz /dev/mapper/vg_root-lv_var  4096

Add Comment

[Martin Ewings (JIRA)]

Martin Ewings updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage Change By: Martin Ewings

PE 2017.2.2 ( facter 3.6.5) RHEL  6.8

The customer is using mountpoints.(parition).available_bytes in conditional logic to determine if software should be installed on a system during Puppet Runs.

Used space seems also to agree where available does not:  used => "5.39 GiB", for facter and 5.4G in df -h

Add Comment

[Owen Rodabaugh (JIRA)]

Owen Rodabaugh updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage

The output of `statfs` and `df` are different which is the reason for this behavior: https://stackoverflow.com/questions/21693673/differ-in-output-of-df-and-statfs It's unclear which one is correct if you attempt to install to those location. Should this instead a request to change facter to be based on `df`? Change By: Owen Rodabaugh CS Priority: Needs Priority Minor CS Impact: In a situation where a customer is relying on facter to determine if there is space to install something this could be frustrating. That said there is are known discrepancies between `statfs` which facter uses and df which the customer is using in this case. CS Severity: 2 - Annoyance CS Business Value: 3 - $$$$ CS Frequency: 2 - 5-25% of Customers

Add Comment

This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage Facter is using the f_bfree value returned by statfs():   https://github.com/puppetlabs/facter/blob/3.9.4/lib/src/facts/linux/filesystem_resolver.cc#L112-L113   Sounds like df uses the f_bavail entry:   https://stackoverflow.com/questions/21693673/differ-in-output-of-df-and-statfs   The difference between the two is that f_bavail reports the space available to non-root users: fsblkcnt_t f_bfree; /* Free blocks in filesystem */ fsblkcnt_t f_bavail; /* Free blocks available to unprivileged user */ The man page also notes that: LSB has deprecated the library calls statfs() and fstatfs() and tells us to use statvfs(2) and fstatvfs(2) instead. http://man7.org/linux/man-pages/man2/statfs.2.html

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Facter / FACT-1814

Mountpoints Fact available space does not accurately reflect disk usage

Change By: Adam Bottchen

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage As Charlie Sharpsteen mentioned, this difference is due to the use of f_bfree rather than f_bavail.  Facter is reporting the free space on the disk whereas df is reporting the amount of space available to unprivileged users.  This was a design decision, so this isn't a bug. It would be a potentially breaking change to switch facter to f_bavail, since customers may be relying on it returning the actual amount of free space available rather than the unprivileged space.  As a result, I suspect the best solution would be to switch this to a feature request asking to add the unprivileged space to the mountpoints structured fact.  That preserves existing functionality while providing the additional correlation with df.

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Facter / FACT-1814

Mountpoints Fact available space does not accurately reflect disk usage

Change By: Adam Bottchen Issue Type: Bug New Feature

Add Comment

[Adam Bottchen (JIRA)]

Adam Bottchen updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage Change By: Adam Bottchen

PE 2017.2.2 ( facter 3.6.5) RHEL  6.8 The customer is using mountpoints.(parition).available_bytes in conditional logic to determine if software should be installed on a system during Puppet Runs. In testing it appears that there is around a 400mb discrepancy between what is reported by facter as available, and what is actually available, which is backed up by df -h. In a test where by continual facter outputs and df outputs where collected during a large file transfer to the target partition,  both facter and df -h tracked upwards in consumed disk space, but never agreed with each other, full attachments below, however a snippet can be seen here: {code:java}

----facter---- 2.39 GiB 58.62%

----system---- Filesystem                 Type  Size  Used Avail Use% Mounted on /dev/mapper/vg_root-lv_var ext4  5.8G  3.4G  2.1G  62% /var

----facter---- 2.39 GiB 58.62%

EDIT:  Add an "unprivileged_free_space" element to the mountpoints structured fact that keys off of b_avail.  This value will reflect what is see in df output, which more closely matches what users expect to see.

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage Although, given statfs() draws a distinction between "available" and "free", our use of the label "available" to represent what statfs() calls "free" was either a poor naming choice or an indicator of a bug.

Add Comment

[Scott McClellan (JIRA)]

Scott McClellan updated an issue

Facter / FACT-1814

Mountpoints Fact available space does not accurately reflect disk usage

Change By: Scott McClellan Team: Platform OS

Add Comment

[Branan Riley (JIRA)]

Branan Riley commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage Agreed this could be a breaking change - but that can be said for every fact change. We sort of have to evaluate the risk on a case-by-case basis. In this situation I think this might be worth going for. I'm not entirely sure what the reasons for choosing f_bfree instead of f_bavail were, and I suspect that f_bavail is more correct in more cases, since most files written on a node are gonna be owned by not-root (one hopes, anyway). I don't have as much insight into how customers use this fact as folks in CS do, though - I very much value your input in evaluating the risk of changing things here. Worst case, we're coming up on Platform 6 and could schedule this for Facter 4. We should at the very least fix Facter to not use the deprecated API (as long as the not-deprecated version is available on all the kernel/libc versions we care about). That sounds like it's probably a different ticket, though.

Add Comment

[Branan Riley (JIRA)]

Branan Riley updated an issue

Facter / FACT-1814

Mountpoints Fact available space does not accurately reflect disk usage

Change By: Branan Riley Labels: breaking linux triaged

Add Comment

This message was sent by Atlassian JIRA (v7.7.1#77002-sha1:e75ca93)

[Branan Riley (JIRA)]

Branan Riley updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage

Change By: Branan Riley Fix Version/s: FACT 4.0.0

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage Since you're messing around with mountpoints...there needs to be some way to exclude mountpoints from fact gathering. Some sites with large storage needs may have 1024+ LUNs and the facter run querying all of them is not a good thing. On the point at hand, why not report both? Add a new fact that reports 'unprivileged space' or something. That way, it's backwards compatible and you don't break things for people that run facter as a non-root user (please don't break things for non-root users).

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage

Trevor Vaughan Check out FACT-718 which implemented the ability to block facts from collection — mountpoints were a pain point specifically targeted. Docs: https://puppet.com/docs/facter/3.10/configuring_facter.html#facts

Add Comment

[Trevor Vaughan (JIRA)]

Trevor Vaughan commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage

Charlie Sharpsteen Thanks! Is there a whitelist mode or do I need to implement my own custom fact to just get the "usual suspects" without running off the rails into insanity?

Add Comment

[Charlie Sharpsteen (JIRA)]

Charlie Sharpsteen commented on FACT-1814

Re: Mountpoints Fact available space does not accurately reflect disk usage

No whitelist at the moment. Current functionality allows adding pre-defined groupings of builtin facts to a blacklist.

Add Comment

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Facter / FACT-1814

Mountpoints Fact available space does not accurately reflect disk usage

Change By: Josh Cooper Fix Version/s: FACT 4.0.0

Add Comment

This message was sent by Atlassian Jira (v8.20.11#820011-sha1:0629dd8)

[Claudia Petty (Jira)]

Claudia Petty updated an issue

Facter / FACT-1814 Mountpoints Fact available space does not accurately reflect disk usage

Change By: Claudia Petty Labels: breaking linux new-feature

Add Comment

This message was sent by Atlassian Jira (v8.20.21#820021-sha1:38274c8)