Filesystem reporting wrong capacity / free- / used space

[arritje nof]

HI,

I recently discovered that our filesystem is reporting erroneous filesystem-statistics.

For instance, i'm using our fork of the loopback filesystem to (re-)mount a subdirectory of a storage-cluster volume, shared through SMB and mounted at /Volumes/UserWorkspaces. 

Normal behaviour would be that this fuse-mounted subdirectory filesystem reports the same statistics as the underlying SMB-share. I've attached images that illustrate the situation. As the images show, the reported statistics seem very odd, and definately not correct.

They're even contradictive, as it's impossible to have over 150 times more space available (61.82 TB) than there is actual capacity (484.04 GB).

i'm not doing anything special in the (i asume) related function(s) in our filesystem.

Am i correctly asuming that this information is returned by statfs?

I've found that this issue only occurs with this specific storage-cluster, but it occurs both when when mounting a volume using SMB AND when using NFS.

Where does it go wrong?

And is there a straightforward way to work around this?

In the text below, you can see 2 SMB volumes, mounted from different clusters. They correspond to the LOOP and SMB fuse-filesystems respectively.

A third mounted volume from the first cluster is mounted using NFS, and corresponds to the fuse-filesystem mounted as NFS

Finally, a forth SMB-share UserWorkspaces,comes from another (non-cluster) server, is a regular Windows SMB-share and corresponds to fuse filesystem SUPER.

This last one reports stats as expected (correctly).

Here some data to illustrate the problem

$ > df -h

Filesystem                                      Size   Used  Avail Capacity   iused      ifree %iused  Mounted on

/dev/disk0s2                                   178Gi  147Gi   31Gi    83%  38538823    8210604   82%   /

devfs                                          275Ki  275Ki    0Bi   100%       951          0  100%   /dev

/dev/disk0s4                                    59Gi   54Gi  4.9Gi    92%  14237211    1285461   92%   /Volumes/HD2

map -hosts                                       0Bi    0Bi    0Bi   100%         0          0  100%   /net

/dev/disk1s2                                   477Gi  444Gi   33Gi    94% 116434354    8565644   93%   /Volumes/data

//MXF...@192.168.180.50/ifs                   125Ti  8.0Ti  117Ti     7%         0 18446744073709551615    0%   /Volumes/ifs

//ad...@macprotest.local/backup                596Gi  501Gi   95Gi    85% 131371050   24878948   84%   /Volumes/backup

loopback@osxfuse0                              451Gi    0Bi   56Ti     0%         0 4294967295    0%   /mnt/LOOP

//Admini...@192.168.180.90/UserWorkSpaces  6.4Ti  383Gi  6.0Ti     6%         0 18446744073709551615    0%   /Volumes/UserWorkSpaces

loopback@osxfuse1                              6.4Ti  383Gi  6.0Ti     6%         0 4294967295    0%   /Volumes/SUPER

//ro...@192.168.180.61/ifs                      106Gi   39Gi   68Gi    37%         0 18446744073709551615    0%   /Volumes/ifs-1

192.168.180.51:/ifs                            125Ti  6.0Ti  117Ti     5% 9454029688 189482930312    5%   /Volumes/ifs-2

loopback@osxfuse2                              1.4Ti  4.3Gi  1.4Ti     1% 864095096  504369288   63%   /Volumes/NFS

and the output of mount :

/dev/disk0s2 on / (hfs, NFS exported, local, journaled)

devfs on /dev (devfs, local, nobrowse)

/dev/disk0s4 on /Volumes/HD2 (hfs, local, journaled)

map -hosts on /net (autofs, nosuid, automounted, nobrowse)

/dev/disk1s2 on /Volumes/data (hfs, local, journaled)

//192.168.180.50/ifs on /Volumes/ifs (smbfs, nodev, nosuid, mounted by arri)

//ad...@macprotest.local/backup on /Volumes/backup (afpfs, nodev, nosuid, mounted by arri)

loopback@osxfuse0 on /mnt/LOOP (mxffusefs, nodev, nosuid, synchronous, mounted by arri)

//ad...@192.168.180.90/UserWorkSpaces on /Volumes/UserWorkSpaces (smbfs, nodev, nosuid, mounted by arri)

loopback@osxfuse1 on /Volumes/SUPER (mxffusefs, nodev, nosuid, synchronous, mounted by arri)

//ad...@192.168.180.61/ifs on /Volumes/ifs-1 (smbfs, nodev, nosuid, mounted by arri)

192.168.180.51:/ifs on /Volumes/ifs-2 (nfs, nodev, nosuid, mounted by arri)

loopback@osxfuse2 on /Volumes/NFS (mxffusefs, nodev, nosuid, synchronous, mounted by arri)

And this is how i mount the fuse filesystem:

# SMB -> LOOP

mkdir -p /Volumes/LOOP

loopback /Volumes/LOOP -omodules=threadid:subdir,volname=LOOP,iosize=1048576,subdir=/Volumes/ifs/UserWorkspaces

etc...

I'm using tha latest osxfuse (2.7.5), but i found that the problem goes back at least to version 2.6.2.

Thanks!

Kind regards.

arri

[Sam Moffatt]

Without taking a look is it possible that the block size is different and the free block count is just being passed through causing a difference in free space?

Cheers,

Sam

--

Sam Moffatt

http://pasamio.id.au

Note: I'm not at my desk, responses may be delayed. Apologies for the typos, smartphones aren't all that smart. 

--
You received this message because you are subscribed to the Google Groups "OSXFUSE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-grou...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

<getinfo.png>

[arritje nof]

Hi Sam,

Thanks for your reply!

The blocksizes reported by the filesystem mounted over SMB, are different from the same filesystem mounted over NFS. In both cases, re-mounting those (SMB-/NFS-) mounts using osxfuse/loopback, gives yet entirely different results. 

To not complicate things too much i'll forget NFS for now and focus on SMB.

Below are the results of statfs on the SMB-mount, followed by the output of statfs on the re-mounted loopback-mountpoint of that SMB-mount.

SMB

                                   block-size (bsize): 16000

                                     io-size (iosize): 1048576

                           total block-count (blocks): 8619477744

                                  free blocks (bfree): 8068507740

         blocks available for non-root users (bavail): 8068507740

                      total nr. of file nodes (files): 18446744073709551615

                              free file-nodes (ffree): 18446744073709551615

                                                  ...

                                         fs-type name: smbfs

                                      mountpoint name: /Volumes/ifs

                                   mounted filesystem: //ro...@192.168.180.51/ifs

-----------

OSXFuse

                                   block-size (bsize): 16384

                                     io-size (iosize): 1048576

                           total block-count (blocks): 29543152

                                  free blocks (bfree): 3773540444

         blocks available for non-root users (bavail): 3773540444

                      total nr. of file nodes (files): 4294967295

                              free file-nodes (ffree): 4294967295

                                                  ...

                                         fs-type name: mxffusefs

                                      mountpoint name: /Volumes/fuseSMB

                                   mounted filesystem: loopback@osxfuse1

As you can see, reported blocksizes are different, but only a little amount, which doesn't don't explain the huge differences in the numbers i see.

What strikes me most, is the number 4294967295 (2^32) which might be the root of the problem.

(I have -D_FILE_OFFSET_BITS=64 defined in my code.)

It seems i'm running into a 32-bit boundary. Where would this limit be hit?

thanks,

kind regards.

arri

[Sam Moffatt]

What that's actually telling me is something more interesting IMHO: SMB is reporting significantly more blocks free than what OSXFUSE is reporting. It's possible that something is 32-bit when it should be 64-bit, there was another thread earlier on how there are places where it's 32-bit in the kext it seems. You could double check that the statfs/statvfs call to the underlying filesystem is behaving properly at the root as it might be that it's getting lost in translation somewhere.

Sam Moffatt
http://pasamio.id.au

[arritje nof]

Hi Sam, 

Thanks for your answer!

Yes, that's exactly the problem i have; different sizes reported.

And appearantly it has been around forever, but it always went unnoticed, as it only shows on this one type/brand storage-cluster; Isilon One FS.

Sofar, I haven't seen this on any other system.

The problem seems to come from a number of fields in struct statvfs that are defined as fsblkcnt_t (__darwin_fsblkcnt_t) and fsfilcnt_t (__darwin_fsfilcnt_t), both of which are unsigned int:

struct statvfs {

unsigned long f_bsize; /* File system block size */

unsigned long f_frsize; /* Fundamental file system block size */

fsblkcnt_t f_blocks; /* Blocks on FS in units of f_frsize */

fsblkcnt_t f_bfree; /* Free blocks */

fsblkcnt_t f_bavail; /* Blocks available to non-root */

fsfilcnt_t f_files; /* Total inodes */

fsfilcnt_t f_ffree; /* Free inodes */

fsfilcnt_t f_favail; /* Free inodes for non-root */

unsigned long f_fsid; /* Filesystem ID */

unsigned long f_flag; /* Bit mask of values */

unsigned long f_namemax; /* Max file name length */

};

The problem becomes appearant when you print stat(v)fs- data, of both the mounted SMB-/NFS-share and the fuse-mounted volume that uses this share as it's backing-store. NB: for completion i've included both statfs- and statvfs-data, as this shows that the problem is really with statvfs and also occurs with non-fuse volumes like in the first example below. (stattest is a small tool that just prints the member-values of the two structs)

Storage-Cluster SMB-share

$ > ./stattest /Volumes/ifs-1/

Successful statfs-ed and statvfs-ed the filesystem!

statfs:

                             block-size (bsize): 16000

                                io-size (iosize): 1048576

                      total block-count (blocks): 8619477744

                             free blocks (bfree): 8060592940

    blocks available for non-root users (bavail): 8060592940

                 total nr. of file nodes (files): 18446744073709551615

                         free file-nodes (ffree): 18446744073709551615

                            filesystem id (fsid): { 805306389, 24 } 

                              mounted by (owner): 501

                          filesysyem type (type): 0x00000018

                  filesysyem subtype (fssubtype): 0x00000004

                                   flags (flags): 0x00000018

                                    fs-type name: smbfs

                                 mountpoint name: /Volumes/ifs-1

                              mounted filesystem: //ro...@192.168.180.50/ifs

statvfs:

   Fundamental file system block size (f_frsize): 16000

                File system block size (f_bsize): 1048576

    Blocks on FS in units of f_frsize (f_blocks): 29543152

                           Free blocks (f_bfree): 3765625644

         Blocks available to non-root (f_bavail): 3765625644

                          Total inodes (f_files): 4294967295

                           Free inodes (f_ffree): 4294967295

             Free inodes for non-root (f_favail): 4294967295

                          Filesystem ID (f_fsid): 805306389

                     Bit mask of values (f_flag): 0x00000002

                Max file name length (f_namemax): 255

done!

Fuse-mount.

$ > ./stattest /Volumes/SMBfuse/

Successful statfs-ed and statvfs-ed the filesystem!

statvfs:

   Fundamental file system block size (f_frsize): 16384

                File system block size (f_bsize): 1048576

    Blocks on FS in units of f_frsize (f_blocks): 29543152

                           Free blocks (f_bfree): 3765608304

         Blocks available to non-root (f_bavail): 3765608304

                          Total inodes (f_files): 4294967295

                           Free inodes (f_ffree): 4294967295

             Free inodes for non-root (f_favail): 4294967295

                          Filesystem ID (f_fsid): 822083612

                     Bit mask of values (f_flag): 0x00000002

                Max file name length (f_namemax): 255

statfs:

                              block-size (bsize): 16384

                                io-size (iosize): 1048576

                      total block-count (blocks): 29543152

                             free blocks (bfree): 3765608304

    blocks available for non-root users (bavail): 3765608304

                 total nr. of file nodes (files): 4294967295

                         free file-nodes (ffree): 4294967295

                            filesystem id (fsid): { 822083612, 25 } 

                              mounted by (owner): 501

                          filesysyem type (type): 0x00000019

                  filesysyem subtype (fssubtype): 0x00000000

                                   flags (flags): 0x0000001a

                                    fs-type name: mxffusefs

                                 mountpoint name: /Volumes/SMBfuse

                              mounted filesystem: loopback@osxfuse0

done!

Could this be a step in the right direction? (i'm not exactly a kernel hacker [yet]..)

If so, where would i need to start fixing this? You mention the kernel, does it need to be fixed there?

Sorry for posting these large chunks, but i hope they effectively illustrate the issue.

Thank's for your time!

Kind regards,

arri

[Sam Moffatt]

It feels like statvfs has a bug in it so my inclination would be to put all of the details together and file a bug with Apple. I'm not much of a kernel hacker either but I'm not sure this can be fixed outside of Apple. 

Cheers,

Sam

--

Sam Moffatt

http://pasamio.id.au

Note: I'm not at my desk, responses may be delayed

[arritje nof]

yek!

that doesn't sound very hopeful with regard to finding a solution sooner rather than later..

I will do some inquiring on the appropriate developer-mailinglist(s) and file a bugreport if applicable. Ofcourse i will keep you informed if there are relevant developments.

In the mean time, i was wondering why the Fuse statfs implementation on Mac OS X is called with a * struct statvfs parameter.

Would there perhaps be a way to simply use a struct stat for moving the data around, and avoid using struct statvfs alltogether?

That might make it possible to create a workaround, while we're waiting for Apple to fix the bug.

cheers,

kind regards,

arri

[arritje nof]

I forgot to mention that i'll also contact the Isilon-support/-engineers to see what they have to say about this.

Again, I will ofcourse forward any relevant information to this list.

gr

arri

[arritje nof]

Following-up on this, a reply from the filesys...@developer.apple.com mailing-list:

> I don’t think you’re doing anything wrong nor is OS X. statvfs is defined by a standard. From man statvfs:

> The statvfs() and fstatvfs() functions conform to IEEE Std 1003.1-2001 (``POSIX.1'').  As standardized, portable applications cannot depend on these functions returning any valid information at all.  This implementation attempts to provide as much useful information as is provided by the underlying file system, subject to the limitations of the specified data types.

> If you look at the published standard [1], it notes that:

> The implementation shall support one or more programming environments in which the widths of blksize_t, pid_t, size_t, ssize_t, suseconds_t, and useconds_t are no greater than the width of type long.

> The result is everything is working appropriately incorrect I think.

> [1] http://pubs.opengroup.org/onlinepubs/009695399/basedefs/sys/types.h.html#tag_13_67

> Matt Bauer

>

i love the phrasing "appropriately incorrect". but what he sais is basically that everything is working correctly according to specifications, but sadly in my case this means that the results are incorrect.

In other words: using statvfs in situations where a filesystem reports block- and inode-counts that exceed the limits of a 32bit unsigned int, is incorrect.

Is there a way to avoid using statvfs? As far as i can see, they're pretty interchangeable.

If the reason to use statvfs was portability, then considering

>  "As standardized, portable applications cannot depend on these functions returning any valid information at all"

maybe using statfs is a better option?

kind regards,

Arjen Keesmaat

[Sam Moffatt]

You could probably change your filesystems implementation to use statfs. This would be in your control since it's within your own file system. This should solve your problem?

Sam Moffatt
http://pasamio.id.au

[arritje nof]

Yes, that is my current workaround; I'm using statfs to make some sense of the numbers and their proportions in order to popupate struct statvfs with values that are as close as possible to what they should be, instead of the random numbers that remaining after chopping half the bits off the original 64bit values.

Ofcourse it's only a workaround, and less than ideal. It makes me wonder why this limitation exists in Mac OSX.

Also, i can hardly be the first to encounter this limitation?

cheers,

arjen

[Sam Moffatt]

The limitation exists because Mac OS X is following the POSIX standard. A few releases ago they had a minor marketing point about being a "true" UNIX due to the POSIX certification and it appears they've kept up the registration.

Sam Moffatt
http://pasamio.id.au

[arritje nof]

Hmm, yes. Seems to me that POSIX compliance doesn't mean up-to-date according latest standards.

Anyway, another question in this context;

I've managed to mitigate my immediate problems of too little reported available-/free-space, by simply assigning a member UINT_MAX whenever the actual value would overflow the 32bit type.

This works fine (file-system is usable again for writing huge files) but results in completely nonsense values.

One way of trying to get our filesystem to report sensible numbers for capacity and used-/free-/available-space, is to use bsize as a sort of wildcard. So for a filesystem with bsize 512 where the value for bsize would be too large, i could instead report bsize 16384 ( 512*32 )  and  blocks/32.

This seems to work more or less, but i imagine this potentially could break a lot of things down the road. 

Is altering the  bsize value a smart this todo? 

And would this be a valid approach in general?

thanks!

arri

[Sam Moffatt]

You could alter bsize for your file system though that might mean that you get writes in larger chunks than your underlying storage likes. Given it looks like you're dealing with a network file system the block size might not be as much of an issue for you. I suspect that you won't have too many issues however you'd have to try and see.

Fundamentally so long as you adhere to the contract of the VFS layer you shouldn't have too many problems.

Cheers,

Sam

Sam Moffatt
http://pasamio.id.au