OSXFUSE lowlevel readdir and 64-bit inodes

[bill...@navimatics.com]

I have developed a file system that uses 64-bit inodes. The file system passes multiple test suites and appears fully functional as long as I restrict inodes to remain in the 32-bit range (i.e. inodes less than 2^32). If however I allow the file system to use the full 64-bit inode range (i.e. inodes larger than 2^32), I am beginning to experience strangeness with the lowlevel readdir call.

For example, mounting the file system at /tmp/root, creating a directory and a file in it and then doing a simple ls -il returns the following:

$ cd /tmp/root

$ touch file

$ mkdir dir

$ ls -il

total 0

17179869192 -rw-r--r--  1 billziss  staff  0 Mar 31 17:06 file

The directory "dir" is missing from the ls output! It is there though:

$ stat -s dir

st_dev=788529165 st_ino=223338299392 st_mode=040755 st_nlink=1 st_uid=501 st_gid=20 st_rdev=0 st_size=0 st_atime=1427846789 st_mtime=1427846789 st_ctime=1427846789 st_birthtime=1427846789 st_blksize=65536 st_blocks=0 st_flags=0

You may notice that "dir" st_ino is 0x3400000000, i.e. the lower 32-bits are zero. This is by design in my file system: directories have their inode's low 32-bits set to 0. However I am now suspecting that readdir does not like such inode numbers, perhaps because the inode number is truncated to 32 bits somewhere between fuse_add_direntry and the user level process. Or it may be that the user level process itself is unable to cope with such inode numbers (still investigating this thought).

I have confirmed using the debugger that both directory entries ("file" and "dir") are reported to OSXFUSE using fuse_add_direntry. I have also confirmed that the stat.st_ino type is __darwin_ino64_t.

I would appreciate any insight on this. Although I could change my file system to use only 32-bit inode numbers, it would severely impact its ability to store large amounts of files and its performance.

Thank you.

[bill...@navimatics.com]

On Tuesday, March 31, 2015 at 5:34:41 PM UTC-7, I wrote:

I have developed a file system that uses 64-bit inodes. The file system passes multiple test suites and appears fully functional as long as I restrict inodes to remain in the 32-bit range (i.e. inodes less than 2^32). If however I allow the file system to use the full 64-bit inode range (i.e. inodes larger than 2^32), I am beginning to experience strangeness with the lowlevel readdir call.

I have done some further research and I believe I have found what causes the issue. The issue appears to be in the OSXFUSE kext and specifically fuse_internal_readdir_processdata().

To better understand the described issue I wrote a small program that performs a readdir and then stats all reported directory entries. For the simple filesystem containing the file "file" and directory "dir" it returned a single entry:

st_ino=0000000400000008, d_ino=0000000000000008, d_type=00, d_name=file

As before the directory entry for "dir" is missing. The directory entry for "file" is there. But notice that st_ino and d_ino defer. This means that stat(2) reports an inode of 0x400000008 and readdir(3) reports an inode of 0x8!

This clearly points to a truncation of the inode number somewhere. After double-checking that the culprit is not my file system I went through the OSXFUSE source code. As mentioned I believe that the problem is in the OSXFUSE kext, function fuse_internal_readdir_processdata(), lines 893-897:

#ifdef _DARWIN_FEATURE_64_BIT_INODE

        de->d_ino = fudge->ino;

#else

        de->d_ino = (ino_t)fudge->ino; /* XXX: truncation */

#endif /* _DARWIN_FEATURE_64_BIT_INODE */

It appears that the symbol _DARWIN_FEATURE_64_BIT_INODE is *not* defined, which results in the line

        de->d_ino = (ino_t)fudge->ino; /* XXX: truncation */

I have confirmed this finding by disassembling the installed OSXFUSE kext on my system. The resulting disassembly only copies the bottom 32 bits effectively throwing away the top 32 ones.

I believe the fix is to simply replace lines 893-897 with:

        de->d_ino = fudge->ino;

This will work whether de->d_ino is 32 or 64 bits.

Thank you for your time. I would very much appreciate any feedback.

Bill

 

[Sam Moffatt]

OSXFUSE is built for 10.5 or later which will default to _DARWIN_FEATURE_64_BIT_INODE not being defined by default on 10.5 as that was the transition platform. Defining _DARWIN_USE_64_BIT_INODE and rebuilding the kext should trigger _DARWIN_FEATURE_64_BIT_INODE to be defined and should cause the kext to behave properly. Alternatively building with any SDK after 10.5 (10.6 or later) should also result in _DARWIN_FEATURE_64_BIT_INODE being defined.

The lstat man page has all of the relevant details here.

Cheers,

Sam

Sam Moffatt
http://pasamio.id.au

--
You received this message because you are subscribed to the Google Groups "OSXFUSE" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osxfuse-grou...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bill...@navimatics.com]

On Tuesday, March 31, 2015 at 10:35:09 PM UTC-7, Samuel Moffatt wrote:

OSXFUSE is built for 10.5 or later which will default to _DARWIN_FEATURE_64_BIT_INODE not being defined by default on 10.5 as that was the transition platform. Defining _DARWIN_USE_64_BIT_INODE and rebuilding the kext should trigger _DARWIN_FEATURE_64_BIT_INODE to be defined and should cause the kext to behave properly. Alternatively building with any SDK after 10.5 (10.6 or later) should also result in _DARWIN_FEATURE_64_BIT_INODE being defined.

Sam:

Your explanation makes sense.

I want to clarify though that opening the kext/osxfusefs.xcodeproj project from Xcode 6.1.1 on OSX 10.10 does not appear to get _DARWIN_FEATURE_64_BIT_INODE defined (at least on my machine). This appears to be because the OSXFUSE build target picks up the sys/cdefs.h file from

    <Xcode>/.../MacOSX10.10.sdk/System/Library/Frameworks/Kernel.framework/Headers/sys/cdefs.h

rather than

    <Xcode>/.../MacOSX10.10.sdk/usr/include/sys/cdefs.h

Yes, the two cdefs.h files are different!

It looks like one has to define __DARWIN_64_BIT_INO_T to 1 to get the definition of _DARWIN_FEATURE_64_BIT_INODE. The _DARWIN_USE_64_BIT_INODE macro from the lstat man page is not mentioned in the Kernel.framework sys/cdefs.h.

It still seems to me then that the simplest fix is to remove the preprocessor conditional and simply use

    de->d_ino = fudge->ino;

Of course any fix that the OSXFUSE maintainers decide on is fine with me.

Bill

 

[bill...@navimatics.com]

On Tuesday, March 31, 2015 at 11:59:09 PM UTC-7, bill...@navimatics.com wrote:

It still seems to me then that the simplest fix is to remove the preprocessor conditional and simply use

    de->d_ino = fudge->ino;

A quick update on this and a workaround.

First it does not seem that the OSXFUSE readdir() fix is as trivial as I originally thought. I am no Darwin kernel expert, but it appears to me that for 64-bit inode numbers to work it requires support for VNODE_READDIR_EXTENDED. Fuse_vnop_readdir() (in fuse_vnops.c) returns EINVAL if it sees the flag VNODE_READDIR_EXTENDED.

Second, I have a workaround to make 64-bit inode numbers work. Simply place -1 in the st_ino field that is passed to fuse_add_direntry:

    stat.st_ino = -1;

    stat.st_mode = 0;

    ret = fuse_add_direntry(req, buf, bufsize, name, &stat, next_ofs);

This will result in a reported d_ino field of 0xffffffff, which lets my file system test suite pass. It is unclear though whether this change would have any other adverse effects in the real world. I understand that FUSE itself does the same thing (reports a d_ino of 0xffffffff) if it receives a NULL stat buffer in fuse_fill_dir_t in the high level interface.

Hope that this is of help to someone else.

Bill Zissimopoulos