Kernel panic in tight mount/unmount loop

[Zach Marquez]

Howdy,

This is the first time I have posted to this group, and the first time I've posted a bug report anywhere, so please excuse me if I leave out pertinent information, but I'll do my best to include what is important.

I'm in the process of writing a filesystem using OSXFUSE 2.4.2, and in testing I have received a kernel panic when I pressed Ctrl+C to send a SIGINT to my application to unmount the filesystem. I have been unable to reproduce this with SIGINT, so I believe it must be some sort of race condition (I sent it almost as soon as I started the program).

To further test, I simply had my application mount the filesystem, then unmount it and exit. I wrote a shell script to loop forever just executing that simple application, and eventually it will always kernel panic as well. (Sometimes after a few hundred executions, sometimes several tens of thousands)

My application is multithreaded: I set up the filesystem in my main thread, and spawn another thread that loops calling fuse_read_cmd and fues_process_cmd, and eventually destroying the filesystem once fuse_read_cmd returns null. I unmount the filesystem from the main thread by fork() and execl() /sbin/umount. (As is done in the default fuse signal handler). I never fork and run umount until after fuse_process_cmd has returned after calling my custom init routine. (Occasionally umount still fails with "umount: MOUNTPOINT: not currently mounted", but my unmount function will loop until the call works, and it always works on either the first or second try).

As a note: Fuse does not give me any error messages during execution of my application, and when I do not have it setup to mount and immediately unmount, it works as expected.

The kernel panic report I get is:
Mon Jun 25 09:49:38 2012
panic(cpu 0 caller 0xffffff7f8079ec96): "OSXFUSE: no private data for mount point?"@/Users/benjamin/Documents/Projekte/OSXFUSE/Repositories/osxfuse/kext/fuse_vfsops.c:809
Backtrace (CPU 0), Frame : Return Address
0xffffff817e853610 : 0xffffff8000220792
0xffffff817e853690 : 0xffffff7f8079ec96
0xffffff817e853760 : 0xffffff7f8079efc1
0xffffff817e853790 : 0xffffff800031b98d
0xffffff817e8537b0 : 0xffffff800030f90f
0xffffff817e853af0 : 0xffffff80003103d0
0xffffff817e853f10 : 0xffffff8000310456
0xffffff817e853f50 : 0xffffff80005cd61b
0xffffff817e853fb0 : 0xffffff80002daa13
      Kernel Extensions in backtrace:
         com.github.osxfuse.filesystems.osxfusefs(2.4.2)[682ECF51-9960-3C45-969E-67A8E5CFD55C]@0xffffff7f80796000->0xffffff7f807abfff

BSD process name corresponding to current thread: mount_osxfusefs

Mac OS version:
11E53

Kernel version:
Darwin Kernel Version 11.4.0: Mon Apr  9 19:32:15 PDT 2012; root:xnu-1699.26.8~1/RELEASE_X86_64
Kernel UUID: A8ED611D-FB0F-3729-8392-E7A32C5E7D74
System model name: iMac12,2 (Mac-942B59F58194171B)

System uptime in nanoseconds: 973264666334
last loaded kext at 723514270341: com.github.osxfuse.filesystems.osxfusefs    2.4.2 (addr 0xffffff7f80796000, size 90112)
last unloaded kext at 240570378086: com.apple.driver.AppleUSBUHCI    4.4.5 (addr 0xffffff7f80a79000, size 65536)
loaded kexts:
com.github.osxfuse.filesystems.osxfusefs    2.4.2
org.virtualbox.kext.VBoxNetAdp    4.1.14
org.virtualbox.kext.VBoxNetFlt    4.1.14
org.virtualbox.kext.VBoxUSB    4.1.14
org.virtualbox.kext.VBoxDrv    4.1.14
com.apple.filesystems.smbfs    1.7.2
com.apple.driver.AppleIntelProfile    85.2
com.apple.driver.AppleHWSensor    1.9.5d0
com.apple.driver.AppleBluetoothMultitouch    70.12
com.apple.filesystems.autofs    3.0
com.apple.driver.AppleMikeyHIDDriver    122
com.apple.driver.AppleUpstreamUserClient    3.5.9
com.apple.driver.AppleMCCSControl    1.0.26
com.apple.driver.AppleMikeyDriver    2.2.0f3
com.apple.driver.AppleHDA    2.2.0f3
com.apple.driver.AudioAUUC    1.59
com.apple.driver.AGPM    100.12.42
com.apple.kext.ATIFramebuffer    7.1.8
com.apple.driver.AppleUSBDisplays    323.3
com.apple.driver.AppleUSBAudio    2.8.5f4
com.apple.driver.AppleSMCPDRC    5.0.0d0
com.apple.iokit.IOUserEthernet    1.0.0d1
com.apple.iokit.IOBluetoothSerialManager    4.0.5f11
com.apple.driver.AppleSMCLMU    2.0.1d2
com.apple.Dont_Steal_Mac_OS_X    7.0.0
com.apple.driver.AudioIPCDriver    1.2.2
com.apple.driver.ACPI_SMC_PlatformPlugin    5.0.0d0
com.apple.ATIRadeonX3000    7.1.8
com.apple.driver.AppleLPC    1.5.8
com.apple.driver.AppleBacklight    170.1.9
com.apple.driver.AppleIntelHD3000Graphics    7.1.8
com.apple.driver.BroadcomUSBBluetoothHCIController    4.0.5f11
com.apple.iokit.SCSITaskUserClient    3.2.0
com.apple.driver.AppleUSBCardReader    3.0.1
com.apple.iokit.IOAHCISerialATAPI    2.0.3
com.apple.driver.AppleIRController    312
com.apple.driver.XsanFilter    404
com.apple.iokit.IOAHCIBlockStorage    2.0.3
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless    1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib    1.0.0d1
com.apple.BootCache    33
com.apple.driver.AppleUSBHub    4.5.0
com.apple.driver.AppleFWOHCI    4.8.9
com.apple.driver.AirPort.Atheros40    504.64.2
com.apple.iokit.AppleBCM5701Ethernet    3.1.2b2
com.apple.driver.AppleAHCIPort    2.3.0
com.apple.driver.AppleUSBEHCI    4.5.8
com.apple.driver.AppleEFINVRAM    1.5.0
com.apple.driver.AppleACPIButtons    1.5
com.apple.driver.AppleRTC    1.5
com.apple.driver.AppleHPET    1.6
com.apple.driver.AppleSMBIOS    1.8
com.apple.driver.AppleACPIEC    1.5
com.apple.driver.AppleAPIC    1.5
com.apple.driver.AppleIntelCPUPowerManagementClient    193.0.0
com.apple.nke.applicationfirewall    3.2.30
com.apple.security.quarantine    1.3
com.apple.driver.AppleIntelCPUPowerManagement    193.0.0
com.apple.driver.AppleProfileTimestampAction    85.2
com.apple.driver.AppleProfileThreadInfoAction    85.2
com.apple.driver.AppleProfileRegisterStateAction    85.2
com.apple.driver.AppleProfileReadCounterAction    85.2
com.apple.driver.AppleProfileKEventAction    85.2
com.apple.driver.AppleProfileCallstackAction    85.2
com.apple.iokit.AppleProfileFamily    85.2
com.apple.driver.AppleBluetoothHIDKeyboard    160.7
com.apple.driver.AppleHIDKeyboard    160.7
com.apple.driver.AppleMultitouchDriver    231.4
com.apple.driver.IOBluetoothHIDDriver    4.0.5f11
com.apple.kext.triggers    1.0
com.apple.driver.DspFuncLib    2.2.0f3
com.apple.driver.AppleSMBusController    1.0.10d0
com.apple.driver.AppleSMBusPCI    1.0.10d0
com.apple.iokit.IOFireWireIP    2.2.4
com.apple.iokit.IOSurface    80.0.2
com.apple.iokit.IOSerialFamily    10.0.5
com.apple.iokit.IOAudioFamily    1.8.6fc17
com.apple.kext.OSvKernDSPLib    1.3
com.apple.driver.AppleHDAController    2.2.0f3
com.apple.iokit.IOHDAFamily    2.2.0f3
com.apple.driver.ApplePolicyControl    3.0.16
com.apple.driver.AppleSMC    3.1.3d8
com.apple.driver.IOPlatformPluginLegacy    5.0.0d0
com.apple.driver.IOPlatformPluginFamily    5.1.0d17
com.apple.driver.AppleGraphicsControl    3.0.16
com.apple.driver.AppleBacklightExpert    1.0.3
com.apple.driver.AppleThunderboltEDMSink    1.1.8
com.apple.driver.AppleThunderboltEDMSource    1.1.8
com.apple.kext.ATI6000Controller    7.1.8
com.apple.kext.ATISupport    7.1.8
com.apple.iokit.IONDRVSupport    2.3.2
com.apple.driver.AppleIntelSNBGraphicsFB    7.1.8
com.apple.iokit.IOGraphicsFamily    2.3.2
com.apple.driver.AppleUSBBluetoothHCIController    4.0.5f11
com.apple.iokit.IOBluetoothFamily    4.0.5f11
com.apple.iokit.IOSCSIMultimediaCommandsDevice    3.2.0
com.apple.iokit.IOBDStorageFamily    1.6
com.apple.iokit.IODVDStorageFamily    1.7
com.apple.iokit.IOCDStorageFamily    1.7
com.apple.driver.AppleThunderboltDPOutAdapter    1.8.2
com.apple.driver.AppleThunderboltDPInAdapter    1.8.2
com.apple.driver.AppleThunderboltDPAdapterFamily    1.8.2
com.apple.driver.AppleThunderboltPCIUpAdapter    1.2.4
com.apple.driver.AppleThunderboltPCIDownAdapter    1.2.4
com.apple.iokit.IOSCSIBlockCommandsDevice    3.2.0
com.apple.iokit.IOUSBMassStorageClass    3.0.1
com.apple.iokit.IOSCSIArchitectureModelFamily    3.2.0
com.apple.iokit.IOUSBHIDDriver    4.4.5
com.apple.driver.AppleUSBMergeNub    4.5.3
com.apple.driver.AppleUSBComposite    4.5.8
com.apple.driver.AppleThunderboltNHI    1.5.5
com.apple.iokit.IOThunderboltFamily    2.0.0
com.apple.iokit.IOUSBUserClient    4.5.8
com.apple.iokit.IOFireWireFamily    4.4.5
com.apple.iokit.IO80211Family    420.3
com.apple.iokit.IOEthernetAVBController    1.0.1b1
com.apple.iokit.IONetworkingFamily    2.1
com.apple.iokit.IOAHCIFamily    2.0.8
com.apple.iokit.IOUSBFamily    4.5.8
com.apple.driver.AppleEFIRuntime    1.5.0
com.apple.iokit.IOHIDFamily    1.7.1
com.apple.iokit.IOSMBusFamily    1.1
com.apple.security.sandbox    177.5
com.apple.kext.AppleMatch    1.0.0d1
com.apple.security.TMSafetyNet    7
com.apple.driver.DiskImages    331.6
com.apple.iokit.IOStorageFamily    1.7.1
com.apple.driver.AppleKeyStore    28.18
com.apple.driver.AppleACPIPlatform    1.5
com.apple.iokit.IOPCIFamily    2.6.8
com.apple.iokit.IOACPIFamily    1.4

Could anyone let me know that this is indeed a bug, or if there is something I am doing that is inherently wrong? (Using fuse multithreaded or perhaps the tight mount/unmount loop, etc.)

Thank you, in advance, for any feedback!

[Benjamin Fleischer]

Hi,

Am 25.06.2012 um 18:34 schrieb Zach Marquez:

> This is the first time I have posted to this group, and the first time I've posted a bug report anywhere, so please excuse me if I leave out pertinent information, but I'll do my best to include what is important.

This is one of the best bug reports so far. Good work! Thanks for the report.

> I'm in the process of writing a filesystem using OSXFUSE 2.4.2, and in testing I have received a kernel panic when I pressed Ctrl+C to send a SIGINT to my application to unmount the filesystem. I have been unable to reproduce this with SIGINT, so I believe it must be some sort of race condition (I sent it almost as soon as I started the program).
>
> To further test, I simply had my application mount the filesystem, then unmount it and exit. I wrote a shell script to loop forever just executing that simple application, and eventually it will always kernel panic as well. (Sometimes after a few hundred executions, sometimes several tens of thousands)

I have not tried to reproduce the issue, yet. But I'm pretty sure your conclusion is correct. Everything points towards a race condition. You are unmounting the file system before the mount has completed. This results in a private data structure being freed while still being needed/accessed from a the mounting thread.

> The kernel panic report I get is:
> Mon Jun 25 09:49:38 2012
> panic(cpu 0 caller 0xffffff7f8079ec96): "OSXFUSE: no private data for mount point?"@/Users/benjamin/Documents/Projekte/OSXFUSE/Repositories/osxfuse/kext/fuse_vfsops.c:809
> Backtrace (CPU 0), Frame : Return Address
> 0xffffff817e853610 : 0xffffff8000220792
> 0xffffff817e853690 : 0xffffff7f8079ec96
> 0xffffff817e853760 : 0xffffff7f8079efc1
> 0xffffff817e853790 : 0xffffff800031b98d
> 0xffffff817e8537b0 : 0xffffff800030f90f
> 0xffffff817e853af0 : 0xffffff80003103d0
> 0xffffff817e853f10 : 0xffffff8000310456
> 0xffffff817e853f50 : 0xffffff80005cd61b
> 0xffffff817e853fb0 : 0xffffff80002daa13
> Kernel Extensions in backtrace:
> com.github.osxfuse.filesystems.osxfusefs(2.4.2)[682ECF51-9960-3C45-969E-67A8E5CFD55C]@0xffffff7f80796000->0xffffff7f807abfff

The stack trace translates to:

panic
fuse_vfsop_getattr
fuse_vfsop_biglock_getattr
vfs_getattr
mount_common
__mac_mount
mount
unix_syscall64
hndl_unix_scall64

> Could anyone let me know that this is indeed a bug, or if there is something I am doing that is inherently wrong? (Using fuse multithreaded or perhaps the tight mount/unmount loop, etc.)

This is definitely a bug. A FUSE file system should not be able to crash the kernel extension under any circumstances.

Regards,
Benjamin

[Zach Marquez]

Benjamin,

Thank you for the quick reply! I'm glad to know my report was useful. I'll keep out of those tight loops for now. (Kernel panics are not very friendly :) )

best,
Zach Marquez

[Benjamin Fleischer]

Hi Stu,

I'm unable to symbolicate those stack trace. It seems Apple has not released a kernel debug kit for this particular build. Could you post a OS X 10.7 (or earlier) panic log?

Regards,

Benjamin

Am 05.07.2012 um 23:27 schrieb Stu:

Wanted to post that I too have had a few kernel panics, building a fuse filesystem in Python for gridfs.

It is not consistent but only occurs when mounting the FS.  I have attached two separate logs from the panics below (running latest 10.8 / Mountain Lion).  Obviously the load base of the kext is different, however it does appear the stack frame within the kext is the same each time.

Cheers,

Stu

Interval Since Last Panic Report:  266807 sec

Panics Since Last Report:          1

Anonymous UUID:                    D78DEA6D-A0CA-49BA-921E-D7855DE398B9

Tue Jul  3 11:44:37 2012

panic(cpu 3 caller 0xffffff80138b7b95): Kernel trap at 0xffffff80138f2a91, type 14=page fault, registers:

CR0: 0x0000000080010033, CR2: 0x00000000000000c0, CR3: 0x000000024ecc207b, CR4: 0x00000000000206e0

RAX: 0x0000000001000001, RBX: 0x0000000000000020, RCX: 0x0000000003000000, RDX: 0xffffff80265a0200

RSP: 0xffffff81042abca0, RBP: 0xffffff81042abcc0, RSI: 0x0000000000000020, RDI: 0xffffff80265a0200

R8:  0x0000000000000000, R9:  0x00000000000003ff, R10: 0xffffffffffffffff, R11: 0x00000000ffffffff

R12: 0xffffff80265e5b70, R13: 0xffffff803fb6f800, R14: 0x0000000000000000, R15: 0x00000000000000c0

RFL: 0x0000000000010282, RIP: 0xffffff80138f2a91, CS:  0x0000000000000008, SS:  0x0000000000000010

Fault CR2: 0x00000000000000c0, Error code: 0x0000000000000000, Fault CPU: 0x3

Backtrace (CPU 3), Frame : Return Address

0xffffff81042ab940 : 0xffffff801381d5f6 

0xffffff81042ab9b0 : 0xffffff80138b7b95 

0xffffff81042abb80 : 0xffffff80138ce4ad 

0xffffff81042abba0 : 0xffffff80138f2a91 

0xffffff81042abcc0 : 0xffffff7f95c1fb70 

0xffffff81042abd10 : 0xffffff8013920bfd 

0xffffff81042abd60 : 0xffffff8013911e84 

0xffffff81042abdd0 : 0xffffff8013908419 

0xffffff81042abe20 : 0xffffff8013b4ae63 

0xffffff81042abe50 : 0xffffff8013b77a83 

0xffffff81042abf50 : 0xffffff8013be17da 

0xffffff81042abfb0 : 0xffffff80138cecf3 

      Kernel Extensions in backtrace:

         com.github.osxfuse.filesystems.osxfusefs(2.4.2)[682ECF51-9960-3C45-969E-67A8E5CFD55C]@0xffffff7f95c1f000->0xffffff7f95c34fff

BSD process name corresponding to current thread: mount_osxfusefs

Mac OS version:

12A256

and

Interval Since Last Panic Report:  14339 sec

Panics Since Last Report:          1

Anonymous UUID:                    D78DEA6D-A0CA-49BA-921E-D7855DE398B9

Tue Jul  3 16:10:09 2012

panic(cpu 3 caller 0xffffff800f2b7b95): Kernel trap at 0xffffff800f2f2a91, type 14=page fault, registers:

CR0: 0x0000000080010033, CR2: 0x00000000000000c0, CR3: 0x000000008279205c, CR4: 0x00000000000206e0

RAX: 0x0000000001000005, RBX: 0x0000000000000020, RCX: 0x0000000003000000, RDX: 0xffffff8021dec930

RSP: 0xffffff813b333ca0, RBP: 0xffffff813b333cc0, RSI: 0x0000000000000020, RDI: 0xffffff8021dec930

R8:  0x0000000000000000, R9:  0x00000000000003ff, R10: 0xffffffffffffffff, R11: 0x00000000ffffffff

R12: 0xffffff8021e0ab70, R13: 0xffffff8034ab6800, R14: 0x0000000000000000, R15: 0x00000000000000c0

RFL: 0x0000000000010282, RIP: 0xffffff800f2f2a91, CS:  0x0000000000000008, SS:  0x0000000000000010

Fault CR2: 0x00000000000000c0, Error code: 0x0000000000000000, Fault CPU: 0x3

Backtrace (CPU 3), Frame : Return Address

0xffffff813b333940 : 0xffffff800f21d5f6 

0xffffff813b3339b0 : 0xffffff800f2b7b95 

0xffffff813b333b80 : 0xffffff800f2ce4ad 

0xffffff813b333ba0 : 0xffffff800f2f2a91 

0xffffff813b333cc0 : 0xffffff7f9160cb70 

0xffffff813b333d10 : 0xffffff800f320bfd 

0xffffff813b333d60 : 0xffffff800f311e84 

0xffffff813b333dd0 : 0xffffff800f308419 

0xffffff813b333e20 : 0xffffff800f54ae63 

0xffffff813b333e50 : 0xffffff800f577a83 

0xffffff813b333f50 : 0xffffff800f5e17da 

0xffffff813b333fb0 : 0xffffff800f2cecf3 

      Kernel Extensions in backtrace:

         com.github.osxfuse.filesystems.osxfusefs(2.4.2)[682ECF51-9960-3C45-969E-67A8E5CFD55C]@0xffffff7f9160c000->0xffffff7f91621fff

BSD process name corresponding to current thread: mount_osxfusefs

Mac OS version:

12A256

[Stuart Carnie]

Hi Benjamin,

I don't have any 10.7 boxes left, but I suspect as Apple is releasing 10.8 this month, we can track it down then.  I will upgrade to 10.8GM and prove logs again.

Cheers,

Stu

Sent from my iPad

[Stu]

Looks like Apple just released the GM seed for ML, along with the associated kernel debug kit.  I'll upgrade to that and post any logs if it should occur again.

Cheers,

Stu

[Benjamin Fleischer]

Hi Stu,

I think I might have found and fixed the cause for your panics. It looks like you issue is caused by the same bug as issue #38. When mount_osxfusefs fails to mount a FUSE volume it calls ioctl FUSEDEVIOCSETDAEMONDEAD to signal that the daemon is dead. As a result the kernel extension signals that the file system is dead. But at this early stage in the mount process the volume has actually not been mounted, yet.

So there is no need for the kernel extension to signal that the file system is dead. In case of OS X 10.8 doing so results in page fault kernel panics like the ones you reported.

Regards,

Benjamin

[Stu]

Nice work, Benjamin!

[Benjamin Fleischer]

Hi Zach,

The just released OSXUFSE 2.5.0 comes with a workaround, that should prevent kernel panics in tight mount/umount loops. At least I have not been able to reproduce this issue with OSXFUSE 2.5.0 on several Macs with different version of OS X. Could you check if you are still getting panics?

Regards,
Benjamin