OSXFUSE + NTFS-3g = disk corruption?

[DualShock]

Hi,

I tested OSXFUSE on my 2008 Mac Pro running Snow Leopard 10.6.8 in preparation for upgrading to Lion. I forced the 64 bit kernel on also. I then installed NTFS-3g 2010.10.2, the latest Mac build offered by Erik Larsson on this page:

http://macntfs-3g.blogspot.com/2010/10/ntfs-3g-for-mac-os-x-2010102.html

However, after testing a few file copies to an internal NTFS drive, I saw disk corruption errors when I ran chkdsk from Windows, and some of the files I had copied had disappeared.

Of note, I never saw disk corruption when using MacFUSE and this same NTFS-3g build, under the 32 bit kernel in Snow Leopard.

Has anyone tested this combination? Is this an OSXFUSE issue, or is this NTFS-3g build just too old and incompatible with OSXFUSE?

[Benjamin Fleischer]

Hi,

Which version of OSXFUSE are you using? 2.3.4?

[DualShock]

Yes, I was testing with 2.3.4.

[Benjamin Fleischer]

Hi,

Am 08.11.2011 um 13:26 schrieb DualShock:

> Hi,
>
> I tested OSXFUSE on my 2008 Mac Pro running Snow Leopard 10.6.8 in preparation for upgrading to Lion. I forced the 64 bit kernel on also. I then installed NTFS-3g 2010.10.2, the latest Mac build offered by Erik Larsson on this page:
>
> http://macntfs-3g.blogspot.com/2010/10/ntfs-3g-for-mac-os-x-2010102.html
>
> However, after testing a few file copies to an internal NTFS drive, I saw disk corruption errors when I ran chkdsk from Windows, and some of the files I had copied had disappeared.

Did you install the MacFUSE compatibility layer when installing OSXFUSE? If you didn't, MacFUSE might still be installed and used. There are a few MacFUSE builds (2.1.7 mostly) that "seemingly" work on 64 bit kernels but will cause data corruption issues eventually.

> Of note, I never saw disk corruption when using MacFUSE and this same NTFS-3g build, under the 32 bit kernel in Snow Leopard.

I wonder if those corruption errors are linked to the 64 bit kernel code or OSXFUSE/NTFS-3G in general. To track this issue down you could boot Snow Leopard in 32 bit kernel mode and "try" to get some data corruption errors using OSXFUSE and NTFS-3g.

But just to make this clear: OSXFUSE does not write to the physical drive. NTFS-3G does. OSXFUSE just passes file system operations along.

Erik, what is your opinion on this? Have you heard of similar issues?

> Has anyone tested this combination? Is this an OSXFUSE issue, or is this NTFS-3g build just too old and incompatible with OSXFUSE?

I'm using NTFS-3G 2010.10.2 on Lion with OSXFUSE 2.3.4 myself. So far I haven't encountered any data corruption problems. But I only use NTFS-3G occasionally. NTFS-3G 2010.10.2 should be well-suited for Snow Leopard as far as I know. On Lion there is a 15 seconds timeout error that can be ignored but other than that it should be fine.

Regards,
Benjamin

[Sam Moffatt]

Some of the early "64-bit compatible" build's for MacFUSE merely
removed the safeguard checks that prevented MacFUSE from running in
64-bit mode. I'd agree that without verifying that exactly one of
those builds wasn't used that first and foremost the issue of
corruption should be placed there.

As an aside you need to be careful that a disk is fully ejected before
unplugging. I've seen people pull out a disk after clicking "eject"
but before it had been removed from the UI causing corruption as it
was still sync-ing data. This was actually a FAT formatted USB stick
so it was unrelated to any third party file system but a good
cautionary tale that we can often be in a hurry but we need to be
careful with these things.

Cheers,

Sam Moffatt
http://pasamio.id.au

[Erik Larsson]

Hi,

Benjamin Fleischer wrote 2011-11-25 18.52:
>> Of note, I never saw disk corruption when using MacFUSE and this same NTFS-3g build, under the 32 bit kernel in Snow Leopard.
> I wonder if those corruption errors are linked to the 64 bit kernel code or OSXFUSE/NTFS-3G in general. To track this issue down you could boot Snow Leopard in 32 bit kernel mode and "try" to get some data corruption errors using OSXFUSE and NTFS-3g.
>
> But just to make this clear: OSXFUSE does not write to the physical drive. NTFS-3G does. OSXFUSE just passes file system operations along.
>
> Erik, what is your opinion on this? Have you heard of similar issues?

No, I haven't. In general, data corruption errors can be on the file
data level or one the file system level. If on file data level (i.e. the
file system itself is fine, evidenced by running chkdsk in Windows),
then OSXFUSE could be to blame (not necessarily, but it's at least
possible).

In this case it's not on the file data level but on the file system
level and in that case, NTFS-3G itself is most likely to blame rather
than OSXFUSE. The only way OSXFUSE could influence NTFS-3G's operation
with regard to maintaining file system integrity is if its libfuse
corrupted the NTFS-3G process memory space, but in that case a crash
would be much more likely than random corruptions written out to disk.

Regards,

- Erik

[Benjamin Fleischer]

Am 26.11.2011 um 06:31 schrieb Erik Larsson:

> Hi,
>
> Benjamin Fleischer wrote 2011-11-25 18.52:
>>> Of note, I never saw disk corruption when using MacFUSE and this same NTFS-3g build, under the 32 bit kernel in Snow Leopard.
>> I wonder if those corruption errors are linked to the 64 bit kernel code or OSXFUSE/NTFS-3G in general. To track this issue down you could boot Snow Leopard in 32 bit kernel mode and "try" to get some data corruption errors using OSXFUSE and NTFS-3g.
>>
>> But just to make this clear: OSXFUSE does not write to the physical drive. NTFS-3G does. OSXFUSE just passes file system operations along.
>>
>> Erik, what is your opinion on this? Have you heard of similar issues?
>
> No, I haven't. In general, data corruption errors can be on the file data level or one the file system level. If on file data level (i.e. the file system itself is fine, evidenced by running chkdsk in Windows), then OSXFUSE could be to blame (not necessarily, but it's at least possible).

If the corruption is on the data level and caused by OSXFUSE we would most likely see similar issues with other file systems, too.

> In this case it's not on the file data level but on the file system level and in that case, NTFS-3G itself is most likely to blame rather than OSXFUSE. The only way OSXFUSE could influence NTFS-3G's operation with regard to maintaining file system integrity is if its libfuse corrupted the NTFS-3G process memory space, but in that case a crash would be much more likely than random corruptions written out to disk.

Thank you, Erik, for clearing this up.

Regards,
Benjamin

[DualShock]

Hi all,

Sorry for the late reply, I had not been following this thread.

Benjamin, I'll try out your suggestion of running OSXFUSE/NTFS-3g in 32 bit mode in Snow Leopard and report back. I can tell you that when I tested out OSXFUSE in 64 bit mode, I first uninstalled the NTFS-3g build I previously mentioned, then MacFUSE (2.0.3,2, which was the latest official build from Amit Singh's site). Then I switched the kernel to 64 bit, rebooted, installed OSXFUSE with the MacFUSE compatibility layer option selected, rebooted, and finally NTFS-3g (with another reboot afterwards).

Also, to answer another post, I have never installed any other MacFUSE build other than the one posted by Amit Singh on the original Google group.

[DualShock]

Hi Benjamin / Erik / others,

I have finally gotten around to re-testing this corruption issue I was seeing previously, as mentioned above. I am still able to reproduce the corruption quite reliably, even after switching to the 32 bit kernel.

To recap, I have a 2008 Mac Pro running Snow Leopard 10.6.8. I am planning to install Lion soon, so I wanted to test OSXFUSE + NTFS-3g in 64 bit mode. This Mac Pro has 3 internal SATA drives, an OS X boot drive (HFS+), a Windows 7 boot drive for Boot Camp (NTFS), and a data drive (also NTFS). The tests I performed were copying files from the OS X boot drive to the data drive, while booted into OS X. (I did NOT perform these tests with an external USB drive, if that makes any difference.)

First I uninstalled NTFS-3g, rebooted, then the old MacFUSE, and rebooted again. Then I switched the kernel to 64 bit, rebooted again. Installed the latest OSXFUSE, 2.3.9, with MacFUSE compatibility layer selected, rebooted, then NTFS-3g 2010.10.2 with UBLIO caching turned on, and finally rebooted again.

The data drive has about 50 existing folders and files in the root directory, with various files and subfolders in each. (FWIW, the drive is 1 TB, with about 15 GB free space.) For the test, from within the Finder I created a new folder called 'ABC' in the root directory, then copied 5 files (plus their associated "._" resource fork files, which get copied automatically) from the OS X boot drive to the data drive, into a DIFFERENT empty folder off the root, NOT the 'ABC' folder first created. The files varied in size from 150 MB - 1 GB. Afterwards, I rebooted into Windows 7 and ran chkdsk /f on the drive.

Test 1:
64 bit kernel, UBLIO turned ON
chkdsk reported corruption. ABC folder and copied files disappeared after chkdsk completed.

Test 2:
64 bit kernel, UBLIO turned OFF
chkdsk reported corruption. ABC folder was still there. 1 file survived the copy + chkdsk, the resource fork files survived, but the other files were truncated to 0 bytes.

Test 3:
32 bit kernel UBLIO turned ON
same results as Test 2

I turned on debug logging in NTFS-3g, but didn't see anything noteworthy, just some messages related to mounting the 2 NTFS drives in my box. I did a few ad hoc tests with just copying files without creating a folder first, and that worked fine sometimes, but not always. I've attached a file containing chkdsk output after running test 3, if that helps out any. (File names in the output edited.)

Hope you guys can help me out with this problem!

[Camilo Aguilar]

What makes you think it is a osxfuse issue? I'd try to rule out first NTFS-3g. Did you ask in their mailing list as well?

[DualShock]

Because I have never seen any corruption issues when using the same NTFS-3g build with Amit Singh's MacFUSE. That being said, OSXFUSE itself may not be to blame, but rather, a combination of the 2.

On Wednesday, April 4, 2012 7:41:13 AM UTC-4, Camilo Aguilar wrote:

What makes you think it is a osxfuse issue? I'd try to rule out first NTFS-3g. Did you ask in their mailing list as well?

[Camilo Aguilar]

That makes sense, sorry if I sounded harsh, it's just that when there are data corruption it's usually the fs implementation. And it's also good to check first with the NTFS-3g authors. I wonder if the Tuxera's POSIX test suite passes fine with the osxfuse and NTFS-3g version you are using. Or if it makes sense to try it out in this case.

[Benjamin Fleischer]

Hi,

Am 04.04.2012 um 10:16 schrieb DualShock:

I have finally gotten around to re-testing this corruption issue I was seeing previously, as mentioned above. I am still able to reproduce the corruption quite reliably, even after switching to the 32 bit kernel.

Since OSXFUSE 2.3.8 the same locking mechanism is used in 32- and 64-bit mode. When using OSXFUSE 2.3.8 or later it shouldn't matter whether you are running a 32- or 64-bit kernel. If you want to test OSXFUSE using the kernel's own locking mechanism you need to install OSXFUSE 2.3.7 or a prior version and boot in 32-bit kernel mode. Just for testing purposes, of course.

To recap, I have a 2008 Mac Pro running Snow Leopard 10.6.8. I am planning to install Lion soon, so I wanted to test OSXFUSE + NTFS-3g in 64 bit mode. This Mac Pro has 3 internal SATA drives, an OS X boot drive (HFS+), a Windows 7 boot drive for Boot Camp (NTFS), and a data drive (also NTFS). The tests I performed were copying files from the OS X boot drive to the data drive, while booted into OS X. (I did NOT perform these tests with an external USB drive, if that makes any difference.)

It would be good if you could verify that the issue is indeed not related to your internal drive. To be absolutely sure use another drive, reformat it using Windows and then try to reproduce the issue.

The data drive has about 50 existing folders and files in the root directory, with various files and subfolders in each. (FWIW, the drive is 1 TB, with about 15 GB free space.) For the test, from within the Finder I created a new folder called 'ABC' in the root directory, then copied 5 files (plus their associated "._" resource fork files, which get copied automatically) from the OS X boot drive to the data drive, into a DIFFERENT empty folder off the root, NOT the 'ABC' folder first created. The files varied in size from 150 MB - 1 GB. Afterwards, I rebooted into Windows 7 and ran chkdsk /f on the drive.

If I run the following command (one line) using NTFS-3G 2010.10.2 + OSXFUSE 2.3.9 I get some weird results:

i=0; while [ $i -lt 50 ]; do let i=$i+1; echo "$i:"; dd if=/dev/urandom of=/Volumes/Test/$i.txt bs=1048576 count=100 2> /dev/null && echo "OK" || echo "Error"; done

This line of code creates 50 random text files that are each 100 MiB in size. /Volumes/Test is the mount point of my NTFS volume. After thirty-something files I get errors, but I havn't had a chance to look into it. BTW, the above command does not return errors using Tuxera's commercial NTFS driver. 

I turned on debug logging in NTFS-3g, but didn't see anything noteworthy, just some messages related to mounting the 2 NTFS drives in my box. I did a few ad hoc tests with just copying files without creating a folder first, and that worked fine sometimes, but not always. I've attached a file containing chkdsk output after running test 3, if that helps out any. (File names in the output edited.)

After the errors I ran chkdsk (Windows 7) but it did not find any inconsistencies or corruption.

At this point I don't know if those issues are caused by OSXFUSE or the dusty version of NTFS-3G. If this was in fact an NTFS issue it might be best to move the discussion over to Tuxera's forum.

Erik, can you shed some light on this? Are there know NTFS-3G issues that might be related? 

Regards,

Benjamin

[Erik Larsson]

Hi,

If the corruption happens with NTFS-3G, I would recommend the NTFS-3G mailing list (the forum is not working so well these days... all the spam has made it quite unusable). Of course if it only happens with OSXFUSE and not MacFUSE then it's strange, but even if OSXFUSE does something wrong it is not in control over the NTFS on-disk state. It is managed by the ntfs-3g process and should not be affected by any weirdness going on in the kernel due to the kernel/userspace separation. If it would be some OSXFUSE problem, I would expect not only NTFS-3G to break, but many other parts of the system (eventually causing a kernel panic).

I think it's more likely to be an NTFS-3G problem, possibly triggered by some subtle difference in how read/write operations are performed (buffer size, alignment, ...).
We have a utility, ntfsmetaclone, which can be used to generate a metadata image of the corrupted volume. So, Mr. DualShock, it might help to post a link to such an image to the NTFS-3G mailing list. (Note that this image will contain all the names of the files and directories, file sizes, permissions, etc. on a volume, so you may want to encrypt it with gpg first and share the key with developers.)
See: http://www.tuxera.com/mac/ntfsmetaclone-1.1.dmg

...and of course there have been many bugfixes since version 2012.10.2. Testing with the latest version would be the first priority of course.

Regards,

- Erik

[DualShock]

Hello Benjamin / Erik,

Thanks so much for the insight regarding my problem. I'd rather not create a metadata dump of my drive, if at all possible. However, what I could try is attempt to recreate the scenario onto another drive, with dummy files and folders of a similar nature as my existing drive. Unfortunately that would take some time because I don't have any free SATA drives lying around. (Same case with Benjamin's suggestion of testing the same scenario on a different drive.) I could probably run a test with an external USB / FireWire drive. Unfortunately I'm a bit tied up with some other things right now and am unable to run more tests. If I can find some time to run more tests I'll post back with my results.

I had been considering Tuxera NTFS as an alternative, but I'd like to try out the free / open source options first, since I've had great results with NTFS-3g / MacFUSE so far.

Thanks again guys!

[Sam Moffatt]

Not sure if this would work but perhaps an idea. Would it be possible
to create a small partition on a physical drive, format it as NTFS
using Windows and then dd the partition to a file on the Mac. Then
mount the file created using dd with NTFS-3G instead of an actual
physical disk. In a sense this would get around the hardware issues
and you could then also use the image (presuming it's small enough) to
hand around. It would be similar to mounting a loop back device is
under Linux, not sure if it is possible via OSXFUSE and NTFS-3G but it
might be worth attempting.

Perhaps more a question for the more experienced NTFS-3G folk on list.

Cheers,

Sam Moffatt
http://pasamio.id.au

[Antoni Beksiak]

Hello,

I hope I can join here. It happened my laptop went sour and I'm using a friend's OS X 10.6.8 iMac with my NTFS hdd connected via USB thru OSXFUSE/NTFS-3G. After a few days I encountered request for repairing the HDD when subsequently connected to a Windows system, lost files afterwards, and - most interesting - failure to boot the laptop from said disk. Which is curious, since http://www.tuxera.com/community/ntfs-3g-faq/#mbr.

Would you have an advice?