fuse.exfat and GoPro Hero 2 SD Cards

[Tom Daly]

Hi,

I am working with GoPro Hero 2 SD cards and fuse.exfat. I am seeing two problems:

#1. Cards taken directly from the camera are unmountable by fuse-exfat. A bitmap size error is reported.

#2. Directories are being corrupted after being read by processes such as scp. I think the issue is with the atime field being modified, and the checksum not be recalculated correctly.

EXAMPLE:

[tom@ingest01 ~]$ cd /card-sdf1/DCIM/

._.DS_Store  .DS_Store    gladiator1/  

[tom@ingest01 ~]$ cd /card-sdf1/DCIM/gladiator1/

[tom@ingest01 gladiator1]$ ls

ls: reading directory .: Input/output error

[tom@ingest01 gladiator1]$ cd ~

[tom@ingest01 ~]$ sudo exfatfsck /dev/sdf1 

[sudo] password for tom: 

exfatfsck 1.0.1

Checking file system on /dev/sdf1.

File system version           1.0

Sector size                 512 bytes

Cluster size                128 KB

Volume size                 117 GB

Used space                   97 GB

Available space              20 GB

ERROR: `20130602081459_GP010001_gladiator1.LRV' has invalid checksum (0x3b1d != 0x3799).

ERROR: failed to open directory `/DCIM/gladiator1'.

Totally 8 directories and 13 files.

File system checking finished. ERRORS FOUND: 2.

[tom@ingest01 ~]$ mount.exfat -v

FUSE exfat 1.0.1

Copyright (C) 2010-2013  Andrew Nayenko

Any ideas?

Regards,

Tom Daly

[Andrew Nayenko]

Hi Tom,

> I am working with GoPro Hero 2 SD cards and fuse.exfat. I am seeing two
> problems:
>
> #1. Cards taken directly from the camera are unmountable by fuse-exfat. A
> bitmap size error is reported.

Could you cite the error message? There is a bitmap size check in the
code, but it's performed on mount, not unmount.

> #2. Directories are being corrupted after being read by processes such as
> scp. I think the issue is with the atime field being modified, and the
> checksum not be recalculated correctly.

If this was the case, fuse-exfat would never work correctly... I'm
afraid I cannot say anything without an FS dump. Please zero out an SD
card, format it in the camera, record a small video (as short as
possible) and send me a compressed dump.


--
Andrew Nayenko <res...@gmail.com>

[Tom Daly]

Hi Andrew,
The error message is:

###
FUSE exfat 1.0.1
ERROR: invalid clusters bitmap size: 121909 (expected at least 121910).
###


I will work on getting you a dump ASAP.

Regards,
Tom

[Tom Daly]

Andrew,
Since I cannot mount the filesystem with exfat, how do you suggest I get
you a dump? Would dd'ing the device node to a file work?

Regards,
Tom

[Andrew Nayenko]

> Andrew,
> Since I cannot mount the filesystem with exfat, how do you suggest I get
> you a dump? Would dd'ing the device node to a file work?

Something like
sudo gzip -c /dev/sdc1 > dump.gz
where sdc1 is your exFAT partition.

--
Andrew Nayenko <res...@gmail.com>

[Tom Daly]

Hi Andrew,

We ran a set of tests on a card today, which I've posted the results below. I have three 6.5GB files representing the starting sectors of the cards in each case, wanted to post this hear to see if you think the dumps will be of use before I post them for download.

- Test card was a 128GB Sandisk SD card.

- Card was inserted into a GoPro Hero2 HD camera, and formatted using the embedded "Format Card" feature.

- The camera was used to record two (2) video clips, and shut down.

- The first test was to insert the card into the Linux box, and attempt to mount it with ExFat. Results are below:

[tom@ingest01 ~]$ sudo dumpexfat /dev/sdg1 

[sudo] password for tom: 

dumpexfat 1.0.1

ERROR: invalid clusters bitmap size: 121909 (expected at least 121910).

sudo gzip -c /dev/sdg1 > dump-from_gopro.gz

- I then used "Disk Utility" in Mac OS X to verify and repair the card.

Verifying volume “Untitled”

Checking file systemChecking volume.

Checking main boot region.

Checking system files.

Checking upper case translation table.

Checking file system hierarchy.

Checking active bitmap.

Active bitmap is too small.

Skipping verification of the active bitmap.

Rechecking main boot region.

Rechecking alternate boot region.

The volume  was found corrupt and needs to be repaired.

Error: This disk needs to be repaired. Click Repair Disk.

- I was now able to mount the card in Linux using fuse-exfat and copy files from the card.

[tom@ingest01 ~]$ sudo dumpexfat /dev/sdf1 

dumpexfat 1.0.1

Volume label                        

Volume serial number      0x01234567

FS version                       1.0

Sector size                      512

Cluster size                  131072

Sectors count              249703424

Free sectors               249628416

Clusters count                975276

Free clusters                 975111

First sector                   32768

FAT first sector               16384

FAT sectors count              16384

First cluster sector           32768

Root directory cluster             4

Volume state                  0x0000

FATs count                         1

Drive number                    0x80

Allocated space                   0%

sudo gzip -c /dev/sdf1 > dump-from_mac.gz

[tom@ingest01 test]$ cp -v -r * /home/tom/test/.

`20130606081610_GOPR0001_test.LRV' -> `/home/tom/test/./20130606081610_GOPR0001_test.LRV'

`20130606081610_GOPR0001_test.MP4' -> `/home/tom/test/./20130606081610_GOPR0001_test.MP4'

`20130606081610_GOPR0001_test.THM' -> `/home/tom/test/./20130606081610_GOPR0001_test.THM'

`20130606081617_GOPR0002_test.MP4' -> `/home/tom/test/./20130606081617_GOPR0002_test.MP4'

`20130606081618_GOPR0002_test.LRV' -> `/home/tom/test/./20130606081618_GOPR0002_test.LRV'

`20130606081618_GOPR0002_test.THM' -> `/home/tom/test/./20130606081618_GOPR0002_test.THM'

- I then unmounted the card using 'umount', then remounted the card using fuse-exfat, and discovered I was unable to read the files:

[tom@ingest01 ~]$ sudo dumpexfat /dev/sdf1 

dumpexfat 1.0.1

Volume label                        

Volume serial number      0x01234567

FS version                       1.0

Sector size                      512

Cluster size                  131072

Sectors count              249703424

Free sectors               249628416

Clusters count                975276

Free clusters                 975111

First sector                   32768

FAT first sector               16384

FAT sectors count              16384

First cluster sector           32768

Root directory cluster             4

Volume state                  0x0000

FATs count                         1

Drive number                    0x80

Allocated space                   0%

[tom@ingest01 ~]$ sudo fsck.exfat /dev/sdf1

exfatfsck 1.0.1

Checking file system on /dev/sdf1.

File system version           1.0

Sector size                 512 bytes

Cluster size                128 KB

Volume size                 119 GB

Used space                   37 MB

Available space             119 GB

ERROR: `20130606081610_GOPR0001_test.THM' has invalid checksum (0x5e32 != 0x5842).

ERROR: failed to open directory `/DCIM/test'.

Totally 5 directories and 7 files.

File system checking finished. ERRORS FOUND: 2.

sudo gzip -c /dev/sdf1 > dump-after_umount.gz

- Lastly, I inserted the card back into the Mac OS X box, and used Disk Utility to repair it. I then mounted the card back under fuse-exfat on Linux. While the card mounted, the directory contents was now empty.

Verifying volume “Untitled 4”

Checking file systemChecking volume.

Checking main boot region.

Checking system files.

Checking upper case translation table.

Checking file system hierarchy.

Incorrect checksum for directory entry set.

Incorrect checksum for directory entry set.

Incorrect checksum for directory entry set.

Incorrect checksum for directory entry set.

Incorrect checksum for directory entry set.

Incorrect checksum for directory entry set.

Checking active bitmap.

The bitmap needs to be repaired.

Rechecking main boot region.

Rechecking alternate boot region.

The volume  was found corrupt and needs to be repaired.

Error: This disk needs to be repaired. Click Repair Disk.

[tom@ingest01 test]$ pwd

/card-sdf1/DCIM/test

[tom@ingest01 test]$ ls -lah

total 256K

drwxr-xr-x 1 root root 128K Jun  6 08:16 .

drwxr-xr-x 1 root root 128K Jun  6 08:15 ..

I took three (3) file system dumps at various steps in the process, but given that I'm using a 128GB card and did not have local space to backup the entire card, I cancelled the operation at about 6.5GB of data. Let me know if this is useful for diagnostics and I will post the dumps. If you need the entire dump, let me know.

Any assistance is appreciated.

Regards,

Tom Daly

[Andrew Nayenko]

Hi Tom,

Disk Utility is agree with me about invalid clusters bitmap. Looks
like camera's format utility incorrectly rounds clusters count
downwards when it creates clusters bitmap. If you have Windows near
by, try to run chkdsk utility against the card formatted in the
camera. If chkdsk also says the FS is corrupt, it's a good cause to
contact manufacturer.

Please share dump-from_gopro.gz and dump-from_mac.gz. I'd like to have
a look at them.

--
Andrew Nayenko <res...@gmail.com>