Safest format for 128GB Android sd card (but also Windows & Linux)

[Vincent Cheng Hoi Chuen]

Cross platform question: Windows Linux Android

Up until today, I had been formatting my 32GB, and 128GB micro
SD cards any way I could.

That is, I don't remember if I formatted them on Linux, or on
Windows or inside the Android phone, because I didn't know
whether it mattered or not - so I would have used any of them,
not knowing that it might matter.

I used the cards interchangeably as a flash card in the slots
in the laptops and in the Android cellphone as a data card.

Recently, I lost all my data (which is the topic of a different
thread) on my 128GB card.

Maybe it's because of the way I formatted it?

Therefore, this thread just asks:
What's the safest way to format a 32GB or 128GB microSD card
you plan on using interchangeably on Linux/Windows/Android?

[Vincent Cheng Hoi Chuen]

Vincent Cheng Hoi Chuen <vcheng...@hotmail.com.hk> wrote in
n10sbh$868$1...@solani.org:

> What's the safest way to format a 32GB or 128GB microSD card
> you plan on using interchangeably on Linux/Windows/Android?

Just to give more data, the oddest things are happening.

A. Linux wouldn't recognize the card yesterday, until I added exfat support
$ sudo apt-get install exfat-fuse exfat-utils
Now Linux recognizes the card, but it only has two directories:
LOST.DIR & .android_secure

The LOST.DIR is like what Windows chkdsk does with fragments, so,
itself, it's not interesting. Likewise, .android_secure is what Android
does with apps "moved" to the SD card (again, not important for my
purpose which is to save my photos).

B. Windows XP SP3 wouldn't recognize the card either, so I added exfat support
https://support.microsoft.com/en-us/kb/955704

But, even after rebooting, Windows doesn't recognize the card.

C. Recuva on Windows therefore doesn't run.
It says the E drive is the wrong "size".

D. Photorec (aka testdisk) on Linux runs but reports no files to be found.
You have to first choose FAT (http://i.imgur.com/JSZOtJh.jpg)
Then it finds a few errors (http://i.imgur.com/CdFy8jH.jpg)
But, in the end, saves no files: (http://i.imgur.com/NxRC64Z.jpg)

All in all, a failure; but I learned something new, which is that I had
better be careful WHERE I format a large microSD card.

Anyone have advice on which is the safest format method?

[Lenny Pinto]

On Fri, 30 Oct 2015 23:35:08 +0000 (UTC), Vincent Cheng Hoi Chuen wrote:

> Anyone have advice on which is the safest format method?

I don't know the answer but I would format the card in the device that will
be using it the most.

If that's android, then there must be a way to format a card in android.

This is san disks web site for formatting.
Maybe it's there
https://www.sdcard.org/downloads/formatter_4/

[Lenny Pinto]

On Fri, 30 Oct 2015 18:12:53 -0700, Lenny Pinto wrote:

> I don't know the answer but I would format the card in the device that will
> be using it the most.

Actually the sandisk sd formatter documentation says to never format in the
device that will be using the card.

They say you must format using their tool.
https://www.sdcard.org/downloads/formatter_4/

But is your card sd or sdhc or sdxc?

Here is what they say about each.
The SD Formatter formats SD Memory Card, SDHC Memory Card and SDXC Memory
Card
(collectively SD/SDHC/SDXC Cards) complying with the SD File System
Specification created by SD
Association (SDA).
The SD Formatter format
s SD/SDHC/SDXC Cards optimizing with these card
’
s performance etc.. It is
strongly recommended to use the SD Formatter to format SD/SDHC/SDXC Cards
rather
than using
formatting tools provided with operating systems. In general,
formatting
tools provided wi
th operating
systems can format various storage media including SD/SDHC/SDXC Cards, but
it may not be
optimized for SD/SDHC/SDXC Cards and it may result in lower performance
than using the SD
Formatter.
SD/SDHC/SDXC Cards have
“
Protected Area
”
in the card
for SD security function. The SD Formatter
will not format the
“
Protected Area
”
. Please use
appropriate
application software or SD host device
which provides SD security function to format the
“
Protected Area
”
in the card.

[Paul]

Vincent Cheng Hoi Chuen wrote:

The most convenient option is FAT32. It works everywhere.
Even MacOSX can read/write FAT32.

Because of the WinXP notion that "32GB is big enough",
you need a third party formatter. The RidgeCrop
formatter can prepare a partition up to 2TB in FAT32.
So if you have a 128GB drive, use Disk Management to
make a partition, make it NTFS if you want, do Quick
Format to reduce write cycles, then when the device has
a drive letter, use the Ridgecrop formatter to change
it to FAT32.

http://www.ridgecrop.demon.co.uk/index.htm?fat32format.htm

http://www.ridgecrop.demon.co.uk/download/fat32format.zip

FAT32 is not protected by a journal. You can use Device Manager
in WinXP, and control whether the thing is set for
"Optimize for Quick Removal". That makes the device safer,
by not leaving huge caches with unwritten files in them.

For safety, even if I've selected that option on a Windows
OS, I still use the "Safely Remove" icon in the notification tray
area. ("Belt and Suspenders" approach to computing.)

It should also be noted, that attacks have been tried. I
don't know if "malware" is the right word for it, because
doing stuff like this is pretty pointless. But it is possible
for someone to write code, which can damage a USB flash device.
Because of that, I use USB flash devices for "transport",
not for archival storage. If you plan to use a USB device
of that sort for an extended period, it would be a good idea
to image (backup) the device occasionally (to a regular
hard drive) for safety.

NTFS - has a journal, trivial damage can be corrected
- Windows and Linux have mature drivers. MacOSX has
had read support, but I'm not sure to this day whether
write support was added. Presumably Apple has some
deal involving Microsoft patents on NTFS.

FAT32 - mature drivers everywhere
- a format used by a lot of devices, like digital
cameras, smartphones, or whatever.
- no journal, must be handled with care, could
get damaged if an OS crashes instead of cleanly
shutting down.
- I haven't had FAT32 damage here for some time,
and am at a loss to explain what has changed. My
handling practices haven't changed, but I seem to
be having less trouble with it.

exFAT - unknown driver maturity on any platform
hard to gauge the safety
- better policies on write behavior, favored
as a means of extending write life
- at the moment, would not be my first choice
for interchange between computers

In fact NTFS is a pretty good format, but it isn't
universal. FAT32 comes the closest to working
everywhere. From a user perspective FAT32 doesn't
support permissions, so pesky details need not
be considered. FAT32 does have a 4GB file size limit,
which makes an inappropriate choice if heaving
around extremely large files.

If you wanted to move a 80GB file on a FAT32 device,
you can use 7ZIP, select "Store" mode (no compression),
set the max file size to 4GB or so, and "save" the
archive to the USB flash. The 80GB file is carved
into 20 numbered files. Presenting the 20 numbered
files to an equivalent Archive Manager elsewhere, should
make the 80GB file recoverable. So there is a way to
get a large file from A to B, using an archiver that
supports part1, part2 etc.

Paul

[Gordon Levi]

The SD Association supplies one
<https://www.sdcard.org/downloads/formatter_4/>.

[J.O. Aho]

On 10/30/2015 11:52 PM, Vincent Cheng Hoi Chuen wrote:
> Cross platform question: Windows Linux Android
>
> Up until today, I had been formatting my 32GB, and 128GB micro
> SD cards any way I could.
>
> That is, I don't remember if I formatted them on Linux, or on
> Windows or inside the Android phone, because I didn't know
> whether it mattered or not - so I would have used any of them,
> not knowing that it might matter.

It don't much matter on which system you format the SDCard, it's more
which file system you are going to use, you need one that all three
supports, which usually ends up with vfat, which ain't the best choice
nowadays when having larger devices.

As the Android gives you more or less not much of choice, format on it
and then try to get your computers to read/write that file system. If
the file system would be exFat, then it can be tricky to get it to work
in Linux as sadly the industry chose to use a closed standard file
system even if there was better open source option for grabs.


--

//Aho

[Jack Ryan]

> NTFS - has a journal, trivial damage can be corrected...
> FAT32 - mature drivers everywhere...
> exFAT - unknown driver maturity on any platform...

You forgot ext3 and ext4 Paul. The filesystems above are proprietary
closed-spec Microsoft filesystems. I know both FAT32 and NTFS both
suffer from chronic fragmentation problems, and probably exFAT too.

So ext4 is a better choice. ext4 is a journeled, supports permission
metadata, and does not suffer from fragmentation. It's also
standardized with an open specification.

[Java Jive]

On Sat, 31 Oct 2015 07:08:43 -0400 (EDT), Jack Ryan
<nor...@remailer.cpunk.us> wrote:

> > NTFS - has a journal, trivial damage can be corrected...
> > FAT32 - mature drivers everywhere...
> > exFAT - unknown driver maturity on any platform...

Agreed, FAT32 seems to be the only choice that can be read by almost
any device of whatever OS and of whatever type.

> So ext4 is a better choice. ext4 is a journeled, supports permission
> metadata, and does not suffer from fragmentation. It's also
> standardized with an open specification.

No, Windows machines can't access it.
--
========================================================
Please always reply to ng as the email in this post's
header does not exist. Or use a contact address at:
http://www.macfh.co.uk/JavaJive/JavaJive.html
http://www.macfh.co.uk/Macfarlane/Macfarlane.html

[Wolf K]

On 2015-10-30 21:19, Lenny Pinto wrote:
> On Fri, 30 Oct 2015 18:12:53 -0700, Lenny Pinto wrote:
>
>> >I don't know the answer but I would format the card in the device that will
>> >be using it the most.
> Actually the sandisk sd formatter documentation says to never format in the
> device that will be using the card.
>
> They say you must format using their tool.
> https://www.sdcard.org/downloads/formatter_4/

[...]

Thanks for the link, but when and why would you format a card? They come
formatted.

TIA,

--
Best,
Wolf K
kirkwood40.blogspot.ca

[The Real Bev]

Every once in a while shit happens :-( We've used various cards/sticks
to transfer stuff from our linux machines to my daughter's windows
machine, and a couple of times the card has become unread/writeable --
possibly because it was removed from her machine before it was ready.
We always eject USB stuff by hand before removing it (higher paranoia
level), but since windows doesn't demand that I think we're blameless.

--
Cheers,
Bev
[] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] [] []
If voting could really change things, it would be illegal.
--Revolution Books, New York, New York

[Aragorn]

On Saturday 31 Oct 2015 13:22, Java Jive conveyed the following to
alt.os.linux...

> On Sat, 31 Oct 2015 07:08:43 -0400 (EDT), Jack Ryan
> <nor...@remailer.cpunk.us> wrote:
>
>> > NTFS - has a journal, trivial damage can be corrected...
>> > FAT32 - mature drivers everywhere...
>> > exFAT - unknown driver maturity on any platform...
>
> Agreed, FAT32 seems to be the only choice that can be read by almost
> any device of whatever OS and of whatever type.
>
>> So ext4 is a better choice. ext4 is a journeled, supports permission
>> metadata, and does not suffer from fragmentation. It's also
>> standardized with an open specification.
>
> No, Windows machines can't access it.

UDF is supported by Windows ─ at least, from Windows 7 onward, I think ─
and also supports POSIX permissions. Nothing designed by Microsoft is
ever going to be a good solution.

--
= Aragorn =

http://www.linuxcounter.net - registrant #223157

[William Unruh]

And not readable on MS systems. So if he wants to use it to exchange
data with Windows, that really is not on.

[Java Jive]

On Sat, 31 Oct 2015 17:23:06 +0100, Aragorn
<thor...@telenet.be.invalid> wrote:
>
> UDF is supported by Windows ? at least, from Windows

XP natively, 2000 with special drivers installed

> onward, I think ?

> and also supports POSIX permissions. Nothing designed by Microsoft is
> ever going to be a good solution.

But it is likely to be the universal solution that is being sought up
thread? How many TVs, Network Media Players, etc, are likely to
support ext4, or even UDF, on a USB stick, straight out of the box?
Not many, if my own experience, as well as anecdotal evidence supplied
by others, is to be relied upon. To my stunned derision, my own QNAP
NMP1000 won't even let you use ext4 for the HD, despite being a Linux
based box - it actually only works properly if the media storage
partition has NTFS format! I have managed to use ext4 on USB sticks
by hacking it, and have thus supplied extra functionality in the form
of optware, but have not been able to solve the HD problem.

[Paul]

The Real Bev wrote:
> On 10/31/2015 05:52 AM, Wolf K wrote:
>> On 2015-10-30 21:19, Lenny Pinto wrote:
>>> On Fri, 30 Oct 2015 18:12:53 -0700, Lenny Pinto wrote:
>>>
>>>> >I don't know the answer but I would format the card in the device
>>>> that will
>>>> >be using it the most.
>>> Actually the sandisk sd formatter documentation says to never format
>>> in the
>>> device that will be using the card.
>>>
>>> They say you must format using their tool.
>>> https://www.sdcard.org/downloads/formatter_4/
>> [...]
>>
>> Thanks for the link, but when and why would you format a card? They come
>> formatted.
>
> Every once in a while shit happens :-( We've used various cards/sticks
> to transfer stuff from our linux machines to my daughter's windows
> machine, and a couple of times the card has become unread/writeable --
> possibly because it was removed from her machine before it was ready. We
> always eject USB stuff by hand before removing it (higher paranoia
> level), but since windows doesn't demand that I think we're blameless.
>

The default for a USB flash device in Windows might be
"Optimize for Quick Removal". And that setting helps a little
bit, if users refuse to use the Safely Remove icon.

You can look in the disk properties in Device Manager
for that setting, amongst other places. Occasionally
a user will modify that setting and... shoot themselves
in the foot.

And different classes of devices have different defaults,
so it's not like that safety setting is applied to everything.
You'd be surprised just how big the Windows write cache
can get in some situations. You could lose 5GB to 10GB
of data, by being careless when removing some kinds
of storage devices inappropriately. "Optimize for Quick Removal"
should prevent a write cache from being used.

Paul

[blind Pete]

Vincent Cheng Hoi Chuen wrote:

Second attempt...

According to my brief read of Wikipedia the 32GB card could be
either SDHC or SDXC, the 128GB card must be SDXC. SDXC requires
exFAT. The firmware on various card reader slots might do
"helpful" things if finds something unexpected, like the wrong
filing system for that size of card. It might refuse to touch
it - or even "fix" it.

<https://en.wikipedia.org/wiki/Secure_Digital>
Search for "reformat the card".

Followup set to alt.os.linux.

--
blind Pete
Sig goes here...

[chris]

On 31/10/2015 12:22, Java Jive wrote:
> On Sat, 31 Oct 2015 07:08:43 -0400 (EDT), Jack Ryan
> <nor...@remailer.cpunk.us> wrote:
>
>>> NTFS - has a journal, trivial damage can be corrected...
>>> FAT32 - mature drivers everywhere...
>>> exFAT - unknown driver maturity on any platform...
>
> Agreed, FAT32 seems to be the only choice that can be read by almost
> any device of whatever OS and of whatever type.

Which is frankly ridiculous. It's ancient and keeps having to be
extended to work with 21st century media. What's next after exFAT, superFAT?

Why didn't Google choose ext3/4 for android? It is so much more robust
than FAT. It wouldn't have taken much to make the existing ext3/4
drivers for windows mature and stable.

Even NTFS would have been better...

[Eef Hartman]

In alt.os.linux chris <ithi...@gmail.com> wrote:
> What's next after exFAT, superFAT?

There is already a "FAT+" in some versions of DR-DOS.

> Why didn't Google choose ext3/4 for android?

ext3 has about the same limits as fat32 (with 512 byte sectors fat32 is
limited to 2 TB, ext3 to 4, 8 or 16 TB as both have at most 2^32 blocks,
in ext3 a block is 2, 4 or 8 sectors).
Only ext4 can go above that.

[J.O. Aho]

For the "industry" had already chosen to go with exFat and it's readable
in that uncool american operating system, while ext4 ain't.
F2FS had been a bit better than FAT, but industry didn't want something
open source, as that didn't mean they could get less license cost for
that uncool OS for the computers...

> Even NTFS would have been better...

Hmmm... I would disagree...


--

//Aho

[chris]

Er, no. FAT32 has a 4GB file size limit and an effective 32GB filesystem
limit as Windows disk tools won't work with bigger FAT32 drives. Hence
why exFAT was cludged.
https://en.wikipedia.org/wiki/Comparison_of_file_systems

ext3 is superior on every level, except cross-platform compatibility.
ext4 is incrementally better.

[chris]

On 01/11/2015 16:55, J.O. Aho wrote:
> On 11/01/2015 11:53 AM, chris wrote:
>> On 31/10/2015 12:22, Java Jive wrote:
>>> On Sat, 31 Oct 2015 07:08:43 -0400 (EDT), Jack Ryan
>>> <nor...@remailer.cpunk.us> wrote:
>>>
>>>>> NTFS - has a journal, trivial damage can be corrected...
>>>>> FAT32 - mature drivers everywhere...
>>>>> exFAT - unknown driver maturity on any platform...
>>>
>>> Agreed, FAT32 seems to be the only choice that can be read by almost
>>> any device of whatever OS and of whatever type.
>>
>> Which is frankly ridiculous. It's ancient and keeps having to be
>> extended to work with 21st century media. What's next after exFAT,
>> superFAT?
>>
>> Why didn't Google choose ext3/4 for android? It is so much more robust
>> than FAT. It wouldn't have taken much to make the existing ext3/4
>> drivers for windows mature and stable.
>
> For the "industry" had already chosen to go with exFat and it's readable
> in that uncool american operating system, while ext4 ain't.

Not quite. Android development started in 2003 [1], exFAT first came out
in 2006 [2] and ext2/3 has been around since the 90s [3].

Google could have saved itself some licence fees by avoiding FAT.
Likewise ext2/3/4 support is in the kernel (now removed in Android).

> F2FS had been a bit better than FAT, but industry didn't want something
> open source, as that didn't mean they could get less license cost for
> that uncool OS for the computers...
>
>> Even NTFS would have been better...
>
> Hmmm... I would disagree...

I wasn't very serious.

[1] https://en.wikipedia.org/wiki/Android_version_history
[2] https://en.wikipedia.org/wiki/File_Allocation_Table#exFAT
[3] https://en.wikipedia.org/wiki/Ext3

[J.O. Aho]

On 11/01/2015 07:13 PM, chris wrote:
> On 01/11/2015 16:55, J.O. Aho wrote:
>> On 11/01/2015 11:53 AM, chris wrote:
>>> On 31/10/2015 12:22, Java Jive wrote:
>>>> On Sat, 31 Oct 2015 07:08:43 -0400 (EDT), Jack Ryan
>>>> <nor...@remailer.cpunk.us> wrote:
>>>>
>>>>>> NTFS - has a journal, trivial damage can be corrected...
>>>>>> FAT32 - mature drivers everywhere...
>>>>>> exFAT - unknown driver maturity on any platform...
>>>>
>>>> Agreed, FAT32 seems to be the only choice that can be read by almost
>>>> any device of whatever OS and of whatever type.
>>>
>>> Which is frankly ridiculous. It's ancient and keeps having to be
>>> extended to work with 21st century media. What's next after exFAT,
>>> superFAT?
>>>
>>> Why didn't Google choose ext3/4 for android? It is so much more robust
>>> than FAT. It wouldn't have taken much to make the existing ext3/4
>>> drivers for windows mature and stable.
>>
>> For the "industry" had already chosen to go with exFat and it's readable
>> in that uncool american operating system, while ext4 ain't.
>
> Not quite. Android development started in 2003 [1], exFAT first came out
> in 2006 [2] and ext2/3 has been around since the 90s [3].

Back in 2003 everyone was happy with vfat, it's first with the larger
storages that exFat became popular and as it worked in the mostly used
OS for desktops, then that was chosen.

> Google could have saved itself some licence fees by avoiding FAT.
> Likewise ext2/3/4 support is in the kernel (now removed in Android).

Are you sure they pay? :P

[Anonymous]

> > So ext4 is a better choice. ext4 is a journeled, supports permission
> > metadata, and does not suffer from fragmentation. It's also
> > standardized with an open specification.
>
> No, Windows machines can't access it.

Sure it can. The Windows admin simply needs to install the FOSS
ext-fs driver for Windows.