Bug: Wrong handling of SPARSE files for copy/moves

[frank55]

As you know there are SPARSE files on NTFS file system.

A huge file with e.g. 1 GB official size could be filled by (currently) only 5 MB and occupies therefore only 5MB on disk.

This can save a lot of disk space.

When I now Copy/Move such a sparse file to another partition on another hard disk

then FastCopy does NOT pay attention to the sparse feature.

Fastcopy reserves the FULL (later) space of 1 GB instead of only 5MB (+sparse flag).

Thats bad, very bad.

I think thats a bug.

Is there a work around?

Can this be fixed in the next FC release?

Thank you

Frank

[SHIROUZU Hiroaki]

It's a specification.

2020年12月22日火曜日 16:03:41 UTC+9 frank55:

[agvu...@gmail.com]

Fast Copy does not perform Windows MOVE operations using the Windows subsystem's built in file moving API functions.  It performs a copy clone operation which then deletes the original file when complete / verified.  There is NO WAY to copy a Sparse File in Windows without the output being a Normal File, without directly reading from and writing to the NTFS subsystem, which you just shouldn't do unless you're performing drive forensics.

What FastCopy CAN do is write the output file in a pseudo-Sparse manner, by identifying large clumps of 0x00 (null) bytes and issuing Sparse write (skip) commands as it discovers them.  The two files might vary slightly on used disk space, but only by a little bit.  The problem is this extra parsing of file contents will consume more CPU and slow down the copy operation rather significantly, in the way that Verify does.  So it would have to be manually enabled by the user.

[SHIROUZU Hiroaki]

If you don't want to  move as copy/move, I recommend to use Explorer :-) 

2021年1月8日金曜日 7:00:46 UTC+9 agvu...@gmail.com:

[agvu...@gmail.com]

It gets tricky because even Explorer cannot move a Sparse File from one disk to another.  Doing so will convert the Sparse File into a Normal File, which inflates the file's "Size on disk" usage.

For example: If you want to move 100 incomplete bittorrent files, Sparse Files, that will eventually total 2 TB when completed but only currently utilize 300 GB of disk space, from a smaller hard drive to a new bigger hard drive, then you REQUIRE a special utility that will make the files Sparse on the new destination disk.  Otherwise they will needlessly occupy 2 TB of disk space when they should only occupy 300 GB presently.  They're mostly 0x00 (null) bytes since they're incomplete downloads.

[SHIROUZU Hiroaki]

I know.

(and Mr.frank55 will say "It is a big bug in the explorer for 30 years!"  :-)

2021年1月8日金曜日 20:04:53 UTC+9 agvu...@gmail.com:

[agvu...@gmail.com]

Less of a bug, more of an oversight.  Microsoft expected Sparse files to be transient and short lived, completing into Normal Files in a matter of minutes or hours, rather than weeks or months.

BONUS:  Any file with a lot of null bytes can be Sparsified to save on disk space.  If you assume even 1% savings, that's a lot of disk space on a 12 or 18 terabyte harddrive.  You can use this as a marketing tool for people to recover unused disk space.  Unlike NTFS Compression, an NTFS Sparse file doesn't use any CPU extra cycles to unpack the file when it is read.

[clsven]

If other programs (like Torrent downloaders like utorrent.exe) can create Sparse files why not Fastcopy?

If Fastcopy detects that the original file is a sparse file then a sparse file should be created on the target partition as well.

Additional runtime for detecting if files are a sparse files are acceptable (if the user enabled new "sparse detection" option).

I use it rarely. But if then its important.