info
Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Copying a number of files one-by-one onto an USB stick slows down to a crawl. How to fix ?
10 views
Skip to first unread message
R.Wieser
Jan 12, 2022, 6:29:23 AM1/12/22
to
Hello all,
I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
that the whole thing slows down to a crawl (about 5 GByte consisting of
50.000 files in over eight hours). I've also tried a another, different USB
stick and see the same thing happen.
Windows task-manager shows my programs procesor usage as zero percent.
Remark: I've used the same program to copy even larger quantities of files
to an USB harddisk, and have not experienced any kind of slowdown there.
I know that an USB stick is rather slow in comparision to a harddisk, but
the above is just ludicrous. :-(
Does anyone have an idea what is going on and how to speed the whole thing
up ?
Regards,
Rudy Wieser
P.s.
I remember having had similar problems while drag-and-dropping the source
folder onto the USB stick. IOW, not a problem with the program itself.
I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
that the whole thing slows down to a crawl (about 5 GByte consisting of
50.000 files in over eight hours). I've also tried a another, different USB
stick and see the same thing happen.
Windows task-manager shows my programs procesor usage as zero percent.
Remark: I've used the same program to copy even larger quantities of files
to an USB harddisk, and have not experienced any kind of slowdown there.
I know that an USB stick is rather slow in comparision to a harddisk, but
the above is just ludicrous. :-(
Does anyone have an idea what is going on and how to speed the whole thing
up ?
Regards,
Rudy Wieser
P.s.
I remember having had similar problems while drag-and-dropping the source
folder onto the USB stick. IOW, not a problem with the program itself.
Paul
Jan 12, 2022, 7:42:35 AM1/12/22
to
As a workaround, try to package the files with 7-ZIP instead.
https://www.7-zip.org/
You can request 7-ZIP to tar the files, and it also has a
segmentation function, so can ensure that output files are less than
4GB to beat the FAT32 limitations on file size for the archive file.
There is a place in the GUI interface to set segment size.
some.zip \
some.zip1 \___ (Bad illustration of beating 4GB file size
some.zip2 / limit of FAT32... You probably know the file
extension for ZIP better than I do :-) )
Do the compression or packaging step on the hard drive. Then
copy across one or more of the large segment files to the USB stick.
That will give a better idea of the speed of the stick. Large
files will do the fewest directory updates and display more of
the sustained transfer rate of the device.
The reason you don't want to point 7-ZIP at the stick directly
for the output step, is to not give it any excuses this time.
7-ZIP can be used later, to do random access within the archive
on the stick, if so desired. If you need NOTE.txt from the 50,000
files, you can get it without unpacking the entire archive.
By preparing the archive file on the hard drive first, you'll be
doing the largest flash block copies possible that way.
You might also want to inspect the file system of the destination
device and make sure the cluster size is 32KB or whatever, rather
than just 512 bytes. An accidental too-small cluster size will
do stuff like that. Flash page sizes are pretty large, and asking
a USB stick to do things in 512 byte clusters, brings out the worst
(read-modify-write at the physical level) in it. That burns up
the flash write life, like crazy. The 32KB cluster size choice,
is getting closer in powers-of-2, to the true page size in the hardware.
This command, agrees with the value displayed when I select "format"
of the associated USB stick partition letter.
wmic volume get driveletter,blocksize
You can add EXFAT support to Windows XP, via an optional download
and install. EXFAT supports stuff like a 1MB cluster or so, and
EXFAT was designed with Flash storage devices in mind. But again,
there's really no excuse for the poor performance. EXFAT is native
to Windows 7, no installing it as an addon there.
I could use a Sandisk Extreme or Sandisk Extreme Pro and this transfer
would be finished in two minutes, tops.
You can put USB3 on a WinXP era machine, using a plugin card with
a NEC USB3 chip on it. They still sell PCIe versions of those
(a poster bought one a month ago). The NEC chip was one of the few
to include WinXP driver support.
The PCI card version of such things (100MB/sec limit) will be
very hard to find. The bridge chip company got bought up and crushed,
and bridge chips might cost 5x as much as they used to, which has
caused distortion of the computer industry products in response.
This is why they don't put bifurcation logic on motherboards any more.
One company made that happen, via piggish practice.
If you wanted USB3 on your Windows XP machine, and it only had PCI
slots, you needed to be upgrading around ten years ago. That
was back when the PCI to PCIe x1 bridge chip might have been
five bucks or so. And then the bridge chip feeds the NEC chip,
on the PCI card.
Paul
Mayayana
Jan 12, 2022, 8:27:34 AM1/12/22
to
"R.Wieser" <add...@not.available> wrote
| I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
| copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
| that the whole thing slows down to a crawl (about 5 GByte consisting of
| 50.000 files in over eight hours). I've also tried a another, different
USB
| stick and see the same thing happen.
|
I don't know any solution. I have noticed that XP doesn't
do USB well. Maybe because it's USB2? Win7 is much faster,
with particular effort. I assume the problem is the USB drivers.
The only other thing I can think of would be if you were using
some kind of AV program. Most of those seem to default to
scanning everything you touch.
| I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
| copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
| that the whole thing slows down to a crawl (about 5 GByte consisting of
| 50.000 files in over eight hours). I've also tried a another, different
USB
| stick and see the same thing happen.
|
do USB well. Maybe because it's USB2? Win7 is much faster,
with particular effort. I assume the problem is the USB drivers.
The only other thing I can think of would be if you were using
some kind of AV program. Most of those seem to default to
scanning everything you touch.
R.Wieser
Jan 12, 2022, 9:00:55 AM1/12/22
to
Mayayana,
> Maybe because it's USB2?
:-) Thats why I mentioned that I also tried an USB HD drive. If it would
be the USB connection itself it would also be slow. It isn't. When I
tried just now it copied the same thing in about 7 minutes. Thats over *60
times* faster. :-|
> The only other thing I can think of would be if you were
> using some kind of AV program.
In that case I should have seen it work its ass off in the task manager.
Which I didn't. 0% usage by any program, 99% idle time.
Regards,
Rudy Wieser
> Maybe because it's USB2?
be the USB connection itself it would also be slow. It isn't. When I
tried just now it copied the same thing in about 7 minutes. Thats over *60
times* faster. :-|
> The only other thing I can think of would be if you were
> using some kind of AV program.
Which I didn't. 0% usage by any program, 99% idle time.
Regards,
Rudy Wieser
R.Wieser
Jan 12, 2022, 9:00:55 AM1/12/22
to
Paul,
Thanks for the suggestions.
The reason why I copy files is that I can than access them using any kind of
filebrowser - even a simple one which doesn't understand the concept of ZIP
folders.
As for the cluster size, the second USB stick was new, and as such I assume
it was pre-formatted with the optimum block size. Also, it was NTFS (I had
hoped it would make a difference, but it looks like it doesn't)
And there is a problem with that suggestion : Years ago, when I thought of
the same (micro-SD connected to a DOS 'puter), I was unable to get that
optimum from the USB stick itself. No matter which drive interrogation
function I looked at, none of them returned the size of the FLASH chips
memory blocks . :-(
> I could use a Sandisk Extreme or Sandisk Extreme Pro and this transfer
> would be finished in two minutes, tops.
I assume my (rather old) machine has USB 2.0 tops, but even there a copy to
an USB HD takes just 7 minutes.
> If you wanted USB3 on your Windows XP machine,
:-) Yup.
But the question is not about the speed of the USB connection, but why the
(USB attached) memory-stick is so much slower (as in: more than 60 times)
than the (USB attached) HD.
I'll re-look into that cluster size though. Who knows ...
Regards,
Rudy Wieser
Thanks for the suggestions.
The reason why I copy files is that I can than access them using any kind of
filebrowser - even a simple one which doesn't understand the concept of ZIP
folders.
As for the cluster size, the second USB stick was new, and as such I assume
it was pre-formatted with the optimum block size. Also, it was NTFS (I had
hoped it would make a difference, but it looks like it doesn't)
And there is a problem with that suggestion : Years ago, when I thought of
the same (micro-SD connected to a DOS 'puter), I was unable to get that
optimum from the USB stick itself. No matter which drive interrogation
function I looked at, none of them returned the size of the FLASH chips
memory blocks . :-(
> I could use a Sandisk Extreme or Sandisk Extreme Pro and this transfer
> would be finished in two minutes, tops.
an USB HD takes just 7 minutes.
> If you wanted USB3 on your Windows XP machine,
But the question is not about the speed of the USB connection, but why the
(USB attached) memory-stick is so much slower (as in: more than 60 times)
than the (USB attached) HD.
I'll re-look into that cluster size though. Who knows ...
Regards,
Rudy Wieser
Sjouke Burry
Jan 12, 2022, 12:20:30 PM1/12/22
to
put the zip file on the stick.
That also increases the life of the stick.
Char Jackson
Jan 12, 2022, 12:41:53 PM1/12/22
to
On Wed, 12 Jan 2022 12:28:35 +0100, "R.Wieser" <add...@not.available>
wrote:
>Hello all,
>
>I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
>copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
>that the whole thing slows down to a crawl (about 5 GByte consisting of
>50.000 files in over eight hours). I've also tried a another, different USB
>stick and see the same thing happen.
>
>Windows task-manager shows my programs procesor usage as zero percent.
>
>Remark: I've used the same program to copy even larger quantities of files
>to an USB harddisk, and have not experienced any kind of slowdown there.
>
>I know that an USB stick is rather slow in comparision to a harddisk, but
>the above is just ludicrous. :-(
>
>Does anyone have an idea what is going on and how to speed the whole thing
>up ?
My totally unscientific observation is that writing to a USB2 device
starts out fast, and when it inevitably slows to a crawl I notice that
the device is blazing hot. If I allow it to cool down, the high write
speed returns until it gets hot again. Is there a correlation between
write speed and heat? *shrug* I don't know, but I can consistently
duplicate that behavior here on my devices.
wrote:
>Hello all,
>
>I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
>copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
>that the whole thing slows down to a crawl (about 5 GByte consisting of
>50.000 files in over eight hours). I've also tried a another, different USB
>stick and see the same thing happen.
>
>Windows task-manager shows my programs procesor usage as zero percent.
>
>Remark: I've used the same program to copy even larger quantities of files
>to an USB harddisk, and have not experienced any kind of slowdown there.
>
>I know that an USB stick is rather slow in comparision to a harddisk, but
>the above is just ludicrous. :-(
>
>Does anyone have an idea what is going on and how to speed the whole thing
>up ?
starts out fast, and when it inevitably slows to a crawl I notice that
the device is blazing hot. If I allow it to cool down, the high write
speed returns until it gets hot again. Is there a correlation between
write speed and heat? *shrug* I don't know, but I can consistently
duplicate that behavior here on my devices.
R.Wieser
Jan 13, 2022, 4:54:11 AM1/13/22
to
Char,
> My totally unscientific observation is that writing to a USB2 device
> starts out fast, and when it inevitably slows to a crawl I notice
> that the device is blazing hot.
I tried checking the sticks temperature (removed the "sleeve") and regulary
touched both the processor as well as the flash chip on the other side. I
would not call it hot by any means, just a bit warm.
> If I allow it to cool down,
I have to try to add a keystroke to halt the copying temporarily and see
what happens.
(@all)
And by the way, it looks like the slowing-down is not gradually (as I would
expect when a device gets hotter), but suddenly. It took 45 minutes to
copy about 4 GByte, and after that each file takes 6 seconds.
Odd to say the least ...
Regards,
Rudy Wieser
> My totally unscientific observation is that writing to a USB2 device
> starts out fast, and when it inevitably slows to a crawl I notice
> that the device is blazing hot.
touched both the processor as well as the flash chip on the other side. I
would not call it hot by any means, just a bit warm.
> If I allow it to cool down,
what happens.
(@all)
And by the way, it looks like the slowing-down is not gradually (as I would
expect when a device gets hotter), but suddenly. It took 45 minutes to
copy about 4 GByte, and after that each file takes 6 seconds.
Odd to say the least ...
Regards,
Rudy Wieser
Paul
Jan 13, 2022, 8:19:06 AM1/13/22
to
if you do the copy from a Command Prompt window.
It might be a side-effect of how File Explorer handles
the file records it is reading. All it takes is around
50,000 files in a folder to "confuse" the File Explorer.
I discovered this, by converting movies to a series of
JPG files (for usage with AVIdemux 2.5). Attempts to
edit the movie, by removing the first thousand files and
deleting them in the trash, resulted in the other 49,000
not refreshing properly in the File Explorer window.
On Windows XP, Robocopy can be installed via a download.
On Win7 it would be a built-in command. Each OS has a
different version of Robocopy, with slight differences in
parameters to be passed.
Robocopy is a folder copying program, so it expects
two folders as arguments.
Paul
R.Wieser
Jan 13, 2022, 11:16:54 AM1/13/22
to
Paul,
> I bet the problem does not occur .... if you do the copy from a Command
> Prompt window.
Actually, my program is console based. :-)
> On Windows XP, Robocopy can be installed via a download.
I've heard of RoboCopy. But alas, I have much more fun trying to create my
own programs for it (as mentioned, using the CopyFile function outof
kernel32).
Also, as mentioned, the problem seems to be related to the USB stick, as
doing the same copy to an USB (spinning rust) disk doesn't show any kind of
slowdown problems.
Today I've been testing a number different things, including throwing quite
a number of files away and restart the copying (which skips files that are
already there). Alas, once the copying starts to get slow (a steady 6
seconds between files) it stays that way. Not even taking the stick out and
letting it "rest" for half an hour (and than restart). A reformat (of the
"quick" type) does allow it to become fast again though.
By the way, I said a second, NTFS formatted stick also got slow. I
just/might/ have been a bit too hasty there. Although /much/ slower than
the USB HD this morning it finished a copy in about 50 minutes.
I've just reformatted the origional stick to NTFS and have started another
copy. I'm currently waiting to see how it goes ...
....
Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
5 GByte) having been copied. As such it does /much/ worse than the second
stick.
I'm currently assuming that I was just unlucky enough to have bought a wonky
USB stick. I think the trash is the correct place for it.
Just why is Murphy so nasty to me ? The last time I assumed that I did my
homework it turned out that I forgot something, and now I'm trying to put
the blame on me (my program not doing its thing) it turns out to be
something I have zero control over ...
Oh wait, IIRC that is one of Murpy's other rules. :-)
Regards,
Rudy Wieser
> I bet the problem does not occur .... if you do the copy from a Command
> Prompt window.
Actually, my program is console based. :-)
> On Windows XP, Robocopy can be installed via a download.
own programs for it (as mentioned, using the CopyFile function outof
kernel32).
Also, as mentioned, the problem seems to be related to the USB stick, as
doing the same copy to an USB (spinning rust) disk doesn't show any kind of
slowdown problems.
Today I've been testing a number different things, including throwing quite
a number of files away and restart the copying (which skips files that are
already there). Alas, once the copying starts to get slow (a steady 6
seconds between files) it stays that way. Not even taking the stick out and
letting it "rest" for half an hour (and than restart). A reformat (of the
"quick" type) does allow it to become fast again though.
By the way, I said a second, NTFS formatted stick also got slow. I
just/might/ have been a bit too hasty there. Although /much/ slower than
the USB HD this morning it finished a copy in about 50 minutes.
I've just reformatted the origional stick to NTFS and have started another
copy. I'm currently waiting to see how it goes ...
....
Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
5 GByte) having been copied. As such it does /much/ worse than the second
stick.
I'm currently assuming that I was just unlucky enough to have bought a wonky
USB stick. I think the trash is the correct place for it.
Just why is Murphy so nasty to me ? The last time I assumed that I did my
homework it turned out that I forgot something, and now I'm trying to put
the blame on me (my program not doing its thing) it turns out to be
something I have zero control over ...
Oh wait, IIRC that is one of Murpy's other rules. :-)
Regards,
Rudy Wieser
Herbert Kleebauer
Jan 13, 2022, 12:37:34 PM1/13/22
to
On 13.01.2022 17:16, R.Wieser wrote:
> Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
> 5 GByte) having been copied. As such it does/much/ worse than the second
> Well, I've aborted the copy after about an hour, with just 1.6 GByte (outof
> stick.
What is the setting for "Enfernungsrichtlinie" in properties of the disk?
[x] Schnelles Enfernen (Standard)
Deaktiviert den Schreibcache auf dem Datenträger und in
Window. Das Gerät kann jedoch auch ohne das Symbol "Sicheres
Entfernen" sicher entfert werden.
[ ] Bessere Leistung
Aktiviert den Schreibcache in Windows ....
Did you try to copy 1 byte files. Maybe without cache the writing
of the directory for each file slows down the copy more than the
data transfer itself.
Char Jackson
Jan 13, 2022, 1:01:54 PM1/13/22
to
On Thu, 13 Jan 2022 10:53:55 +0100, "R.Wieser" <add...@not.available>
wrote:
>Char,
>
>> My totally unscientific observation is that writing to a USB2 device
>> starts out fast, and when it inevitably slows to a crawl I notice
>> that the device is blazing hot.
>
>I tried checking the sticks temperature (removed the "sleeve") and regulary
>touched both the processor as well as the flash chip on the other side. I
>would not call it hot by any means, just a bit warm.
Thanks for checking. On at least two of my USB2 sticks, they get too hot
to touch if I do extended writes, but it sounds like you've disproved my
theory.
wrote:
>Char,
>
>> My totally unscientific observation is that writing to a USB2 device
>> starts out fast, and when it inevitably slows to a crawl I notice
>> that the device is blazing hot.
>
>I tried checking the sticks temperature (removed the "sleeve") and regulary
>touched both the processor as well as the flash chip on the other side. I
>would not call it hot by any means, just a bit warm.
to touch if I do extended writes, but it sounds like you've disproved my
theory.
R.Wieser
Jan 13, 2022, 3:56:53 PM1/13/22
to
Herbert,
> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
When the memory stick was FAT32 formatted stick it was the first. Later I
formatted it to NTFS, which a choice you only get when the second is
enabled.
In short, I have had both for that stick. Didn't make much of a difference.
Regards,
Rudy Wieser
> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
formatted it to NTFS, which a choice you only get when the second is
enabled.
In short, I have had both for that stick. Didn't make much of a difference.
Regards,
Rudy Wieser
R.Wieser
Jan 13, 2022, 3:56:54 PM1/13/22
to
Char,
> Thanks for checking.
Thank you for mentioning it. I had not considered it as a possible cause.
> On at least two of my USB2 sticks, they get too hot to touch
> if I do extended writes, but it sounds like you've disproved
> my theory.
Too many different companies making them. I've had a a few of a brand which
would crap out after just a few uses. On the other hand I've still got a 1
GByte stick which got weekly use and now is over a decade old.
Regards,
Rudy Wieser
> Thanks for checking.
Thank you for mentioning it. I had not considered it as a possible cause.
> On at least two of my USB2 sticks, they get too hot to touch
> if I do extended writes, but it sounds like you've disproved
> my theory.
would crap out after just a few uses. On the other hand I've still got a 1
GByte stick which got weekly use and now is over a decade old.
Regards,
Rudy Wieser
Charlie Gibbs
Jan 13, 2022, 8:24:48 PM1/13/22
to
really sucked, but I stuck with it because FAT32 only stores time stamps to
the nearest 2 seconds, making time stamp comparison (and makefiles) problematic.
However, as you noted earlier, I also noticed that copying large numbers of
small files to a flash drive runs like molasses through a pinhole in January,
so I resorted to the solution that has already been mentioned here: zip the
files on your hard drive and then copy the .zip file to the flash drive.
Since zip files preserve time stamps to the second, there was no longer
any reason not to revert the flash drive to FAT32 and get even better speed.
Zipping directly to the flash drive causes many small work files to be
created and scratched on it, again resulting in abominable performance.
--
/~\ Charlie Gibbs | Microsoft is a dictatorship.
\ / <cgi...@kltpzyxm.invalid> | Apple is a cult.
X I'm really at ac.dekanfrus | Linux is anarchy.
/ \ if you read it the right way. | Pick your poison.
Ant
Jan 13, 2022, 9:52:44 PM1/13/22
to
In alt.windows7.general Charlie Gibbs <cgi...@kltpzyxm.invalid> wrote:
> On 2022-01-13, R.Wieser <add...@not.available> wrote:
> > Herbert,
> >
> >> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
> >
> > When the memory stick was FAT32 formatted stick it was the first. Later I
> > formatted it to NTFS, which a choice you only get when the second is
> > enabled.
> >
> > In short, I have had both for that stick. Didn't make much of a difference.
> I had a thumb drive formatted as NTFS for quite some time. The performance
> really sucked, but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and makefiles) problematic.
What about exFAT? NTFS is a problem to use on devices that don't know it.
> On 2022-01-13, R.Wieser <add...@not.available> wrote:
> > Herbert,
> >
> >> What is the setting for "Enfernungsrichtlinie" in properties of the disk?
> >
> > When the memory stick was FAT32 formatted stick it was the first. Later I
> > formatted it to NTFS, which a choice you only get when the second is
> > enabled.
> >
> > In short, I have had both for that stick. Didn't make much of a difference.
> I had a thumb drive formatted as NTFS for quite some time. The performance
> really sucked, but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and makefiles) problematic.
--
Slammy new week as expected. Lots of spams again! 2022 isn't any better and different so far. :(
Note: A fixed width font (Courier, Monospace, etc.) is required to see this signature correctly.
/\___/\ Ant(Dude) @ http://aqfl.net & http://antfarm.home.dhs.org.
/ /\ /\ \ Please nuke ANT if replying by e-mail.
| |o o| |
\ _ /
( )
R.Wieser
Jan 14, 2022, 3:59:09 AM1/14/22
to
Charlie,
> but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and
> makefiles) problematic.
I encountered a similar problem : even a just-copied file could be a number
of seconds different from the origional. As I'm not in the habit of running
backups while working on my machine I decided to allow for a 10 second
difference in timestamps.
That, and problems with summer/wintertime differences. :-\
> However, as you noted earlier, I also noticed that copying large numbers
> of small files to a flash drive runs like molasses through a pinhole in
> January,
Hmmm... Although comperativily it takes an USB stick quite a bit longer
than a USB HD, I do not really see any slow-down over the duration of the
copying itself.
> so I resorted to the solution that has already been mentioned here: zip
> the files on your hard drive and then copy the .zip file to the flash
> drive.
I also considered that, but that would have thrown quite a spanner in what
my program was actually build for : checking for changed files and
updating/deleting them on the copy.
Also, I wanted to be sure that regardless of how much space I had left on
the source I would always be able to make a backup or update it.
Next to that I also considered the possibility of a write or read error (bad
sector) could trash a compressed file, with little-to-no chance of recovery.
> Zipping directly to the flash drive causes many small work files to be
> created and scratched on it, again resulting in abominable performance.
Not only zipping to an USB stick. I also noticed a much worse performance
when extracting from a ZIP stored on an USB stick.
But yes, the effective need for an intermediate, to-be-transferred ZIP file
is what I ment with my "how much space I had left on the source" remark.
Regards,
Rudy Wieser
> but I stuck with it because FAT32 only stores time stamps to
> the nearest 2 seconds, making time stamp comparison (and
> makefiles) problematic.
of seconds different from the origional. As I'm not in the habit of running
backups while working on my machine I decided to allow for a 10 second
difference in timestamps.
That, and problems with summer/wintertime differences. :-\
> However, as you noted earlier, I also noticed that copying large numbers
> of small files to a flash drive runs like molasses through a pinhole in
> January,
than a USB HD, I do not really see any slow-down over the duration of the
copying itself.
> so I resorted to the solution that has already been mentioned here: zip
> the files on your hard drive and then copy the .zip file to the flash
> drive.
my program was actually build for : checking for changed files and
updating/deleting them on the copy.
Also, I wanted to be sure that regardless of how much space I had left on
the source I would always be able to make a backup or update it.
Next to that I also considered the possibility of a write or read error (bad
sector) could trash a compressed file, with little-to-no chance of recovery.
> Zipping directly to the flash drive causes many small work files to be
> created and scratched on it, again resulting in abominable performance.
when extracting from a ZIP stored on an USB stick.
But yes, the effective need for an intermediate, to-be-transferred ZIP file
is what I ment with my "how much space I had left on the source" remark.
Regards,
Rudy Wieser
Frank Slootweg
Jan 14, 2022, 10:16:09 AM1/14/22
to
Wieser <add...@not.available> wrote:
> Herbert,
>
> > What is the setting for "Enfernungsrichtlinie" in properties of the disk?
Google Translate broken!? :-) (Yes, GT can handle Herbert's small spelling
> Herbert,
>
> > What is the setting for "Enfernungsrichtlinie" in properties of the disk?
error, "Entfernungsrichtlinie".)
It means "removal policy".
<https://translate.google.com/?sl=de&tl=en&text=Enfernungsrichtlinie&op=translate>
But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
fast performance) removal policy, at least they don't on my Windows 8.1
system.
(Probably USB *disks* have a (fast removal versus fast performance)
removal policy, but I haven't checked.)
[...]
Char Jackson
Jan 14, 2022, 11:37:42 AM1/14/22
to
On Fri, 14 Jan 2022 09:57:44 +0100, "R.Wieser" <add...@not.available>
wrote:
One of the reasons that I like to use containers, such as zip or rar, is
that data corruption is much easier to spot. With bare files that aren't
in a container, bit rot can go undetected for a long period of time and
once found, it can be very difficult to correct.
My main tool for detecting bit rot, and then *correcting* it, is
Quickpar. http://www.quickpar.org.uk/
I use it almost daily and highly recommend it. It works equally well on
non-containerized files, but I mostly use it with multipart rar files.
<snip>
wrote:
that data corruption is much easier to spot. With bare files that aren't
in a container, bit rot can go undetected for a long period of time and
once found, it can be very difficult to correct.
My main tool for detecting bit rot, and then *correcting* it, is
Quickpar. http://www.quickpar.org.uk/
I use it almost daily and highly recommend it. It works equally well on
non-containerized files, but I mostly use it with multipart rar files.
<snip>
R.Wieser
Jan 14, 2022, 12:54:50 PM1/14/22
to
Char,
> One of the reasons that I like to use containers, such as zip or rar,
> is that data corruption is much easier to spot.
You mean, the first time you try to use /any/ file outof it it barfs and
refuses to let you access most, if not all of its contained files ? :-|
> With bare files that aren't in a container, bit rot can go undetected
> for a long period ...
True. But than I only lose the affected file(s), not the whole (or
many/most of) the container.
... and once found, it can be very difficult to correct.
Harder than fixing an bit-rotted container ? :-p
But granted, the sooner you become aware of a problem with a backup the
better.
> My main tool for detecting bit rot, and then *correcting* it,
> is Quickpar.
:-) You make it sound as if you can just throw a file at it and it will
detect and correct bitrot in it. Which is ofcourse not quite the way it
works.
Though granted, pulling the files thru such a program (which adds
detecting/recovery information to the file) before saving them to a backup
does enable the files to survive a decent amount of bitrot or other similary
small damage.
Regards,
Rudy Wieser
> One of the reasons that I like to use containers, such as zip or rar,
> is that data corruption is much easier to spot.
refuses to let you access most, if not all of its contained files ? :-|
> With bare files that aren't in a container, bit rot can go undetected
True. But than I only lose the affected file(s), not the whole (or
many/most of) the container.
... and once found, it can be very difficult to correct.
Harder than fixing an bit-rotted container ? :-p
But granted, the sooner you become aware of a problem with a backup the
better.
> My main tool for detecting bit rot, and then *correcting* it,
> is Quickpar.
detect and correct bitrot in it. Which is ofcourse not quite the way it
works.
Though granted, pulling the files thru such a program (which adds
detecting/recovery information to the file) before saving them to a backup
does enable the files to survive a decent amount of bitrot or other similary
small damage.
Regards,
Rudy Wieser
R.Wieser
Jan 14, 2022, 12:54:53 PM1/14/22
to
Frank,
> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.
On my XPsp3 machine I can choose either. Though when I set it to "fast
removal" I can't select NTFS when formatting (and vise-verse). Which, long
ago, confused the heck outof me. :-o
Regards,
Rudy Wieser
P.s.
The way you quoted it made it look as if /I/ posted that "what if" line ...
> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.
removal" I can't select NTFS when formatting (and vise-verse). Which, long
ago, confused the heck outof me. :-o
Regards,
Rudy Wieser
P.s.
The way you quoted it made it look as if /I/ posted that "what if" line ...
Herbert Kleebauer
Jan 14, 2022, 1:09:14 PM1/14/22
to
On 14.01.2022 16:16, Frank Slootweg wrote:
> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.
in explorer right-click on the external usb drive
> But AFAIK/AFAICT, USB memory sticks do not have a (fast removal versus
> fast performance) removal policy, at least they don't on my Windows 8.1
> system.
select "Properties"
on the Hardware tab, select your device and select "Properties"
select "Change Settings" on the General tab.
select the "Policies" tab
Frank Slootweg
Jan 14, 2022, 2:27:01 PM1/14/22
to
I could remember that functionality, but could not find it again.
In hindsight, I apparently overlooked several things, both on the
'Hardware' tab and on the second level 'Properties' popup/window.
So thanks again.
Frank Slootweg
Jan 14, 2022, 2:27:02 PM1/14/22
to
R.Wieser <add...@not.available> wrote:
> Frank,
[...]
> P.s.
> The way you quoted it made it look as if /I/ posted that "what if" line ...
Sorry, a case of sloppy reading! I thought you didn't understand
Herbert's German text.
I will now crawl back under my rock. :-)
> Frank,
[...]
> P.s.
> The way you quoted it made it look as if /I/ posted that "what if" line ...
Herbert's German text.
I will now crawl back under my rock. :-)
Char Jackson
Jan 14, 2022, 2:54:01 PM1/14/22
to
On Fri, 14 Jan 2022 18:54:24 +0100, "R.Wieser" <add...@not.available>
wrote:
>Char,
>
>> One of the reasons that I like to use containers, such as zip or rar,
>> is that data corruption is much easier to spot.
>
>You mean, the first time you try to use /any/ file outof it it barfs and
>refuses to let you access most, if not all of its contained files ? :-|
I suppose so, but I'd say that's preferable to the alternative, which is
not knowing. With individual files, I suppose each file would have to be
loaded in some way, checking for obvious corruption or error messages,
or in the case of system files like .dll and .exe just give it your best
shot, but all of the major container types give you an easy method to
detect corruption of the container itself. That's a huge benefit.
>> With bare files that aren't in a container, bit rot can go undetected
>> for a long period ...
>
>True. But than I only lose the affected file(s), not the whole (or
>many/most of) the container.
You don't actually have to lose any data. Quickpar can usually recover
that stuff, if it's used correctly.
>... and once found, it can be very difficult to correct.
>
>Harder than fixing an bit-rotted container ? :-p
Good point. Much harder to spot damage in individual files, but once
spotted, probably equally easy to correct if you've planned ahead.
>But granted, the sooner you become aware of a problem with a backup the
>better.
>
>> My main tool for detecting bit rot, and then *correcting* it,
>> is Quickpar.
>
>:-) You make it sound as if you can just throw a file at it and it will
>detect and correct bitrot in it. Which is ofcourse not quite the way it
>works.
Right, you have to plan ahead. You have to ask yourself, are these files
something that I care about? If so, how much damage do I want to be
prepared to correct? The answer to the second question, using Quickpar,
can be anywhere from 0% to 100%, but it's a decision that has to be made
before any damage occurs.
>Though granted, pulling the files thru such a program (which adds
>detecting/recovery information to the file) before saving them to a backup
>does enable the files to survive a decent amount of bitrot or other similary
>small damage.
Quickpar creates additional files (parity files) that you can use to
detect and correct damage. The actual files that you care about are not
touched in any way unless and until it becomes necessary to repair them.
You also don't have to store the parity files alongside the files that
they're protecting. You can, of course, but you can also store them, or
a second copy of them since they're relatively small, in another
location.
wrote:
>Char,
>
>> One of the reasons that I like to use containers, such as zip or rar,
>> is that data corruption is much easier to spot.
>
>You mean, the first time you try to use /any/ file outof it it barfs and
>refuses to let you access most, if not all of its contained files ? :-|
not knowing. With individual files, I suppose each file would have to be
loaded in some way, checking for obvious corruption or error messages,
or in the case of system files like .dll and .exe just give it your best
shot, but all of the major container types give you an easy method to
detect corruption of the container itself. That's a huge benefit.
>> With bare files that aren't in a container, bit rot can go undetected
>> for a long period ...
>
>True. But than I only lose the affected file(s), not the whole (or
>many/most of) the container.
that stuff, if it's used correctly.
>... and once found, it can be very difficult to correct.
>
>Harder than fixing an bit-rotted container ? :-p
spotted, probably equally easy to correct if you've planned ahead.
>But granted, the sooner you become aware of a problem with a backup the
>better.
>
>> My main tool for detecting bit rot, and then *correcting* it,
>> is Quickpar.
>
>:-) You make it sound as if you can just throw a file at it and it will
>detect and correct bitrot in it. Which is ofcourse not quite the way it
>works.
something that I care about? If so, how much damage do I want to be
prepared to correct? The answer to the second question, using Quickpar,
can be anywhere from 0% to 100%, but it's a decision that has to be made
before any damage occurs.
>Though granted, pulling the files thru such a program (which adds
>detecting/recovery information to the file) before saving them to a backup
>does enable the files to survive a decent amount of bitrot or other similary
>small damage.
detect and correct damage. The actual files that you care about are not
touched in any way unless and until it becomes necessary to repair them.
You also don't have to store the parity files alongside the files that
they're protecting. You can, of course, but you can also store them, or
a second copy of them since they're relatively small, in another
location.
R.Wieser
Jan 14, 2022, 5:01:15 PM1/14/22
to
Frank,
> Sorry, a case of sloppy reading! I thought you didn't understand
> Herbert's German text.
I'm actually from his west-side next-door neighbour, the Netherlands.
A lot of "nachsynchronisierter schaukasten films" (hope I wrote that right)
in my youth and a year getting at school enabled me to read most of it.
> I will now crawl back under my rock. :-)
Ah, you do not need to go /that/ far... :-p
Regards,
Rudy Wieser
> Sorry, a case of sloppy reading! I thought you didn't understand
> Herbert's German text.
A lot of "nachsynchronisierter schaukasten films" (hope I wrote that right)
in my youth and a year getting at school enabled me to read most of it.
> I will now crawl back under my rock. :-)
Regards,
Rudy Wieser
R.Wieser
Jan 14, 2022, 5:01:15 PM1/14/22
to
Char,
> I suppose so, but I'd say that's preferable to the alternative, which
> is not knowing
Hmmm... not knowning versus losing everything. I think I would go for not
knowing.
> I suppose each file would have to be loaded in some way
...
magic. A difference between lots of small parts versus one big blob.
Though the multiple small parts would need a seperate file to store all the
checksums. Either that, or add them as an alternate data stream to the
files themselves (which ofcourse would need the storage to be NTFS or
comparable in this regard)
> Right, you have to plan ahead. You have to ask yourself, are these
> files something that I care about?
Well, that is why I'm making backups of them, aren't I ? :-)
> If so, how much damage do I want to be prepared to correct?
I'm not prepared to repair anything.
On the other hand, I'm making multiple backups of stuff thats important to
me. If one even /hints/ of failing I make a new copy from it (or from one
of the others) - and compare it against one of the others to verify its
integrity.
> Quickpar creates additional files
Ah yes, ofcourse. For some reason I was thinking of the origional data
being wrapped in the parity/checksom/recovery data. Or that the data was
stored in a before mentioned alternate data stream.
Regards,
Rudy Wieser
> I suppose so, but I'd say that's preferable to the alternative, which
> is not knowing
knowing.
> I suppose each file would have to be loaded in some way
> but all of the major container types give you an easy method to
> detect corruption of the container itself
Which happens by reading the whole container and than doing some checking
> detect corruption of the container itself
magic. A difference between lots of small parts versus one big blob.
Though the multiple small parts would need a seperate file to store all the
checksums. Either that, or add them as an alternate data stream to the
files themselves (which ofcourse would need the storage to be NTFS or
comparable in this regard)
> Right, you have to plan ahead. You have to ask yourself, are these
> files something that I care about?
> If so, how much damage do I want to be prepared to correct?
On the other hand, I'm making multiple backups of stuff thats important to
me. If one even /hints/ of failing I make a new copy from it (or from one
of the others) - and compare it against one of the others to verify its
integrity.
> Quickpar creates additional files
Ah yes, ofcourse. For some reason I was thinking of the origional data
being wrapped in the parity/checksom/recovery data. Or that the data was
stored in a before mentioned alternate data stream.
Regards,
Rudy Wieser
Char Jackson
Jan 14, 2022, 8:31:53 PM1/14/22
to
On Fri, 14 Jan 2022 23:00:46 +0100, "R.Wieser" <add...@not.available>
wrote:
>Char,
>
>> I suppose so, but I'd say that's preferable to the alternative, which
>> is not knowing
>
>Hmmm... not knowning versus losing everything. I think I would go for not
>knowing.
That's a false choice, obviously. :-)
The choice is knowing or not knowing. It's much, much easier to check a
container (and by doing so, check all of its contents in one fell
swoop), than to check individual files.
It sounds like you have a solution that works for you, though, so it's
all good.
<snip>
wrote:
>Char,
>
>> I suppose so, but I'd say that's preferable to the alternative, which
>> is not knowing
>
>Hmmm... not knowning versus losing everything. I think I would go for not
>knowing.
The choice is knowing or not knowing. It's much, much easier to check a
container (and by doing so, check all of its contents in one fell
swoop), than to check individual files.
It sounds like you have a solution that works for you, though, so it's
all good.
<snip>
R.Wieser
Jan 15, 2022, 2:44:22 AM1/15/22
to
Char,
> That's a false choice, obviously. :-)
>
> The choice is knowing or not knowing.
I don't think so. But lets not pursue this further, shall we.
> It's much, much easier to check a container (and by doing so,
> check all of its contents in one fell swoop), than to check
> individual files.
:-) Its not you doing the checking, but a program. And just as you can let
your containerizing program do the checking for you so can I start my
program to trawl thru all those files of mine and do the same.
> It sounds like you have a solution that works for you, though, so
> it's all good.
I was thinking the same.
Thanks for bringing your methods up though. It gave me other possibilities
to consider. Which is always good.
Regards,
Rudy Wieser
> That's a false choice, obviously. :-)
>
> The choice is knowing or not knowing.
> It's much, much easier to check a container (and by doing so,
> check all of its contents in one fell swoop), than to check
> individual files.
your containerizing program do the checking for you so can I start my
program to trawl thru all those files of mine and do the same.
> It sounds like you have a solution that works for you, though, so
> it's all good.
Thanks for bringing your methods up though. It gave me other possibilities
to consider. Which is always good.
Regards,
Rudy Wieser
R.Wieser
Jan 15, 2022, 8:04:00 AM1/15/22
to
Steve,
> Sometimes, especially on the XP machine, the USB connection drops in
> the middle of a transfer, and then I need to run CHKDSK on the flash
> drive, otherwise it slows down or might lose data.
Thanks for the suggestion. Someone else mentioned that his stick heats up
while writing and causes problems like I described. Maybe yours has the
same problem ?
After I posted my above question and not getting a "Thats a well-known
problem" response I've done a number of tests. As it turns out the problem
is that specific USB stick (see the further posts I made)
As such I decided to effectivily bin it (I can still muck around with it,
but it won't ever be used for storage again)
Regards,
Rudy Wieser
> Sometimes, especially on the XP machine, the USB connection drops in
> the middle of a transfer, and then I need to run CHKDSK on the flash
> drive, otherwise it slows down or might lose data.
Thanks for the suggestion. Someone else mentioned that his stick heats up
while writing and causes problems like I described. Maybe yours has the
same problem ?
After I posted my above question and not getting a "Thats a well-known
problem" response I've done a number of tests. As it turns out the problem
is that specific USB stick (see the further posts I made)
As such I decided to effectivily bin it (I can still muck around with it,
but it won't ever be used for storage again)
Regards,
Rudy Wieser
Frank Slootweg
Jan 15, 2022, 8:07:29 AM1/15/22
to
R.Wieser <add...@not.available> wrote:
> Frank,
>
> > Sorry, a case of sloppy reading! I thought you didn't understand
> > Herbert's German text.
>
> I'm actually from his west-side next-door neighbour, the Netherlands.
Yes, I know that, like me, you're a Dutchie. In the beginning your
> Frank,
>
> > Sorry, a case of sloppy reading! I thought you didn't understand
> > Herbert's German text.
>
> I'm actually from his west-side next-door neighbour, the Netherlands.
lastname had me thinking you were German, but that was a long time ago.
> A lot of "nachsynchronisierter schaukasten films" (hope I wrote that right)
> in my youth and a year getting at school enabled me to read most of it.
German. And - in news programs - neither is Joe Biden. There's this
thing called 'subtitles', you should try it some time! :-(
> > I will now crawl back under my rock. :-)
>
> Ah, you do not need to go /that/ far... :-p
R.Wieser
Jan 15, 2022, 12:14:30 PM1/15/22
to
Frank,
> Yes, I know that, like me, you're a Dutchie. In the beginning your
> lastname had me thinking you were German, but that was a long time ago.
Well, I've been told that my great-granddad (or one above that) came from
Germany to try his luck here, and just never left. IOW, do seem to have
some german blood in me.
> I hate(d) those films. Earth to Germany: John Wayne isn't (wasn't) a
> German. And - in news programs - neither is Joe Biden.
nowerdays I have the same dislike towards televised movies. Not much of a
problem with having some dubbed stuff coming thru the news though.
> There's this thing called 'subtitles', you should try it some time! :-(
Gee, what an innovative idea. :-p
But yes, I take that over audio dubbing any time. Than again, I've always
found subtitles handy to support my understanding of what gets spoken.
>> > I will now crawl back under my rock. :-)
>>
>> Ah, you do not need to go /that/ far... :-p
>
> Well, it's actually quite cosy! :-)
Well yeah, but it hurts my back when I have to bow down that deep to look at
you when I want to talk to you. :-)
Regards,
Rudy Wieser
> Yes, I know that, like me, you're a Dutchie. In the beginning your
> lastname had me thinking you were German, but that was a long time ago.
Germany to try his luck here, and just never left. IOW, do seem to have
some german blood in me.
> I hate(d) those films. Earth to Germany: John Wayne isn't (wasn't) a
> German. And - in news programs - neither is Joe Biden.
problem with having some dubbed stuff coming thru the news though.
> There's this thing called 'subtitles', you should try it some time! :-(
But yes, I take that over audio dubbing any time. Than again, I've always
found subtitles handy to support my understanding of what gets spoken.
>> > I will now crawl back under my rock. :-)
>>
>> Ah, you do not need to go /that/ far... :-p
>
> Well, it's actually quite cosy! :-)
you when I want to talk to you. :-)
Regards,
Rudy Wieser
Brian Gregory
Jan 15, 2022, 9:23:13 PM1/15/22
to
On 12/01/2022 13:52, R.Wieser wrote:
> Paul,
>
> Thanks for the suggestions.
>
> The reason why I copy files is that I can than access them using any kind of
> filebrowser - even a simple one which doesn't understand the concept of ZIP
> folders.
>
> As for the cluster size, the second USB stick was new, and as such I assume
> it was pre-formatted with the optimum block size. Also, it was NTFS (I had
> hoped it would make a difference, but it looks like it doesn't)
>
> And there is a problem with that suggestion : Years ago, when I thought of
> the same (micro-SD connected to a DOS 'puter), I was unable to get that
> optimum from the USB stick itself. No matter which drive interrogation
> function I looked at, none of them returned the size of the FLASH chips
> memory blocks . :-(
As a rule of thumb always format USB flash drives with 32K allocation
blocks. A possible alternative is to remember what size blocks were used
when it was brand new. By default Windoze tends to format USB flash with
blocks that are much too small.
--
Brian Gregory (in England).
> Paul,
>
> Thanks for the suggestions.
>
> The reason why I copy files is that I can than access them using any kind of
> filebrowser - even a simple one which doesn't understand the concept of ZIP
> folders.
>
> As for the cluster size, the second USB stick was new, and as such I assume
> it was pre-formatted with the optimum block size. Also, it was NTFS (I had
> hoped it would make a difference, but it looks like it doesn't)
>
> And there is a problem with that suggestion : Years ago, when I thought of
> the same (micro-SD connected to a DOS 'puter), I was unable to get that
> optimum from the USB stick itself. No matter which drive interrogation
> function I looked at, none of them returned the size of the FLASH chips
> memory blocks . :-(
As a rule of thumb always format USB flash drives with 32K allocation
blocks. A possible alternative is to remember what size blocks were used
when it was brand new. By default Windoze tends to format USB flash with
blocks that are much too small.
--
Brian Gregory (in England).
R.Wieser
Jan 16, 2022, 4:03:16 AM1/16/22
to
Brian,
> As a rule of thumb always format USB flash drives with 32K allocation
> blocks.
Thanks. I've heard that number before, but never thought of it as an
universal one (true for sticks of any make and size). I've always assumed
that the flash-chips block size could/would change with the chips size.
> A possible alternative is to remember what size blocks were used when it
> was brand new.
You mean that I have to store those values for all my different (and
changing) sticks in a document somewhere (or a sticking it on the stick
itself) ? Yeah, I thought about that too. But as I cannot even seem to
be able to uniquely mark an individual stick (so I can look it up in a list)
...
> By default Windoze tends to format USB flash with blocks that are much too
> small.
And there I was, assuming that MS would know more about this that I could
ever hope to do, and therefore use some sensible default. :-(
Thanks for the info.
Regards,
Rudy Wieser
> As a rule of thumb always format USB flash drives with 32K allocation
> blocks.
universal one (true for sticks of any make and size). I've always assumed
that the flash-chips block size could/would change with the chips size.
> A possible alternative is to remember what size blocks were used when it
> was brand new.
changing) sticks in a document somewhere (or a sticking it on the stick
itself) ? Yeah, I thought about that too. But as I cannot even seem to
be able to uniquely mark an individual stick (so I can look it up in a list)
...
> By default Windoze tends to format USB flash with blocks that are much too
> small.
ever hope to do, and therefore use some sensible default. :-(
Thanks for the info.
Regards,
Rudy Wieser
Mayayana
Jan 16, 2022, 8:39:21 AM1/16/22
to
"R.Wieser" <add...@not.available> wrote
| > As a rule of thumb always format USB flash drives with 32K allocation
| > blocks.
|
| Thanks. I've heard that number before, but never thought of it as an
| universal one (true for sticks of any make and size). I've always
assumed
| that the flash-chips block size could/would change with the chips size.
|
It's not universal. Block size only relates to the limits
of 4-byte integers. I don't know whether the counting is
an unsigned or signed number, but the blocks have to
be big enough that the number of them doesn't end up
more than the limit of a long integer on a 32-bit system.
| > As a rule of thumb always format USB flash drives with 32K allocation
| > blocks.
|
| Thanks. I've heard that number before, but never thought of it as an
| universal one (true for sticks of any make and size). I've always
assumed
| that the flash-chips block size could/would change with the chips size.
|
of 4-byte integers. I don't know whether the counting is
an unsigned or signed number, but the blocks have to
be big enough that the number of them doesn't end up
more than the limit of a long integer on a 32-bit system.
Paul
Jan 16, 2022, 9:29:24 AM1/16/22
to
On 1/16/2022 8:39 AM, Mayayana wrote:
> "R.Wieser" <add...@not.available> wrote
>
> | > As a rule of thumb always format USB flash drives with 32K allocation
> | > blocks.
> |
> | Thanks. I've heard that number before, but never thought of it as an
> | universal one (true for sticks of any make and size). I've always
> assumed
> | that the flash-chips block size could/would change with the chips size.
> |
>
> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.
I hoped to have two examples (using dead USB sticks),
> "R.Wieser" <add...@not.available> wrote
>
> | > As a rule of thumb always format USB flash drives with 32K allocation
> | > blocks.
> |
> | Thanks. I've heard that number before, but never thought of it as an
> | universal one (true for sticks of any make and size). I've always
> assumed
> | that the flash-chips block size could/would change with the chips size.
> |
>
> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.
but one of them, the datasheet is untraceable.
**********************************************
Lexar S73 32GB (Not exactly fast)
29f128g08cfaaa 16GBx2
Open NAND Flash Interface (ONFI) 2.2-compliant
Multiple-level cell (MLC) technology
Organization
Page size x8: 8640 bytes (8192 + 448 bytes) <=== the 448 bytes could be ECC
Block size: 256 pages (2048K + 112K bytes)
Plane size: 2 planes x 2048 blocks per plane
Device size: 64Gb: 4096 blocks;
128Gb: 8192 blocks; <===
256Gb: 16,384 blocks;
512Gb: 32,786 blocks
Endurance: 5000 PROGRAM/ERASE cycles
Array performance
Read page: 50μs (MAX)
Program page: 1300μs (TYP)
Erase block: 3ms (TYP)
Synchronous I/O performance
Up to synchronous timing mode 5
Clock rate: 10ns (DDR)
Read/write throughput per pin: 200 MT/s
**********************************************
I would think a cluster size of 8K or larger
would be a start, to reduce write amplification.
Note that Windows 7 preparation of the stick, should
align the beginning of the partition to 1MB (0x100000)
boundaries. And by aligning the partition to binary-power-of-two,
that avoids the OS trying to write "the end of one page and
a bit onto the second page".
Don't format the sticks on Windows XP, for fear or
screwing up the alignment.
Then do the experiment again.
The second chip I wanted to research was this,
but I got zero reasonable leads for this search.
***************************************
This is a 128GB miniature USB flash stick, with only
a single Ball Grid Array flash chip on it. Can't find a
datasheet, to get a page size. The marking on the chip
of "NW605" is not a part number, but seems to reference
the MT29F part number below.
Lexar S23-128 plastic barrel, 10MB/sec writes
3XA22 NW605 (BGA)
MT29F1T08CUEABH8-12:A
NAND Flash 1TBIT 83MHZ 152LBGA
This is one of those "really shouldn't be a USB3 product" designs.
The plastic barrel for the USB stick was a disaster, allowing
one of the USB3 high speed pins to snap off during insertion,
and dooming the device to reading at USB2 rates.
***************************************
I have other USB sticks that are still working, but on those
the trick to disassembling them without damaging them is not obvious
to me.
Paul
R.Wieser
Jan 16, 2022, 9:50:18 AM1/16/22
to
Mayayana,
> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.
Are you sure ? I mean, why would a flash chip do anything with individual
byte addressing over its whole range ?
As far as I know the reading/writing of the flash chip is broken up in two
parts : one command with an address to indicate which block needs to be
read/written, and another command with its own address to read bytes
from/write bytes to the flash-chips internal buffer.
As long as that buffer is smaller than what the OS can handle everything is
hunky-dory. Which might well be why the blocksize is limited to "just" 32
KByte.
Regards,
Rudy Wieser
P.s.
I just checked, and the largest "allocation size" I can choose is a mere
4096 bytes ....
> It's not universal. Block size only relates to the limits
> of 4-byte integers. I don't know whether the counting is
> an unsigned or signed number, but the blocks have to
> be big enough that the number of them doesn't end up
> more than the limit of a long integer on a 32-bit system.
byte addressing over its whole range ?
As far as I know the reading/writing of the flash chip is broken up in two
parts : one command with an address to indicate which block needs to be
read/written, and another command with its own address to read bytes
from/write bytes to the flash-chips internal buffer.
As long as that buffer is smaller than what the OS can handle everything is
hunky-dory. Which might well be why the blocksize is limited to "just" 32
KByte.
Regards,
Rudy Wieser
P.s.
I just checked, and the largest "allocation size" I can choose is a mere
4096 bytes ....
Frank Slootweg
Jan 16, 2022, 11:07:31 AM1/16/22
to
R.Wieser <add...@not.available> wrote:
> P.s.
> I just checked, and the largest "allocation size" I can choose is a mere
> 4096 bytes ....
Maybe that's an XP limitation.
> P.s.
> I just checked, and the largest "allocation size" I can choose is a mere
> 4096 bytes ....
FWIW, on my Windows 8.1 system for a 4GB SD-card (i.e. not an USB
memory stick) the choices range from 1024 bytes to 32 kilobytes for
FAT32 and from 512 bytes to 64 kilobytes for NTFS.
What was your stick-size and type of filesystem again? (I lost track
in this long thread.)
R.Wieser
Jan 16, 2022, 11:49:00 AM1/16/22
to
Frank,
> What was your stick-size and type of filesystem again? (I lost
> track in this long thread.)
Its an 8 GByte stick currently NTFS formatted.
Regards,
Rudy Wieser
> What was your stick-size and type of filesystem again? (I lost
> track in this long thread.)
Regards,
Rudy Wieser
Charlie Gibbs
Jan 16, 2022, 1:42:49 PM1/16/22
to
On 2022-01-16, R.Wieser <add...@not.available> wrote:
>> By default Windoze tends to format USB flash with blocks that are much too
>> small.
>
> And there I was, assuming that MS would know more about this that I could
> ever hope to do, and therefore use some sensible default. :-(
Sensible default? Microsoft? Thanks - I needed a laugh this morning.
>> By default Windoze tends to format USB flash with blocks that are much too
>> small.
>
> And there I was, assuming that MS would know more about this that I could
> ever hope to do, and therefore use some sensible default. :-(
Paul
Jan 16, 2022, 2:13:19 PM1/16/22
to
On 1/16/2022 1:42 PM, Charlie Gibbs wrote:
> On 2022-01-16, R.Wieser <add...@not.available> wrote:
>
>>> By default Windoze tends to format USB flash with blocks that are much too
>>> small.
>>
>> And there I was, assuming that MS would know more about this that I could
>> ever hope to do, and therefore use some sensible default. :-(
>
> Sensible default? Microsoft? Thanks - I needed a laugh this morning.
>
For formatting large FAT32 volumes (up to 2.2TB), the Ridgecrop
> On 2022-01-16, R.Wieser <add...@not.available> wrote:
>
>>> By default Windoze tends to format USB flash with blocks that are much too
>>> small.
>>
>> And there I was, assuming that MS would know more about this that I could
>> ever hope to do, and therefore use some sensible default. :-(
>
> Sensible default? Microsoft? Thanks - I needed a laugh this morning.
>
fat32formatter can do that. This solves the ~32GB limit Microsoft
has on fat32 partitions. I think there might be some cluster
size options as well.
https://web.archive.org/web/20200424145132/http://www.ridgecrop.demon.co.uk/index.htm?fat32format.htm
https://web.archive.org/web/20200410224838if_/http://www.ridgecrop.demon.co.uk/download/fat32format.zip
*******
Windows 10 has some extra-large cluster size options for
NTFS, but those are not backward compatible with NTFS
on earlier OSes, and I can't really recommend using
a 1MB cluster on Win10 and then not being able to
read it on Windows 7. Clusters up to 64KB might be
a bit more compatible across the OSes.
*******
4KB clusters are used on C: , so that Compression and
EFS encryption will work. Windows 10 C: used to support
having 64KB clusters on it. But after the third or
fourth upgrade, it stopped "tolerating" that choice.
I only discovered this by accident, by taking an older
partition and installing Windows 10 in it... without
reformatting first. Oops :-)
Since you are not normally installing Windows 10 C: on
a USB flash drive, this 4KB choice is not an issue.
The Windows To Go thing, might have been the reason
there was some room for non-standard clusters at first.
Once WTG was removed as an option, that would make it
easier for them to insist on 4KB clusters.
https://docs.microsoft.com/en-us/windows/deployment/planning/windows-to-go-overview
Paul
Brian Gregory
Jan 17, 2022, 6:50:35 AM1/17/22
to
Brian Gregory
Jan 17, 2022, 6:56:53 AM1/17/22
to
On 16/01/2022 08:20, R.Wieser wrote:
> Brian,
>
>> As a rule of thumb always format USB flash drives with 32K allocation
>> blocks.
>
> Thanks. I've heard that number before, but never thought of it as an
> universal one (true for sticks of any make and size). I've always assumed
> that the flash-chips block size could/would change with the chips size.
It probably does change. But we're not needing an exact match to get a
> Brian,
>
>> As a rule of thumb always format USB flash drives with 32K allocation
>> blocks.
>
> Thanks. I've heard that number before, but never thought of it as an
> universal one (true for sticks of any make and size). I've always assumed
> that the flash-chips block size could/would change with the chips size.
speed benefit.
Having a big block on the drive divided into many small allocation
blocks seems to be what can slow things down.
R.Wieser
Jan 17, 2022, 11:18:54 AM1/17/22
to
Brian,
> If you want it to work fast don't use NTFS. Use FAT or exFAT.
I only reformatted to NTFS to check if the problem was perhaps files-system
related. It doesn't seem to be.
> It probably does change. But we're not needing an exact match to get a
> speed benefit.
Speed is one thing. Not needing reading-modify-write the same flash-block
is another. And that will happen if the allocation size is not the same
as, or a multiple of, the flash-block size.
> Having a big block on the drive divided into many small allocation blocks
> seems to be what can slow things down.
:-) you or the stick needing to do multiple read-modify-write actions will
/definitily/ slow the stick down more than a simple write action - even if
you just divide the flash block in (an uneven) two.
IOW, I do understand the mechanics behind it. Which is why I'm (still) a
bit surprised you cannot ask the stick for its optimum allocation size ...
Regards,
Rudy Wieser
> If you want it to work fast don't use NTFS. Use FAT or exFAT.
related. It doesn't seem to be.
> It probably does change. But we're not needing an exact match to get a
> speed benefit.
is another. And that will happen if the allocation size is not the same
as, or a multiple of, the flash-block size.
> Having a big block on the drive divided into many small allocation blocks
> seems to be what can slow things down.
/definitily/ slow the stick down more than a simple write action - even if
you just divide the flash block in (an uneven) two.
IOW, I do understand the mechanics behind it. Which is why I'm (still) a
bit surprised you cannot ask the stick for its optimum allocation size ...
Regards,
Rudy Wieser
Mike S
Jan 18, 2022, 2:26:23 AM1/18/22
to
happens.
R.Wieser
Jan 18, 2022, 6:33:40 AM1/18/22
to
Mike,
> Might be interesting to open it up and add a heat sink and see what
> happens.
I thought of that too. But after I stopped the copying, ejected it, left it
alone for an hour (letting it cool down), restarted the copy and than saw it
was directly going back into its one-file-each-six-seconds rithm I didn't
think that that would give me any more information.
Regards,
Rudy Wieser
> Might be interesting to open it up and add a heat sink and see what
> happens.
alone for an hour (letting it cool down), restarted the copy and than saw it
was directly going back into its one-file-each-six-seconds rithm I didn't
think that that would give me any more information.
Regards,
Rudy Wieser
Brian Gregory
Jan 18, 2022, 10:12:21 AM1/18/22
to
as an allocation block size.
Brian Gregory
Jan 18, 2022, 10:16:42 AM1/18/22
to
On 12/01/2022 11:28, R.Wieser wrote:
> Hello all,
>
> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
> that the whole thing slows down to a crawl (about 5 GByte consisting of
> 50.000 files in over eight hours). I've also tried a another, different USB
> stick and see the same thing happen.
If you can try with XCOPY.
XCOPY is good at optimizing the writes to the directory.
I think it opens and creates all the files first, then copies the
contents one by one, and finally closes all the files.
> Hello all,
>
> I've got a simple program using CopyFile (kernel32) on an XPsp3 machine to
> copy a number of files to a 8GByte, FAT32 formatted USB stick, and notice
> that the whole thing slows down to a crawl (about 5 GByte consisting of
> 50.000 files in over eight hours). I've also tried a another, different USB
> stick and see the same thing happen.
If you can try with XCOPY.
XCOPY is good at optimizing the writes to the directory.
I think it opens and creates all the files first, then copies the
contents one by one, and finally closes all the files.
R.Wieser
Jan 18, 2022, 11:44:43 AM1/18/22
to
Brian,
> I think the flash used often has much larger pages than would make sense
> as an allocation block size.
There is nothing much I can say to that, as its as unspecific as it comes
...
Regards,
Rudy Wieser
> I think the flash used often has much larger pages than would make sense
> as an allocation block size.
...
Regards,
Rudy Wieser
Paul
Jan 18, 2022, 1:04:25 PM1/18/22
to
that ensures proper cluster alignment for flash memory storage.
Doesn't matter what file system you use, if the partitions
start on 1MB boundaries, that should help a lot with aligning
the clusters (which are power-of-two), with the flash hardware
addressing (which is power-of-two favoring).
If you cleaned the stick and prepared it on WinXP, then bits of
the geometry are divisible by 63, and that is not a power of two
number, and that doubles the amount of work for writes (assuming
lots of small files, not just one big file).
I've made this kind of error before with hard drives - prepped
them with WinXP, then notice that "something is wrong" on Windows 7.
And eventually, by accident, I might discover my alignment error.
That preparing on Windows XP was a mistake. This is an issue with
512e drives (512 virtual, 4K physical) and not an issue with
512n drives (512 virtual, 512 physical).
I have one drive on this machine right now, WD Gold 4TB, and it is
one of the largest 512n drives you can get. And that means it
doesn't have alignment issues internally (no edge cases, no
read-modify-write). It was purchased in case I needed the drive for
Windows XP, but my Windows XP machine is dead, so now I can use
the 512n drive(s) I have for other things.
Paul
J. P. Gilliver (John)
Jan 18, 2022, 1:32:34 PM1/18/22
to
On Tue, 18 Jan 2022 at 13:04:09, Paul <nos...@needed.invalid> wrote (my
responses usually follow points raised):
[]
How did the 63 size come about? It seems a very odd choice in computing.
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf
Mike Jackson |\ _,,,---,,_
and Squeak /,`.-'`' -. ;-;;,_ Shame there's no snooze button
[1998] |,4- ) )-,_..;\ ( `'- on a cat who wants breakfast
zzz '---''(_/--' `-'\_)
responses usually follow points raised):
[]
>If you cleaned the stick and prepared it on WinXP, then bits of
>the geometry are divisible by 63, and that is not a power of two
>number, and that doubles the amount of work for writes (assuming
>lots of small files, not just one big file).
[]
>the geometry are divisible by 63, and that is not a power of two
>number, and that doubles the amount of work for writes (assuming
>lots of small files, not just one big file).
How did the 63 size come about? It seems a very odd choice in computing.
--
J. P. Gilliver. UMRA: 1960/<1985 MB++G()AL-IS-Ch++(p)Ar@T+H+Sh0!:`)DNAf
Mike Jackson |\ _,,,---,,_
and Squeak /,`.-'`' -. ;-;;,_ Shame there's no snooze button
[1998] |,4- ) )-,_..;\ ( `'- on a cat who wants breakfast
zzz '---''(_/--' `-'\_)
Paul
Jan 18, 2022, 1:42:44 PM1/18/22
to
On 1/18/2022 1:31 PM, J. P. Gilliver (John) wrote:
> On Tue, 18 Jan 2022 at 13:04:09, Paul <nos...@needed.invalid> wrote (my responses usually follow points raised):
> []
>> If you cleaned the stick and prepared it on WinXP, then bits of
>> the geometry are divisible by 63, and that is not a power of two
>> number, and that doubles the amount of work for writes (assuming
>> lots of small files, not just one big file).
> []
> How did the 63 size come about? It seems a very odd choice in computing.
It was a choice in CHS.
> On Tue, 18 Jan 2022 at 13:04:09, Paul <nos...@needed.invalid> wrote (my responses usually follow points raised):
> []
>> If you cleaned the stick and prepared it on WinXP, then bits of
>> the geometry are divisible by 63, and that is not a power of two
>> number, and that doubles the amount of work for writes (assuming
>> lots of small files, not just one big file).
> []
> How did the 63 size come about? It seems a very odd choice in computing.
https://en.wikipedia.org/wiki/Cylinder-head-sector
63 sectors/track
Windows 7 uses unconventional-looking values in the partition table.
You can see these with ptedit32.exe . A utility you should have
acquired back when it was easy to get. The FTP server it was on (for free),
closed some time ago. And since archive.org does not archive FTP sites,
the info is easily lost when files are hosted that way.
Paul
Paul
Jan 18, 2022, 1:51:30 PM1/18/22
to
If just means using a custom value, rather than accepting
the default value.
You can take a look through the Micron catalog for
values on some TLC, but then, you won't know which
of the chips is used for USB flash sticks. Presumably
some of the flash is less well tested and suited
to making USB flash sticks. As Micky would say "what
do you expect for $1.50". Someone has to cater to that
market, of "cheap as chips".
I don't think they really throw anything away.
There's always someone vending defective flash, somewhere.
With the bazaar selling model, anything is sustainable.
The TSOP or quad flatpack style chips, might be used
in long-ish USB flash sticks. Wheres the fine pitch BGA
could be used for the stubby USB flash sticks. I only have
one stubby one here, and it sucks. That's the one where I
couldn't find a datasheet for the chip.
Paul
Paul
Jan 19, 2022, 4:02:06 AM1/19/22
to
On 1/18/2022 1:51 PM, Paul wrote:
> On 1/18/2022 11:42 AM, R.Wieser wrote:
>> Brian,
>>
>>> I think the flash used often has much larger pages than would make sense
>>> as an allocation block size.
>>
>> There is nothing much I can say to that, as its as unspecific as it comes
>> ...
>>
>> Regards,
>> Rudy Wieser
>
> 8KB is a good value for a 32GB flash stick.
>
Wikipedia says the page size is 4KB to 16KB.
> On 1/18/2022 11:42 AM, R.Wieser wrote:
>> Brian,
>>
>>> I think the flash used often has much larger pages than would make sense
>>> as an allocation block size.
>>
>> There is nothing much I can say to that, as its as unspecific as it comes
>> ...
>>
>> Regards,
>> Rudy Wieser
>
> 8KB is a good value for a 32GB flash stick.
>
https://en.wikipedia.org/wiki/Flash_memory
The pages can't be too large, because that
would affect the performance of the chip.
Paul
R.Wieser
Jan 19, 2022, 6:20:14 AM1/19/22
to
Paul,
> Wikipedia says the page size is 4KB to 16KB.
Which, in short, means three choices : 4K, 8K or 16K , as those numbers need
to be powers of two.
Though that page also mentions something about an "erase page", which size
can be multiple millions of bytes. And somehow that reads as it doesn't
matter if your allocation size matches the page size (unless it matches the
"erasure page" size) ...
Regards,
Rudy Wieser
> Wikipedia says the page size is 4KB to 16KB.
to be powers of two.
Though that page also mentions something about an "erase page", which size
can be multiple millions of bytes. And somehow that reads as it doesn't
matter if your allocation size matches the page size (unless it matches the
"erasure page" size) ...
Regards,
Rudy Wieser
Paul
Jan 19, 2022, 7:45:10 AM1/19/22
to
device, but as long as it has the usual flavor of wear leveling,
the pages can be recycled as needed. Maybe it costs you two writes,
to do 4K clusters on 8K pages, the second write being 8KB of data.
The second half of a page should be write-able, even if it wasn't
used on the first attempt. But if the stick did that, the storage
would be hard to manage.
I would say, having a larger cluster is likely a better idea.
And you can use the RidgeCrop formatter, if you want to customize
the FAT32 on a 128GB USB stick (as Microsoft doesn't do FAT32 above
a certain size).
In the case of NTFS, you can easily select 64KB clusters if you want, but if
you're doing 50000 files, be aware of the "wasted space" problem. I've
used 64K clusters on a backup partition before. For no particular
reason except to try it.
One other thing, regarding the heat issue. It seems some of the
flash chips are 3.3V ones, the USB stick is running off a 5V supply,
and there could be a dropping regulator of some sort to power the flash.
That might be where some of the heat comes from.
Paul
Brian Gregory
Jan 20, 2022, 9:03:34 PM1/20/22
to
what we're discussing?
R.Wieser
Jan 21, 2022, 5:16:19 AM1/21/22
to
Brian,
> Why would any size except the "erase page" size be at all relevant to what
> we're discussing?
Good question. Why don't you start with telling us why it doesn't. ? :-)
Regards,
Rudy Wieser
> Why would any size except the "erase page" size be at all relevant to what
> we're discussing?
Regards,
Rudy Wieser
Brian Gregory
Jan 26, 2022, 8:58:03 AM1/26/22
to
On 21/01/2022 10:05, R.Wieser wrote:
> Brian,
>
>> Why would any size except the "erase page" size be at all relevant to what
>> we're discussing?
>
> Good question. Why don't you start with telling us why it doesn't. ? :-)
Any other size mismatch can be easily worked around without having to
> Brian,
>
>> Why would any size except the "erase page" size be at all relevant to what
>> we're discussing?
>
> Good question. Why don't you start with telling us why it doesn't. ? :-)
wait for the Flash chips to do anything.
R.Wieser
Jan 26, 2022, 10:12:18 AM1/26/22
to
Brian,
>>> Why would any size except the "erase page" size be at all relevant to
>>> what
>>> we're discussing?
>>
>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>
> Any other size mismatch can be easily worked around without having to wait
> for the Flash chips to do anything.
Than you have a problem. I do not know of any computer, Windows or
>>> Why would any size except the "erase page" size be at all relevant to
>>> what
>>> we're discussing?
>>
>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>
> Any other size mismatch can be easily worked around without having to wait
> for the Flash chips to do anything.
otherwise, which has a cluster size of a few megabytes. IOW, you will
/always/ have a mismatch in regard to the erase-page size.
Also, *what* other size that can be mismatched ?
And by the way : try to imagine what needs to be done when someone writes a
block matching some other smaller block size into a erase-block sized page
which causes the need to have bits changed from a Zero to a One.
Regards,
Rudy Wieser
Brian Gregory
Jan 26, 2022, 4:21:24 PM1/26/22
to
On 26/01/2022 15:11, R.Wieser wrote:
> Brian,
>
>>>> Why would any size except the "erase page" size be at all relevant to
>>>> what
>>>> we're discussing?
>>>
>>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>>
>> Any other size mismatch can be easily worked around without having to wait
>> for the Flash chips to do anything.
>
> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.
>
> Also, *what* other size that can be mismatched ?
If your cluster size is smaller than the minimum block size you can read
> Brian,
>
>>>> Why would any size except the "erase page" size be at all relevant to
>>>> what
>>>> we're discussing?
>>>
>>> Good question. Why don't you start with telling us why it doesn't. ? :-)
>>
>> Any other size mismatch can be easily worked around without having to wait
>> for the Flash chips to do anything.
>
> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.
>
> Also, *what* other size that can be mismatched ?
or write then you'll need to work around that sometimes but I don't
think you need to erase and re-write anything extra compared to if the
read/write size matches the allocation block size.
> And by the way : try to imagine what needs to be done when someone writes a
> block matching some other smaller block size into a erase-block sized page
> which causes the need to have bits changed from a Zero to a One.
allocation block size means your erase size blocks get "fragmented" less.
R.Wieser
Jan 26, 2022, 8:29:00 PM1/26/22
to
Brian,
> If your cluster size is smaller than the minimum block size you can read
> or write then you'll need to work around that
That also needs explanation I'm afraid. What problem (do you think)
(c|w)ould occur and what would the work-around be ?
> but I don't think you need to erase and re-write anything extra compared
> to if the read/write size matches the allocation block size.
Really ? Lets assume that :
1) I write (a cluster) to the latter part of the first allocation block and
the first part of the second allocation block
2) both allocation blocks already contain different data than whats written
to them
3) both allocation blocks are in the same erasure page.
What would the flash chip do?
Now imagine that the cluster size exactly matches the allocation block (and
is aligned with it).
> But it does happen less if you make the allocation blocks bigger. Bigger
> allocation block size means your erase size blocks get "fragmented" less.
While its true its also a red herring. Its not something you or I have
control over. Also, why do you think the producer of those flash chips did
not think of that ? I mean, just make the allocation block the same size as
the erasure page size and presto! no more fragmentation ... :-p
Regards,
Rudy Wieser
> If your cluster size is smaller than the minimum block size you can read
> or write then you'll need to work around that
(c|w)ould occur and what would the work-around be ?
> but I don't think you need to erase and re-write anything extra compared
> to if the read/write size matches the allocation block size.
1) I write (a cluster) to the latter part of the first allocation block and
the first part of the second allocation block
2) both allocation blocks already contain different data than whats written
to them
3) both allocation blocks are in the same erasure page.
What would the flash chip do?
Now imagine that the cluster size exactly matches the allocation block (and
is aligned with it).
> But it does happen less if you make the allocation blocks bigger. Bigger
> allocation block size means your erase size blocks get "fragmented" less.
control over. Also, why do you think the producer of those flash chips did
not think of that ? I mean, just make the allocation block the same size as
the erasure page size and presto! no more fragmentation ... :-p
Regards,
Rudy Wieser
Frank Slootweg
Jan 27, 2022, 10:43:50 AM1/27/22
to
R.Wieser <add...@not.available> wrote:
[...]
> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.
Not that it matters one bit for your problem/discussion, but exFAT on
Windows (at least on 8.1) can have allocation unit sizes of upto 32768
kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
16, 32MB).
[...]
[...]
> Than you have a problem. I do not know of any computer, Windows or
> otherwise, which has a cluster size of a few megabytes. IOW, you will
> /always/ have a mismatch in regard to the erase-page size.
Windows (at least on 8.1) can have allocation unit sizes of upto 32768
kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
16, 32MB).
[...]
R.Wieser
Jan 27, 2022, 1:09:00 PM1/27/22
to
Frank,
> but exFAT on Windows (at least on 8.1) can have allocation unit sizes
> of upto 32768 kilobytes, i.e. 32 megabytes.
That doesn't even sound odd to me when looking at the ammount of memory a
current computer can have. IOW, they can probably handle it. Would not
want to try to use that FS on my old C64 though .... :-)
> "a few megabytes" starts at 2MB (then 4, 8, 16, 32MB).
I have to remember that. Many of my files are quite a bit smaller than
that, which would fill up such an FS a bit faster than you would expect
(lots of slack space).
Thanks for the info.
Regards,
Rudy Wieser
> but exFAT on Windows (at least on 8.1) can have allocation unit sizes
> of upto 32768 kilobytes, i.e. 32 megabytes.
current computer can have. IOW, they can probably handle it. Would not
want to try to use that FS on my old C64 though .... :-)
> "a few megabytes" starts at 2MB (then 4, 8, 16, 32MB).
that, which would fill up such an FS a bit faster than you would expect
(lots of slack space).
Thanks for the info.
Regards,
Rudy Wieser
Mayayana
Jan 27, 2022, 1:22:17 PM1/27/22
to
"Frank Slootweg" <th...@ddress.is.invalid> wrote
|
| Not that it matters one bit for your problem/discussion, but exFAT on
| Windows (at least on 8.1) can have allocation unit sizes of upto 32768
| kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
| 16, 32MB).
Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)
|
| Not that it matters one bit for your problem/discussion, but exFAT on
| Windows (at least on 8.1) can have allocation unit sizes of upto 32768
| kilobytes, i.e. 32 megabytes. "a few megabytes" starts at 2MB (then 4, 8,
| 16, 32MB).
Frank Slootweg
Jan 27, 2022, 1:45:46 PM1/27/22
to
MB, not GB.
Paul
Jan 27, 2022, 2:36:21 PM1/27/22
to
It doesn't mean that it is practical, to select the ExFAT high valued ones.
NTFS on Windows 10 has also been given some larger options.
64KB clusters should work just about anywhere. But the format
menu has some >64KB options only on Windows 10. And I won't be
selecting those, any time soon... unless the choice is backward
compatible all the way back to Win2K. It's no good having partitions
that only mount in Windows 10, and no where else. For one thing,
you want the option of using the CHKDSK from the other OSes,
on occasion.
You can use outlandish choices like that, if your USB stick holds
only MRIMG files from Macrium backups. That's the scenario where
it makes sense.
Paul
Mayayana
Jan 27, 2022, 3:54:15 PM1/27/22
to
"Frank Slootweg" <th...@ddress.is.invalid> wrote
| > Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)
|
| Gimme some of what you're smoking! :-)
|
| MB, not GB.
Woops. Sorry, you're right. So I could save 31,000
small text files. But I couldn't fit the 42K files I have
on my 3 GB backup drive on that 1 TB drive.
| > Great. So you can fit up to 31 100 KB files on a 1 TB disk. :)
|
| Gimme some of what you're smoking! :-)
|
| MB, not GB.
small text files. But I couldn't fit the 42K files I have
on my 3 GB backup drive on that 1 TB drive.
Brian Gregory
Jan 28, 2022, 5:09:34 AM1/28/22
to
On 27/01/2022 01:28, R.Wieser wrote:
> Brian,
>
>> If your cluster size is smaller than the minimum block size you can read
>> or write then you'll need to work around that
>
> That also needs explanation I'm afraid. What problem (do you think)
> (c|w)ould occur and what would the work-around be ?
>
>> but I don't think you need to erase and re-write anything extra compared
>> to if the read/write size matches the allocation block size.
>
> Really ? Lets assume that :
>
> 1) I write (a cluster) to the latter part of the first allocation block and
> the first part of the second allocation block
What are you meaning by allocation block?
> Brian,
>
>> If your cluster size is smaller than the minimum block size you can read
>> or write then you'll need to work around that
>
> That also needs explanation I'm afraid. What problem (do you think)
> (c|w)ould occur and what would the work-around be ?
>
>> but I don't think you need to erase and re-write anything extra compared
>> to if the read/write size matches the allocation block size.
>
> Really ? Lets assume that :
>
> 1) I write (a cluster) to the latter part of the first allocation block and
> the first part of the second allocation block
I was using the name "allocation block" to mean what also gets called
cluster - the filing system allocation block size.
I guess you mean the block size in the flash.
Firstly I'm assuming the sizes are binary multiples, so no cluster
crosses a boundary between flash blocks.
Secondly I'm assuming that the storage system is managing storage space
and doing wear levelling and will try and arrange things so that when
possible (hopefully most of the time in many cases) block writes are
directed to areas of flash that are in the erased state (you can have a
flash block that is divided up into clusters and leave unused clusters
in the erased state.
R.Wieser
Jan 28, 2022, 8:28:10 AM1/28/22
to
Brian,
> What are you meaning by allocation block?
:-) What did *you* mean with it ?
> I was using the name "allocation block" to mean what also gets called
> cluster - the filing system allocation block size.
Really ? Than what do you call a *flash* (not FS) read/write block (no, not
the erase-page one) ? 'cause that is what we are talking about, right ?
Also, your message <j4l3s0...@mid.individual.net> (17-01-22 12:56:47)
seems to contradict the above ...
And why /did/ you chose to use an ambigue name like "allocation block" when
there is, in regard to a harddisk, a perfectly good (and rather distinctive)
"cluster" definition* available ?
* a rather good definition, as I could as easily assume that an "allocation
block" on a harddisk would be referring to a sector.
> I guess you mean the block size in the flash.
Lol ? Which "block size in the flash" are you here referring to ? The
read/write one ? the erase-page one ? Maybe yet another ?
> Firstly I'm assuming the sizes are binary multiples, so no cluster crosses
> a boundary between flash blocks.
See above. It fully depends on what you are referring to with that "flash
block".
On the off chance that you're referring to the flashes read/write page than
the opposite is true - the read/write page matches or is smaller than the FS
cluster thats projected ontop of it.
> Secondly I'm assuming that the storage system is managing storage space
> and doing wear levelling
Its *hard* to respond to an example, isn't it ? It forces you to actually
/think/ about the truths you hold as evident. Better make some
assumptions to make that kind of effort go away. :-\
But as you seem to have no intention to actually answer me it makes no sense
to me to continue our conversation. So, goodbye.
Regards,
Rudy Wieser
> What are you meaning by allocation block?
> I was using the name "allocation block" to mean what also gets called
> cluster - the filing system allocation block size.
the erase-page one) ? 'cause that is what we are talking about, right ?
Also, your message <j4l3s0...@mid.individual.net> (17-01-22 12:56:47)
seems to contradict the above ...
And why /did/ you chose to use an ambigue name like "allocation block" when
there is, in regard to a harddisk, a perfectly good (and rather distinctive)
"cluster" definition* available ?
* a rather good definition, as I could as easily assume that an "allocation
block" on a harddisk would be referring to a sector.
> I guess you mean the block size in the flash.
read/write one ? the erase-page one ? Maybe yet another ?
> Firstly I'm assuming the sizes are binary multiples, so no cluster crosses
> a boundary between flash blocks.
block".
On the off chance that you're referring to the flashes read/write page than
the opposite is true - the read/write page matches or is smaller than the FS
cluster thats projected ontop of it.
> Secondly I'm assuming that the storage system is managing storage space
> and doing wear levelling
/think/ about the truths you hold as evident. Better make some
assumptions to make that kind of effort go away. :-\
But as you seem to have no intention to actually answer me it makes no sense
to me to continue our conversation. So, goodbye.
Regards,
Rudy Wieser
Brian Gregory
Jan 28, 2022, 8:32:12 PM1/28/22
to
On 28/01/2022 13:27, R.Wieser wrote:
> Brian,
>
>> What are you meaning by allocation block?
>
> :-) What did *you* mean with it ?
>
>> I was using the name "allocation block" to mean what also gets called
>> cluster - the filing system allocation block size.
>
> Really ? Than what do you call a *flash* (not FS) read/write block (no, not
> the erase-page one) ? 'cause that is what we are talking about, right ?
>
> Also, your message <j4l3s0...@mid.individual.net> (17-01-22 12:56:47)
> seems to contradict the above ...
>
> And why /did/ you chose to use an ambigue name like "allocation block" when
> there is, in regard to a harddisk, a perfectly good (and rather distinctive)
> "cluster" definition* available ?
>
> * a rather good definition, as I could as easily assume that an "allocation
> block" on a harddisk would be referring to a sector.
Windows calls what we used to call clusters "Allocation blocks".
> Brian,
>
>> What are you meaning by allocation block?
>
> :-) What did *you* mean with it ?
>
>> I was using the name "allocation block" to mean what also gets called
>> cluster - the filing system allocation block size.
>
> Really ? Than what do you call a *flash* (not FS) read/write block (no, not
> the erase-page one) ? 'cause that is what we are talking about, right ?
>
> Also, your message <j4l3s0...@mid.individual.net> (17-01-22 12:56:47)
> seems to contradict the above ...
>
> And why /did/ you chose to use an ambigue name like "allocation block" when
> there is, in regard to a harddisk, a perfectly good (and rather distinctive)
> "cluster" definition* available ?
>
> * a rather good definition, as I could as easily assume that an "allocation
> block" on a harddisk would be referring to a sector.
J:\>
J:\>
J:\>
J:\>
J:\>chkdsk
The type of the file system is NTFS.
Volume label is Local Disk 1.
WARNING! F parameter not specified.
Running CHKDSK in read-only mode.
CHKDSK is verifying files (stage 1 of 3)...
127232 file records processed.
File verification completed.
13407 large file records processed.
0 bad file records processed.
0 EA records processed.
25 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)...
139456 index entries processed.
Index verification completed.
0 unindexed files scanned.
0 unindexed files recovered.
CHKDSK is verifying security descriptors (stage 3 of 3)...
127232 file SDs/SIDs processed.
Security descriptor verification completed.
6113 data files processed.
CHKDSK is verifying Usn Journal...
37122192 USN bytes processed.
Usn Journal verification completed.
Windows has checked the file system and found no problems.
1023998975 KB total disk space.
999706604 KB in 63950 files.
36132 KB in 6114 indexes.
0 KB in bad sectors.
263007 KB in use by the system.
65536 KB occupied by the log file.
23993232 KB available on disk.
4096 bytes in each allocation unit.
255999743 total allocation units on disk.
5998308 allocation units available on disk.
J:\>
J:\>
J:\>
J:\>
See.
>
>> I guess you mean the block size in the flash.
>
> Lol ? Which "block size in the flash" are you here referring to ? The
> read/write one ? the erase-page one ? Maybe yet another ?
>
>> Firstly I'm assuming the sizes are binary multiples, so no cluster crosses
>> a boundary between flash blocks.
>
> See above. It fully depends on what you are referring to with that "flash
> block".
>
> On the off chance that you're referring to the flashes read/write page than
> the opposite is true - the read/write page matches or is smaller than the FS
> cluster thats projected ontop of it.
If you think that those blocks are smaller then no read modify write is
required.
>
>> Secondly I'm assuming that the storage system is managing storage space
>> and doing wear levelling
>
> Its *hard* to respond to an example, isn't it ? It forces you to actually
> /think/ about the truths you hold as evident. Better make some
> assumptions to make that kind of effort go away. :-\
>
> But as you seem to have no intention to actually answer me it makes no sense
> to me to continue our conversation. So, goodbye.
0 new messages