The "HPFS gives you more space than FAT" FAQ
David Charlap
a typical hard drive than the FAT file system. You may have heard
that HPFS wastes space unless you're on a huge partition. Well, it's
not true. Here's why:
On HPFS, the partition's control structures consume:
16 sectors for boot code
2 sectors for anchor blocks (superblock and spare block)
2 sectors for first code page
101 sectors for hot-fixes
80 sectors for emergency directory b-tree expansion (20 DirBlocks)
4 sectors per every 8MB of disk space (0.025% of total space)
Plus pre-allocated directory storage. This storage is 20
sectors (10KB) for every 1M of disk space, with a minimum of
200 sectors (100KB) and a maximum of 16000 sectors (8MB).
Directory entries in HPFS take up 36 bytes (for files whose
names are 5 characters or less) plus extra space (in multiples
of 4 bytes) for longer names.
So, for 8.3 filenames (keeping things comparable to FAT), each
directory entry will consume 44 bytes (36 plus 8 for the extra
7 chars). These directory entries will come out of the
pre-allocated directory storage space, where possible.
On FAT, the control structures consume:
1 sector for boot code
up to 1024 sectors for 2 copies of the FAT tables
up to 32 sectors for the root directory
slack to round everything to an even cylinder boundary
FAT directory entries consume 32 bytes each. Root directory
entries consume no extra space, since their storage is
pre-allocated.
In general, one can ignore the HPFS pre-allocated directory storage
and the FAT root directory space. This is because:
- It's not much space - FAT's 32 sectors is only 16KB. On HPFS, it's
more space, but it's still only 10K for every MB of disk - 1% of the
total space. And for large drives (over 800MB), it stops growing.
- As files are allocated, this space is used. For instance, 8MB of
directory 8.3 HPFS directory entries is approximately 186000 files!
This means you can create 186000 8.3 files without allocating any
additional sectors for directory storage.
If you want to count this storage in the free-space estimates, you
should then count the amount of space 190000 FAT files would
consume. (5.7MB) Even counting FAT's 16K of root directory in the
mix, it's still a lot of extra storage.
For the general calculations, I'll ignore directory storage
(pre-allocated, root, or otherwise) in my calculations. At the end of
the article is a re-calculation of my example cases, taking directory
storage into account.
So, if you have a partition size of p, your total amount of free space
on it will be approximately (assuming 512 byte sectors, of course):
FAT: p - (1024 sectors = 512K)
HPFS: (p * 0.99975) - (201 sectors = 100K)
This is what is often advertised. And it becomes clear that HPFS
doesn't give you as much free space, since its control structures grow
in proportion with the partition size, whereas FAT's don't.
But, when you start adding files, this changes:
FAT's cluster size is 2K for partitions under 128MB,
4K for partitions over 128MB but under 256MB,
8K for partitions over 256MB but under 512MB,
and 16K for partitions over 512MB but under 1GB,
etc...
HPFS's cluster size is always 512 bytes.
On average, 1/2 of a cluster (slack space) is wasted for any file.
Additionally, HPFS consumes one sector (512 bytes) per file as an
Fnode. So, if you have f files on a partition, the amount of wasted
space (from slack) is:
HPFS: (256 bytes + 1 sector = 768 bytes) * f
FAT: 1K * f (for partitions under 128MB)
2K * f (for partutions over 128MB but under 256MB)
etc...
Taking both together, you can compute how much space you will gain or
lose for a given partition to switch from HPFS to FAT (or vice versa):
In all cases,
unusable space = control structures + slack
free space = total space - unusable space - data.
Since data is what you're storing, we can discount it. That will be
consumed equally, regardless of what the file system is. We want to
minimize unusable space, since that space is waste. We can compre
the efficiency of the file system easily, by comparing the wasted
space.
For example, I have two partitions on my hard drive. One is 65MB, and
the other is 466MB. The 65MB partition has about 1700 files and
directories on it. The 466MB partition has about 7200 files and
directories on it. (These are actual figures from the computer on
my desk, next to me right now.)
So, the amount of unusable space (in KBytes) for the 65MB partition is
about:
FAT: 512 + (1700 * 1) = 2212 (about 2.2MB)
HPFS: (.00025 * 66560) + 100 + (1700 * .75) =
18 + 100 + 1275 = 1393 (about 1.4MB)
In this case, HPFS would allow an extra 819K data to be stored
on the drive. (0.00025 * 66560 = 16.64, but we have to round
it up to a 2K increment.) This is about 1.2% of the total
partition size.
On the 466MB drive, the wasted space is about:
FAT: 512 + (7200 * 4) = 29312 (about 28.6MB)
HPFS: (.00025 * 477184) + 100 + (7200 * .75) =
120 + 100 + 5400 = 5620 (about 5.5MB)
In this case, HPFS would allow 23.1MB more data to be stored on
the drive. This is about 5% of the total partition size.
As you can see, the amount of free space gained over FAT increases
rapidly as the partition size increases. As an extreme, one could
calculate the wasted space on a 2GB partition with 25000 files on it.
(Not uncommon among file servers):
FAT: 512 + (25000 * 16) = 400512 (about 391.1MB)
HPFS: (.00025 * 2097152) + 100 + (25000 * .75) =
526 + 100 + 18750 = 19106 (about 18.7MB)
Here, HPFS allows an extra 372MB to be stored. This is 18% of
the total partition size!
-----------------
But, you say, "you've been ignoring the pre-allocated directory space
all this time". OK. While coming up with the equation for the
general case is a bit ugly, I'll recompute it for the three examples I
used above. In all cases, assume that the filenames are exactly 8.3
characters (12 chars - don't forget to include the .) long. In both
cases, I'll assume that directories are packed as tightly into
pre-allocated space as possible. I'm ignoring slack in directories,
since directories are files, and the slack's average will be computed
in the average slack for the file system.
Of course, sub-optimal packing of directory names on the FAT drive
will end up wasting more directory space. And use of longer HPFS
names will use up more of the pre-allocated dir space (but judging by
the size of it, I don't think additional directory storage would be
allocated anyway.)
So, for a 65MB partition with 1700 files:
FAT: FAT tables = 512K
Root DIR = 16K (512 filenames)
Extra 1188 dir entries = 37K (32 bytes per entry)
Slack space = 1700K (2K clusters)
------------------------------
Total "waste" = 2265K (about 2.2MB)
HPFS: Fixed structures = 100K
Variable structures = 18K (9 8MB bands - 36 sectors)
Pre-allocated dirs = 650K (10K per 1MB. 15127 filenames)
F-nodes = 850K (512 bytes per file)
Slack space = 425K (256 bytes per file)
---------------------------
Total "waste" = 2043K (about 2MB)
In this case, HPFS allows an extra 222KB of data to be stored. This
is about 0.3% of the partition's total size.
-----
For the 466MB drive with 7200 files on it:
FAT: FAT tables = 512K
Root DIR = 16K (512 filenames)
Extra 6688 dir entries = 209K
Slack space = 28800K (8K clusters)
-------------------------------
Total "waste" = 29537K (about 28.8MB)
HPFS: Fixed structures = 100K
Variable structures = 118K (59 8MB bands - 236 sectors)
Pre-allocated dirs = 4660K (108450 filenames)
F-nodes = 3600K
Slack space = 1800K
----------------------------
Total "waste" = 10278K (about 10MB)
In this case, HPFS allows an extra 19259K (about 18.8MB) of data to be
stored. This is about 4% of the partition's total size.
-----
For the extreme case of a 2GB partition with 25000 files on it:
FAT: FAT tables = 512K
Root DIR = 16K (512 filenames)
Extra 24488 dir entries = 765K
Slack space = 400000K (32K clusters)
---------------------------------
Total "waste" = 401293 (about 391.9MB)
HPFS: Fixed structures = 100K
Variable structures = 512K (256 8MB bands - 1024 sectors)
Pre-allocated dirs = 8000K (max size - 186181 filenames)
F-nodes = 12500K
Slack space = 6250K
----------------------------
Total "waste" = 27362K (about 26.7MB)
In this case, HPFS allows an extra 373931K (about 365MB) of data to
be stored. This is about 17.8% of the partition's total size.
---------------------------------------
In other words, even when you complicate the mathematics with the
directory structures, HPFS still gives you more free space than FAT
volumes. On the small (65MB) partition, it's not much (only 222K) but
as the partition gets larger and the number of files increases, the
difference in storage becomes incredible.
---------------------+--------------------------------------------------------
David Charlap | The contents of this message are not the opinions of
da...@visix.com | Visix Software, or of anyone besides myself.
Visix Software, Inc. +------------------------------------------------+-------
Member of Team-OS/2 | Quantum mechanics: The dream stuff is made of. |
---------------------+------------------------------------------------+
Roderick W. Smith
es:
>
> This article explains how and why HPFS gives more usable free space on
> a typical hard drive than the FAT file system. You may have heard
> that HPFS wastes space unless you're on a huge partition. Well, it's
> not true. Here's why:
[much good information snipped]
Nice job. I'm sure it'll clear up a lot of questions in peoples' minds. I'd
add a few points, though:
1) Extended Attributes will impact a FAT system more than an HPFS system, and
your calculations don't take this into account. Under FAT, each file that
has EAs chews up an additional allocation block (or multiple allocation
blocks, if the EAs for the file are large). This is why there was a thread
a few weeks ago about huge EA DATA. SF files on a FAT system, since the
system in question had a large partition, and thus the EA DATA. SF file
was chewing up quite a few megs of storage all by itself, though the
actual data in that file was probably under a single megabyte. HPFS
includes space for a few bytes of EAs (I don't recall precisely how much)
in its directory structures, so for many files containing minimal EAs,
there's ZERO additional impact. For files with more EAs, they get
allocated using the HPFS 512-byte (0.5K) allocation block size, so in this
(worst-case) scenario, the allocation block size "whammy" is a DOUBLE win
for HPFS.
2) Hard drives typically have a large number of very small files. I don't
have any exact figures, but I do recall hearing that the average file size
is about 32K, and I just checked my main partition, and for that, it comes
out to 34K. Now, consider that this average includes stuff like my 20MB
swap file, several multi-hundred K executables, etc. Anybody who's taken
a basic statistics course should realize that the AVERAGE (mean) value is
a bit deceptive in some ways for this sort of situation. When you've got
huge values (like 20MB) inflating the estimate, you've got to have larger
numbers of really small values (e.g., less than the average) to balance
that. The upshot of this is that, especially for larger FAT drives, the
slack space estimates computed by formulae involving number of files will
tend to UNDERESTIMATE the total amount of slack space on the drive. I've
seen a number of posts indicating that 1GB drives tend to lose about 300MB
to slack under FAT and OS/2.
3) Programs exist for DOS to measure slack space in FAT precisely. I don't
know of any similar programs for OS/2, or ones that will take the special
nature of EA slack into consideration. It'd be nice to have an OS/2
program that would scan a hard drive and compute both wasted slack and
what WOULD be wasted if the same files were on the other type of
filesystem. This probably wouldn't be too difficult to do, either....
Does anybody know about system calls to get allocation block sizes, EA
sizes, etc.? ;-)
--Rod Smith
RSM...@PSYCH.COLORADO.EDU
pdwa...@hk.super.net
>3) Programs exist for DOS to measure slack space in FAT precisely. I don't
> know of any similar programs for OS/2, or ones that will take the special
> nature of EA slack into consideration. It'd be nice to have an OS/2
> program that would scan a hard drive and compute both wasted slack and
> what WOULD be wasted if the same files were on the other type of
> filesystem. This probably wouldn't be too difficult to do, either....
> Does anybody know about system calls to get allocation block sizes, EA
> sizes, etc.? ;-)
I happen to have a dos program right here. I run it once and a while to
give myself that warm feeling I get from knowing the benifits of using
hpfs on my larger disks.
The program is called "WASTED.COM" v1.0 by lui...@m-net.arbornet.org
dated 1994.
I think that I found the program on Simtel in my DOS days. Useful to
keep around for a comparison.
Here is an example of the output generated by the program:
---
C:\>wasted f:
Wasted, Version 1.0 by lui...@m-net.arbornet.org
F:
Drive Cluster Size FAT Type Total Used Total Wasted Total Size
F 16384 FAT-16 287.4 Meg 46.6 Meg 516.2 Meg
Cluster Largest Wasted in Wasted in Total Wasted Per Cent Existing
Size Partition Files Directories Space Wasted Partitions
2048 128.0 Meg 4.6 Meg 0.5 Meg 5.1 Meg 1.8
4096 255.9 Meg 9.1 Meg 1.0 Meg 10.1 Meg 3.5
8192 511.9 Meg 19.5 Meg 2.1 Meg 21.6 Meg 7.5
16384 1023.8 Meg 42.3 Meg 4.3 Meg 46.6 Meg 16.2 <= F
32768 2047.5 Meg 90.0 Meg 8.8 Meg 98.8 Meg 34.4
---
Drive F is actually an HPFS drive, so the larger hpfs filenames are
being missed.
- Paul
///////////////////////////////////////////////////////////////////////
// Paul D. Walker Consultant //
// pdwa...@HK.super.NET HK Computer Solutions //
///////////////////////////////////////////////////////////////////////
Paul S. Gazo
>
>In <4auv1q$a...@lace.colorado.edu>, rsm...@psych.colorado.edu (Roderick W. Smith) writes:
>>3) Programs exist for DOS to measure slack space in FAT precisely. I don't
>> know of any similar programs for OS/2, or ones that will take the special
>> nature of EA slack into consideration. It'd be nice to have an OS/2
>> program that would scan a hard drive and compute both wasted slack and
>> what WOULD be wasted if the same files were on the other type of
>> filesystem. This probably wouldn't be too difficult to do, either....
>> Does anybody know about system calls to get allocation block sizes, EA
>> sizes, etc.? ;-)
>
>I happen to have a dos program right here. I run it once and a while to
>give myself that warm feeling I get from knowing the benifits of using
>hpfs on my larger disks.
>
>The program is called "WASTED.COM" v1.0 by lui...@m-net.arbornet.org
>dated 1994.
>
>I think that I found the program on Simtel in my DOS days. Useful to
>keep around for a comparison.
Well, it's not exactly what you ask for, but I have written Waste/2. I
haven't taken the effort to throw in EA checks, but the program itself is
OS/2 native, and scans FAT or HPFS (or, actually, any FS OS/2 supports). It
just recursively goes through sub-dirs, tallies, and reports the results. It
determines the cluster size for any given FS as well, and reports how much
space should be saved by using HPFS over the other FS.
It should be noted that EAs don't really alter the picture significantly,
when it comes to space loss. First off, the actual EA data size is no
smaller or larger between two file-systems. Just as the actual file size
doesn't change from FS to FS, neither does the EA size. What does change for
files, is the space the FS allocates to create a file. On FAT, one file is
created, in total, to hold EAs. On HPFS, no files are created. The EAs are
actually stored in the same cluster as the file data, directly pre-pended (I
may be wrong on this... the structure may be appended). Either way, the
margin for error is one times the cluster size on the non-HPFS partition.
If anyone wants Waste/2 for any reason, just reply, and I'll upload it to
Hobbes. It's a text-mode app, written in BCOS/2 1.x (I could re-compile with
2.0 to get Warp binary compression, but would lose 2.x support... opinions on
this?), and runs in at a few k, plus docs. It's freeware.
+========================================================================+
| Life is like an onion; layer upon layer upon layer, it leaves a bad |
| taste in your mouth, and it makes you want to cry. -Pink |
+=============== http://server.uwindsor.ca:8000/~reaume ==== Team OS/2 ==+
Roderick W. Smith
>
> It should be noted that EAs don't really alter the picture significantly,
> when it comes to space loss. First off, the actual EA data size is no
> smaller or larger between two file-systems. Just as the actual file size
> doesn't change from FS to FS, neither does the EA size. What does change for
> files, is the space the FS allocates to create a file. On FAT, one file is
> created, in total, to hold EAs. On HPFS, no files are created. The EAs are
> actually stored in the same cluster as the file data, directly pre-pended (I
> may be wrong on this... the structure may be appended). Either way, the
> margin for error is one times the cluster size on the non-HPFS partition.
This isn't quite correct. It is true that EAs are stored in a single file
under FAT, but the file is really there just to keep DOS from mucking with
the EAs. Each file that has EAs will consume 1x the allocation block size
for the partition, assuming there are relatively few EAs; or some multiple of
the allocation block size if there are more EAs. Essentially, though the EAs
are stored in a single file, the EAs should be thought of as being stored in
separate files for slack space purposes.
As to HPFS, EAs are stored in two ways. First, the file header information
(which isn't technically part of the file storage, but is more akin to a
directory entry for FAT) has space for a certain amount of EAs (roughly 200
bytes, if I recall correctly). Since this file header is required, files
with a few bytes of EAs won't consume any extra disk space because of them.
When the EAs exceed this limit, they start getting storage space that's
allocated in much the same way as the files to which they are attached, but
the EAs aren't attached in the sense of taking up the same allocation blocks
or anything.
In sum, EAs CAN make a big difference, but a utility would be required to
measure this precisely, and I don't know of such a utility. I do know that
there were some threads just a few weeks ago about relatively new OS/2 users
who had FAT partitions and multi-megabyte EA DATA. SF files on their ~1GB FAT
hard drives. Because a certain amount of EA storage comes "free" with HPFS,
I can't imagine that these systems would use more than a couple hundred K in
extra EA storage under HPFS, but as I say, a utility would be required to
measure this accurately.
> If anyone wants Waste/2 for any reason, just reply, and I'll upload it to
> Hobbes. It's a text-mode app, written in BCOS/2 1.x (I could re-compile with
> 2.0 to get Warp binary compression, but would lose 2.x support... opinions on
> this?), and runs in at a few k, plus docs. It's freeware.
Please do. This sounds like a useful utility, even if it doesn't measure
EAs. It might be nice if the DOS version gave an HPFS estimate, too; then
DOS users considering OS/2 could see how it'd benefit their disk space. ;-)
--Rod Smith
RSM...@PSYCH.COLORADO.EDU
David Charlap
>
>when it comes to space loss. First off, the actual EA data size is
>no smaller or larger between two file-systems. Just as the actual
>file size doesn't change from FS to FS, neither does the EA size.
>What does change for files, is the space the FS allocates to create a
>file. On FAT, one file is created, in total, to hold EAs. On HPFS,
>no files are created. The EAs are actually stored in the same
>cluster as the file data, directly pre-pended (I may be wrong on
>this... the structure may be appended). Either way, the margin for
>error is one times the cluster size on the non-HPFS partition.
Actually, it's stranger than this.
On FAT, yes, the data is all stored in one file per partition. But
each file's EA data is padded to fill a cluster. This way it can be
accessed more quickly. So a file with 1 byte of EA data (not
possible, of course, since the EA's name must be stored as well) will
consume one cluster of disk space.
on HPFS, small amounts of EA data (I forget how much) is stored
directly in the file's F-node. Which means zero additional storage is
used. If there is more EA data than will fit in the F-node,
additional sectors are used as necessary. I don't know if these
sectors are appended to the file, prepended or if they're stored a
parallel structure that's independant from the file data.
I haven't included this information in the FAQ yet because I need a
good way to categorize it. On FAT, any file with EAs will burn
clusters. On HPFS, files with only small numbers of EAs will consume
zero extra space for them. I'll have to get some figures on the
amount and size of EAs for typical file systems to calculate all
this. Perhaps the next revision.
> If anyone wants Waste/2 for any reason, just reply, and I'll
>upload it to Hobbes. It's a text-mode app, written in BCOS/2 1.x (I
>could re-compile with 2.0 to get Warp binary compression, but would
>lose 2.x support... opinions on this?), and runs in at a few k, plus
>docs. It's freeware.
If you make a descriptive README file, go ahead and upload it. It
could be useful. Don't use the binary compression, though. There are
still plenty of OS/2 2.x users out there. (I, for one, haven't
upgraded my home PC yet, because I want to buy a CD-ROM drive first.)
David Gaskill
In message <4b4ght$9...@tst.hk.super.net> - pdwa...@hk.super.net writes:
>If anyone wants Waste/2 for any reason, just reply, and I'll upload
>it to Hobbes. It's a text-mode app, written in BCOS/2 1.x (I could re-compile with
>2.0 to get Warp binary compression, but would lose 2.x support... opinions on
>this?), and runs in at a few k, plus docs. It's freeware.
Yes please, (recompile would be nice if you have nothing better to do
over Christmas!)
David
Ian D. Kent
In article <4bcd4p$1...@shelby.visix.com>, da...@visix.com says...
--snip--
>
>on HPFS, small amounts of EA data (I forget how much) is stored
>directly in the file's F-node. Which means zero additional storage is
>used. If there is more EA data than will fit in the F-node,
>additional sectors are used as necessary. I don't know if these
>sectors are appended to the file, prepended or if they're stored a
>parallel structure that's independant from the file data.
As you said, the EA data is stored in the files F-node, which in turn is
allocated in the directory band by default. If no free sector is available
in the directory band (area reserved during partition formatting) to create
the F-node, then it will be created in free space. If the EA data grows too
large to be contained in the F-node (sorry, I can't remember how much room
is available in the F-node for EA data either) then additional sectors will
be allocated as required from free space. It is not implied in the HPFS
documentation I have that any attempt is made to locate these multi-sector
EA files near to their `parent' data file.
I'll see if I can find out how much space is available for EA data in the
F-node.
If you can point me to any additional sources of information on HPFS I would
be grateful.
Ian
//////////////////////////////////////////////////////////////////////////
// Ian D. Kent [Team OS/2] //
// Home: Ian....@muc.de //
// CI$ : 10004...@compuserve.com //
// Work: Ian....@dlr.de //
//////////////////////////////////////////////////////////////////////////