Optimizing the file system for a large number of small files
Dave August
I'm building an application that will have at least 100,000 (maybe
1,000,000) small files, probably under 1K in size.
These files will be mostly static data but will have a small amount of data
that changes. The changes will only be happening to about 1000 files at any
one time. This will be a 24/7 system.
I've have a few questions.
1: What file system would you recommend NTFS or FAT and why.
2: How well will ether file system handle 1 million file names in one
folder, as relates to finding, opening and closing times.
3: If putting all the files in one folder is not a good idea, what's a good
maximum number for any given folder and how deep is a good maximum number of
folders in a tree.
4: What are the ramifications of reducing the cluster/allocation size to 1K
(or less) to get maximum use of the disk space.
5: What else can be done to 'prepare' the file system for this kind use
besides preallocating all 1,000,000 million files on a freshly formatted
drive (I have no problem with doing that).
6: Are there any serious ramifications if this is on a RAID array and/or a
mirrored volume set.
7: Assuming opens are done with exclusive access, read/modify/write, (or
just seek and write) and close on every update, what are the issues with
backing and restoring up this many files.
8: Should I forget about using a traditional file system and just do direct
accsess to the disk, and if so is using a handle to the physical device (via
CreateFile(\\.\PHYSICALDRIVEx....) the best way, and if so what about
backing up.
FWIW as a bit of background, about 5 years ago I worked on a 24/7 system
that was NTFS based. We were taking in roughly 100,000 files an hour,
writing them in 200 separate folders (not all files in every folder but
based on some criteria writing each file in an appropriate folder) , keeping
around 10 old ones and purging the oldest after that (rotating LIFO). After
about 24 hours the B-trees and free space links in NTFS got so unbalanced
that the file system ground to a halt. This system won't have that
problem, these files will all stay and only every so often will a new file
get created. (imagine a manufacturing system collecting data on every new
device made) but I wonder what problems it will have.
Dave August
Pat [MSFT]
This will be very, very slow as the file count increases. NTFS uses a
modified b-tree so, 1 million files will take ~20 lookups to find (2 million
will need 21), so the overhead is very flat.
2) NTFS will handle it OK. Make sure you disable 8.3 name creation (which
gets very expensive with large file # counts due to the number of similar
files and re-tries and may have actually been the problem you were seeing
previously). Also, if your application does not require it, you can disable
the update last access/modified times which lowers overhead further.
3) 100k files / folder. You can do more (unlimited), but that seems to be
about right. Folders are used as indices and can speed the lookup up quite
a bit.
4) Fragmentation for the files that are >4k.
5) The MFT will be storing the files <1k (which would make lookup very, very
fast). I would recommend modifying the MFT reserve size from 1/8 storage
capacity to something like 1/4 or larger when the volume is created (will
minimize MFT fragmentation). Also, if you are running any AV software, with
this file count it will be a non-trivial in its CPU/Memory resources. So,
you may need an extra 256-->512MB just for the AV software.
6) Possibly. I assume you mean HW RAID here. Depending on how robust the
RAID driver is, I have seen instances where the DPC's were queued up very
badly under stress. Otherwise things should be fine. I would strongly
recommend a large controller cache and battery backup (on the controller) if
you go down this path.
7) With 1mil files, the primary overhead of a backup is actually disk seek
times. If your disk subsystem has an avg seek of 3ms (which would be very
good), you would end up with .003 * 1,000,000 = 3000 sec. (50 min) used just
in the seeks. Personally, I would recommend the volume shadow copy
mechanism instead.
8) I wouldn't go down that route. It would open a can of worms w.r.t.
maintenance (as you point out), and definitely increase your long term costs
significantly.
Disclaimer -
Obviously, I work for MS and have a vested interest here.
A couple of other thoughts. I would recommend going with Win2k3 (which will
also get you the volume shadow copy). We did a lot of work on the Memory
Manager (which also controls file caching) and improved some of the low
level NTFS task efficiencies (not new features so much as making the
features present more efficient). Also, depending on the file count to be
opened (concurrently) you may want to look at the use of a x64/AMD64
solution. Even running in 32bit mode (which you would likely need to do
unless all of the NIC, RAID, etc. OEMs provided native 64bit drivers) the
memory access times (cache lookup for this conversation) alone made it worth
while. You should also have a minimum of 2GB RAM in the box, and I would
(personally) recommend more (4+GB). These types of apps are all about
caching, generally, and any RAM not used directly for system operations is
used for file caching.
Pat
"Dave August" <aug...@NOSPAMacmesi.com> wrote in message
news:ubbduUSY...@tk2msftngp13.phx.gbl...
Dave August
Thanks for the detailed response. I do appreciate it.
These files would be the only things on this drive.
Could I up the MFT to 1/2 or 3/4 or even the whole drive?
The box is spec'd to be XP Pro with 2GB physical. I'll see about raising
that to 4GB and going to 2k3, as you point out if for no other eason than
'volume shadow'. Doubt I can get them to go 64bit, and I suspect that less
than 1000 files will be open at any one time, in all probability it'll be
100 or so.
I suspicion all names in this system will be 8.3 names and totally unique
(can you say 0000001.DAT to 1000000.DAT )
It'll be on a private net no connection to the outside world so no AV
junkware will be running.
Thanks again
Dave August
"Pat [MSFT]" <patf...@online.microsoft.com> wrote in message
news:eAhkq1eY...@TK2MSFTNGP10.phx.gbl...
Pat [MSFT]
the 'reserved' space and not the actual allocation. MFT will only take what
it needs, but will create a reserved buffer space around itself to limit
fragmentation. If the drive gets low on space, the MFT reserve space will
be made available.
The # files open is fine.
Pat
"Dave August" <aug...@NOSPAMacmesi.com> wrote in message
news:%23o4rIAh...@TK2MSFTNGP09.phx.gbl...
Andy Lyakhovetskiy
We have installed (upgraded) W2K3 server with 2 large
volumes 1TB each. First volume has about 600,000 files in
66,000 directories - 450GB, Second about 250,000 files in
43,000 directories - 300GB. The mounting time for the
first volume is around 30 sec, for the second about 20 sec.
The lookups are very slow. We tried to defrag volumes and
tweaked some parameters but the improvements were not
significant (1-5%). The server hardware - IBM x345 with
1.5GB RAM, Dual Xeon 2.7. The CPU utilization is not more
then 15-20% and RAM utilization 50%. On average about 100
users are connected to the server.
The backup software is Veritas Netbackup 5.0 and backup
performance is 2 times slower then for other identical
servers with smaller volumes.
Any ideas?
Thanks
Andy
>.
>
Pat [MSFT]
not the files). A slow mounting usually means that there are other issues.
Also, the CPU and memory usage seems high. With that many files and such a
small user base (relative to the file count), I would expect to see the CPU
max out at 20% and average 5-->10.
Have you checked TaskMan to see which process is using the most memory
(Virtual Memory column) and/or CPU? Is there anything in the
System/Application event logs?
Pat
"Andy Lyakhovetskiy" <anon...@discussions.microsoft.com> wrote in message
news:45d701c47347$76672540$a601...@phx.gbl...