Is Defragmentation Still Necessary

[Toney Talbot]

HoweverIn 64 bit systems, we have an address space of virtually unlimited size. To prevent overuse of the RAM or swap space, the free function could just return the memory to the OS when an unused block is of size bigger than 4096 (Page size) and never use that address again. So while I know freeing RAM is important, I'm saying that defragmentation is not.

So, my question is if memory defragmentation is still relevant. If not, then perhaps we can have a different sort of malloc/free for 64 bit systems, which might be more efficient if it does not try to spend time defragmenting memory. At least for applications that we know are not gonna use the whole 256 TB of address space. What are your opinions?

It'll always be relevant, but not because of the lack of address space, but because of memory coherency. Thanks to the 32bit systems we also already have many algorithms for low fragmentation allocators and free-space coalescing allocators, and there is no reason why they would just be ignored on 64bit systems, they don't add much overhead at all and the amortized overhead may even drop on a 64 bit system).

I've looked around and found multiple answers around the web on this. I turned off auto defrag on my Windows 7 machine because some people have told me it creates more wear and tear on the disk (using a regular magnetic platter hd), and that its also not as necessary because Windows has better file system optimization now that creates less fragments.

I have not found any reasons to believe that there is less fragmentation in modern windows than there was back in win9? or xp, they do say that, but I spent a lot of times looking at the data tossed in clusters, I assume they did not. There is no evidence to show that happening. It does get all fragmented, and even tossed all over the disk.

Much (and many) of the OS files are locked up (when in that OS). Locked files will not be defrag while the system is running, making the on-line defragmenting that windows own defragmenter does, not really efficient. And off-line defrag, one done before the OS runs is much more complete.

This pic shows the locked files in Red, from my system, which has a number of things turned off, Give you a good idea of just how many things will not get defragged when the OS is still running. This cluster view is zoomed up, to show the data area not the unused areas.

Does it matter? Between the large caches, the larger caches on the disk hardware itself, the speed of disks nowdays even the lower RPM drives because of increased data density, and improved methods, plus the pre-loading, the percieved performance does not change that much. Sequential reading is still much faster.

The pre-fetcher system that MS made, parts of it are dependant on the defrag process itself, allowing for the aligning of that data to be read sequentially, without that it would not be as effective at all.

One thing people like to forget when discussing the need for defragging, is disk recovery, it is much easier to un-delete files, or recover file items when the data is sequential in the next clusters. When the file tables are all gone, and your recovery software is of the lesser type the data being sequential could improve recovery chances by a lot. (That doesn't replace needing a proper back-up)

It is false. Fragementation affects the performance of your hard drive and it will do it even more as time passes by. A hard drives is continuously reading, writing and deleting data. So it can happen that logical connected data does not appear always in the same place. If you are opening a file, where your hard drive has to search for the data, it is obvious that is going to take longer.

Defragmentation is responsible for arranging the logical connected blocks again in order. It is no coincidence that Microsoft enabled auto defragmentation for hard drives by default. If you got a solid state drived installed, this option should be disabled, because it will unnecessary stress it.

Dear Lifehacker,

I hear people talk about "defragging" their computers all the time as a way to make it faster, but I'm not really sure what that means. What does defragging do and is it something I need to do to my computer? How often?

Most hard drives have spinning platters, with data stored in different places around that platter. When your computer writes data to your drive, it does so in "blocks" that are ordered sequentially from one side of the drive's platter to the other. Fragmentation happens when those files get split between blocks that are far away from each other. The hard drive then takes longer to read that file because the read head has to "visit" multiple spots on the platter. Defragmentation puts those blocks back in sequential order, so your drive head doesn't have to run around the entire platter to read a single file. Image by XZise.

If you have a solid-state drive (SSD) in your computer, you do not need to defragment it. Solid-state drives, unlike regular hard drives, don't use a spinning platter to store data, and it doesn't take any extra time to read from different parts of the drive. So, defragmentation won't offer any performance increases (though SSDs do require their own maintenance).

Windows 7 and Windows 8 automatically defragment your hard drives for you on a schedule, so you shouldn't have to worry about it yourself. To make sure everything's running smoothly, open up the Start menu or Start screen and type "defrag." Open up Windows' Disk Defragmenter and make sure it's running on a schedule as intended. It should tell you when it was last run and whether your drives have any fragmentation.

Note: A lot of you are finding that Windows 7's "automatic" defrag leaves a lot to be desired. All the more reason you should check in with Disk Defragmenter every once in a while and make sure it's doing its job! Windows 8 seems to be much better about running it regularly.

Note that in Windows 8, you'll see your SSDs in the Disk Defragmenter, but it doesn't actually defrag them; it's just performing other SSD-related maintenance. So don't worry if it's checked off along with the other drives.

If you're on Windows XP, you'll need to defragment your drives yourself. Just open the Start menu, click Run, type Dfrg.msc and press Enter. You'll open up the Disk Defragmenter, from which you can defragment each of your drives individually. You should do this about once a week or so, but if you want, you can set it to run on a schedule using Windows' Task Scheduler.

We've talked a bit before about the best defragmenting tools, since Windows' built-in Disk Defragmenter isn't the only one. However, for the vast majority of people, Windows' built-in tools are just fine. Third-party tools are useful if you want to see which files are fragmented or defragment system files (like if you're trying to shrink a drive), but are otherwise unnecessary for most users. So kick back, let your scheduled defragger do it's thing, and forget about it!

"ext2, ext3, and ext4 file systems [...] allocates files in a more intelligent way. Instead of placing multiple files near each other on the hard disk, Linux file systems scatter different files all over the disk, leaving a large amount of free space between them."

-How To Geek

"Modern Linux filesystem(s) keep fragmentation at a minimum by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Some filesystems, like ext3, effectively allocate the free block that is nearest to other blocks in a file. Therefore it is not necessary to worry about fragmentation in a Linux system."

-TLDP

Fragmentation is always a concern no matter what the filesystem. Ext3/Ext2, though, have minimized the problem to the point where most admins just operate as if it's not a concern. This is minimized by use of a competent I/O scheduler and by the filesystem leaving trailing space after each file to accommodate growing. On top of that, additional fragments are stored closer together so even when there is fragmentation, the effects are severely muted. Ext4 switching to extents was also intended to help mitigate fragmentation even farther.

Fragmentation still exists, ext2 had an offline defragmentation tool, ext4 is getting an online one and btrfs (just because it's related to the question) already has an online defrag. If you're on ext3 your only real choice is either to convert to ext4 (via tune2fs) or try to see how much luck you have with user space tools like shake.

For the question about FAT32, fragmentation doesn't happen when you just operate on a file, it happens when you write to the file. To answer your question directly, fragmentation is an aspect of the filesystem design, so your FAT32 filesystem has the potential for fragmentation whenever you write something to it.

Yes, it can get fragmented. It's not a matter if Windows or Linux or anything else is accessing it, it's how each filesystem deals with file size, how it stores all those blocks, whether it leaves empty spaces between each file...

Truecrypt containers are a file, for all pratical matters. When you create a container using TrueCrypt, you're creating a file of the size that you choose. For your operating system, it's just like any other file. If you make copies of it, etc, perhaps the copies might end up being saved fragmented.

And if you're worried about truecrypt security about fragmentation, do some research about using truecrypt with SSDs: since the SSD try to lever the wear of all memory inside it, you might have the same effect of copying the container around.

ext2 and FAT file systems are both methods to record which blocks on disk belong to which file. When not all block of a file are contiguous, the file is called fragmented. But as should be obvious, fragmentation is caused by the block allocation strategy when writing a file, not by how you then record those blocks.

What does make a difference is how far ahead the file system drivers can look. If the driver knows there are 10 more blocks to write, it can pick a contiguous free area and avoid future fragmentation. This depends on the driver architecture, which is OS-specific. But the exact same free space search can be done on FAT and ext2.

3a8082e126