How Do You Defragment a Linux System?
Peter T. Breuer
> in alt.os.linux, Peter T. Breuer wrote:
>> Why should it? It just tells you the system doesn't care about file
>> fragmentation! Reads from the disk are put into an optimal sequence
>> before getting anywhere near the disk heads, so who cares if the
>> file on disk is mished up everywhere? The disk heads will still seek
>> only once across the disk in reading it! Yeah, sure they read the file
>> blocks in order 7561248390 off the disk in that one pass, but they're
>> presented to you in 01234565789 order!
> While reading all the chunks out of order if far better than jumping
> around for them, it isn't quite as good as reading them all without
> any other data between them (short disk seeks take less time than
> long disk seeks). Therefore defragmentation will still be of some
> benefit - just not as much as is the case with MS filesystems.
It probably won't be a benefit, because a single elevator seek across
the disk will cover calls to many different files from many different
processes in a multiuser, mutitasking operation. Well, that's the
theory, anyway. You can practically guarantee a write from syslog
to be mixed in there, at least.
The logical problem (that you didn't notice) in my para is that you
still have to wait to read the blocks, no matter what order they are
read in. So if block 1 is out of order, you have to wait to the
end of the elevator pass to get it. Don't you? But, no you don't.
The system does readahead and caches. So block 1 was collected last time
one went anywhere near it.
>> So stop worrying. Accept that computer science 25 years ago decided
>> that fragmentation was (a) irrelevant and even good, and (b) doesn't
>> happen if you follow a sensible allocation strategy.
>>
>> I've explained (a) partially to you. There's another half of the answer
>> to (a) that I haven't given. I haven't explained (b) to you at all.
>> I'm not going to. You can bother to look it up.
> I have yet to hear of this "sensible allocation strategy" that makes
> it so that fragmentation "doesn't happen." Could you give me a hint
> as to where you wish me to search to find this wonder?
Well, the first bit of it is "pre-allocation" - reserving a bit of
space for file growth when you make the file. The real trick however
is in not placing new files in the first hole on disk in which you
could put them, but instead spreading them out.
> Searching on "linux defragmentation" seems to confirm my belief that
> the ext2 filesystem is affected by fragmentation, but not as badly as
It's not affected.
> MS systems, and seems to refute your statement that fragmentation has
> no effect. Example: http://cbbrowne.com/info/defrag.html says"
> "There is a program called defrag that defragments ext2 filesystems."
This doesn't refute anything. That program has been unused for years,
and only existed "for fun".
> ...
> "According to the documentation for defrag, random access is only
> likely to be assisted by a few percent; possibly up to 25 for
Absolutely. Forget it.
> sequential access if files are highly fragmented. This isn't the
Single file access is improbable.
> ``black mark'' one might expect: it merely indicates that ext2
> is not affected very adversely by fragmentation. ext2fs (the
> most commonly used file system under Linux) is somewhat resilient
> against fragmentation, and more importantly is not severely
> affected by it when it does happen. This is quite different
But of course.
> from the ``MS-DOS'' experience, where fragmentation has
> significant deleterious effects on system performance."
But nobody can explain why! :-)
> From what I read in my search, fragmentation has a slight
> (but not zero) negative effect on ext2 filesystems, Linux
Nothing you should worry about.
> is quite good at avoiding fragmentation, Linux gets better
> at avoiding fragmentation when it has more free space on a
> disk, Linux takes quite a bit less of a hit (but not zero)
Of course - this is a generic statement. It doesn't apply to linux
only!
> in performance when there is fragmentation, there is a utility
> called defrag which defragments a linux disk, Defrag is risky
But it doesn't. The utility does not exist. It only existed once
as a fun piece of code, and it hasn't been maintained since. I
probably still have it somewhere on my system ...
oboe:/usr/oboe/ptb% locate defrag
/usr/local/sbin/ddefrag
/usr/local/sbin/defrag.sh
...
but there is no way I would dare to run it nowadays! My code is from
1997.
> if the power goes out during defragmentation, and defrag is
> generally not worth the effort of running. I am sure that
Indeed. It's certainly not worth the risk even if the power
stays on! I wouldn't dare run it, IF you could get hold of and compile
the code! I bet you can't.
> those with more experience will correct the above if I got
> it wrong... :)
Peter
Kevin Easton
> In alt.os.linux Guy Macon <GuyMacon@http://www.guymacon.com/home.html> wrote:
>
>> in performance when there is fragmentation, there is a utility
>> called defrag which defragments a linux disk, Defrag is risky
>
> But it doesn't. The utility does not exist. It only existed once
> as a fun piece of code, and it hasn't been maintained since. I
> probably still have it somewhere on my system ...
>
> oboe:/usr/oboe/ptb% locate defrag
> /usr/local/sbin/ddefrag
> /usr/local/sbin/defrag.sh
> ...
>
> but there is no way I would dare to run it nowadays! My code is from
> 1997.
>
>> if the power goes out during defragmentation, and defrag is
>> generally not worth the effort of running. I am sure that
>
> Indeed. It's certainly not worth the risk even if the power
> stays on! I wouldn't dare run it, IF you could get hold of and compile
> the code! I bet you can't.
The most recent release I can find was 0.73pjm1 by Peter J. Moulder
<p...@bofh.asn.au>, back in mid 2001. This is for the entire "defrag"
package which includes defrag programs for minix, ext and ext2.
From reading the documentation with it, it seems that it wasn't
originally written "for fun" - it was written for the original extended
filesystem because it suffered from fragmentation and inefficient
allocation problems. The package was later extended to include a
defragger for the second extended filesystem, even though it is quite
good at avoiding fragmentation on it's own.
- Kevin.
Peter T. Breuer
> The most recent release I can find was 0.73pjm1 by Peter J. Moulder
Peter Moulder might well have written it. As I recall it was part of
the e2compr effort, and he had once to repack before compressing. But that
was later abandoned as racy.
> <p...@bofh.asn.au>, back in mid 2001. This is for the entire "defrag"
> package which includes defrag programs for minix, ext and ext2.
Well, minix and ext have not been used since about 1994, I would guess!
Boot disks used to come in minix, but that was all (it has low
overhead, like fat). Ext2 has never needed defragmentation in itself.
> From reading the documentation with it, it seems that it wasn't
> originally written "for fun" - it was written for the original extended
Then you read it wrong.
> filesystem because it suffered from fragmentation and inefficient
> allocation problems. The package was later extended to include a
I think you misaprehend the development. As I recall people did have
some defraggers for the then unused minix system, and that code formed
the basis for the largely-for-fun defrag utility that briefly appeared
for ext2, and at some later point that (also unused code) evidently has
got folded back into the originals, and perhaps become maintained by
someone after being orphaned.
But I wouldn't put any money on it.
> defragger for the second extended filesystem, even though it is quite
> good at avoiding fragmentation on it's own.
It's better than good - it fragments deliberately and it doesn't matter.
More or less. I wish you would accept the truth of a proposition and
then rationalise based on that, instead of trying to find truths that
support your prior rationalizatoon.
Peter
Kevin Easton
>> The most recent release I can find was 0.73pjm1 by Peter J. Moulder
>
> Peter Moulder might well have written it. As I recall it was part of
> the e2compr effort, and he had once to repack before compressing. But that
> was later abandoned as racy.
Stephen C. Tweedie was the original maintainer of the defrag package of
programs.
>> <p...@bofh.asn.au>, back in mid 2001. This is for the entire "defrag"
>> package which includes defrag programs for minix, ext and ext2.
>
> Well, minix and ext have not been used since about 1994, I would guess!
> Boot disks used to come in minix, but that was all (it has low
> overhead, like fat). Ext2 has never needed defragmentation in itself.
>
>> From reading the documentation with it, it seems that it wasn't
>> originally written "for fun" - it was written for the original extended
>
> Then you read it wrong.
In that case, I shall quote it here where you and other interested
bystanders may decide for themselves. Note that when this text refers
to the "extended file system" it is talking about the original extfs,
not the second extended file system ext2fs.
[begin quote]
The extended file system stores a list of unused disk blocks in a
series of unused blocks scattered over the disk (the "free list").
When blocks are required to store data, they are removed from the head
of the list, and are added back when released (by unlinking or
truncating a file).
However, only the free blocks stored at the head of the list are
available to the extfs at any time. This means that not all the free
space is known to the extfs when it tries to find a free block; as a
result, it does not always find the most efficient way to use free
space.
The resulting poorer performance over time of the extended file system
is unfortunate, because the larger partitions and longer filenames it
supports are useful to have around.
So, the first release of the Linux defragmenter was specifically
designed to overcome this major failing of the extended filesystem.
It allows you to recover all that lost performance from your extfs
partition.
This is in contrast to the other standard Linux filesystems --- the
minix, ext2fs and xiafs file systems --- in which free space is stored
in a single bitmap, and the file system can allocate free space from
anywhere on the disk. These filesystems are very much less prone to
severe fragmentation that extfs is. The ext2fs filesystem also
contains extra code to keep fragmentation reduced even in very full
filesystems. However, over time, even the best filesystem will
eventually become more and more fragmented.
* Note for ext2fs users - Ext2fs divides up partitions into "block
groups", and it tries to balance the amount of data in each group in
an effort to control fragmentation. Defragmenting an ext2fs partition
involves a compromise between making files contiguous and filling
groups: some groups may simply contain far too many files for all
those files to be stored within a single group. So, don't be too
surprised if, after defragmentation, "frag" reports that a very few of
your larger files still are not entirely contiguous.
[end quote]
>> filesystem because it suffered from fragmentation and inefficient
>> allocation problems. The package was later extended to include a
>
> I think you misaprehend the development. As I recall people did have
> some defraggers for the then unused minix system, and that code formed
> the basis for the largely-for-fun defrag utility that briefly appeared
> for ext2, and at some later point that (also unused code) evidently has
> got folded back into the originals, and perhaps become maintained by
> someone after being orphaned.
>
> But I wouldn't put any money on it.
>
>> defragger for the second extended filesystem, even though it is quite
>> good at avoiding fragmentation on it's own.
>
> It's better than good - it fragments deliberately and it doesn't matter.
> More or less. I wish you would accept the truth of a proposition and
> then rationalise based on that, instead of trying to find truths that
> support your prior rationalizatoon.
I'm not the previous poster you were responding to - that was my first
post in this thread, mainly because I saw you say that the programs
haven't been maintained since "about 1997" and would be hard to find,
and I knew this to be false.
- Kevin.
Peter T. Breuer
>> Then you read it wrong.
> In that case, I shall quote it here where you and other interested
> bystanders may decide for themselves. Note that when this text refers
Fine. What I am saying however, is that you aren't aware of the
context in which it exists and in which those words (whatever they are,
I ahven't read them yet) are written. I am trying to give you the
perspective to read it with that you may lack!
> to the "extended file system" it is talking about the original extfs,
> not the second extended file system ext2fs.
> [begin quote]
> The extended file system stores a list of unused disk blocks in a
> series of unused blocks scattered over the disk (the "free list").
> When blocks are required to store data, they are removed from the head
> of the list, and are added back when released (by unlinking or
> truncating a file).
> However, only the free blocks stored at the head of the list are
> available to the extfs at any time. This means that not all the free
Well, that's easily fixed.
> space is known to the extfs when it tries to find a free block; as a
> result, it does not always find the most efficient way to use free
> space.
True. It uses the FAT method - first hole available.
> The resulting poorer performance over time of the extended file system
> is unfortunate, because the larger partitions and longer filenames it
> supports are useful to have around.
> So, the first release of the Linux defragmenter was specifically
> designed to overcome this major failing of the extended filesystem.
But do you KNOW when ext was used? I would guess it was abandoned
as the standard before version 1.1 of the kernel came out! I certainly
never ever saw it used, and I go back at least that far!
> It allows you to recover all that lost performance from your extfs
> partition.
> This is in contrast to the other standard Linux filesystems --- the
> minix, ext2fs and xiafs file systems --- in which free space is stored
> in a single bitmap, and the file system can allocate free space from
> anywhere on the disk. These filesystems are very much less prone to
I.e. they use a global strategy.
> severe fragmentation that extfs is. The ext2fs filesystem also
> contains extra code to keep fragmentation reduced even in very full
> filesystems. However, over time, even the best filesystem will
> eventually become more and more fragmented.
Well, not necessarily. Try removing a lot of large files and replacing
them with a lot of small ones. Instant defragmentation! Indeed, merely
removing will do the defrag!
> * Note for ext2fs users - Ext2fs divides up partitions into "block
> groups", and it tries to balance the amount of data in each group in
Yes, we know. This amounts to "spreading the files around". Or
"minimizing the size of the largest hole".
> an effort to control fragmentation. Defragmenting an ext2fs partition
> involves a compromise between making files contiguous and filling
> groups: some groups may simply contain far too many files for all
> those files to be stored within a single group. So, don't be too
Sure, in other words a large file will be "fragmented" anyway.
> surprised if, after defragmentation, "frag" reports that a very few of
> your larger files still are not entirely contiguous.
Indeed. It's what it tries to do - fragment large files in order to
avoid filling too many holes.
> [end quote]
>> It's better than good - it fragments deliberately and it doesn't matter.
>> More or less. I wish you would accept the truth of a proposition and
>> then rationalise based on that, instead of trying to find truths that
>> support your prior rationalization.
> I'm not the previous poster you were responding to - that was my first
OK, I apologize for that. I didn't follow (or remember) the names.
> post in this thread, mainly because I saw you say that the programs
> haven't been maintained since "about 1997" and would be hard to find,
> and I knew this to be false.
Well, I'm not sure it's false. There is a difference between patching a
program up so that it once again compiles, and seems to run, and
actually using deep knowledge of the FS innards in order to write a
semantically correct program that's as robust as everything else
connected with ext2. Ext2 itself still has minor bugs, btw, so that's
the why I wouldn't touch this utility with a bargepole, no matter
how recently it's been packaged (again?).
Peter
Robert E A Harvey
> In alt.os.linux Guy Macon <GuyMacon@http://www.guymacon.com/home.html> wrote:
...
> > Searching on "linux defragmentation" seems to confirm my belief that
> > the ext2 filesystem is affected by fragmentation, but not as badly as
>
> It's not affected.
>
> > MS systems, and seems to refute your statement that fragmentation has
> > no effect. Example: http://cbbrowne.com/info/defrag.html says"
>
> > "There is a program called defrag that defragments ext2 filesystems."
>
> This doesn't refute anything. That program has been unused for years,
> and only existed "for fun".
Some years ago I worried about this a bit, and spent some time
thinking about it and trying to read about it. But with little
serious outcome. In the end I adopted a darwinian approach:
If fragmentation mattered then, given the nature of the unix
community, there would be hundreds of free and commercial solutions to
it. And religious wars about which was "best". There aren't, and
that alone is sufficient evidence to suppose it doesn't matter in the
slightest.
It is reported. But then, when you do a mkfs that lists the block
numbers it is using. No-one expects you to take any notice of that
either.
Peter T. Breuer
>>In alt.os.linux Guy Macon <GuyMacon@http://www.guymacon.com/home.html> wrote:
>>> ... seems to confirm my belief that the ext2 filesystem is
>>> affected by fragmentation, but not as badly as MS systems,
>>
>>It's not affected.
> You have evidence of this? "Almost no effect" or "very small effect"
> I can believe, but "not affected"? That seems improbable.
"No effect" means both of the things you said. I.e. "forget about it".
AKA "there are other things that will slow you down a lot more, so
worry about those if you want to worry about something". AKA "use a
measure that doesn't show the effect if you don't want to see the effect".
There really are operating circumstances in which fragmentation is good for
you so try and stay in that zone!
I am trying to tell you that the effect is not perceptible in any truly
meaningful way. You can find it if you look for it hard enough, but
the question of whether you have to worry about it if you have to look
for it in order to find it I leave to your good judgement. In my
judgment, you could probably speed things up (if you care) by choosing
a larger blocksize for your FS, or having fewer superblocks, etc.
Peter
Unknown
<p...@oboe.it.uc3m.es> wrote:
>> from the ``MS-DOS'' experience, where fragmentation has
>> significant deleterious effects on system performance."
>
>But nobody can explain why! :-)
>
On the contrary, it's well understood as to why. It's true that ext2
is less affected by fragmentation and is less prone to fragmentation
than say NTFS. However, such resiliency does not come for free. The
overhead involved in searching for the "correct" available hole to
place the file in inflicts a performance penalty on ext2 that does not
exist in NTFS.
So while it's true that NTFS is more susceptible to degraded
performance due to fragmentation, It's also true that when
defragmented it is much faster than ext2. With a moderate level of
fragmentation the two probably perform about the same.
Not exactly a scientific controlled study, and not NTFS vs. ext2, but
representative of Linux file system philosophy compared to Windows
implementation as it relates to performance:
**********************
http://www.theregister.co.uk/content/4/23755.html
I'm using Win-XP on FAT and SuSE on ReiserFS, both installed clean and
subsequently patched.
.
.
Copying that directory to another location on the same disk took 3
minutes, 13 seconds in Windows and 3 minutes, 51 seconds in Linux,
using the GUI file browsers.
.
.
Copying the directory took 3 minutes, 57 seconds using the Bash shell
(cp -r) and two minutes, three seconds using the DOS shell (copy).
*********************
--
Regards
T.G. Reaper
***********
Peter T. Breuer
> On Wed, 11 Dec 2002 03:47:08 +0100, "Peter T. Breuer"
> <p...@oboe.it.uc3m.es> wrote:
>>> from the ``MS-DOS'' experience, where fragmentation has
>>> significant deleterious effects on system performance."
>>
>>But nobody can explain why! :-)
> On the contrary, it's well understood as to why. It's true that ext2
I meant that all the people moaning about fragmentation seem not to ask
the question "why" of msdos. Instead they seem to take it as a natural
phenonemen of file systems in general and then ask "why doesn't it" of
fragmentation in linux.
They would do well to realize that their behaviour is somewhat irrational.
> is less affected by fragmentation and is less prone to fragmentation
> than say NTFS. However, such resiliency does not come for free. The
Actually, as far as I recall, it is virtually for free and there is no
searching, in that the FS maintains a quite detailed multi-queue of free
holes of various sizes. The overhead is in doing the accounting, not
in "searching".
Feel free to correct me if you know different.
> overhead involved in searching for the "correct" available hole to
> place the file in inflicts a performance penalty on ext2 that does not
> exist in NTFS.
> **********************
> http://www.theregister.co.uk/content/4/23755.html
Hmmm .. seems to be down. We are probably having networking problems
somewhere on the backbone.
> I'm using Win-XP on FAT and SuSE on ReiserFS, both installed clean and
> subsequently patched.
> .
> .
> Copying that directory to another location on the same disk took 3
> minutes, 13 seconds in Windows and 3 minutes, 51 seconds in Linux,
> using the GUI file browsers.
Try it again, this time after having catted the files to /dev/null
first, and thus primed the caches. You should find it's twice as fast.
That's just so you get a feel for the parameters of the problem.
Remember that in normal use, the caches are indeed warm.
And also remember that the time depends on whether the output is
buffered or not. You're just seeing when the o/s chooses to signal that
it has scheduled the writes, not when the writes themselves finish!
I.e. the time you are seeing is very variable.
Anyway don't look at the wall-clock, instead look at the output from
"time ...". This will give you the system time and user time, which you
should add up. If it's 50% of the wall-clock time, then you will now
that you only got the system for 50% of the time. It IS a multitasking
O/S, you know!
> Copying the directory took 3 minutes, 57 seconds using the Bash shell
> (cp -r) and two minutes, three seconds using the DOS shell (copy).
FS overhead should have been insignificant if the directories are not
overpopulated. To prove it, try using ext2 instead.
I believe you are merely seeing the difference between multitasking and
single-tasking.
Peter
Jack Kelly Dobson
> On Wed, 11 Dec 2002 03:47:08 +0100, "Peter T. Breuer"
> <p...@oboe.it.uc3m.es> wrote:
>
>
>>> from the ``MS-DOS'' experience, where fragmentation has
>>> significant deleterious effects on system performance."
>>
>>But nobody can explain why! :-)
>>
>
> So while it's true that NTFS is more susceptible to degraded
> performance due to fragmentation, It's also true that when
> defragmented it is much faster than ext2. With a moderate level of
> fragmentation the two probably perform about the same.
>
> Not exactly a scientific controlled study, and not NTFS vs. ext2, but
> representative of Linux file system philosophy compared to Windows
> implementation as it relates to performance:
>
NTFS/HPFS also suballocates small files directly into the FAT table itself
to improve performance.
That's one of the major reasons that a corrupt NTFS volume is so difficult
to repair.
j-