Defrag Software For Sco
scott
SCO box?
I know I can pull the data off the drive to tape, delete the data, then
restore it back to the drives; but it take too long. A defrag solution like
win 95 would be nice even without the GUI.
Bill Vermillion
scott <sc...@hsmc-ul.com> wrote:
It really shouldn't take that long. There used to be commercial
degrag utilites for many Unix systems, but since the migration to
FFS [fast file system] type of file systems they seemed to have
gone away slowly.
Be aware that just deleting the data and restoring it wont take you
back to what you would get in a fresh install, or even the way MS
would do it. That's because Unix being smarter overall uses
a 'free list' when it allocates new files, as opposed to an MS way
of looking for the lowest empty sector available [guaranteed to
fragment a system as soon as you start using it - partily because
of the MS dynamic swap].
So in the Unix world you should remake the filesystem after backing
it up.
The leading leading supertar utilites, eg BackupEdge and LoneTar
for example, make this extremly easy as their recovery disks
[RecoverEdge from BackupEdge and AirBag with LoneTar] give you a a
two-boot disk solution [at the lowest level] to enable you to boot
from floppy, remaked the systems, and restore. Totally painless. I
have clients on both - I've inherited some with one or the other,
and for some the Unix variant on the system dicated what product to
use. Their interfaces are quite different from each other
I've never had a problem with either and both have saved systems
that failed in far less time that any other solutions.. Go to their
respective web sites, www.microlite.com or www.cactus.com and make
your own choices.
Unless you have an extremely active system you won't see "instant
fragmentation" like the MS systems. For most instances it is just
not the problem it is in the MS world.
--
Bill Vermillion - bv @ wjv . com
Roberto Zini
If your "SCO box" is an UnixWare7/OpenUnix8 one, have a look at the
"fsadm" system utility.
Best,
Roberto
--
---------------------------------------------------------------------
Roberto Zini email : r.z...@strhold.it
Technical Support Manager -- Strhold Evolution Division R.E. (ITALY)
---------------------------------------------------------------------
"Has anybody around here seen an aircraft carrier?"
(Pete "Maverick" Mitchell - Top Gun)
Tony Lawrence
> Unless you have an extremely active system you won't see "instant
> fragmentation" like the MS systems. For most instances it is just
> not the problem it is in the MS world.
There's also the point that multiuser systems are apt to be jumping all
around the hard drive anyway just because the users need data from different
places.
--
Tony Lawrence (to...@aplawrence.com)
SCO/Linux articles, help, book reviews, tests,
job listings and more : http://www.pcunix.com
Bill Vermillion
>> Unless you have an extremely active system you won't see "instant
>> fragmentation" like the MS systems. For most instances it is just
>> not the problem it is in the MS world.
>There's also the point that multiuser systems are apt to be jumping
>all around the hard drive anyway just because the users need data
>from different places.
And in a decent system - eg SCSI on Unix - the elevator seeking
mechanism is surely more efficient than the 'jump all over the
place' methods used in the MS world.
Which now brings me to ask this. Are any of the IDE system capable
of doing this? Elevator seeks? Is anyone building add-on controllers
that do this?
This 'jump all over the place' is one reason why the seek time seemed
to be one of the major requisites for performance in the MS arena.
Since the base design sprang from a single-user mentality I can see
this would be a logical paradigm to follow given the heritage of the
original OS [if you can really call the original a 'system'] upon
which it was based.
Bill Campbell
Why do you think you need to defrag? The Unix file systems generally don't
have problems with this. I've never seen over 10% fragmentation, even on
development machines where I'm always creating and deleting files.
Bill
--
INTERNET: bi...@Celestial.COM Bill Campbell; Celestial Software LLC
UUCP: camco!bill PO Box 820; 6641 E. Mercer Way
FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
URL: http://www.celestial.com/
To say that UNIX is doomed is pretty rabid, OS/2 will certainly play a role,
but you don't build a hundred million instructions per second multiprocessor
micro and then try to run it on OS/2. I mean, get serious.
-- William Zachmann, International Data Corp
Fulko Hew
Bill Vermillion <bi...@wjv.com> wrote:
But now-a-days with SCSI drives and IDE drives that 'virtualize' the
physical characteristics of the drive, elevator seeking algorithms
are broken by the drives themselves. When a cylinder used to be a
cylinder, and incrementing that number actually put you on a different
head, everybody was happy and fast. But today, everything is cached
by the drive, and you never 'really' know when the drive will cross
a physical cylinder boundary on your behalf.
So elevator seeks only worked when you had control, in SCSI you only
specify the logical sector, and in IDE you only 'think' you know
what cylinder your on.
Correct me if I'm wrong.
Fulko
-------------------------------------------------------------------------------
Fulko Hew, Voice: 905-681-5570
Senior Engineering Designer, Fax: 905-681-5556
SITA Airport Services Email: fu...@wecan.com
777 Walkers Line,
Burlington, Ontario, Canada, L7N 2G1
Bill Campbell
>Bill Vermillion wrote:
>
>> Unless you have an extremely active system you won't see "instant
>> fragmentation" like the MS systems. For most instances it is just
>> not the problem it is in the MS world.
>
>There's also the point that multiuser systems are apt to be jumping all
>around the hard drive anyway just because the users need data from different
>places.
Which is why SCSI drives are still far superior to IDE for busy multi-
tasking systems since they do a lot to optimize disk access in this
environment.
Bill
--
INTERNET: bi...@Celestial.COM Bill Campbell; Celestial Software LLC
UUCP: camco!bill PO Box 820; 6641 E. Mercer Way
FAX: (206) 232-9186 Mercer Island, WA 98040-0820; (206) 236-1676
URL: http://www.celestial.com/
``Liberty don't work as good in practice as it does in speeches.''
Will Rogers
John DuBois
>So elevator seeks only worked when you had control, in SCSI you only
>specify the logical sector, and in IDE you only 'think' you know
>what cylinder your on.
Note that with SCSI you can have multiple outstanding block requests, which
gives the drive itself the liberty to optimize their gathering in whatever
manner is most effective.
John
--
John DuBois spc...@armory.com. KC6QKZ/AE http://www.armory.com./~spcecdt/
Robert Carnegie
Well, afaik "elevator seek" doesn't refer to fragmentation-resistant
design of the filesystem, where (I vaguely understand) a cylinder
(perhaps not one _whole_ cylinder) is reserved for a file to grow
into - with favourable implication for performance as well - but
to collecting several disk access requests and sorting them into
physical disk order before getting the data.
Since the disk data space is still flat, just with cylinder boundaries
in different places along the way, I'd figure that both the filesystem
layout itself and the elevator seek strategy are still pretty efficient.
But I haven't done the maths, or the tests.
The disk space is _not_ flat where a SCSI disk has found bad data tracks
and has quietly re-mapped them to store data elsewhere on the disk, and
so optimised performance won't be optimal when those blocks are read or
written - but on an important system there should be few enough bad
blocks so that most of the time, naive assumptions about the disk's
geometry _still_ don't particularly hurt performance.
Robert Carnegie
Glasgow, Scotland
Bill Vermillion
Robert Carnegie <rja.ca...@excite.com> wrote:
>Fulko Hew <fu...@wecan.com> wrote in message news:<101090509...@wecan.com>...
>> Bill Vermillion <bi...@wjv.com> wrote:
>>
>>
>> > Tony Lawrence <to...@pcunix.com> wrote:
>> > >Bill Vermillion wrote:
>> > >> Unless you have an extremely active system you won't see
>> > >> "instant fragmentation" like the MS systems. For most
>> > >> instances it is just not the problem it is in the MS world.
>> > >There's also the point that multiuser systems are apt to be
>> > >jumping all around the hard drive anyway just because the
>> > >users need data from different places.
>> > And in a decent system - eg SCSI on Unix - the elevator seeking
>> > mechanism is surely more efficient than the 'jump all over the
>> > place' methods used in the MS world.
[rest of my comments deleted - no need to clutter the thread - wjv]
>> But now-a-days with SCSI drives and IDE drives that 'virtualize' the
>> physical characteristics of the drive, elevator seeking algorithms
>> are broken by the drives themselves.
SCSI devices have ALWAYS done that. Write ordering is still a
performance gain so you don't write block 1234, 7823241, 11134,
123535, etc. but do it in ascending order so you make one seek
across the disk.
Decent SCSI drive will also have the write-back enabled so the
drive can do the ordering, but if you select write-thru or force
things to write 'sync' then you have potentially created a big
performance sink.
>> When a cylinder used to be a
>> cylinder, and incrementing that number actually put you on a different
>> head, everybody was happy and fast. But today, everything is cached
>> by the drive, and you never 'really' know when the drive will cross
>> a physical cylinder boundary on your behalf.
Has nothing to do with ordering the writes ahead of time. As noted
above cylinder/sectors have no bearing on SCSI. When SCSI went
with ZBR [zone bit recording] where the number of sectors per track
change as the diameter of the written track changes you could no
longer count on this. You can see this in the specs on the high
end drive where disk performance is speced as minimum xfer of
250Mbit/sec to a maxium of 500Mbit/sec [for example]. You will
also not number of sectors is quoted as an average. We stopped
worry about cyl/head/sector when we put the smarts in the drive.
>> So elevator seeks only worked when you had control, in SCSI you only
>> specify the logical sector, and in IDE you only 'think' you know
>> what cylinder your on.
>Well, afaik "elevator seek" doesn't refer to fragmentation-resistant
>design of the filesystem, where (I vaguely understand) a cylinder
>(perhaps not one _whole_ cylinder) is reserved for a file to grow
>into - with favourable implication for performance as well - but
>to collecting several disk access requests and sorting them into
>physical disk order before getting the data.
>Since the disk data space is still flat, just with cylinder
>boundaries in different places along the way, I'd figure that both
>the filesystem layout itself and the elevator seek strategy are
>still pretty efficient. But I haven't done the maths, or the tests.
That is dependant on the OS. I either have not seen, or if I have
I do not recall the layouts in the HTFS. In a typical fast file
system the disk space is not 'flat' if by 'flat' you mean you start
at one end of the drive and fill it up until you get to the far
end, which was the original MS approach. [If you notice even MS
has gotten away from that, or conversely any MS systems I've used
in the past few years has had second party software to place
programs and data strategically aross the drive.
ISTR that SCO first went with a FFS aproach when implement AFS ??
[The Acer File System] which was I understood inspired by the
original Berekely Fast File System from the '80s. I put both the
original 51S file system and a new FFS system on the same disk and
saw about a 10 times performance improvement on the FFS formatted
sections as opposed to the 'xenix' format which is what it was
refered to then.
In the FFS environement you build cylinder groups. When files are
written the OS attempts to spread them across the entire dis and
attempts to keep larger files entirely within their own cylider
group. At the lowest level files are stored in much larger pieces
so that each read gets many more sectors as a group than the older
file systems. [Do NOT confuse this with the MS block sizes in their
16 bit FS as those were put there ONLY to enable the system to use
large drives as the amount of 'allocation units' was hard coded in.
The 32 bit system let you got back to small blocks.]
[An interesting aside and an experiment you might perform when you
have nothing to do is to take an MS 16 bit fat and look at it with
your typical degframenting tool so you can see how it looks. Then
do a conversion to a 32 bit FAT, and before doing anything else go
back and look at it. You will see [typically] one block allocated
and 7 empty blocks, and that same patter will repeat and repeat
throught the disk. That's why you must always defrag one of those
after conversion, because you have created a drive that will have
poor response and get worse progressively faster as almost every
new file will be fragmented].
But in a typical fast file implmentation, as I said above, the files
are allocated in larger pieces, and when you have file pieces that
wont fill a full 8K [for example] block, then the piece is stored
elsewhere. When you add to that file the piece is gathered up
along with new data and the file is also extended in 8K units.
Since the files are spread across the disk to start with, you even
have a decent chance of getting the new write tagged on the the end
of the current file to make it fully contiguous.
To sum that up - 'flat' access went away for modern Unix file
systems a long time ago.
>The disk space is _not_ flat where a SCSI disk has found bad data
>tracks and has quietly re-mapped them to store data elsewhere on
>the disk, and so optimised performance won't be optimal when those
>blocks are read or written - but on an important system there
>should be few enough bad blocks so that most of the time, naive
>assumptions about the disk's geometry _still_ don't particularly
>hurt performance.
Whether bad blocks being remapped affect drive performance will
depend on how that is implemented at the drive level. Bad tracks
can be different.
One scheme I saw implemented recovery in this manner.
There were spare sectors for each track set up during formatting.
If a sector were found bad - sector 21 on that track for example -
the ECC mechanism would kick in, the data in that sector would be
read in, and the track would be reformatted with the back sector
now being labeled sector 0 and sector 1 would start on sector 22.
Since the 0th sector is invisible to the OS and the only
performance hit would be on a continous read across several tracks
there would be a very slight rotational delay associated with 22
sectors passing under the head before sector 1 is found.
Now you don't even worry about delays waiting for the first data
sector to come under the head because with zero latency the drive
starts reading as soon as the head settles on the track and then
the sectors are ordeded in the internal buffer before they are
delivered to the controller and on to the system.
The idea of having performance problems when there are lots of bad
tracks probably comes from the days when the OS took care of that.
Remeber the bad sector routines for recovery during the fsck?
That's gone and no drive manufacturer would dream of puttnig the
spare tracks at the end, but when the OS managed it that was all
you really could do.
But this is all low level that you don't have to worry about.
There still exist many myths from the past based on performance of
drives when we had to know the drive geometry and things passed
along as gospel because that's how they worked in the MS world, and
these still wont die.
I just saw a refrence recently to a reformatting to 'refresh' the
magnetism and that myth has never been true. Where it probably
started was that when you'd see errors go away by reformatting, was
in the era of drives with stepper motors. Think of rack and pinion
- gears, etc. As the drive was used these could wear and track
positioning would change. A reformat made the drive work to
current head postioning. [In the early days of floppies you had to
realign them].
But with the advent of dedicated servos this pretty much went away
- but you would see the thermal recal kick in as the drive got warm
to make up for expansion of the head stalks as there was one servo
head on the the bottom platter. Now with embedded servos,
automatic reallocation of bad sectors, large buffers, you don't
have to worry about the mechanical details.
You now can worry about the things that affect performance higher
up - eg good data base designs, good OS designs, etc.
At one time in a design for their highest end drives, IBM drives
would notice a power failure, and covert the spinlde drive motor
into a small generator powered by the angular momentum from the
rapidly spinning platters. This gave the drive enough time to flush
it's 1MB or so drive cache to disk. I don't know if this one ever
was in production or perhaps limited production, but
Anyway 'flat' data space and user manipulation went away for the
most part a long time ago.
Jeff Hyman
> > Correct me if I'm wrong.
>
> Well, afaik "elevator seek" doesn't refer to fragmentation-resistant
> design of the filesystem, where (I vaguely understand) a cylinder
> (perhaps not one _whole_ cylinder) is reserved for a file to grow
> into - with favourable implication for performance as well - but
> to collecting several disk access requests and sorting them into
> physical disk order before getting the data.
>
> Since the disk data space is still flat, just with cylinder boundaries
> in different places along the way, I'd figure that both the filesystem
> layout itself and the elevator seek strategy are still pretty efficient.
> But I haven't done the maths, or the tests.
>
> The disk space is _not_ flat where a SCSI disk has found bad data tracks
> and has quietly re-mapped them to store data elsewhere on the disk, and
> so optimised performance won't be optimal when those blocks are read or
> written - but on an important system there should be few enough bad
> blocks so that most of the time, naive assumptions about the disk's
> geometry _still_ don't particularly hurt performance.
>
> Robert Carnegie
> Glasgow, Scotland
I'm not trying to start an argument here... but if hard disk
fragmentation is not a big issue with UNIX or SCSI, then why is it that
when I use a super-tar and its crash recovery component to restamp all
HD partition info then restore all data.... that my system speeds
up by around 20%? P2, 7.2ms HD, SCSI 256k RAM. I do this twice
a year weither I need it or not. Biggest increase is noticable in
the Informix Database Querys.... and the entire system in the
multiuser environment has noticeable speed increase. I've seen no
arguments to convince me to stop this procedure. Speed is Speed.
--
Best Regards,
Jeff Hyman, President
.--.
__________________________ .-. | | __________________________________
Lone Star Software Corp. | | | | .-. E-Mail: je...@cactus.com
Cactus International, Inc. | |_| | | | Sales: 800.525-8649
509 E. Ridgeville Blvd. _ |___ |_| | Support: 301.829-1622
Mt. Airy, MD 21771 _| ~- | ___| Fax: 301.829-1623
http://www.CACTUS.com \, _} | | http://www.LONE-TAR.com
------------------------- \( -- | | -----------------------------------
| |
Scott Trowbridge
I agree completely. We had to something similar. Pull all the data off
our data drives and put it on tape, remake the filesystem, then restore the
data. We had about the same improvement (20%) and it needs to be done at
least every three to four months.
Someone needs to write a program to do a defrag live!
Scott
"Jeff Hyman" <sco...@cactus.com> wrote in message
news:m15fQUM...@cactus.com...
Michael Krzepkowski
Jeff Hyman wrote:
You must be using Informix SE which stores data in regular files.
Using Informix utilities to re-create your largest database tables and
re-creating indexes should give you even better results.
Or you could switch to Informix Online, which manages raw
partitions by itself.
My $0.02
Michael
Ken Wolff
I agree. I unload/divvy/reload our filesystems once a quarter and see the
same performance improvement.
--------------------------------------------------------------
Ken Wolff
Phone: 616-957-4949 Ext: 1111
FAX: 616-957-1614
--------------------------------------------------------------
Fulko Hew
Jeff Hyman <sco...@cactus.com> wrote:
> I'm not trying to start an argument here... but if hard disk
> fragmentation is not a big issue with UNIX or SCSI, then why is it that
> when I use a super-tar and its crash recovery component to restamp all
> HD partition info then restore all data.... that my system speeds
> up by around 20%? P2, 7.2ms HD, SCSI 256k RAM. I do this twice
> a year weither I need it or not. Biggest increase is noticable in
> the Informix Database Querys.... and the entire system in the
> multiuser environment has noticeable speed increase. I've seen no
> arguments to convince me to stop this procedure. Speed is Speed.
and Scott Trowbridge <sc...@hsmc-ul.com> responded:
> Someone needs to write a program to do a defrag live!
I know it wasn't on a Unix box, but I once evaluated a live
defragger for VAX/VMS. What I discovered was:
a) yes the file system fragmented.
b) yes the defraged system ran faster
but I also discovered that:
c) after defragging the system got back to the same level
of fragmentation in less than a day (it was a busy system)
And by the way... it took this defragger about 1 day of CPU time
do do its thing (offline), Online it couldn't keep up.
CPU's and disks were slow back then. :-(
Conclusions:
1/ The performance hit of the fragged system was very tolerable
I seem to recall about 5%.
2/ I'd have to run the defragger every other day to make a difference.
3/ the defragger couldn't keep up.
So based on that, I said "why bother".
I (think) I get the same kind of performance from my SCO systems too.
YMMV
Ken Wolff
<snip>
>You must be using Informix SE which stores data in regular files.
>Using Informix utilities to re-create your largest database tables and
>re-creating indexes should give you even better results.
>Or you could switch to Informix Online, which manages raw
>partitions by itself.
>
>My $0.02
>
>Michael
Using raw partitions would make an unload/reload pretty much useless
wouldn't it? I don't believe any Super-tar product (or tar/cpio) can
backup raw partitions.
However, your database on a raw partition can still become fragmented
depending on your block/segment sizes and the growth of tables. The only
difference is instead of using a backup utility you would need to use your
database backup utility, recreate the DB and then reload the DB.
The database we use (DataServer 7.0 from Unify Corporation) supports raw
partitions but we still use the file method.
Ken
Jean-Pierre Radley
|
| Using raw partitions would make an unload/reload pretty much useless
| wouldn't it? I don't believe any Super-tar product (or tar/cpio) can
| backup raw partitions.
The BackEDGE manual certainly says it can back up raw filesystem partitions.
--
JP
Bob Meyers
"Jeff Hyman" <sco...@cactus.com> wrote in message
news:m15fQUM...@cactus.com...
> ------------ clipped ----------
> fragmentation is not a big issue with UNIX or SCSI, then why is it that
> when I use a super-tar and its crash recovery component to restamp all
> HD partition info then restore all data.... that my system speeds
> up by around 20%?
I have done this too and seen similar results! I always wondered why. If
anyone has an urge, install and NEW SCO from CD, then supertar it off, blow
HD away, restore from tape. I have seen my own, homegrown benchmark jump
20-30% by doing this. I started thinking I must be making some mistake - how
could a NEW install be so fragmented?
Bob Meyers
"Michael Krzepkowski" <mich...@sqlcanada.com> wrote in message
news:3B991EF3...@sqlcanada.com...
> You must be using Informix SE which stores data in regular files.
> Using Informix utilities to re-create your largest database tables and
> re-creating indexes should give you even better results.
> Or you could switch to Informix Online, which manages raw
> partitions by itself.
>
Nope, not in my tests. Mine was pretty much a standard SCO install from CD.
- bill -
The Lone-Tar manual says it can back up raw filesystems, but that they
need to be unmounted first.
--
-bill-
Technical Service Systems - bi...@TechServSys-garbage.com
Jeff Hyman
Michael,
Its been some time since I did an ascii unload and reload. :-)
but it does help as you mentioned. The fact that I did NOT do
this and experienced the speed increase even further supports
my claim.
Best Regards,
Jeff Hyman
.--.
__________________________ .-. | | __________________________________
Lone Star Software Corp. | | | | .-. Email: je...@cactus.com
Cactus International, Inc. | |_| | | | Sales: (800) 525-8649
509 E. Ridgeville Blvd. _ |___ |_| | Support: (301) 829-1622
Mt. Airy, MD 21771 _| ~- | ___| Fax: (301) 829-1623
http://www.CACTUS.com \, _} | |
------------------------- \( -- | | -----------------------------------
| |
Jeff Hyman
> At 07:25 PM 9/7/01 +0000, Michael Krzepkowski wrote:
>
> <snip>
>
> >You must be using Informix SE which stores data in regular files.
> >Using Informix utilities to re-create your largest database tables and
> >re-creating indexes should give you even better results.
> >Or you could switch to Informix Online, which manages raw
> >partitions by itself.
> >
> >My $0.02
> >
> >Michael
>
> Using raw partitions would make an unload/reload pretty much useless
> wouldn't it? I don't believe any Super-tar product (or tar/cpio) can
> backup raw partitions.
The super-tar products can backup RAW partitions. Look up the
-zDEV flag. This flag can also be used to backup a DOS partition
as well. You can also have the compression flag turned ON for the
RAW partition backup. Whats even of more value is you can do a
byte-by-byte verification of the RAW partition after its backed up.
Here's the problem... a RAW partition is backed up in raw-blocks as
compared to individual files... so the restore will leave the RAW
partition in its original state. IOW: Its a block-by-block backup
as compared to a file-by-file backup. I still cannot help but believe
that even a RAW partition can (and will) get data scattered in the
blocks allocated to the RAW partition. Does the Informix On-Line
have its own method of organizing blocks and re-organizing them
as the blocks get scattered?
Best Regards,
Jeff Hyman
.--.
__________________________ .-. | | __________________________________
Lone Star Software Corp. | | | | .-. Email: je...@cactus.com
Cactus International, Inc. | |_| | | | Sales: (800) 525-8649
509 E. Ridgeville Blvd. _ |___ |_| | Support: (301) 829-1622
Mt. Airy, MD 21771 _| ~- | ___| Fax: (301) 829-1623
http://www.CACTUS.com \, _} | |
------------------------- \( -- | | -----------------------------------
| |
>
Michael Krzepkowski
This is nice that it can backup raw partition, but restoring them does
not reorganize the data. And, yes I know that most of you guys know that.
Michael
Michael Krzepkowski
Jeff mentioned that his Informix queries were faster. Informix SE uses
standard file system to hold data and index information. If you backup-restore
these you will likely get better performance. Normally they would be kept
on a separate file system anyways. Informix offers utilities to unload table,
drop it and re-load whcih not only will defragment data, but also reorganize
indexes. This a DBA job and should be performed regularly.
Database aside, I use systems where there is very little other file activity,
so my customers are not faced with issue of fragmented files and slowdown.
Michael
Jeff Hyman
There are 'before' and 'after' scripts that allow unmounting and
remounting RAW partitons (amoung many other things). These scripts
also have the capability to make darn sure the raw partition is
re-mounted in any combination of events.
1. backup successful - verification successful
2. backup successful - verification failed
3. backup failed - verification aborted
4. backup successful - No verification scheduled
5. backup failed - No verification scheduled
--
Best Regards,
Jeff Hyman, President
.--.
__________________________ .-. | | __________________________________
Jeff Hyman
If a file is loaded to a hard drive in a compressed format, then
uncompressed... the free list will get fragmented because the uncompressed
file will be occupying blocks different then that of the compressed file.
The compressed file then gets removed and the fragmentation begins.
Michael Krzepkowski
Depending on the size of your database, you should look iinto
dropping and re-creating indexes every few months. It makes sense
for tables larger than a couple hundred thousand records and up.
Many database servers experience very little of "regular" file activity,
so backup and re-install of the system is an overkill and it does not
help clean unused space in datafiles (after delete) or re-sort indexes.
If you want to squeeze more performance from your Informix database,
post in comp.databases.informix and we'll go from there.
Regards,
Michael Krzepkowski
SQL Systems Inc.
Michael Krzepkowski
Jeff Hyman wrote:
Informix Online is quite good an managing free space and allocation
of contiguos space, but it does not perform defragmentation by itself.
It offers utilities to do so. Backup of raw partition and restore changes
nothing - but simplifies system recovery dramatically.
As someone mentioned on a multiuser system allocation/deallocation
of blocks is the fact of life, so your OS better be able to live with it.
Michael
Ken Wolff
Cool. I was not aware of that.
But then again, backing up a raw partition and restoring it wouldn't defrag
anything, correct? You still need to use your database utilities to
accomplish that.
Michael Krzepkowski
Ken Wolff wrote:
> At 08:40 PM 9/7/01 +0000, Jean-Pierre Radley wrote:
> >Ken Wolff propounded (on Fri, Sep 07, 2001 at 07:57:00PM +0000):
> >|
> >| Using raw partitions would make an unload/reload pretty much useless
> >| wouldn't it? I don't believe any Super-tar product (or tar/cpio) can
> >| backup raw partitions.
> >
> >The BackEDGE manual certainly says it can back up raw filesystem partitions.
> >
> >--
> >JP
>
> Cool. I was not aware of that.
>
> But then again, backing up a raw partition and restoring it wouldn't defrag
> anything, correct? You still need to use your database utilities to
> accomplish that.
You bet.
Michael
Bill Vermillion
Jeff Hyman <sco...@cactus.com> wrote:
> I'm not trying to start an argument here... but if hard disk
>fragmentation is not a big issue with UNIX or SCSI, then why is it that
>when I use a super-tar and its crash recovery component to restamp all
>HD partition info then restore all data.... that my system speeds
>up by around 20%? P2, 7.2ms HD, SCSI 256k RAM. I do this twice
>a year weither I need it or not. Biggest increase is noticable in
>the Informix Database Querys.... and the entire system in the
>multiuser environment has noticeable speed increase. I've seen no
>arguments to convince me to stop this procedure. Speed is Speed.
The discussion started off with a person wanting a defrag program
like that on Win95 because he said backing up data and restoring it
after deleting the files would take too long.
Campbell said that he had never seen more than 10% fragmentation on
busy systems hes used. The heart of the original discussion was
that the Unix systems don't fragment as badly or as rapidly as the
MS systems.
You said " if .. fragementation is not a big issue with Unix or
SCSI .." and the proceeded to say that you have a 20% speed
increase. You also said you do this twice a year. No problem with
that. The Winxx systems need to have it run almost continually,
and if you recall the settings on the defrag utilities one option
is to run every Friday night. That's 25 times more often than your
twice a year. No one said to stop defragging.
I think the fact that the defrag programs that used to be available
for SCO and the other commercial defrag and optimization utilities
for other Unix system no longer a being marketed, points out that
defragmentation is not the major problem that it is in the MS
world.
The one caveat I mentioned in my first post was also that the
person needed to remake the filesystem to get it back into good
shape, and I think I mentioned your AirBag and Tom's RecoverEdge.
If not I was amiss. The commercial programs attempted to analyze
the useage and move the more often used files to the front of the
disk.
That was back in the days of 40ms drives being the norm, under 1Mb
second transfer rates, and other slow things we have gladly given
up. In those days disks were still pretty much accessed front to
back, and the orginization off the free block list played a part
too, so that often people would use cron tod perform an fsck -S
[capitol S], which would reorder the free block list ONLY if
everyting else checked out OK. IOW it was the only safe method
to run an fsck. This helped keep the fragementation down as the
freelist then because after the intial allotment of blocks the free
list was populated by blocks associated with removed files and were
just put on in a LIFO order as I recall. That led to fragmentation
so the sorting of the list help keep performance up a bit longer.
I do NOT miss those days in the least bit.
Bill Vermillion
- bill - <bi...@TechServsys-garbage.com> wrote:
>Jean-Pierre Radley wrote:
>>
>> Ken Wolff propounded (on Fri, Sep 07, 2001 at 07:57:00PM +0000):
>> |
>> | Using raw partitions would make an unload/reload pretty much useless
>> | wouldn't it? I don't believe any Super-tar product (or tar/cpio) can
>> | backup raw partitions.
>> The BackEDGE manual certainly says it can back up raw filesystem
>> partitions.
>The Lone-Tar manual says it can back up raw filesystems, but that they
>need to be unmounted first.
But raw systems aren't mounted as the application uses the disk
directly. Mounted disks use the system file systems. Or am I
missing something completely here?
Bill Vermillion
Ken Wolff <ke...@cscc.maximus.com> wrote:
>At 08:40 PM 9/7/01 +0000, Jean-Pierre Radley wrote:
>>Ken Wolff propounded (on Fri, Sep 07, 2001 at 07:57:00PM +0000):
>>| Using raw partitions would make an unload/reload pretty much
>>| useless wouldn't it? I don't believe any Super-tar product (or
>>| tar/cpio) can backup raw partitions.
>>The BackEDGE manual certainly says it can back up raw filesystem partitions.
>
>Cool. I was not aware of that.
>But then again, backing up a raw partition and restoring it
>wouldn't defrag anything, correct? You still need to use your
>database utilities to accomplish that.
But databases quite often handle disks in ways that are not
intuitive. They quite often spread data all across the disk to
start with.
Think of it like this.
You are starting a large business and you have a lot of paper to
file. You estimate that in a years time you will fill up 25 file
cabinets, but on the first day they are all empty.
You do not start with the first cabinet and put in 20+ alphabetic
dividers in the first drawer. And then in the middle of the week
split those among two drawers. And then next Monday move some to
the third drawer and mid-week move them again to the 4th drawer.
Week three starts you on the second 4-drawer file cabinet.
The above example shows that you would fill up one four-drawer file
cabinet every two weeks.
What you would most logically do is figure out the alpha
distribution and label all the file cabinets the first day and put
the dividers in the appropriate drawers. By the end of the year
you have all the file cabinets full and if you predicted things
accurately you would never have to shuffle papers.
Some database desings work along those same line so you can see you
would NOT want to defragment a raw partition. Since the database
has access to the complete raw partition [quite often a dedicated
drive] it knows where it placed every piece of data it has no need
for indexes, directories, even file systems. It just needs
a disk formatted at the lowest level and then it can do it's own
thing.
Bill
Bill Vermillion
<sco...@cactus.com> wrote:
>Recently, Bob Meyers wrote:
>
>> "Jeff Hyman" <sco...@cactus.com> wrote in message
>> news:m15fQUM...@cactus.com...
>>
>> > ------------ clipped ---------- I'm not trying to start an
>> > argument here... but if hard disk fragmentation is not a big
>> > issue with UNIX or SCSI, then why is it that when I use a
>> > super-tar and its crash recovery component to restamp all HD
>> > partition info then restore all data.... that my system speeds
>> > up by around 20%?
>> I have done this too and seen similar results! I always wondered
>> why. If anyone has an urge, install and NEW SCO from CD, then
>> supertar it off, blow HD away, restore from tape. I have seen my
>> own, homegrown benchmark jump 20-30% by doing this. I started
>> thinking I must be making some mistake - how could a NEW install
>> be so fragmented?
> If a file is loaded to a hard drive in a compressed format,
>then uncompressed... the free list will get fragmented because the
>uncompressed file will be occupying blocks different then that of
>the compressed file. The compressed file then gets removed and the
>fragmentation begins.
Jeff - here's the chance to shine by building an intelligent file
compression system.
Let's make an easy example with a 2:1 compression. Compressed the
file would occupy 100 segments [abirtrary - could be 100K chunks or
100 byte chunks for purposes of this discussion].
Assume that we have save a few bytes up at the front of the file
[like an index] that is updated after compression so that we know
exactly how large the file will be when uncompressed.
Now lets invert our thinking. We next allocate all the blocks
needed to fully hold the expaned file and we include the blocks
used in the compressed file as part of that.
Now we start decomprssing the file from back to front and as we
decompress the file we write to the end of the pre-allocated
sections and then work forward, so that at the end, the
decompression has overwritten the compressed archive and is in it's
proper place and ready to run.
This would mean you could take a 100MB file that would expand to
200MB and install it on a drive that has only 200MB free space
instead of needing 300MB to start with.
If you can have it done by Christmas we'll all be happy, and we'll
name it jzip - Jeff's Zip - in your honor :-)
Just because decompressing a file causes fragementation now doesn't
mean it always has to be that way :-).
Tony Lawrence
> >The Lone-Tar manual says it can back up raw filesystems, but that they
> >need to be unmounted first.
>
> But raw systems aren't mounted as the application uses the disk
> directly. Mounted disks use the system file systems. Or am I
> missing something completely here?
I think that what they are referring to here is the ability to take a file
system and put it to tape * as a file system *, like the old "dump" programs.
--
Tony Lawrence (to...@aplawrence.com)
SCO/Linux articles, help, book reviews, tests,
job listings and more : http://www.pcunix.com
Bill Vermillion
>> >The Lone-Tar manual says it can back up raw filesystems, but
>> >that they need to be unmounted first.
>> But raw systems aren't mounted as the application uses the disk
>> directly. Mounted disks use the system file systems. Or am I
>> missing something completely here?
>I think that what they are referring to here is the ability to take
>a file system and put it to tape * as a file system *, like the old
>"dump" programs.
Thanks. That was not clear. 'raw filesystem' makes sense in that
case, but somewhere in the discussion data bases came up and those
basically use a raw disk, but in that case it is not a filesystem.
Brian K. White
> I'm not trying to start an argument here... but if hard disk
> fragmentation is not a big issue with UNIX or SCSI, then why is it that
> when I use a super-tar and its crash recovery component to restamp all
> HD partition info then restore all data.... that my system speeds
> up by around 20%? P2, 7.2ms HD, SCSI 256k RAM. I do this twice
> a year weither I need it or not. Biggest increase is noticable in
> the Informix Database Querys.... and the entire system in the
> multiuser environment has noticeable speed increase. I've seen no
> arguments to convince me to stop this procedure. Speed is Speed.
I think the claim is not that modern unix filesystems don't have
fragmentation, but that the unix filesystems are such that they never get
beyond a certain level of overall fragmentation. There is no way to always
stay 100% unfragmented, but if the system is more intelligent than M$, then
it is possible to hover at a level that is pretty good.
By comparison, M$ can only ever get worse and worse and worse with every
erase and write.
You can reset yourself to 100% unfragmented by using the backup and restore
method, but it is more or less pointless IMO, since it goes quickly right
back to where it was as soon as you start using the system, and does not get
any worse as time goes on.
But there are other things besides fragmentation that do impact performance
and do sometimes get worse as time goes on, depending on the activities of
your applications, and are also reset back to optimum performance by a
backup & restore. and it is one of these other things that actually needs a
fresh re-created filesystem and not just a bckup-erase-restore in order to
get the speed improvement. I'm talking about directory entries. when a file
is created in a directory, an entry is created in that directory. when a
program checks the status of the directory it must scan through all such
entries. a new directory has no entries and the scan is instantanious.
create 50,000 files in the directory and the scan is very slow. delete files
and the entries do not go away, they are blanked but not removed, so the
scan is still very slow. most directories don't have a problem with this
because usually files get created and erased and new ones created, and blank
entries get re-used, and the total number of entries stays sane, but all it
takes is one incident, one spike in the normal activity that creates a
zillion files for some reason one day, and that directory becomes "slow" and
stays that way until it is removed and re-created. Over time, the normal
randomness of activity in any given directory will cause any directory that
ever sees activity to become slower than it was at birth, but generally not
a lot, and generally it reaches a certain level and doesn't get worse.
However some directories are very likely to see spikes of activity that slow
them down greatly, and they are often directories that are used a lot by
many parts of the system, so the system as a whole will feel slower.
consider /tmp, consider web and ftp server directories that might hold a lot
of images or something, consider database or other application directories
that might hold a lot of little files, or might have held a lot of files for
a few seconds once. the cannonical example is news spools but I don't think
many of us run nntpd on our office servers.
Brian K. White -- br...@aljex.com -- http://www.aljex.com/bkw/
+++++[>+++[>+++++>+++++++<<-]<-]>>+.>.+++++.+++++++.-.[>+<---]>++.
filePro BBx Linux SCO Prosper/FACTS AutoCAD #callahans Satriani
Bill Vermillion
Brian K. White <br...@aljex.com> wrote:
>"Jeff Hyman" <sco...@cactus.com> wrote in message
>news:m15fQUM...@cactus.com...
>> I'm not trying to start an argument here... but if hard disk
>> fragmentation is not a big issue with UNIX or SCSI, then why is
>> it that when I use a super-tar and its crash recovery component
>> to restamp all HD partition info then restore all data.... that
>> my system speeds up by around 20%? P2, 7.2ms HD, SCSI 256k
>> RAM. I do this twice a year weither I need it or not. Biggest
>> increase is noticable in the Informix Database Querys.... and the
>> entire system in the multiuser environment has noticeable speed
>> increase. I've seen no arguments to convince me to stop this
>> procedure. Speed is Speed.
>I think the claim is not that modern unix filesystems don't have
>fragmentation, but that the unix filesystems are such that they
>never get beyond a certain level of overall fragmentation. There is
>no way to always stay 100% unfragmented, but if the system is more
>intelligent than M$, then it is possible to hover at a level that
>is pretty good. By comparison, M$ can only ever get worse and worse
>and worse with every erase and write.
Totally dependand on what you are doing. If it did hover at a certain
level it would be wonderful. Then you design the system for that
level of performance and never worry about it again.
>You can reset yourself to 100% unfragmented by using the backup and
>restore method, but it is more or less pointless IMO, since it goes
>quickly right back to where it was as soon as you start using the
>system, and does not get any worse as time goes on.
Again - application dependant. It can get worse as time goes on.
Take for example a database program that dynamically grows the
files and also has index functions. New data is added, and the
file grows. If you have an automatic indexing function those will
also grow - albeit slower as the index records are most probably
very short and a data file will expand many records for each
expansion of an index file.
That means that at one point when the data is extended, then an
index file will be extended, and depending on the file system
design and of course state of the free list and many other
variables, the new blocks for the index can be added right after
the last contingous data block. Next time data is added the data
section is fragemented. The more indexes and the more files in the
database the more rapidly this occurs.
Some databases expand automatically, and other require you to
expand the data. As much as I disliked the perfromance I saw on
systems running the BBX based systems, at least the one that seemed
prevalent in this area required user expansion. That meant you
minimized the fragmentation.
When I used to do a lot of filePro programming on slower systems,
when I'd backup the system to remake the file system and gain
performance I'd remove ALL the index files. I'd then load the
files one at a time and pre-expand them to the size that I would
estimate they would grow in a years time, and then and only then
rebuild all the automatic indexes. This meant that fragmentation
would be postponed for a year. [there are other ways to make sure
you minimize this but I covered all this on the FP list years ago]
>But there are other things besides fragmentation that do impact
>performance and do sometimes get worse as time goes on, ... and it
>is one of these other things that actually needs a fresh re-created
>filesystem and not just a bckup-erase-restore in order to get the
>speed improvement. I'm talking about directory entries.
The fresh file system takes you back to ground zero as far as the
free blocks and free list are concerened. But you do not have to
remake a file system to restore performance on a directory.
>when a file is created in a directory, an entry is created in that
>directory. when a program checks the status of the directory it
>must scan through all such entries. a new directory has no entries
>and the scan is instantanious.
And of course there is no reason to can an empty directory is there
:-)
>create 50,000 files in the directory and the scan is very slow.
Anyone who creates a program that makes 50,000 entries in a
standard derived Sys V directory gets what they deserve. The reason
is that the directory [which is a file itself] goes past the size
that will be allocated from block in the inode and then the rest of
the directory entries will be in blocks that are indirect links so
instead of 2 access to find the directory pieces you will have to
make at least 4. It is the multiple access that cause it to be
slow, plus the fact that you are probably looking at the directory
with 'ls'. Since that by default sorts everything you are making
many many more accesses than a directory of normal length.
Depending on the system, the OS version, etc, performance starts
falling off in the 500 to 1000 entry range.
> delete files and the entries do not go away, they are blanked but
>not removed, so the scan is still very slow.
That's something that really needs to be taken care of. ISTR that
at least some *ix OSes take care of that. Then there is Irix with
it's xts that has no problem with a million directory entries, but
that's another story. Perhaps the ports of xtfs into the Linux
world [courtesy of SGI] will propagate.
>most directories don't have a problem with this because usually
>files get created and erased and new ones created, and blank
>entries get re-used, and the total number of entries stays sane,
>but all it takes is one incident, one spike in the normal activity
>that creates a zillion files for some reason one day, and that
>directory becomes "slow" and stays that way until it is removed and
>re-created. Over time, the normal randomness of activity in any
>given directory will cause any directory that ever sees activity to
>become slower than it was at birth, but generally not a lot, and
>generally it reaches a certain level and doesn't get worse.
You never can tell. But one thing you can do to optimize directory
performance - without making a new file system - is to backup the
hierarchy and remove the directory and then restore it.
The reason for this is if you remember in directory saves and
restores, you go to the lowest level of the directory and save
coming back up the tree. This is because of file permissions.
If you created top down and set the permissions there you could
create diretories that you could not access as the permissions
would be set at that time.
The restoration from the lowest level creates all the directories
first in a given tree, and on each level back up the tree the
permissions are set, so at that point the backup program might not
have further access. This also has the function of making all
sub-directories being the first in any directory hierarchy so
searching down the tree will be faster - except for directories
added later. You can actually perform the directory restructure
on your system if you have enough space to store you tar file of
the tree for example.
>However some directories are very likely to see spikes of activity
>that slow them down greatly, and they are often directories that
>are used a lot by many parts of the system, so the system as a
>whole will feel slower.
Sound like bad design/planning :-)
>consider /tmp, consider web and ftp server directories that might
>hold a lot of images or something,
If you system is quiescent for any time you can easily rename
/tmp to /tmp.old, and make a new /tmp, then remove /tmp.old
As to web and server directories containing many files that should
be a matter of pre-planning and organization during construction
of the applications/web-pages.
>consider database or other application directories that might hold
>a lot of little files, or might have held a lot of files for a few
>seconds once. the cannonical example is news spools but I don't
>think many of us run nntpd on our office servers.
Well the directory problems with the older news systems, usally
when you were unspooling things, was helped with some of the newer
news system designs. expire used to create problems. The worst
I'd have was when the net police would go on a rampage and they'd
start mass cancels and file up the control directory so the
performance would degrade [as you desribed above]. Many cured this
by ignorning controls, which meant that canceled articles hung
around much to some authors chagrin. :-)
No - I don't run nntpd - I still run cnews as I only get about 3000
articles a day - which is in excess of 98% of my news wants - and I
only have to chase strays on the outside world. Reading news
locally spoils you for anything else.
Bill
Ken Wolff
No offence Brian...but your message was sooo long I didn't read all of it.
Yes, many little files can cause problems. That's not our case.
Over and over some folks on this group state fragmentation may happen on
UNIX filesystems but they shouldn't effect performance. All I know is
that our system's response time and backup time grows and grows over
time. Everyone's mileage may vary.
If I backup the file systems, run divvy and recreate the file systems and
then restore from backup I get back to my original performance. I gage
this from the elapsed time of our BackupEdge Master backup and verify
time. It starts at about 7 hours and when it gets to 9-10 hours (backing
up the same amount of data) I do a couple of backups and then
divvy/restore. Then it goes back to 7 hours.
Honestly I belive the best defrag method is backup/divvy/restore. I'm not
looking for any type of other option.
I don't care what type of disk I'm using (IDE or SCSI), what geometrics
they have, what RAID level is used or not used.
Ken
Bill Vermillion
In article <GJD6K...@wjv.com>, Bill Vermillion <b...@wjv.com> wrote:
I really screwed this paragraph up!
>That's something that really needs to be taken care of. ISTR that
>at least some *ix OSes take care of that. Then there is Irix with
>it's xts that has no problem with a million directory entries, but
>that's another story. Perhaps the ports of xtfs into the Linux
>world [courtesy of SGI] will propagate.
XFS - XFS - XFS - NOT XTFS. What a maroon I am. [to many
acronyms]. Besides the performance on directory lookups not being
substantiall different from 100 to 10,000,000 files in a directory,
the logic block sizes can bet over 1MB long. Considering it
supported media files and the higher end works station could
edit broadcast video in near-real-time - which means multi-hundred
MB files - large block sizes would work well there.
But the XFS is being ported to Linux by SGI. They fell badly
on moving to iNTEL and NT - even though the first NT ports were on
SGI's on MIPS chips - but I'd not be suprised if they disappeared
in the next couple of years.
Sorry about the screwup.
Jeff Hyman
-------- clipped ---------
Bill,
Christmas of what year? :-) I understand what you are saying.
Interestingly, the super-tars make every effort, and take advantage
of every opportunity, to avoid writting temporary files to disk during
compression and decompression. Staying in tune with the defragmentation
topic of this thread, a super-tar compressed file will UNcompress
"on-the-fly" through a "virtual pipe" (not touching the HD) then get
written to HD when its already uncompressed. This is the special
event (of not creating temporary files on the HD) that causes the
speed increase (deframentation). Amount of available free RAM plays
into this too. It would probably take up more overhead to accomplish
your mentioned request then to leave things as they are. Also, to
maintain my humbel and honest reputation <g> I do not have the
background nor knowledge to write such a compression product. I am
however very lucky to be surrounded by friends that are much smarter
then I.
PS: I have been seeing a big increase in users NOT using software
compression and instead using the compression chip built into
the tape drive itself. Its faster and uses less CPU cycles.
Only concern is you need a compatible tape drive with a compatible
compression chip to restore the data.
--
Best Regards,
Jeff Hyman, President
.--.
__________________________ .-. | | __________________________________
Bill Vermillion
Jeff Hyman <sco...@cactus.com> wrote:
>Recently, Bill Vermillion wrote:
>-------- clipped ---------
>> Jeff - here's the chance to shine by building an intelligent file
>> compression system.
> Christmas of what year? :-) I understand what you are saying.
It was an intesting though I had. There is could be a real problem
with it as it is doing it in place, just like self-modifying code.
You could wind up with a mess :-)
>Interestingly, the super-tars make every effort, and take advantage
>of every opportunity, to avoid writting temporary files to disk during
>compression and decompression.
Now the do. I surely remember the old days when if you had to
compress files you had to write them to disk and then spin them out
to tape. Thank goodness those days are gone.
>It would probably take up more overhead to accomplish
>your mentioned request then to leave things as they are.
Doesn't really belong in the super-tar products. Just an intesting
idea on how to uncompress a file on a file system that doesn't have
enought space for the uncomrpressed file and the compressed file.
> Also, to maintain my humbel and honest reputation <g> I do not
>have the background nor knowledge to write such a compression
>product. I am however very lucky to be surrounded by friends that
>are much smarter then I.
>PS: I have been seeing a big increase in users NOT using software
> compression and instead using the compression chip built into
> the tape drive itself. Its faster and uses less CPU cycles.
> Only concern is you need a compatible tape drive with a compatible
> compression chip to restore the data.
Thankfully - unless someone is still using the old drives where the
schemes were incompatible, we don't have that problem anymore.