Fragmentation, how do you measure it without buying something?
Rob Schaeffer
someone? Or what do you look for?
I will post a summary.
Thanks.
Rob
"Putting magnets on the T.V. distorts the picture and
makes it more real."
Phil Daly, JAC, 665 Komohana St, Hilo HI 96720, USA
turn rate should be zero --- if it's not then the files are very fragmented.
Note that when I say zero I mean just that (0.0). If it's 0.01 you've
got a fragmented disk. Leave MONITOR running for a couple of days before
making a decision.
"Anything I say should be taken at your own risk with a hefty pinch of salt"
+------------------------------------------------------------------------------+
| | |
| | Telephone : +1 808 969 6568 (direct) |
| P N Daly | +1 808 961 3756 x268 (reception) |
| Joint Astronomy Centre | Facsimile : +1 808 961 6516 |
| 665 Komohana Street | Telenet DTE : 31528080905310 |
Phil Daly, JAC, 665 Komohana St, Hilo HI 96720, USA
Mats G L|fdahl
Phil Daly wrote:
Look at the WINDOW TURN RATE using MONITOR DISK (I think). The window
turn rate should be zero --- if it's not then the files are very fragmented.
Note that when I say zero I mean just that (0.0). If it's 0.01 you've
got a fragmented disk. Leave MONITOR running for a couple of days before
making a decision.
I entered MONITOR FCP (Yes, I read your correction), because my
computer often complains about "page file badly fragmented", and I
thought this might have something to do with it. It has been running
for a couple of hours now, and the current reading is
CUR AVE MIN MAX
Window Turn Rate 0.00 0.00 0.00 1.93
I'm not sure if you meant the average or the maximum value, but if you
did mean the MAX value, I seem to be in trouble. What can I do about
it?
/Mats.
--
-----------------------------------------------------------------------------
Mats Lofdahl, Stockholm Observatory, S-133 36 Saltsjobaden | +46 - 8 16 44 75
-----------------------------------------------------------------------------
Internet: lof...@astro.su.se | Bitnet: grodan@sekth | Sunet: royacs::lofdahl
-----------------------------------------------------------------------------
Ed Wilts
> How do you measure disk fragmentation, with out having to buy software from
> someone? Or what do you look for?
>
Send a mail message to VMS...@FHCRCVAX.Bitnet with the line:
SEND FIND.PACKAGE
When you have built the package (you'll need a C compiler), do the following:
$ FIND DUAn: /EXTENT=20/DISPLAY=(USED,ALLOC,EXTENT)
to display all files with at least 20 extents. Obviously, you'll need to be
privileged (at least read access to INDEXF.SYS).
You can also search for files with multiple headers as well as a whole slew of
other options. Once you have found the files, you can then determine how
important it is to you that they're fragmented.
A really nice way to use this utility is to search for all directories that are
over 127 blocks (ie. those that aren't cached).
FIND DUAn: /NAME=*.DIR;1/ALLOC=127/MATCH=AND/DISPL=(USED,ALLOC)
Many thanks to Joe Meadows for writing this excellent utility.
--
.../Ed Preferrred: Ed.W...@BSC.Galaxy.BCSystems.Gov.BC.CA
Ed Wilts Alternate: EdW...@BCSC02.BITNET (604) 389-3430
B.C. Systems Corp., 4000 Seymour Place, Victoria, B.C., Canada, V8X 4S8
Phil Daly, JAC, 665 Komohana St, Hilo HI 96720, USA
The original message I saw had a subject field and text
> Subject: Fragmentation, how do you measure it without buying something?
> How do you measure disk fragmentation, without having to buy
> software from someone? Or what do you look for?
Clearly this refers to disk fragmentation and my posting stands (i.e.
look at the window turn rate and backup/restore the volume to defragment it).
A poster then replied with
> I entered MONITOR FCP (Yes, I read your correction), because my
> computer often complains about "page file badly fragmented", and I
> thought this might have something to do with it. It has been running
> for a couple of hours now, and the current reading is ...
Clearly this refers to pagefile fragmentation which has nothing to do with
disk fragmentation.
The pagefile can be thought of as a `virtual disk' and this can become
fragmented when too much activity occurs whether or not the disk is itself
fragmented. Lessen the activity or increase the size of your `virtual disk'
will cure the problem. So will a reboot as this `virtual disk' gets
`initialized' after every reboot (hence there is no fragmentation if the file
is dumped after a reboot). Installing a secondary pagefile has the same
effect.
This will also do as a reply to the poster who posted
> Subject: PAGEFILE.SYS badly fragmented - help
Ehud Gavron misunderstood me, so I guess others did too.
Wittwer Fritz
It gives you a list of all fragmented files, including the number of fragments
and other directoryinformation of every file. There is nod defragmentation tool
in it.
You may use SWING (another shareware) to defragment single files, but at your
own risk.
I use this tool to check my disks since two yars.
If you have interest in it, i may look where it is and mail it to you.
Fritz
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Fritz Wittwer Phone: + 41 31 63 42 85 +
+ Ascom Tech AG Fax: + 41 31 55 52 11 +
+ Freiburgstrasse 370 E-Mail: wit...@tech.ascom.ch +
+ CH-3018 Bern 18 PSI-Mail: 02284643510211::Wittwer +
+ Switzerland (* Disclaimer: this may be a joke or not *) +
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--------------------------------------------------------------------
| Wittwer Fritz, Ascom Tech Ltd, Freibugstr. 370, CH-3018 Bern |
| Switzerland Phone + 41 31 63 42 85 FAX +41 31 55 52 11 |
--------------------------------------------------------------------
Ehud Gavron 602-885-7700x.2546
[Talking about the pagefile fragmentation]
#
#My suggestion: reboot first. If you start getting fragmentation problems
#again then you can only due one of two things:
#
#1) Backup and retore the disk using standalone backup (full defragments
# any volume);
#2) Run and online defragmentation package
Pagefile fragmentation refers to the internal blocks
allocated within the file. It does not refer to the
file structure on disk.
There is *NOTHING* you can do with BACKUP/RESTORE or
any package whatsoever to solve pagefile fragmentation.
THE ONLY THING TO DO is to increase the size of the
file, or add a secondary pagefile. In the former
case, you will need to reboot for this to take effect.
In the latter case, if the fragmentation is not too
bad then merely stopping processes using up the first
pagefile and then recreating them will help. (My
favorite to pick on are REMACP, NETACP, EVL, the
MultiNet_Server for TCP/IP in VMS, SYMBIONT_nnnn
for queues, and users who abuse the REPLY/ALL command :^)
#It may be that your CPU has been up and running too long.
This is not a documented restriction with VMS. THere
is no such thing as a "CPU been up and running too
long" error. If your pagefile is sized appropriately
then you should not get a page file fragmentation error
(or the more sever page file critical error).
#You can, of course, always add a secondary page file on another disk to
#balance the load.
You can of course realize that your problem is a pagefile
which is too small so you can solve it by
a. increasing the size of the current file
b. adding another file on the same disk
c. adding another file on a different disk
None of these are "better" choices than others -- depending
on the type of paging/swapping that goes on at your system.
# P N Daly Telephone : +1 (808) 969-6568
# Hilo HI 96720-6030 InterNET : PND @ JACH.HAWAII.EDU
Ehud
--
Ehud Gavron (EG76)
gav...@vesta.sunquest.com
I want to so HAPPY, the VEINS in my neck STAND OUT!!
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cool quotes at the end produced courtesy of Vnews 1.4
(anon ftp at arizona.edu, kudos to Joel M. Snyder)
Phil Daly, JAC, 665 Komohana St, Hilo HI 96720, USA
hold the file entirely then they need to be refreshed - a window turn.
Typically, the buffers can't map the file if the disk is fragmented
hence my reply. The `page file badly fragmented' can occur for a similar
reason but a simple reboot can cure this problem.
My suggestion: reboot first. If you start getting fragmentation problems
again then you can only due one of two things:
1) Backup and retore the disk using standalone backup (full defragments
any volume);
2) Run and online defragmentation package
It may be that your CPU has been up and running too long. You can, of
course, always add a secondary page file on another disk to balance the
load.
"Anything I say should be taken at your own risk and with a hefty pinch of salt"
P N Daly Telephone : +1 (808) 969-6568
Joint Astronomy Centre Facsimile : +1 (808) 961-6516
665 Komohana Street SPAN : 19527::54273::PND
Hilo HI 96720-6030 InterNET : PND @ JACH.HAWAII.EDU
U S A JANET : PND @ UK.AC.ROE.STARLINK
Terry Kennedy, Operations Mgr.
> I entered MONITOR FCP (Yes, I read your correction), because my
> computer often complains about "page file badly fragmented", and I
> thought this might have something to do with it. It has been running
> for a couple of hours now, and the current reading is
That's a different kind of fragmentation. The system is telling you that
the page file has very few contiguous areas of free space, and that it is
spending most of it's time shuffling the file.
There are some known VMS problems which can lead to this problem in un-
usual circumstances, as well as being caused by a mis-tuned system. The
simplest thing to do is to increase the size of the file. Use the SYS$UP-
DATE:SWAPFILES command procedure to change the sizes, or use AUTOGEN if
you prefer. Be sure to heed the warning about not deleting the old files
before rebooting!
Terry Kennedy Operations Manager, Academic Computing
te...@spcvxa.bitnet St. Peter's College, US
te...@spcvxa.spc.edu (201) 915-9381
j...@cmkrnl.uucp
JAC, 665 Komohana St, Hilo HI 96720, USA) writes:
> Look at the WINDOW TURN RATE using MONITOR FCP . The window
> turn rate should be zero --- if it's not then the files are very fragmented.
> Note that when I say zero I mean just that (0.0). If it's 0.01 you've
> got a fragmented disk. Leave MONITOR running for a couple of days before
> making a decision.
Er. A window turn rate of >0 indeed indicates fragmented files. But you can
have files that are fragmented sufficiently to cause performance problems,
while still seeing a window turn rate of 0.
Window turns indicate those transfers which can't be mapped in the current
window. But the current window can map several extents (discontiguous pieces)
of a file. Window turns indicate only that the file is sufficiently fragmented
that all of the extents that are needed won't fit in the mapping window at
once.
A more sensitive indicator is the "split transfer rate" in the MONITOR IO
display. This shows transfers that start within one extent and end in
another-- even if they don't involve fetching more mapping pointers. These are
not quite as expensive as window turns but they still hurt.
Window turns don't always imply split transfers, either - if you need a chunk
of a file that's all within one extent, but the extent is not mapped in the
current window, you get a window turn but you don't get a split transfer. So
window turns are not a subset of split transfers.
And of course MONITOR doesn't tell you which disks the window turns and split
transfers are occuring on, either.
That said, these two numbers are the best tools VMS gives you for determining
if fragmented files are causing performance problems. I would amend the advice
to leave MONITOR running for a couple of days, and say that you should leave it
running but into a recording file and then only look at your primary usage
periods; otherwise, 8 hours a day of no activity is going to skew the averages.
Include the IO class in the data collection and you may have some hope of
figuring out which disks the split I/Os and window turns are happening on.
There are programs available from DECUS which will look at the files on your
disk and tell you how many are fragmented how badly, and others which will tell
you how badly the free space on the disk is fragmented. At worst you can look
at a few files with DUMP/HEADER to see how many extents they have. But this
sort of "static" information is not particularly informative, because it
doesn't tell you whether the fragments are actually causing performance
problems. SYS$SYSTEM:SYS.MAP could be "maximally fragmented" at one disk
cluster per extent for all I care; I don't think I've looked at it for months.
This is an extreme case but it is not at all irrelevant -- I know some sites
(mostly using their systems for software developement, it's all compile, link,
and debug) that found *no* perceivable performance gains after completely
defragging all of their disks.
Many of the companies that sell defraggers offer free or very-low-cost trial
versions of their software, and their software usually includes some better
monitoring utilities than VMS offers. There is also a product from Touch
Technologies (619-455-7404) called I/O Plus which is *strictly* a file
performance monitoring tool, allowing you to identify "problem files". Many
sites don't need to defragment every file they have plus all of the free space
on every disk, they only need to make a few key files contiguous, and once
these files are identified they can be fixed up manually with CONVERT and/or
COPY/CONTIGUOUS. I'm pretty sure that Touch offers a trial version of this
product.
(Disclaimer: I do some work for Touch Technologies. I don't have anything to
do with I/O Plus, though.)
--- Jamie Hanrahan, Kernel Mode Consulting, San Diego CA
Chair, VMS Internals Working Group, U.S. DECUS VAX Systems SIG
Internet: j...@dcs.simpact.com, hanr...@eisner.decus.org, or j...@crash.cts.com
Uucp: ...{crash,eisner,uunet}!cmkrnl!jeh
weis...@dfwdsr.sinet.slb.com
> I entered MONITOR FCP (Yes, I read your correction), because my
> computer often complains about "page file badly fragmented", and I
> thought this might have something to do with it.
> -----------------------------------------------------------------------------
> Mats Lofdahl, Stockholm Observatory, S-133 36 Saltsjobaden | +46 - 8 16 44 75
> -----------------------------------------------------------------------------
> Internet: lof...@astro.su.se | Bitnet: grodan@sekth | Sunet: royacs::lofdahl
> -----------------------------------------------------------------------------
Messages about "page file badly fragmented" have nothing to do with over-all
disk fragmentation. Rather, it is the system telling you that your pagefile
needs to be increased in size because there are not enough contiguous blocks
available WITHIN the file to satisfy its needs.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Bill Weissborn
Schlumberger Well Services
Data Services Region -- System Support Staff Engineer
12770 Coit Rd. Suite 200 weissborn%dfw...@hub.sinet.slb.com
Dallas, Tx. 75251 or try: weissborn%ntc...@hub.sinet.slb.com
(214) 980-7924
"I got to play baseball...I got a chance to hit..." Ted Williams
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
James A. Harvey
>A file is cached into seven buffers and if these seven buffers cannot
>hold the file entirely then they need to be refreshed - a window turn.
>Typically, the buffers can't map the file if the disk is fragmented
>hence my reply. The `page file badly fragmented' can occur for a similar
>reason but a simple reboot can cure this problem.
NO NO NO NO NO! The "page file badly fragmented, system continuing" and
"page file space critical, system trying to continue" messages have
ABSOLUTELY NOTHING TO DO WITH FRAGMENTED DISKS. When the page file becomes
nearly full, the algorithm that the system for page file space allocation
results in the allocated bits and pieces becoming fragmented. Hence, it's
the page file SPACE that's fragmented, not the disk space belonging to the
file. You can even get the message when using a contiguous page file,
because it has nothing to do with disk space allocation for the file itself.
I really wish Digital would change this message to "page file SPACE is badly
fragmented" so that it would confuse fewer novice system managers (maybe).
Seeing these messages means that you are running out of page file space.
You need to add more page file space by either increasing the size of your
present page file(s) or adding one or more new page files on other disks.
Rebooting doesn't "cure" the problem, any more than shooting a screaming
patient in the head "cures" them of their illness (it does makes them stop
screaming though :-). Occasionally you may be faced with a situation where
the reason your page file has filled up is because of an abnormal load and
you would rather take action to avoid such a load again, instead of adding
more page file space (like when an instructor tells a class of 50 students
to type "MACSYMA" and press return). Your experiences may vary.
There's a real good source for answers to these types of "what does this
message mean" questions. It's called VMS System Messages and Recovery
Procedures, Parts I and II. Quoting from page 197 of Volume II of the FM:
PAGECRIT, page file space critical, system trying to continue
Facility: SYSTEM, VMS System Services
Explanation: The system is running out of page file space. This message
is more critical than the PAGEFRAG message.
User Action: Create a new page file with more space. See the Guide to
Setting Up a VMS System.
PAGEFRAG, page file badly fragmented, system continuing
Facility: SYSTEM, VMS System Services
Explanation: The system is running out of page file space.
User Action: Create a new page file with more space. See the Guide to
Setting Up a VMS System.
...............
...
Concerning the totally unrelated question about how one can determine how
badly fragmented one's disks are, most commercial producers of VMS disk
defragmentation products happily give away (or nearly) disk analysis programs
that will show you how awfully fragmented your disks are. It helps to sell
the merchandise. Sometimes a modest investment for a demo version of their
software will suffice.
Jim Harvey, IUPUI Computing Services, ija...@iupui.edu or har...@indiana.edu
BITNET: IJAH400@INDYVAX UUCP: {...backbones...}!iuvax!iugate!harvey
"Pleeze spel 'mnemonic' corectly"
d...@ttisms.uucp
> computer often complains about "page file badly fragmented", and I
Hi,
"Page file badly fragmented" has nothing to do with a disk drive being
fragmented. This message happens when the in-memory bit map for a
pagefile is fragmented. Normally this happens if your page file(s) are
too small. During peak times *each* of your page files should be about
50% free (not "reserved", but "free").
When your page file bit-map gets badly fragmented, the system will be
*very* sluggish...and if the problem gets worse, the system will
deadlock (stop!!).
Dan E.
j...@cmkrnl.uucp
> hold the file entirely then they need to be refreshed - a window turn.
No. The file's *mapping pointers* -- the values that relate ranges of virtual
block numbers within the file to logical block numbers on the disk -- are
stored in a structure, the window control block (WCB), that can hold seven
entries, ie, seven extents (it's seven by default; more on this later). Your
choice of language implies that the file's contents are cached and this is not
the case.
Furthermore, fragmentation can hurt you even if you don't get window turns. A
read or write that starts near the end of one extent and ends in another
extent requires extra head movement and rotational delay to get from one piece
of the file to the other. This is not as costly as a window turn but it is
obviously more costly than an operation that can be completed all within a
logically contiguous range of blocks.
> My suggestion: reboot first.
rebooting will not help disk file fragmentation problems.
Tom B. O'Toole
> CUR AVE MIN MAX
> Window Turn Rate 0.00 0.00 0.00 1.93
>
>I'm not sure if you meant the average or the maximum value, but if you
>did mean the MAX value, I seem to be in trouble. What can I do about
>it?
some misleading information posted... For one thing, someone made a reference
to 'seven buffers used to map the file'. Actually, there are seven (by
default, controlled by sysgen parameter ACP_WINDOW) pointers maintained
by the file system which point to file extents (contiguous chucks of the
file). These are used by the IO subsystem to map a virtual I/O operation
to a logical/physical one. The effects of this mapping is to take what looked
to the user like a continuous stream of blocks and map them to possibly
scattered areas all over the disk. This mapping, once set up when the file
is opened, can be done without file system intervention, IF the virtual block
can be mapped into one of the seven extents referenced by the pointers.
Otherwise, the pointers have to be updated to point to some different extents,
and the file system has to be brought into the matter. This is your window
turn, and as you can imagine, operating on fragmented files with MANY extents
cause lots of window turns. Your window turn rate is pretty low, given a
moderate level of file activity. Note that these extent pointers have nothing
at all to do with I/O or RMS buffers.
Another thing, the problem you are having with your pagefile has nothing at
all to do with file fragmentation, but with internal fragmentation within the
pagefile. The best solution might be to add another pagefile, or increase the
size of the one you are currently using. You may have to reboot to clear the
current condition. You may also want to take a look at your workload and see if
users are running several large-memory jobs concurrently. What version of VMS
are you running. There is a problem with 5.0 and 5.1 (I think it's fixed in
5.2), where you WILL eventually run out of space in your pagefile. In this case
the person who said 'maybe your system has been up too long' is actually
right!
--
Tom O'Toole - ecf_...@jhuvms.bitnet - JHUVMS system programmer
Homewood Computing Facilities, Johns Hopkins University, Balto. Md. 21218
ease!Trim!eeeaaaassse!trimtrimtrimeeeeeeaaaaassetrimease!trim!ease!trimeaase
Randy V Gritter
I have been having a problem getting the Oracle Export Utility and the VMS
Backup program to write to the same tape. Basically what happens is if I put
the backup save set on the tape first EXP gives me a device positioning error
and if I put the Export on first BACKUP says the tape is not it ANSI format.
I have tried a boatload of other things which I won't go into here but I would
appreciate any ideas you folks might have.
Randy
PS Reply by email if possible as I don't always have time to read news.
Stephen Tihor
Does Oracle export?
Is your tape drive one that properly supports ANSI format tapes with all the
positioning forward and backwards that can imply (some 8mm drives need
special configuration settings to do this well or at all.)
David Hazledine
> I have been having a problem getting the Oracle Export Utility and the VMS
> Backup program to write to the same tape. Basically what happens is if I put
> the backup save set on the tape first EXP gives me a device positioning error
> and if I put the Export on first BACKUP says the tape is not it ANSI format.
> I have tried a boatload of other things which I won't go into here but I would
> appreciate any ideas you folks might have.
To the best of my knowledge VMS BACKUP writes an ANSI labelled tape whereas
Oracle's EXPORT utility writes an unlabelled tape (starting at the beginning). I
don't think it is possible to mix'n'match these two on the same tape.
--------------------------------------------------------------------------------
David Hazledine EMBL Data Library
Database Administrator PF 10.2209
EMBL Data Library 6900 Heidelberg, Germany
Internet: Hazl...@EMBL-Heidelberg.DE
--------------------------------------------------------------------------------
Jerry Weiss
>Randy Gritter writes:
>
>> I have been having a problem getting the Oracle Export Utility and the VMS
>> Backup program to write to the same tape. Basically what happens is if I put
>> the backup save set on the tape first EXP gives me a device positioning error
>> and if I put the Export on first BACKUP says the tape is not it ANSI format.
>> I have tried a boatload of other things which I won't go into here but I would
>> appreciate any ideas you folks might have.
>
>To the best of my knowledge VMS BACKUP writes an ANSI labelled tape whereas
>Oracle's EXPORT utility writes an unlabelled tape (starting at the beginning). I
>don't think it is possible to mix'n'match these two on the same tape.
>
I did this regularly under VMS 4.x and Oracle V5.x. When I incremented
both versions something changed and I also had some difficulty. If I
recall correctly you have to look more carefully into the export options
under Oracle V6.x. You have to use the command line method and NOT
the interactive method (sic).
Jerry
--
Jerry S. Weiss Dept. Medicine, Northwestern Univ. Medical
jweiss @casbah.acns.nwu "When the goings get tough, the tough Beam UP!"