DECUServe Journal March 1995
Brian McMahon, Info-VAX Refugee
=====================
March, 1995
From the Editor's Keyboard
==========================
March, which certainly came in like a lion and is showing signs of
going out like a lamb as scheduled, spent the middle part trying to
make up its mind one way or the other. This edition of the Journal
also seemed to be stuck on dead center at times, though that had
more to do with a particularly enthusiastic flu bug that's been
going around our community than anything else. (But we sure are
looking forward to spring.)
Looking ahead a bit, we expect the April issue to be much like this
one, a general survey issue, but around May or June, we'd like to
run another special-topic issue. The "mature hardware" issue last
December seemed to go over well, so we thought we'd tap in to
another area of DECUServe expertise. We haven't settled firmly on
this yet, but we're seriously considering the virtues of an issue
dedicated to DEC Notes setup and maintenance. (DECUServe may well
be the best publicly-available resource on Notes, after all.)
As always, your comments, feedback, and suggestions will be warmly
welcomed. Contact information for the editors is at the end of each
issue.
The DECUServe Journal March, 1995 Page 2
Table of Contents
Table of Contents
=================
CONTENTS
From the Editor's Keyboard . . . . . . . . . . . . . 1
Table of Contents . . . . . . . . . . . . . . . . . 2
News Performance on VMS . . . . . . . . . . . . . . 3
How to Do 24x7 . . . . . . . . . . . . . . . . . . 25
Phone Line Trouble . . . . . . . . . . . . . . . . 31
How Many Users? . . . . . . . . . . . . . . . . . 36
Frame Relay Feedback . . . . . . . . . . . . . . . 40
Text from TPU to Micro . . . . . . . . . . . . . . 42
DECterm Timeout Problem . . . . . . . . . . . . . 46
What Will Get Purged? . . . . . . . . . . . . . . 49
Holding Jobs -- Why? . . . . . . . . . . . . . . . 55
Can't See Added Memory . . . . . . . . . . . . . . 57
Mixed-architecture Cluster . . . . . . . . . . . . 59
About the DECUServe Journal . . . . . . . . . . . 60
Contact Information . . . . . . . . . . . . . . . 61
The DECUServe Journal March, 1995 Page 3
News Performance on VMS
News Performance on VMS
=======================
DECUServe, of course, has system management "experiences" of its
own. In fact, most tasks are performed remotely by various
volunteers. In late January to early February, the high volume of
incoming network news caused serious problems. DECUServe users
being the way they are, the original discussion of the trouble and
the effort to resolve it in the DECUSERVE_FORUM conference spun off
a long and informative thread in the INTERNETWORKING conference,
discussing the problems (is "challenges" a better word?) of a
netnews implementation on VMS.
This thread is a bit longer than we usually run, but we think it's
worth it. The discussion that follows touches on UNIX, RMS,
ALL-IN-1, e-mail, design philosophy, and more. No matter where
you're coming from, there's a decent chance that you'll find
something of interest here.
Participants: John Briggs, Dale Coy, Mike Durkin, Jamie Hanrahan,
Bob Hassinger, Brent Hutto, Terry Kennedy, David Mischler, Pat
Scopelliti, Mark Shumaker, Brian Tillman, Don Vickers.
Note 312.0, 9-Feb-1995
Hassinger: NEWS on VMS systems
------------------------------
I posted the following in another conference as part of a discussion of
the problems we are having here on DECUServe with Usenet News
performance. Dale Coy suggested starting a technical discussion in this
conference.
The subject of processing needs for Usenet News and its various
implementations seems interesting. There is a particularly
interesting aspect too in that some suggest the file systems under
some other operating systems are better suited to the needs of News
than is VMS and its file system.
<<< EISNER::$2$DIA7:[NOTES$HIVOL]DECUSERVE_FORUM.NOTE;1 >>>
-< DECUServe_Forum >-
================================================================================
Note 385.95 NEWS or NEWS READER problems 95 of 105
EISNER::HASSINGER "Bob Hassinger" 9 lines 7-FEB-1995 22:43
-< Does News not like VMS? Why? >-
--------------------------------------------------------------------------------
Is there a technical discussion anywhere here on DECUServe about the
issues involved in News performance on VMS? Particularly as contrasted
with other operating systems and their file systems?
I have never looked in any detail into what is going on and why News
seems to have so much trouble on VMS systems. I am told news
applications don't have as much difficulty when implemented on some
other systems. If there is any truth in that I think it would make an
interesting technical discussion.
The DECUServe Journal March, 1995 Page 4
News Performance on VMS
Note 312.1, 9-Feb-1995
Kennedy: Summary
----------------
I thought I discussed this before, but it may have been in the ANU News
newsgroup.
> I have never looked in any detail into what is going on and why News
> seems to have so much trouble on VMS systems. I am told news
> applications don't have as much difficulty when implemented on some
> other systems. If there is any truth in that I think it would make an
> interesting technical discussion.
The way the Unix news packages work is to have lots of little files and a
single database file to track things. The only VMS server implementation
(ANU News) copied this model. However, the VMS filesystem isn't particularly
happy about lots of small files. We also run into problems with RMS indexed
files with lots of insert/delete activity. VMS does both of these things
slower than the Unix systems the design came from. Let's not turn this into
a VMS-vs-Unix thing, but just note that sometimes VMS's added features aren't
needed by some applications.
Correcting this requires either a new model for storing news or enhancing
the performance of the VMS features that News needs. The new file system
might help with the latter. For the former, a number of methods have been
proposed and none of them are particularly attractive. I've heard of text
libraries, implementing a Unix-like filesystem on a raw disk, and a bunch
of others.
However, there is a serious shortage of people developing for VMS - I think
there are only a handful of folks working with ANU News, none of which have
the time for this sort of project. It's the same problem I had with VMS C-
Kermit - lots of users, no developers.
Note 312.2, 9-Feb-1995
Coy: What direction to look, first?
-----------------------------------
Yes, this isn't a "VMS versus Unix" thing, at all. It's somewhat a
matter of how a _particular_ implementation that was chosen for putting
News on VMS platforms, has some unfortunate problems.
To maintain the paradigm of News, it's necessary that each posting have
a unique identification across *ALL* newsgroups. [That is, if
something is posted to multiple groups, it is identifiable as the same
item so that you don't have to read it more than once -- unlike Notes].
There are other key differences that will probably emerge from this
topic.
Anyhow, a "natural" implementation for News (where "natural" means that
it's one that would come easily to mind) would be to implement the
The DECUServe Journal March, 1995 Page 5
News Performance on VMS
newsgroup heirarchy as directories and subdirectories:
[COMP]
[COMP.ORG]
[COMP.ORG.DECUS]
...
[COMP.SYS]
[COMP.SYS.WHATEVER]
...
with individual files for the individual postings -- and with another
set of "control" files to keep track of the other necessary
information.
That means that there are a (relatively) few directories, and many of
the directories contain hundreds or thousands of files, and lots and
lots of additions/deletions are done every hour.
The files are _usually_ very small, but _some_ are quite large (even as
large as megabytes, for some groups).
What does that mean? For one thing, it means fragmentation.
======================
What filenames? Well, a "natural" thing would be to make the file name
the same as the "item number" in the local newsgroup. Now, that's just
an arbitrary number, but it's useful [more discussion topics here].
So, the files get names assigned as they are added to the system.
1234.ITM
1235.ITM
1236.ITM
This has implications for performance on the VMS file system. For one
thing, having all of the "current" filenames very similar to each other
results in slow lookup. [more discussion topics here]
Note 312.3, 9-Feb-1995
Tillman: MX solved this, albeit on a smaller scale
--------------------------------------------------
The SMTP package MX has to consider this situation as well, but on a
somewhat smaller scale. The answer was to create subdirectories for
items based on the last digit of the message number (queue entry) so
that no one directory received tons of entries while others were
sparsely populated. The second thing that was done was to change the
control file structure from indexed to sequential with a fixed record
size. This allowed the benefits of direct access by key (the message
number), but without the overhead of an index and the need to
periodically reclaim empty buckets. The speed-up in message handling
was dramatic.
The DECUServe Journal March, 1995 Page 6
News Performance on VMS
Note 312.4, 9-Feb-1995
Coy: Understand before solving
------------------------------
I believe that all of the problems are solvable, if someone were doing
a lot of work on News implementations for VMS. [Although I also
suspect that the solutions to some problems would make other problems
worse...]
However, for very good reasons, there isn't much development work going
on for News on VMS. [This does not have to do with the "rumored death
of VMS", at all -- it's just more cost-effective to use an existing
implementation for another platform as a server. VMS clients are
another thing, entirely]
Bob's question, though, was "why are there problems with the existing
implementation". I believe that it is an excellent question.
Note 312.5, 9-Feb-1995
Hassinger: Parallels with ALL-IN-1?
------------------------------------
> The SMTP package MX has to consider this situation as well, but on a
> somewhat smaller scale. The answer was to create subdirectories for
> items based on the last digit of the message number (queue entry) so
> that no one directory received tons of entries while others were
> sparsely populated.
That sounds at least somewhat similar to something ALL-IN-1 does.
There may be some interesting parallels in the design issues and
solutions. At least ALL-IN-1 _has_ had a good bit of design work and
study to try to get good performance on VMS systems.
Note 312.6, 9-Feb-1995
Hassinger: Background on the UNIX file system features in question?
-------------------------------------------------------------------
> The way the Unix news packages work is to have lots of little files and a
>single database file to track things. The only VMS server implementation
>(ANU News) copied this model. However, the VMS filesystem isn't particularly
>happy about lots of small files. We also run into problems with RMS indexed
>files with lots of insert/delete activity. VMS does both of these things
>slower than the Unix systems the design came from.
I presume the basic requirements for News processing derive from what
looked like viable designs from the point of view of UNIX. That is,
the people designing News said how can I design it so I will be able to
make it work well on my UNIX system. Right?
So, knowing very little about UNIX file systems, how do they differ
from VMS WRT the way they can support lots of little sequential files?
Also, on a UNIX system how is the indexed file function mentioned above
The DECUServe Journal March, 1995 Page 7
News Performance on VMS
handled? I was under the impression that vanilla UNIX systems
typically did not come with a standard indexed file system, and there
was no one really standard add-on indexed file system. Am I wrong, or
is some other sort of design used that does not call for an indexed
file capability.
Note 312.7, 9-Feb-1995
Coy: I suspect it just happened
-------------------------------
> I presume the basic requirements for News processing derive from what
> looked like viable designs from the point of view of UNIX. That is,
> the people designing News said how can I design it so I will be able to
> make it work well on my UNIX system. Right?
I don't think so (in terms of _basic_ requirements). NEWS (as one of
the two fundamental types of conferencing systems) has requirements.
Implementing it with a per-message transport imposes other
requirements. Those actually are independent of the operating system.
Given a "modern" file architecture that is heirarchical, the natural
approach was taken. On UNIX, it happened to work well.
> So, knowing very little about UNIX file systems, how do they differ
> from VMS WRT the way they can support lots of little sequential files?
I'll wait for an expert to give a good answer. The "short" answer is
that "all files are alike".
Note 312.8, 9-Feb-1995
Kennedy:
---------
> I presume the basic requirements for News processing derive from what
> looked like viable designs from the point of view of UNIX. That is,
> the people designing News said how can I design it so I will be able to
> make it work well on my UNIX system. Right?
I think they just took the obvious route. The current news servers (including
ANU) were designed when a "big news day" was thousands or some small tens of
thousands of postings. Just about any storage methodology (possibly excluding
3 x 5 cards 8-) would have worked then.
> So, knowing very little about UNIX file systems, how do they differ
> from VMS WRT the way they can support lots of little sequential files?
The limit to I/O performance with lots of small files on a Unix system is
generally the seek times of the disk(s) involved. VMS imposes additional
layers between the driver and the application which News processing doesn't
want. To demonstrate this, try the IOZONE benchmark on a Unix box and a VMS
system. It just writes sequential files and times the operation. Perhaps the
new file system will help VMS - Digital certainly has a lot of effort going
The DECUServe Journal March, 1995 Page 8
News Performance on VMS
into it.
> Also, on a UNIX system how is the indexed file function mentioned above
> handled? I was under the impression that vanilla UNIX systems
> typically did not come with a standard indexed file system, and there
> was no one really standard add-on indexed file system. Am I wrong, or
> is some other sort of design used that does not call for an indexed
> file capability.
At least on BSD-ish systems, there are 3 common "culturally compatible"
indexed access methods, dbm, db, and dbz. dbz is a subset of db designed for
access to news article information (the features of db that weren't needed
were omitted) and the remaining pieces were optimized for the things news
did.
These generally work by having a text file containing the data, and one or
more files (often with the suffixes .dir and .pag for one implementation, or
_.db for another) which contain hashed pointers to records and/or subrecords
in the text file, which are accessed by lseek().
This brings up another VMS issue - the C RTL isn't particularly good at
I/O. That may account for at least part of the problem.
Note 312.9, 9-Feb-1995
Hanrahan: Musings on news item text storage
-------------------------------------------
Does anyone have any idea as to where the time is really going? ie if we were
attacking this piecemeal, would it be better to devise a better scheme for item
text storage, or redo the index files first?
One of the things NEWS has to do regularly is DELETE a lot of those item text
files. On a typical news directory this takes my VS 3100/76 about a second per
file. (Note that we are generally deleting the oldest files, which means
deleting the first files in the directory in lexical order, which is the worst
case for VMS!) There are only 86,400 seconds in a day, so it's a good thing
that I'm not getting anywhere close to a full feed. As it is, the nightly
skim takes about five or six hours.
Text libraries are out, as they don't allow shared access -- you'd have to hide
them behind a server process that single-threads all the accesses (more
coding). They're also moderately slow, and they require period compression.
Actually, ANY scheme other than individual item text files requires periodic
compression, or reorganization, or whatever, to recover disk space under
some conditions. Suppose you have some sort of container file (indexed file,
text library, whatever) per newsgroup. It's a high-volume group and you're
running low on disk space, so you cut its retention time from 10 to 5 days.
Now in the current scheme, you get back half the space the group's items were
using with the next nightly skim. But with any sort of "container", you have
to do something else to make the container release its now-unused buckets
back to the free space on the volume.
The DECUServe Journal March, 1995 Page 9
News Performance on VMS
ANU actually did not inherit one of the characteristics of the Unix
design: On Unix, crossposted items are represented by links, while on VMS,
crossposted items simply appear as duplicate .ITM files. I don't know how much
of a space penalty this actually incurs.
My friend Dan E. reports that indexed files with variable-length records are
actually a pretty good way to store text on VMS. You could have one such file
per newsgroup, with the keys being the item number and line number. Not unlike
NOTES, in fact! Now the prospect of re-orging thousands of these files every
night may seem daunting, but wait: There would probably be no need to re-org
them all every night; it could be done on a rotating basis.
But If I were tackling this project I'd rather go with a special-format
container file, probably using chains of relative block numbers or some such,
optimized for fast additions and deletions rather than fast access to the text.
After all, most of the text is never read...
The alternative to one container file per newsgroup seems to be to use a very
small number of large container files. Note that any reasonable implementation
would allow for more than one, to allow the news data to be spread among
multiple disks.
Chuck von Rospach once suggested that there was no real need to do anything
with a news batch in terms of creating individual item files. He argued, why
not just leave the batch as it is? The index files would simply contain
file names and offsets into the various batch files. You would keep a
"reference count" for each batch file indicating how many news item index
entries pointed to it; delete the batch when the count drops to zero. It
seems to me that this would work fine as long as all articles on a system
expired at about the same rate, but not so well if you had batches containing
items for newsgroups with widely varying expiry periods -- in effect all groups
would expire as slowly as the slowest ones.
Note 312.10, 10-Feb-1995
Hassinger:
-----------
> One of the things NEWS has to do regularly is DELETE a lot of those item text
> files. On a typical news directory this takes my VS 3100/76 about a second
per
> file. (Note that we are generally deleting the oldest files, which means
> deleting the first files in the directory in lexical order, which is the worst
> case for VMS!)
Can someone quantify further this issue of the
effort/time/resources/whatever it takes to delete the first file in a
directory as compared with say, the last file?
If it is a really significant difference in typical News cases, then
perhaps there is a fairly simple hack to just alter the file naming
scheme so files end up at the beginning of the directory and the oldest
ones are at the end?
The DECUServe Journal March, 1995 Page 10
News Performance on VMS
Of course, the flip side of this is to compare delete performance with
add performance. Do you lose on the add side when you gain on the
delete side and do they take similar amounts of effort (e.g. the change
would be a wash)?
Note 312.11, 10-Feb-1995
Coy: No free lunch
------------------
> Can someone quantify further this issue of the
> effort/time/resources/whatever it takes to delete the first file in a
> directory as compared with say, the last file?
Let's assume that the directory contains 2 files: 1233.ITM and 1234.ITM
In a nutshell, VMS keeps the names of the files in a contiguous "list"
in sorted order [broad general concept]. If something is added to the
end of the list, that's very easy (just extend the list). So, adding
1235.ITM to a list where the last thing is 1234.ITM takes hardly any
work.
If we delete 1233.ITM, all of the other things in the list have to
"move up one place". Of course, if we instead add 1232.ITM, then
everything in the list has to "move down one place".
[Having them in contiguous sorted order makes it really convenient to
do other things -- for instance, directory displays are always sorted
with no work done by the system.]
There is also a more-technical problem that occurs because of the way
VMS caches directories, associated with the large number of files that
may be in a single news directory *AND* the fact that the names of the
files are so nearly-identical. That one's really nasty and really
technical.
Note 312.12, 10-Feb-1995
Briggs: file deletion overhead by directory end
-----------------------------------------------
> Can someone quantify further this issue of the
> effort/time/resources/whatever it takes to delete the first file in a
> directory as compared with say, the last file?
For a file deletion that does not empty out a directory block, you do
one I/O to write the updated directory block. This is independent
of whether the file is at the beginning or end of the directory.
(One can assume that the directory block was cached during file lookup).
For a file deletion at the front of a directory that is n blocks in size
which empties the first block, you do between n and 2*n I/O
operations (depending on cache efficiency). The contents of all the
subsequent blocks are moved up by one block. This is done with
The DECUServe Journal March, 1995 Page 11
News Performance on VMS
block-at-a-time I/O. Thus, one read and one write per directory block.
The reads may be satisfied from cache. Plus, you've got to update the
directory's file header (one or two additional I/O's).
For a file deletion at the tail of a directory which empties the block,
all you've got to do is update the directory's file header (one or two
I/O's).
For the files typically used in NEWS, I think you can get about 20
files per directory block.
Thus, the penalty for front-end deletions is 1 or 2 times directory
size divided by 20 I/O's per file.
In addition to this I/O to update the bottom level directory, there
may be some I/O associated with file lookup. (One I/O for the file
header and one or more I/O's per directory level, all of which may
be satisfied from cache). And there will be one I/O to update the file
header to indicate that the file has been deleted. The updates to
the bitmap and index file bitmap are write-back cached and are likely
to be deferred.
Thus, the fixed I/O cost is 2 I/O's per file (assuming 100% efficient
directory cache and 0% efficient file header cache and ignoring
bitmap updates).
I think I've got this right. But I'm willing to accept corrections.
Note 312.13, 10-Feb-1995
Coy: Caching?
-------------
John, I think you've got it technically right EXCEPT for the REAL
situation with caching, in directories with lots of files with
very-similar names.
My ramblings were intended to be a "generic" description of the
situation. Thanks for doing the technical details.
There is an _excellent_ article on DSNlink about this particular
situation (with lots of files that all start with the same one or two
characters).
To give a "generic" description of the problem, which I'm sure somebody
will flesh out with details -- the caching hash algorithm uses some
number of characters from the beginning of the filename. If there are
only a few filenames in the directory, a large number of characters can
be used (memory fails me - 5? 6?). As the number of files in a
directory increases (the depth), the number of characters that can be
used in the hash algorithm is reduced. For reasonably large numbers of
files (such as seen in NEWS), only the first character of the filename
is used in the hash algorithm. That's fine if your files are named
AAAA and BBBB and CCCC. But if they are named 1234, 1235, 1236...
The DECUServe Journal March, 1995 Page 12
News Performance on VMS
So, you get a "hit" on the caching, too. Did I get that right?
Note 312.14, 10-Feb-1995
Kennedy: One idea I'd had in the past
-------------------------------------
> If we delete 1233.ITM, all of the other things in the list have to
> "move up one place". Of course, if we instead add 1232.ITM, then
> everything in the list has to "move down one place".
One quick fix might be to use versions instead of file names. I believe
VMS stores multiple versions in a more compact manner than unique file
names. Of course, this would prevent you from having more than 32K articles
in a newsgroup at any given time, and you'd have to keep a pointer to what
version was the "head of the list", since you'd be much better to wrap from
32K to 0 than trying to shuffle the versions around so that the highest
version was always the newest item.
This would also reduce the size of the directories enough that the could
probably be cached and also not suffer expansion problems.
Note 312.15, 10-Feb-1995
Hassinger:
-----------
How would that interact with the caching Dale is talking about?
Note 312.16, 13-Feb-1995
Briggs: Directory index cache
-----------------------------
Badly. The cache that Dale is talking about is the "Directory Index"
cache. This cache is used as a quick way to determine which directory
block contains the file you are interested in. It uses the first 15
characters of the file name as the index into the cache. If all files
use the same name, the first 15 characters will all be the same and
this cache will not be effective.
There are other caches from which the actual directory data is read
(the directory data cache, the RMS path cache and the RMS directory
cache). These caches are either made more effective or are unaffected
by Terry's scheme.
I found the cache information on DSNlink using keywords "DIRECTORY" and
"CACHE" in an article entitled "[OPENVMS] Commonly Asked Questions
About Directory (.DIR) Performance"
Terry's scheme would reduce directory size by a factor of three to 1
(assuming 8 byte file names and directories with at least 4 or 5 files).
This would seem to be a net win.
The DECUServe Journal March, 1995 Page 13
News Performance on VMS
Note 312.17, 13-Feb-1995
Hassinger:
-----------
Could a file name hashing algorithm that took, say the article number
Dale spoke of and uniquely turn it into a different form make all this
work significantly better? That is, for example, at all the points in
News where the file name is currently formed directly from the item
number, insert the hashing routine to transform it before it is used.
Clearly, don't know enough about the Directory Index cache scheme to
really understand the prospects for this idea (and my DSNlink access is
nearly non-existant to do research there), but it this sort of idea
seems to present an opportunity to control the file names so as to best
interact with VMS. Also obviously I have never looked in the News code
to know if it is reasonably feasible to find and modify the appropriate
points in this way, so I don't know at this point if it is reasonable
to suggest.
However, it seems interesting to ask the above question to see if a
significantly better file naming scheme can be designed. If so, then
it would be worth a look to see if it was reasonable to work it into
the News code.
Note 312.18, 13-Feb-1995
Coy: Yes and no
---------------
> Could a file name hashing algorithm that took, say the article number
> Dale spoke of and uniquely turn it into a different form make all this
> work significantly better? That is, for example, at all the points in
> News where the file name is currently formed directly from the item
> number, insert the hashing routine to transform it before it is used.
There could be some improvement in the "fit" with directory caching.
I don't see that there is any hope for the file add/delete problem.
That is -- anything that helps the efficiency of file deletes, hurts
the efficiency of file additions in equal measure. Unless you intend
to do more additions than deletions (or the reverse)...
And my belief is that the add/delete problem is the bigger difficulty.
Note 312.19, 13-Feb-1995
Durkin: Simplistic, but probably not viable
-------------------------------------------
Is there any merit in implementing a scheme in which the news
adds were spread across more than one spindle ? I'm assuming the
news feed software could be stopped, redirected to the next disk,
The DECUServe Journal March, 1995 Page 14
News Performance on VMS
and restarted with little or no detrimental effects. Of course this
requires the news reader(s) to be modified to handle the news items
spread across n spindles, which should be feasible using linked-list
logicals. This certainly implies a willingness to spend some money
on disk storage and software modifications in order to preserve the
VMS OS underneath it all. Ask yourself whether it will be worth it
all in the end.
Note 312.20, 13-Feb-1995
Briggs: More idle speculation on NEWS item storage
--------------------------------------------------
Assuming that directory access is, in fact, a bottleneck here, one
obvious solution would be to get rid of the directories:
Don't enter the item text files in any directory. Store file
id's in the news database.
Personally, I like Jamie's idea of roll-your-own management of large
container files. You end up implementing just what you need without
the baggage of a complete, general purpose file system.
Note 312.21, 14-Feb-1995
Hassinger: The ends and the middle?
-----------------------------------
> Personally, I like Jamie's idea of roll-your-own management of large
> container files. You end up implementing just what you need without
> the baggage of a complete, general purpose file system.
It sounds as if that might well turn out to be what it takes in the
end.
OTOH, On the helping adds hurts deletes in equal measure thread:
IF the effect is linear from one extreme to the other then yes, but...
I can believe that at the ends of the spectrum the effects are equal
and opposite. As I understand it we are pretty much at one end of the
spectrum - adding should be optimum and deleting is worst case.
The question that comes to mind is how it works out in the middle
between the two extremes. If you had something like a totally random
assignment of file name, would it turn out to be a wash as compared
with the extremes, or would there be a net gain? Sometime with this
sort of thing it turns out the extremes go down hill exponentially
rather than linearly, so that the middle is actually much better than
the average of the extremes.
For example, is the time to add or delete a file in the middle exactly
half the time at the worst case end and exactly twice the time at the
best case end?
The DECUServe Journal March, 1995 Page 15
News Performance on VMS
Note 312.22, 14-Feb-1995
Coy: Linear
-----------
> For example, is the time to add or delete a file in the middle exactly
> half the time at the worst case end and exactly twice the time at the
> best case end?
No, there is a "constant" and a variable in the equation. That is, it
always takes a certain amount of work, regardless of positioning.
However, with regard to the variable part -- on one end it is zero.
And the time in the middle is half that at the other end.
In other words -- it's linear.
Note 312.23
Shumaker: A modest proposal.
----------------------------
If the 'adds' can take place at any time during the day, but the 'deletes'
are normally done outside of prime time -- as in an overnight cleanup --
it seems to me that there would be a net advantage during peak-usage
times from favoring the 'adds'.
Hashing might not be necessary; how about just reversing the order of the
digits? Consecutively-added files "12345.NEWS" and "12346.NEWS" would
then become "54321.NEWS" and "64321.NEWS", and should be located relatively
far apart in the directory -- not perfect, probably not optimum, but should
be relatively easy to implement and easy for humans to read if need be.
Note 312.24, 14-Feb-1995
Coy: Which free lunch would you prefer?
---------------------------------------
> If the 'adds' can take place at any time during the day, but the 'deletes'
> are normally done outside of prime time -- as in an overnight cleanup --
> it seems to me that there would be a net advantage during peak-usage
> times from favoring the 'adds'.
Which is exactly what is done in the current implementation.
> Hashing might not be necessary; how about just reversing the order of the
> digits? Consecutively-added files "12345.NEWS" and "12346.NEWS" would
> then become "54321.NEWS" and "64321.NEWS", and should be located relatively
> far apart in the directory -- not perfect, probably not optimum, but should
Which would also make it NOT favor either adds or deletes.
The DECUServe Journal March, 1995 Page 16
News Performance on VMS
Note 312.25, 14-Feb-1995
Hanrahan:
----------
Creating the files directory-less would indeed help a lot as far as news itself
was concerned, but a disk with 100,000 or a million "lost files" would create
new problems. You'd never be able to run analyze/disk on it, for example...
Also, going to a container file approach would permit easy spreading of
the news item text over multiple disk spindles, without bothering with
multivolume sets or with stripe sets.
Note 312.26, 14-Feb-1995
Hassinger:
-----------
If the containers were, say one per news group per day it might work
out rather nicely in at least one respect. Typically I think the
granularity of the retention times is a day. So it might be possible
to reduce the deletes to just one container file per day per group. I
think there are complicating factors, but the basic idea is attractive.
I rather suspect there is a good bit more work in implementing the
containers than it would be to fiddle with name schemes and the like.
BTW - Is there any way to delete a full directory without having to go
through the file at a time process?
Note 312.27, 14-Feb-1995
Vickers: Don't try this at home, kids
-------------------------------------
Sure. Just do SET FILE/NODIRECTORY to the directory file and then
delete it. Two commands, no waiting. (-;
Of course, the files in the (former) directory are unaffected and lost
unless they had been entered as 'alias' entries in another directory.
As the help says:
SET
FILE
/NODIRECTORY
/NODIRECTORY
Use with extreme caution.
Removes the directory attributes of a file and allows you to
delete the corrupted directory file even if other files are
contained in the directory. When you delete a corrupted directory
file, the files contained within it are lost.
The DECUServe Journal March, 1995 Page 17
News Performance on VMS
Use ANALYZE/DISK_STRUCTURE/REPAIR to place the lost files in
[SYSLOST]. You can then copy the lost files to a new directory.
This qualifier is valid only for the Files-11 On-Disk Structure
Level 2 files. For more information about the Verify Utility, see
the VMS Analyze/Disk_Structure Utility Manual.
Note 312.28, 15-Feb-1995
Briggs: More arm chair designer's pipe dreams
---------------------------------------------
> If the containers where, say one per news group per day it might work
> out rather nicely in at least one respect. Typically I think the
> granularity of the retention times is a day. So it might be possible
> to reduce the deletes to just one container file per day per group. I
> think there are complicating factors, but the basic idea is attractive.
One complicating factor is the fact that individual articles can
specify an explicit expiration time. This is typically used for things
like FAQ's that are posted monthly. Another complication is cancels.
So you still need a way to delete individual articles out of a
container.
Larger container files would seem better -- if you have just a few
containers, they can all be kept permanently open. The news posting and
news deleting processes wouldn't have to bother with file opens and
closes. (I figure file lookup is the next bottleneck waiting after
all us smart arm-chair designers fix the file deletion problem.)
Note 312.29, 15-Feb-1995
Hassinger:
-----------
> One complicating factor is the fact that individual articles can
> specify an explicit expiration time. This is typically used for things
> like FAQ's that are posted monthly. Another complication is cancels.
That is the one I was thinking of. I so rarely use these features that
I don't know much about them. And when I was getting a News feed at my
site the cancels never seemed to work or make sense anyway.
Note 312.30, 15-Feb-1995
Hassinger:
-----------
> Sure. Just do SET FILE/NODIRECTORY to the directory file and then
> delete it. Two commands, no waiting. (-;
Not quite what I was thinking of ... ;-)
The DECUServe Journal March, 1995 Page 18
News Performance on VMS
Of course it is possible to organize a set of delete operations so they
are ordered in reverse lexical order. make of list of what you want to
delete then sort it in descending order, or start from the end of the
list. How hard would it be to get News to do this, and would it be a
significant gain?
I suppose the answer to that last part is really asking for some
measure of Dale's constant part vs the best and worst case variable
parts of the operation. If the difference between the best and worst
case part is relatively big compared to the constant part, then the
effort to delete in reverse lexical order might be a significant win.
Note 312.31, 15-Feb-1995
Coy: Minor
----------
> Of course it is possible to organize a set of delete operations so they
> are ordered in reverse lexical order. make of list of what you want to
> delete then sort it in descending order, or start from the end of the
> list. How hard would it be to get News to do this, and would it be a
> significant gain?
In the GENERAL case, that's a help. Not always a big win, but always a
win.
In the case of NEWS, unless some other measures were taken also, we are
faced with the problem of deleting "all of the stuff at the top". Now,
is a _small_ gain by deleting the top stuff from the "end of the
list". But not a big gain.
In any case, I thought the objective here was mostly to understand
what's going on -- not to redesign it. It is highly UNprobable that
anybody is going to do a lot of work to recode the NEWS server on VMS.
As far as I can tell, there isn't a *MUCH* better way of doing the file
stuff with VMS.
Note 312.32, 15-Feb-1995
Hassinger:
-----------
RENAME suffers from the same problems, right?
Note 312.33, 15-Feb-1995
Coy: Yep
--------
> RENAME suffers from the same problems, right?
Of course.
The DECUServe Journal March, 1995 Page 19
News Performance on VMS
Note 312.34, 16-Feb-1995
Hassinger:
-----------
So, we are being pretty successful at showing that what News needs is
hard for VMS to provide efficiently. I am not sure we have seen a good
comparison with alternative OSs that demonstrates what it is about them
that makes them inherently far better at providing the require
capabilities.
I am wondering if there are fundamental differences in design, or if it
is a question of better News implementations that do a better job of
working around fundamental problems that are common to all systems, or
maybe the other OSs just have things like bigger directory caches and
if so then is it a tuning issues for VMS or are there fundamental
limitations in VMS that make it impossible to provide big enough caches
or whatever - that sort of thing.
I suspect a lot of this is obvious to people who have worked more
closely with RMS and Files-11 internals - to the point where it is not
even apparent to them that the issues even need to be explained and
discussed. Some do need more background though. And I think with
that, then we need comparisons with the same factors on other OSs.
At some fundamental level any OS that has files, directories, and disk
I/O has to deal with the same set of issues and performance trade-offs.
But it has been suggested that some other OSs manage to do dramatically
better than VMS, even on significantly less well endowed hardware
platforms.
Where exactly does VMS go wrong? What is the fatal flaw?
Note 312.35, 16-Feb-1995
Coy: It's not wrong
-------------------
Whoa. Nobody said VMS went "wrong". Nobody said there is a "fatal
flaw".
Note 312.36, 16-Feb-1995
Hassinger: In _this case_, maybe yes, maybe no...
-------------------------------------------------
If "other systems" can do far better, reportedly with less resources,
then it would seem that in terms of the specific needs of News, either
the VMS implementation of News is significantly less efficient than the
implementations on those other systems, or VMS is not as "good" *WRT
the particular qualities Usenet news requires* in some important way.
The DECUServe Journal March, 1995 Page 20
News Performance on VMS
Note 312.37, 16-Feb-1995
Kennedy:
---------
See the "IOZONE" paragraph of .8 for starters. For C programs I've simply
recompiled on Unix, I see elapsed time improvements of from 40x to several
hundred x when moving from VMS (VAX 4000-500) to Unix (486-class PC). Almost
all of this is from faster disk I/O, although malloc() is also faster on the
Unix boxes, but not by as much.
Note 312.38, 16-Feb-1995
Coy: Swiss Army Knives are not good hammers
-------------------------------------------
At the very FIRST, over-simplistic, somewhat-incorrect-but-useful,
viewpoint of a Unix file system: [**NOTE** No criticism intended here]
o All files are the same -- just a stream of bytes.
o Directory files are files. See rule 1 above.
o Directory files don't have their contents sorted.
o There aren't version numbers (affects filesystem overhead).
o There aren't "file extensions" (affects filesystem overhead).
o The filesystem does not provide locking.
o The filesystem is not extremely robust.
Notice the lack of overhead. Notice the lack of features.
NEWS on Unix is a rather primitive and simplistic implementation (not a
criticism). It fits well with a rather primitive and simplistic
filesystem. The "good fit" is mostly an accident (IMO).
> implementations on those other systems, or VMS is not as "good" *WRT
> the particular qualities Usenet news requires* in some important way.
You seem to still be hunting some criticism of the VMS implementation
of NEWS, or some criticism of VMS.
The simple fact is that NEWS doesn't "require" much of a file system at
all. But VMS doesn't supply a "brain-dead" file system. I guess one
could say that VMS's failure to provide a featureless filesystem is
where VMS is not as good as Unix.
Note 312.39, 16-Feb-1995
Hutto: If Unix weren't Unix, then News wouldn't be News.
--------------------------------------------------------
I take an evolutionary viewpoint. If Unix, the operating system of
choice for News hosts, had a file system as slow as RMS at doing
the sorts of thing News implementations do, then News as we know it
would not be out there handling thousands upon thousands of messages
per day. The whole News system would have fallen apart years ago.
In that event, either something built on a different paradigm would
The DECUServe Journal March, 1995 Page 21
News Performance on VMS
have replaced News as the most popular way of exchanging that sort
of data, or the whole thing would have died out.
In other words, if the most freely available, least-common-denominator
multiuser system throughout the world had been VMS and DECnet, nothing
*like* News would be operating today on the *scale* of News (as we
know it). To paraphrase Steven Jay Gould, "News is a contingent fact
of history, arising because of the file system and networking that
are built into Unix".
Note 312.40, 16-Feb-1995
Kennedy:
---------
> At the very FIRST, over-simplistic, somewhat-incorrect-but-useful,
> viewpoint of a Unix file system: [**NOTE** No criticism intended here]
None taken 8-)
> o The filesystem does not provide locking.
Using this as an example, though it covers many of the previous points as
well, the Unix concept is that the system shouldn't burden every process with
things the majority of them probably don't need. For example, taking the
stream-of-bytes file structure as an example, many utilities and applications
are quite happy with that representation of the data. Those that aren't (for
example, the News index files) use an additional code layer to build a keyed-
access file.
VMS does the same thing - the last hardware/software I know of that imple-
mented file structure as actual disk attributes was the IBM 3x0 CKD format
disks. With VMS, RMS does this - without RMS everything is just disk blocks.
> o The filesystem is not extremely robust.
I've heard this for some time. I've been running Unix heavily since 4.2BSD
and I've run other flavors before then. By 4.2BSD the Berkeley Fast File System
was pretty solid. I've never lost a filesystem due to a crash or panic, and
the most complaints I see from fsck on a crash or power failure are the equiv-
alents of "File not found in a directory" or "File marked for deletion" from
ANALYZE/DISK.
AIX (from IBM) has a rather revolutionary filesystem where the sizes of disk
partitions can be changed on the fly, etc. and it seems to have received high
praise from users and reviewers alike.
While DEC hasn't confirmed this, VMS's "The New File System", TNFS, sounds
almost exactly like 4.4BSD's Log-structured File System (LFS). LFS wasn't
fully debugged at the time of the 4.4 release, and moving it to a new OS is a
major undertaking, but it would be somewhat humorous if TNFS is really a polish-
ed LFS.
The DECUServe Journal March, 1995 Page 22
News Performance on VMS
Note 312.41, 16-Feb-1995
Scopelliti: News would still be News
------------------------------------
> I take an evolutionary viewpoint. If Unix, the operating system of
> choice for News hosts, had a file system as slow as RMS at doing
> the sorts of thing News implementations do, then News as we know it
> would not be out there handling thousands upon thousands of messages
> per day. The whole News system would have fallen apart years ago.
Nope. Just would have been implemented using different rules - perhaps
some similar to those discussed earlier in this topic.
Note 312.42, 17-Feb-1995
Hutto: Alternative evolutionary paths
-------------------------------------
>> then News as we know it
>> would not be out there handling thousands upon thousands of messages
>> per day. The whole News system would have fallen apart years ago.
^^^^^^^^^^^^^^^^^^
> Nope. Just would have been implemented using different rules - perhaps
> some similar to those discussed earlier in this topic.
That's what I mean by "as we know it". The rules for implementation
would have evolved to suit the predominant operating system platform
on which it was deployed.
But then I meant to say "...per day. Or else, the whole...", rather
than the wording I actually used.
Note 312.43, 17-Feb-1995
Mischler: BSD is fine, but UNIX...
----------------------------------
> I've heard this for some time. I've been running Unix heavily since 4.2BSD
>and I've run other flavors before then. By 4.2BSD the Berkeley Fast File System
>was pretty solid. I've never lost a filesystem due to a crash or panic, and
>the most complaints I see from fsck on a crash or power failure are the equiv-
>alents of "File not found in a directory" or "File marked for deletion" from
>ANALYZE/DISK.
I agree that the BSD FFS survives system and power crashes very well at this
point. OTOH, if SCO UNIX was the only product you had seen you would conclude
that the file system is *intended* to be corrupted at the slightest hiccup.
But that is all AT&T SVR3 code, anyway.
The DECUServe Journal March, 1995 Page 23
News Performance on VMS
Note 312.44, 17-Feb-1995
Tillman: Sure they have
-----------------------
> While DEC hasn't confirmed this, VMS's "The New File System", TNFS, sounds
>almost exactly like 4.4BSD's Log-structured File System (LFS). LFS wasn't
>fully debugged at the time of the 4.4 release, and moving it to a new OS is a
>major undertaking, but it would be somewhat humorous if TNFS is really a
polish-
>ed LFS.
Someone from Digital said flat out that this was true when I attended
the "TNFS" session at DECUS '94/New Orleans.
Note 312.45, 17-Feb-1995
Coy: I forgot this was a religious war
--------------------------------------
>> I've heard this for some time. I've been running Unix heavily since 4.2BSD
>>and I've run other flavors before then. By 4.2BSD the Berkeley Fast File
System
>>was pretty solid. I've never lost a filesystem due to a crash or panic, and
>>the most complaints I see from fsck on a crash or power failure are the equiv-
>>alents of "File not found in a directory" or "File marked for deletion" from
>>ANALYZE/DISK.
I was _trying_ to give some comparative information.
If one decides, on the basis of "I have not had any big problems", what
the technical facts are, then one may be "misguided". This is
particularly true in low-probability events. For instance, up until
recently, the Japanese could claim that they hadn't seen any big
problems with their earthquake-resistant design...
The simple fact is that the (basic) Unix filesystem is not built as
robustly as the VMS file system.
The simple fact is that *NO* filesystem is totally robust in the face
of all possible events.
The simple fact is that the "bad events" don't happen very often, and
that in case of "bad events" there is STILL a low probability that you
will see problems. For instance, if your filesystem (whatever OS) is
"idled" -- and all caches (if any) have been written -- you can pull
the power plug and it matters *NOT* how robust the system is, you won't
observe a problem.
The more-than-Unix robustness designed into the VMS file system often
prevents ---
>>the most complaints I see from fsck on a crash or power failure are the equiv-
>>alents of "File not found in a directory" or "File marked for deletion" from
>>ANALYZE/DISK.
The DECUServe Journal March, 1995 Page 24
News Performance on VMS
--- from happening. It also often prevents other "no big deal"
difficulties, and it means that you don't have to run ANAL/DISK/REPAIR
every time you boot the system (but with Unix you run the equivalent at
reboot if you are smart). Again, that's NOT THE POINT. The point is
that the added robustness in VMS **INCREASES OVERHEAD**. And there's
no way in VMS to say "well, I don't care as much about the NEWS files,
so I'll take a .001 bigger risk, so get rid of the
robustness/overhead".
I apologize for saying:
>>> o The filesystem is not extremely robust.
when I should have said:
o The VMS filesystem has a lot of overhead associated with
robustness, and that overhead is not present in most basic Unix
filesystems.
Note 312.46, 17-Feb-1995
Kennedy: Sorry, wasn't trying to argue (much 8-)
------------------------------------------------
> I was _trying_ to give some comparative information.
As was I. My experience on very busy production systems in both environments
is that the Unix filesystem is at least as solid as the VMS one, and thus the
claim that:
> when I should have said:
> o The VMS filesystem has a lot of overhead associated with
> robustness, and that overhead is not present in most basic Unix
> filesystems.
isn't necessarily true. I'd agree with "the VMS filesystem has a lot of
overhead".
Let's consider the case of sequential reads to a large contiguous file on
a disk mounted read-only. Many of the checks needed for "robustness" (no
matter how that is defined) can be skipped in this case. Yet it looks as if
the VMS filesystem has a lot of overhead even in that case. Perhaps that's
due to generalizations about where "robustness" is needed in the code, per-
haps it's due to RMS overhead, but it isn't due to a lack of something in
the Unix filesystem design.
Studying this VMS issue might have useful results for many applications.
For example, many years ago someone I know worked on an imaging system being
developed on VMS. The biggest overhead was writing the completed images to
tape. After he switched to physical (vs. logical) I/O to the tape drive he
gained an order of magnitude in performance. Likewise, comparing the speed
of a BACKUP/PHYSICAL to tape with the speed of a BACKUP/IMAGE, it would seem
that if improvements could be made, BACKUP (/IMAGE) users would benefit.
The DECUServe Journal March, 1995 Page 25
How to Do 24x7
How to Do 24x7
==============
The next stream of notes, taken from the SITE_MANAGEMENT conference,
considers the various issues involved in creating a 24-hour by 7-day
operation, particularly when it is necessary to introduce 24x7
support to an existing organization. The discussion offers tips,
advice, and a look at how some major vendors handle things
internally.
Participants: David Campen, Ta Fuh Chiam, Matt Holdrege, Ed Kozam,
Chris March, Pat Scopelliti, Harrison Spain.
Note 234.0, 23-FEB-1995
Spain: Building a 24hr x 7 day support organization
---------------------------------------------------
We are planning to convert from a 12 hr x 5 day support organization to
a 24 hr x 7 day support organization (yikes!).
Does anyone have any suggestions on how to go about doing this? We
support operating systems (about 7) in conjunction with our product
line.
I'd like to hear how you support the light hours vs the heavy hours,
suggested shifts, shift differentials, hiring, war stores :-), etc.
Note 234.1, 23-FEB-1995
Holdrege: the simple answer
---------------------------
I did 24x7 support for about 5 years. You just learn to make sure that
nothing will go wrong at night. If someone does have to pull an
"all-nighter" you give them the day off.
Note 234.2, 23-Feb-1995
March: 4-10 hour days?
----------------------
All the shops I've been in have been 7x24.
The best implementation of this was a 4-day work week, half of each
shift worked Sun-Wed, the other half Wed-Sat. If I remember correctly,
shifts ran 7am-5pm, 3pm-1am, and 12am-10am.
It worked well because there were several people who didn't mind
working Friday and Saturday night, and everyone was in on Wednesday
to deal with staff meetings and such.
After I left, it was decided that people would rotate so the same
people wouldn't have to work every Friday and Saturday. I'm not
sure how they implemented it.
The DECUServe Journal March, 1995 Page 26
How to Do 24x7
The compensation was 10% for everyone on the 4-day week (anything over
8 hours/day was considered overtime, so 2 hours a day was OT), plus 10%
for evenings (3pm-1am), and plus 15% for the night shift (12am-10am).
Of course, everybody on these shifts was also hourly, not salary. I'm
not sure how you could work the salary stuff in there.....
Note 234.3, 23-Feb-1995
Spain: Thanks for the help so far!
----------------------------------
I'm not sure I can get folks to handle 10 hrs of phone support :-).
I'd like to leave the day shift alone which is nice and stable and runs
from 5AM to 5PM. Unfortunately covering the other 12 hours Mon-Fri is
relatively easy while the weekends present a real problem.
Does anyone know how Digital, Sun, or HP do this?
Note 234.4, 23-Feb-1995
Scopelliti: Love my VCS!
------------------------
On tours I've had of various "lights out" facilities, they used a lot of
VCS, DECalert, and other such products.
We use VCS and once it saves you at 2:00 A.M. from having to get up,
get dressed, drive 1/2 hour, get into the building, walk over to the
console, type B<RETURN>, get out of the building, drive 1/2 hour, get
undressed and back into bed, you'll never question the license cost!
Note 234.5, 23-Feb-1995
Campen: 10 days on, 4 days off.
-------------------------------
A scheme we use (but for something other than computer operations) is
10 days on, 4 days off. Each shift is 8 hours with 3 shifts per day.
There is then a 4th "relief" shift that fills in when the person(s) who
normally have a shift are on their days off. You are still short by one
8 hour shift out of every 21; this and sick time and holidays are made
up with overtime.
Note 234.6, 24-Feb-1995
Holdrege: on call
-----------------
many of Digital's support people are on call from their homes. If you
have 24x7 support on a product like DEC EDI, and you call after hours,
you will get a call back from an engineer at home. You can even hear
The DECUServe Journal March, 1995 Page 27
How to Do 24x7
their kids yelling in the background.
When 3COM was growing up, they had one engineer per week who carried
the "on call" beeper for after hours calls. This one engineer had to
handle calls on their entire product line. That didn't work very well
when they grew to their current size.
Note 234.7, 24-Feb-1995
Chiam:
-------
I believe Digital has different ways of handling support after hours
depending on which products.
HP after hours support in the U.S. is handled by their support folks in
Australia, if I am not wrong.
Note 234.8, 26-Feb-1995
Holdrege: global support
------------------------
I can vouch for that. I even FTP'd a patch from Australia during a late
night HP upgrade once.
Note 234.9, 26-Feb-1995
Kozam: Staffing for 24 hour service
-----------------------------------
Perhaps you could tell us more about how you hope to achieve in going
24x7:
- Is there a particular customer that needs 24x7?
- Is this going to be a premium cost service?
- Do you intend to have "business as usual" at 3AM or
just the essentials for emergency issues?
- Do you expect a steady stream of work or many quiet hours
peppered with a few calls?
- Are most of your support calls brief and resolved in one
phone call or is it more typical that you need to do
research?
- Are you expecting an expanded volume of calls or do you hope to
shift the daytime workload into the night? (Reduce equipment
needs, office space, etc.)
Not knowing your constraints, here are some ideas:
1. Rather than going directly to 24x7, consider extended hours,
say 5AM until 11PM.
2. Consider "emergency only" support after hours.
3. Consider several layers of support. Some available
immediately, some available via pager.
Here are some thoughts in the meantime.
The DECUServe Journal March, 1995 Page 28
How to Do 24x7
I think that the most important thing your organization can do in
making this transition is to give serious thought to the impact of
this change on your employees' lives. Has anyone asked your existing
support staff what they think of this idea? You ought to do that
right away.
Selling an extended business day is pretty easy. Non-traditional work
hours have lots of benefits that people readily realize. This
includes easier commuting, off peak shopping, the ability to see do
daytime tasks such as shopping, dealing with the MVA, seeing a doctor.
If your work force is diverse in terms of age, marital status, and
children, then staffing from 5AM until about 11PM should be easy.
If you think getting your employees to work 10 hour shifts is going to
be hard, then wait 'til you try to get some people to 'go nocturnal'
and cover the late night hours, weekends, holidays. You're asking
your employees to make a fairly large sacrifice and in one way or
another, they will expect compensation (money, time off, promotion,
special projects, flexibility in scheduling). Of course, even the
most bizarre schedules may appeal to some, but you can't count on it.
What this all translates to is that people don't really want to work
bad shifts, so you should probably attempt to minimize what happens
after hours. You will always be able to provide better service during
regular hours (more people available, more experienced support
people/developers on hand), so it isn't unreasonable to encourage
people to call during regular hours, for non-urgent issues.
Fortunately, the workload naturally drops off after regular hours.
You can also actively discourage after hours calls, perhaps with an
automated message such as "Our offices are closed for the day. If this
is an emergency, please stay on the line". I've sometimes booked
airline tickets at 2AM, but I could have just as well waited until
the morning - this was a call for my convenience, hardly a necessity.
Now to this nuts and bolts of how round the clock staffing (adjusted
for work load) is handled.
There are two basic ways to handle the unpopular shifts. One is the
free market approach, e.g. sweeten the "bad" shifts with a pay
differential or extra time off. The other is more in the spirit of
Karl Marx in which everyone is required to pull part of the load -
e.g. everyone does one weekend a month. It may take a combination of
the two to get the job done. You might want to take seniority into
account. Think about what you'll do if someone refuses to work some
weekends. You're going to have a pretty hard time selling a Karl Marx
approach to existing employees since they didn't bargain on working
nights, weekends, and holidays. New employees might have that as an
expectation of the job. Some places do some initial training then
put new employees on the night shift with the expectation that after
they pay their dues, they will get preference for better scheduling.
You will make everyone happier by letting employees have as much
The DECUServe Journal March, 1995 Page 29
How to Do 24x7
flexibility as possible. My experience with schedules is that how
things finally get staffed hardly looks like the original schedule.
It IS important to have ONE master schedule so that when it 11PM rolls
around and no one shows up, it is clear who to call. For each change,
get TWO signatures, one from the person giving up the shift and
another from the person taking it. "I'll take your midnight shift" is
VERY different from "I'll CONSIDER taking your midnight", but it is
amazing how easy it is to get things confused. We tend to hear what
we want to.
I wouldn't try to get a permanent night or weekend shift, although if
there is one person just in love with being the night owl, so be it.
Once the novelty of being up all night and sleeping all day wears off,
most people realize that being permanently out of sync() with the rest
of the world is less than ideal. It's hard to have a social life or
family life when you need to sleep when everyone else wants to play.
You can pull it off for a while, with lots of coffee, but trouble is
coming soon. Further, you don't want to isolate anyone from the
informal exchange of information that is often very important. Lots
of real problems get solved during informal conversations.
Whew! That's already too long. Hope it is helpful.
Note 234.10, 27-Feb-1995
Spain:
-------
> - Is there a particular customer that needs 24x7?
Yes, but we expect to sell this to other customers once the service is
in place.
> - Is this going to be a premium cost service?
We expect to have the customer pay a premium for the extended hours.
Our estimate and other vendors (like HP and DEC) are surprisingly
similar.
> - Do you intend to have "business as usual" at 3AM or
> just the essentials for emergency issues?
We are expected to have "business as usual" although I suspect (like
Digital) our customer would not expect the phone to be answered on the
first ring like during the day. We are hoping to work out a dispatch
type arrangement for slow hours.
> - Do you expect a steady stream of work or many quiet hours
> peppered with a few calls?
Good question, we just don't know. I expect it to be fairly slow. On
the other hand, I expect weekends to be a bit busier since folks tend
to upgrade during these hours.
The DECUServe Journal March, 1995 Page 30
How to Do 24x7
> - Are most of your support calls brief and resolved in one
> phone call or is it more typical that you need to do
> research?
It varies. Since we support OS calls, our model would be very close to
a SUN/HP/DEC type support organization. We lean heavily on a massive
symptom solution database which leverages our expertise and allows us
to close calls quickly.
> - Are you expecting an expanded volume of calls or do you hope to
> shift the daytime workload into the night? (Reduce equipment
> needs, office space, etc.)
I expect the daytime calls to increase but not dramatically. This
could be a mistake in planning since when we go to 24hrs, the calls
that we would have expected at night might very well come in during the
day.
> Not knowing your constraints, here are some ideas:
> 1. Rather than going directly to 24x7, consider extended hours,
> say 5AM until 11PM.
We may do this short term but 24 x 7 was paid for and 24 x 7 will have
to be delivered after 90 days.
> 2. Consider "emergency only" support after hours.
I'm hoping I can get away with this. Time will tell if this is
acceptable.
> 3. Consider several layers of support. Some available
> immediately, some available via pager.
Good suggestion, thanks!
> Here are some thoughts in the meantime.
>
> I think that the most important thing your organization can do in
> making this transition is to give serious thought to the impact of
> this change on your employees' lives. Has anyone asked your existing
> support staff what they think of this idea? You ought to do that
> right away.
Already done :-).
> Whew! That's already too long. Hope it is helpful.
Outstanding! Thank you! :-)
The DECUServe Journal March, 1995 Page 31
Phone Line Trouble
Phone Line Trouble
==================
If your duties include system management, you've probably found that
there's a lot more to the job than either the personnel department
or your mother told you about. Frequently, system managers are
expected to be not only experts on their own systems, but on any
systems your users may possibly want to connect to, including the
telephone network. More than one full-fledged expert in things
relating to computer networks (the kind who eats RFCs for breakfast)
has been reduced to gibbering incoherence by the prospect of having
to deal directly with one of the various descendants of Ma Bell.
Well, okay, so we may be exaggerating a LITTLE bit. But for many
computer professionals, telephone-speak is an alien language. But
that is exactly what the TELECOMMUNICATIONS conference is there to
help fix! The following topic deals with one of the most
potentially maddening telecom problems of all time -- line noise.
Participants: Barton Bruce, Linwood Ferguson, Terry Kennedy, Jeff
Killeen, Brian Tillman.
Note 95.0, 14-Feb-1995
Killeen: Digital TELCOM links
-----------------------------
Currently I have...
Office Switchboard
------------------
!
!<---17 mile FX line ---> Office Phone
!
!<---17 mile FX line ---> Office Modem
The quality of office phone line on long distance calls is poor. We have been
through all possible fixes with Telco - it is a cascading line loss problem.
Question is there a digital link on the market that I could use between the
two locations? Ideally one that lets you set the volume levels. Even better
would be something that would let me multiplex the voice and data over one
line rather than two.
Note 95.1, 14-Feb-1995
Kennedy: Odd
------------
> The quality of office phone line on long distance calls is poor. We have been
> through all possible fixes with Telco - it is a cascading line loss problem.
That's odd - I have one of these here. It's delivered on regular pair from
my CO to SPC. Between the CO and the remote CO it's digital, and at the far
end it turns back into a regular pair. It has better response than the local
The DECUServe Journal March, 1995 Page 32
Phone Line Trouble
lines - 3db loss, essentially flat across the usable frequency range.
Perhaps you've got some weird kind of circuit?
Note 95.2, 14-Feb-1995
Ferguson: FX line can arrive as 4-wire local loop
-------------------------------------------------
When they put in an FX line for me here, they ran it as a 4-wire
circuit to inside my building, and then converted it there. They said
this was because of the local loop length to allow them to provide
better quality.
I know nothing of the details, but it was always "just like being
there". it was not a FX line into a switchboard, just a remote
dialtone. But I was surprised they did it that way (a 4-wire loop
instead of a regular pair).
Note 95.3, 14-Feb-1995
Killeen:
---------
FX lines to another central office is a different issue.
Note 95.4, 14-Feb-1995
Killeen:
---------
> Perhaps you've got some weird kind of circuit?
No - and the issue isn't just the loss on the FX link - it is the loss through
switch board plus the loss on the FX link. When have had a couple very
knowledgeable folks look at this and they agree it is inherit in the system.
The problem is far worse with long distance calls than it is with local calls.
Note 95.5, 14-Feb-1995
Kennedy: It *is* possible
-------------------------
I still think there's something funny going on. Your "switchboard" has a
fixed loss. The line to your house (actually OPX, not FX) has a fixed loss.
The sum of those is also a fixed loss.
Depending on the type of circuit you ordered, you have either something
like a metallic pair (probably what you really have), a 2-wire transmission
circuit, a 4-wire transmission circuit, or a digital circuit.
With the metallic pair, you're on your own. It has the advantage of being
just like at the office, but with greater loss. Depending on the type of
phone system you have, you might be able to modify a line card for higher
The DECUServe Journal March, 1995 Page 33
Phone Line Trouble
audio levels (this generally only works on electronic phone systems, not on
1A-like systems).
On the 2-wire or 4-wire systems (which are all 4-wire inside the telco
transmission plant - the difference is whether the hybrids and equalizers
are at the local CO or at your facility), the telco passes audio from end
to end and detects ringing current and on/off hook states and passes it
to the remote end. You can buy hardware to do this which runs over a 3002-
type circuit (which Barton will tell you is obsolete). These circuits can
be adjusted over a wide range of gain/loss - you can get 0dB if you want.
Digital is just that - digital. It's like the 4-wire system except that
the transmission between CO's is done digitally. Again, you can buy hard-
ware that will do this given a DS0 between your home and the office.
There are a number of ways to solve this, depending on how much you're
willing to spend. You could get a DS0 and do it digitally, or you could
get something faster and bridge your Ethernet to your house *and* get a
few phone extensions out of it as well.
Note 95.6, 14-Feb-1995
Bruce:
-------
Will they tell you how much loss they have in the SOP circuit?
If it is 4 wire to your site, there are all those convenient little dip switches
and you can 'tune' loss/gain in each direction as you please when they are not
looking.
> Question is there a digital link on the market that I could use between the
> two locations? Ideally one that lets you set the volume levels. Even better
> would be something that would let me multiplex the voice and data over one
> line rather than two.
Sure! And MANY terminal sessions statmuxed and bridged ethernet, and conversion
to/from LAT, and compressed voice (5.6kb, 8, 12, 16, 30kb),
demodulated/remodulated FAX passed as data on the link and as FAX modem tones at
far ends.
And ALL that might be on a 9.6 line, but really starts being usable on a 56kb
line. Box prices are typically calculated on 1/2 what they save you on a
Calcutta to Chicago link over 3 years :-) Seriously, they are NOT cheap,
but do work. There has to be a simpler solution for your problem.
Note 95.7, 14-Feb-1995
Killeen: DS0?
-------------
> With the metallic pair, you're on your own. It has the advantage of being
> just like at the office, but with greater loss. Depending on the type of
> phone system you have, you might be able to modify a line card for higher
The DECUServe Journal March, 1995 Page 34
Phone Line Trouble
> audio levels (this generally only works on electronic phone systems, not on
> 1A-like systems).
It is a Northern Telecom Meridian Norstar DR3
> Digital is just that - digital. It's like the 4-wire system except that
> the transmission between CO's is done digitally. Again, you can buy hard-
> ware that will do this given a DS0 between your home and the office.
What is DS0?
Note 95.8, 14-Feb-1995
Killeen:
---------
> Sure! And MANY terminal sessions statmuxed and bridged ethernet, and
conversion
> to/from LAT, and compressed voice (5.6kb, 8, 12, 16, 30kb),
> demodulated/remodulated FAX passed as data on the link and as FAX modem tones
at
> far ends.
Where would I find such boxes?
Note 95.9, 15-Feb-1995
Bruce:
-------
> Where would I find such boxes?
The the Network trade show in Boston for the next 3 days. At least one of
the offending companies must be there.
The over advertised unit everyone thinks of is the Micom Marathon.
It is in reality a statmux that grew up a tad more so they can still sell it.
I would prefer to NOT get all the games they try to throw in. Sadly, on
a 56kb line the MAX SYNC speed Micom gives you for other things is 38.4
even with all voice channels idle. If you can use the difference for
compressed voice and terminal traffic and such the loss may be ok. Their
version of integral terminal server and integral ethernet bridge can,
IMHO, be better done with something like Gandalf compressing bridges and
generic terminal servers.
A better brand probably is PCSI. They can give you all but maybe 1.8k
lost to overhead when all your voice channels are on hook. They change the
SYNC channel clock rate. Cisco and Gandalf and many other unit can handle
the 'variable' rate SYNC connection.
Micom may now go past 64kb. PCSI does!
Even PCSI does have some 'weirdness' (the PM has promised a fix in maybe
The DECUServe Journal March, 1995 Page 35
Phone Line Trouble
3-6 months) but is only relevant if you have a few low speed voice
channels on a 128kb line. Seems a programmer had too few bits allocated and
so can't expand a table that needs some more entries. You might not
now get all 126+kb on your 128kb line when everyone is onhook. But if you had
8 voice channels at 8kb each or 4 at 16kb each, everything would be fine.
If you just had 3 at 5.6kb you would be wasting over 40kb. They will fix that.
At 64kb and below for your link, everything will work as expected with
not needless waste.
At the slightly higher end :-) you stick several T1s of PCM voice and
any video and frame relay you have into some large ATM box.
With good cisco router/bridges now in the $2k range speaking frame
relay, if you have several sites you need to interconnect here in MA
there are some nice opportunities. OTOH, those voice compression boxes don't
yet work on F/R links.
Note 95.10, 15-Feb-1995
Kennedy: What DS0 is
--------------------
> What is DS0?
DS0 is the name for a 56Kb "chunk" of bandwidth. It might be one of 24 slots
in a T1, or a circuit by itself.
In general, the "T" names label specific transmission gear and interfaces,
while the "DS" names just describe bandwidth/bit rate.
24 DS0's to a T1, 28 T1's to a T3. Also (but less frequently seen) 4 DS1's
to a DS2.
Note 95.11, 15-Feb-1995
Tillman: Try Data Race
----------------------
See 92.2 in this conference for the name of a company that can supply
equipment that can do data and voice over the same lines.
Note 95.12, 16-Feb-1995
Bruce:
-------
> DS0 is the name for a 56Kb "chunk" of bandwidth. It might be one of 24 slots
or 64kb
The 56 is simply a 64 you can only use 7 of the 8 bits on for assorted
unfortunate reasons.
The DECUServe Journal March, 1995 Page 36
Phone Line Trouble
Note 95.13, 17-Feb-1995
Killeen:
---------
BTW - I forgot to mention the problem with these lines is not hearing the
other party but the other party hearing me. Are there any off the shelf boxes
that allow you to boast you transmit levels?
Note 95.14, 18-Feb-1995
Kennedy: You're out of range (90 kilofeet!)
-------------------------------------------
> BTW - I forgot to mention the problem with these lines is not hearing the
> other party but the other party hearing me. Are there any off the shelf boxes
> that allow you to boast you transmit levels?
Now we're getting somewhere. When the telco provides dialtone in rural
areas, they increase the battery voltage to accomodate the additional loss
in the cabling. [These days, this is done by selecting a range (in feet)
for the circuit when it's ordered, so many telco folks are no longer aware
of this fact].
If your circuit is a classic metallic pair, nobody is providing batter
on it, and your "PBX" (which I assume is actually something like a 1A key
system, possibly electronic) is just attaching your extension to the CO
line at your main office. That's 90kilofeet, which is well beyond most
CO boost anyway, even if they were doing any, which they aren't.
With specs like that, the circuit should never have been provisioned in
the first place, and anybody who's looked at it should have said the same
thing. If you show this to them, they'll probably remember this.
There are amplified handsets (mouthpiece, earpiece, or both) but they
work by pulling current from the line, and that's one thing you're short
of in this circuit, so they may actually make it worse.
As I said in an earlier note, there are other circuits that would work
better for you, but without seeing exactly what sort of PBX you have, it's
hard to suggest something exactly.
How Many Users?
===============
Once you've got your line noise problems solved, you may start
wondering just how many users your cleaned-up line can support.
Capacity planning for comm links can often feel like stumbling
around in the dark; only rarely are you given enough information to
make accurate calculations, and of course everyone's mileage will
vary. This next stream from the TELECOMMUNICATIONS conference might
supply a few useful reference points.
The DECUServe Journal March, 1995 Page 37
How Many Users?
Participants: Rick Bowen, Dale Coy, Linwood Ferguson, Matt
Holdrege, David Mischler, Ray Whitmer, Billy Youdelman.
Note 106.0, 13-Oct-1994
Ferguson: How many users can a 56K circuit support?
---------------------------------------------------
We're going to monitor some "real" traffic and also set up some user
tests, but I've been asked to see if I can find some real situations:
How many active VT-style users can a 56K circuit support? Say with
a terminal server at one end and bridges?
Obviously this is very subjective, depending on what users will put up
with.
Obviously it depends on what they are doing, how often the repaint
screens versus just enter short fields over and over.
But any background of real situations will lend some reality to what
we are doing.
For what it's worth what we are investigating is how much bandwidth
we would need to support sites of varying size from about 10 users to
60 users, on VT type terminals, using anything from head-down data
entry programs (i.e. alot of continuous data but little screen
painting) to widely varying inquiry routines (lots of screen painting),
all with character-by-character key entry (i.e. no block mode). We
also have to allow for printing but I can come up with bandwidth
used for those pretty easily.
Our people are guessing anywhere from substantial fractions of a T1 (or
equivalent with whatever technology -- that's not the question, yet) to
"one 56K can handle at least 40-50 users with no trouble at all".
Note 106.1, 13-Oct-1994
Bowen: How many users can a 6610 support?
-----------------------------------------
The question depends a lot upon how active they are, what protocol they
are using, and if you are doing any large file transfer or print
jobs. We have run offices of up to 150 people over 56kb lines which supported
on average 20-30 people communicating to a remote system at any one time.
However, interactive performance would seriously lag when a file transfer or
print job was in progress. You can also get improved capability if
you use cost of service type packet prioritization and compression.
If you send me some expected traffic I can run some simulations.
The DECUServe Journal March, 1995 Page 38
How Many Users?
Note 106.2, 14-Oct-1994
Ferguson: More details
----------------------
Some things I know, some I don't.
We anticipate these to be VT style terminals connected to terminal
servers running TCP/IP connected to RS6000 hosts. The terminal servers
will also be handling serial printers with aggregate bandwidth ranging
widely but probably averaging 1200 lpm (obviously we will need to do
this kind of arithmetic for each site).
They will be quite active; historically (whether smart or not) we have
made people share terminals if they were not keeping them busy most of
the time (managers who make these decisions are exceptions, of course,
they get their own and don't keep it busy). Some are real production
stations, scanning merchandise with bar code readers as fast as they
can pass them over the scanner (the scan itself is only a few
characters but the screen updates are maybe a quarter to half a screen
each time).
As to file transfers (other than print files) that's a killer also.
We plan to have a few PC's at each site (like 2-5). Normally these
will be either in terminal emulation mode or running in client server
mode with reasonably controlled levels of transfer. But they will be
able to do things like suck up a print file or transfer spreadsheets or
documents around, so this is unpredictable, and I suspect not
prioritizable(?).
And the users are demanding. This is a new, wonderful,
3-year-in-progress replacement system. We have to plan for adequate
band width to be "just like being there".
The technology has not been bought yet, though. So whether we use
dedicated lines, frame rely, or tin cans and long string is an open
question. But I am kind of assuming the real question you have to
answer first is what kind of speeds you need?
By the way: we know the cost will change over this three year period,
and I can buy more bandwidth for less 3 years from now. But you have
to start somewhere, and very conservative management won't buy "don't
worry about network cost 3 years from now, it will be cheap" as a
cost analysis. :-)
Note 106.3, 14-Oct-1994
Coy: Simplistic number
----------------------
OK, let me just give a raw number for others to shoot at. Based on
only a lot of experience and what you just wrote about the application
characteristics.
I would plan to use 56K bandwidth for every 8 users.
The DECUServe Journal March, 1995 Page 39
How Many Users?
Note 106.4, 14-Oct-1994
Holdrege: options
-----------------
For normal character cell users, generally plan for 2-4 kilobits per
user. I have had up to 100 people on a 56K, but they were not that
active.
If you are printing to a printserver, or doing client/server where the
application resides on the server, your 56K will be eaten up totally
during each transfer.
A way to coordinate this is to use different protocols for character
cell and file transfer. Then assign a class of service on your
bridge/router.
The easy way is to use LAT for terminal sessions and TCP/IP for bursty
traffic. Then give LAT a priority on your bridge. Check your
bridge/router documentation for clues. Some devices can assign class of
service to a finer degree than others.
Note 106.5, 15-Oct-1994
Youdelman:
-----------
> But they will be
> able to do things like suck up a print file or transfer spreadsheets or
> documents around, so this is unpredictable, and I suspect not
> prioritizable(?).
PSI (the net vendor) offers what they call a permanent virtual circuit
wherein one can, say, keep some part of the total bandwidth for outgoing
traffic even when incoming traffic would ordinarily make it hard to get
out. I don't know the details but something like this seems like it'd
be useful here..
Note 106.6, 15-Oct-1994
Whitmer: That reminds me of the time...
---------------------------------------
This does not really add to the conversation, but I can't resist. Back
in the early 1980's a law firm using our VAX CPU for word processing
and heavy ALL-IN-1-like OA stuff had their 9600 baud modem go out --
they were stat-muxing 7 users and a diablo printer (for all their
correspondence) over the line. Since they were behind in their bills,
to send them a message, we replaced the 9600 baud modem (fairly
expensive back then) with a 2400 baud modem. They went on this way for
couple of months without comment or apparent reduced use. When they
settled their bills, we replaced the modem. But they were surprised to
hear that there was any kind of an equipment failure. We were about
ready to take the 9600 baud modem back and ask for our money back.
Moral: it is greatly a matter of user expectations?
The DECUServe Journal March, 1995 Page 40
How Many Users?
Note 106.7, 20-Oct-1994
Mischler: TCP vs. LAT on slow links
-----------------------------------
> The easy way is to use LAT for terminal sessions and TCP/IP for bursty
> traffic. Then give LAT a priority on your bridge.
I have found that telnet terminal sessions merely get slow in
situationsrt)ﾂ Date: 20 Feb 1995 03:00:06 -0600ﾂ Reply-To: lett...@mcs.netﾂ Followup-To: alt.fan.lettermanﾂ Organization: MCSNet ServicesﾂWSummary: This posting contains a list of Frequently Asked Questions (andtheir answers) ﾂH about the Late Show/Late Night with David Letterman.New readers of the ﾂE alt.fan.letterman newsgroup should read thisFAQ list before posting.ﾂ&Approved: news-answe...@MIT.Eduﾂ NNTP-Posting-Host: venus.mcs.comﾂ Lines: 1359loom-picayune.MIT.EDUﾂ