Speaking About Crashes and Doing Dumb Things
TELECOM Digest Editor
Last Sunday night I got on line about 10:00 p.m. here to do some work
on the Digest and I had a bright idea about a new script I wanted to
try out. Well the script flubbed, which was not anything unusual for
scripts that I write or try to hack on, but the main annoyance was
it left me with a directory full of about a hundred .h, .c. and .o
files to clean out when I decided to quit the experiment.
Now, I try to be smart with potentially disasterous commands like
'rm' and I personally have 'rm' aliased to 'rm -i' meaning to not
erase a file without asking for confirmation. The problem is, if
you have a whole directory full of garbage files to get rid of
then if you go to that directory and do 'rm *' it will stop over
and over again, asking about each file. The command 'rm -f' will
NOT overrride 'rm -i' on this machine at least, although 'rm -f'
will work in a script running in the background with its own shell
regardless of what arguments I happen to have attached to 'rm'
for my use in the foreground.
So far so good. Instead of having to answer 'y' a 120 times for
every garbage file in the garbage directory I am abolishing, I
decided just this one time I would unalias 'rm' instead. So I
did 'unalias rm' then I did 'rm *' -- but the trouble is I had
** forgotten to change directories to the one I wanted **.
Ooops!
Well all I can say is thank goodness for the Info-Mac Archives
and other major users of the resources and bandwidth here, because
massis was **very slow** at the time and after a couple seconds
I got a message saying 'bin is a directory', and this awful chill
went up my spine. As quick as I could, I slapped the keyboard and
and banged every key I could find to stop the job. Fortunatly I
only lost about a third of my main directory; none of the 'dot files'
were lost but I did lose one of the MAJOR processing scripts used
in the Digest -- one that takes the finished Digest and bursts it
into individual messages for comp.dcom.telecom.
I went to every site I am on looking for a spare copy of it with
no results. I did find spare copies of some of the other scripts
I lost in this debacle. In email to the admin at MIT I pleaded,
"In a dumbo raid at about 10 pm Sunday night, an ill advised
use of rm * caused the loss of several files. Can you please
use the Saturday night backup to restore my missing files."
I chatted with him on the phone early Monday morning about it
also, and he was rather annoyed by it. It seems Monday was a bad
day in his office also and he had his hands full with a dozen
things going on at once. When time permitted, he had one of his
associates, Mary Ann Ladd fetch the files from the tape which
had been run Saturday night. Later he send me email and gave
me hell:
"In the future we will not use tape backups to restore
individual files. We will use them only for restoration
after a disk failure or other major system problem. I have
400 other users to support here besides you."
My scripts were waiting for me in a /tmp directory and I grabbed
them immediatly. Now everything is back to normal and I can sit
here and laugh about it, but Monday morning I did not feel like
laughing, believe me you. Now all that remains is one of these
days I should make a pilgrimage to Boston, where after an
appropriate time of worship, I will offer myself as a humble
sacrifice to the gods at lcs.mit.edu who work so patiently
with me day after day and long into the night as well sometimes.
Losing all the processing scripts in a clumsy session at the
keyboard! What a way to wind up fifteen years of this Digest!
And this weekend approaching does mark fifteen years of it and
the start of year sixteen. Thanks to my heroes at lcs, I'll be
around to annoy you, abuse you and otherwise amuse you for
another year I guess.
PAT
Sam Ismail
TELECOM Digest Editor (ptow...@massis.lcs.mit.edu) wrote:
> Last Sunday night I got on line about 10:00 p.m. here to do some work
> on the Digest and I had a bright idea about a new script I wanted to
> try out. Well the script flubbed, which was not anything unusual for
> scripts that I write or try to hack on, but the main annoyance was
> it left me with a directory full of about a hundred .h, .c. and .o
> files to clean out when I decided to quit the experiment.
<blah blah>
Duh. Can you say "backup"? DON'T LEAVE ALL YOUR IMPORTANT FILES IN
JUST ONE PLACE.
(Sorry, PAT. Don't mean to be so rash but you need to learn your
lesson :)
Computer Historian, Programmer, Musician, Philosopher, Athlete, Writer, Jackass
Matthew B Landry
In article <telecom...@massis.lcs.mit.edu> TELECOM Digest Editor wrote:
> laughing, believe me you. Now all that remains is one of these
> days I should make a pilgrimage to Boston, where after an
I've wondered for a while now ... why is the Digest hosted on a site
at MIT, when you're in Chicago? It's not like Chicago is lacking in
quantity of local ISPs, and I'm sure any number of them would clamor
for the right to have their domain stamped on every issue. MIT doesn't
need publicity to survive ... most ISPs do.
> And this weekend approaching does mark fifteen years of it and the
> start of year sixteen. Thanks to my heroes at lcs, I'll be around to
> annoy you, abuse you and otherwise amuse you for another year I
> guess.
Congratulations. The Digest's history truly is an impressive
achievement, of which you should be proud. However much some of us may
disagree with some of your opinions, we all appreciate the work you
put in to make this medium work, and keep quality up.
(BTW, if that Zip+4 with no address trick really works, you
should see a $20 bill from me in the next couple of days. That is the
"suggested subscription fee", right? I just figured ... why not kill
two birds with one stone ... check out a neat hack, and finally get
off my arse and pay my share to help the Digest and c.d.t.)
Matthew Landry
Well, yeah. Actually I do sometimes speak
for Msen. But not from THIS account. O-
[TELECOM Digest Editor's Note: Thanks for your kind note of support.
The 'suggested amount' of $20 is because only about one out of every
twenty readers send anything. About five percent of the readership
participates in all. I'd be happy if everyone who reads the Digest
sent a dollar per month. Seriously; I'd be happy if half of them
sent a dollar a year. If I were to give a Treasurer's Report, it
would go like this: my income is about $1200-1500 per month, and
this comes from (1) the ITU grant; (2) royalties from the CD-ROM;
and (3) reader support. It is hard to live in the rich, white,
northern suburbs of Chicago on that sort of income. Having two
heart attacks in two years with an accumulated indebtedness to
Rush/North Shore Medical Center of about $51,000 did not help.
I'm told they wrote that off under Hill-Burton, and from time to
time I go see the village food pantry and social worker. I am
the village's favorite poor person; they get to practice their
liberal social policies on me, and I don't complain. They do
mumble occassionally when the water bill is not paid on time.
I am fully aware that MIT has no need of me, and I have a very great
need of them. MIT has housed the archives since 1988 (they were at
Boston University when the archives were much smaller) and although I
have done my production work from other locations (including bu.edu
in the beginning and at Northwestern University for quite a few years)
I happen to like MIT and the great technical support they give me.
I've got a Sparc-20 workstation pretty much to myself (massis) and
*lots* of storage space for the archives; more than I will use for
awhile. I am not really interested in being on the .com domain
although to hear Mr. Pfieffer tell it, that's the only way to go. I
could move Digest production and distribution to an account at bu.edu
anytime I wanted to do so or to berkeley.edu at any time, as I am a
user at both sites. PAT]
Mike Morris
In article <telecom...@massis.lcs.mit.edu> ptow...@massis.lcs.
mit.edu (TELECOM Digest Editor) writes:
> Last Sunday night I got on line about 10:00 p.m. here to do some work
> on the Digest and I had a bright idea about a new script I wanted to
> try out. Well the script flubbed, which was not anything unusual for
> scripts that I write or try to hack on, but the main annoyance was
> it left me with a directory full of about a hundred .h, .c. and .o
> files to clean out when I decided to quit the experiment.
> Now, I try to be smart with potentially disasterous commands like
> 'rm' and I personally have 'rm' aliased to 'rm -i' meaning to not
> erase a file without asking for confirmation. The problem is, if
> you have a whole directory full of garbage files to get rid of
> then if you go to that directory and do 'rm *' it will stop over
> and over again, asking about each file. The command 'rm -f' will
> NOT overrride 'rm -i' on this machine at least, although 'rm -f'
> will work in a script running in the background with its own shell
> regardless of what arguments I happen to have attached to 'rm'
> for my use in the foreground.
(rest clipped to save space)
Yeah, I've had bad cases of brain fade too.
I used to have a shell account and the admin had rm aliased to a
script that moved the files into a /save-for-delete directory (in the
users own space), and a chron job that flushed anything over 14 days
old.
I don't know enough shell script to write something like that, but
maybe you do, or know somebody who can. (If you do, please email me a
copy -- I'll be getting a Linux machine runnig here in a month or two
and would like to have that functionality).
Mike
David W. Tamkin
[TELECOM Digest Editor's Note: David wrote me to say that creating
an alias which was the name of the command itself was a very bad
arrangement. That letter is not available now, however I wrote
back to him asking why it was bad. PAT]
ptownson responded to dattier:
> What is the 'bad arrangement' where aliasing rm -i to rm is concerned?
> It has caught me a couple times when I almost erased something in error.
It's a very good idea to have an alias for "rm -i" and to be in the
habit of using the alias instead of the name of the executable most of
the time. It's shortsighted, though, to name the alias "rm". The
time can come when:
(1) you're using a different system and don't have your aliases in place;
(2) your own system for some reason didn't source your alias definitions;
(3) somehow your aliases got unset; or
(4) you're advising someone else who doesn't have your aliases in his or
her environment,
and you'll type or dictate "rm" in expectation of your tame substitute but
get the feral /bin/rm instead.
That's why the alias for "rm -i" should be named something like "delete" or
"del" or "erase"; you won't get into the habit of typing "rm" to mean some-
thing that isn't the same as rm(1). If any situations like those four exam-
ples come up, the worst that will happen is a "delete: not found" message.
As a general rule it's not advisable to make an alias name supersede a pro-
gram's name, but with something destructive like rm it's especially risky.
Because I was used to typing "scratch" for local files on my Commodore
equip- ment, I named my alias "scr". Sheer dumb luck (I didn't know
at the time that aliases outrank names of executables, so I didn't
know that it was possible to alias "rm -i" to "rm") kept me from
doing it; I can't claim to have acted out of knowledge or foresight.
But if ever my aliases weren't in place for some reason and I were to
type "scr *", I'd get "scr: not found" and lose no files.
Of course now, after years of typing "rm" to get "rm -i", if you
changed it suddenly you'd still be in the habit of typing "rm" instead
of your new alias, and then you'd still end up with feral /bin/rm.
[I know that this is not what happened to you on Monday. "/bin/rm *"
in the wrong directory will have the same results no matter what your
alias for "rm -i" is called.]
Scot E. Wilcoxon
You neglected to mention if you finally decided to make backups of your
recovered files.
Scot E. Wilcoxon sew...@fieldday.mn.org
[TELECOM Digest Editor's Note: Backups? What are those? PAT]
Christopher J. Whaley
The way around the rm -i alias is to use the direct path to the
command, e.g., "/bin/rm *". That bypasses the alias. Two other
suggestions. First, change the permissions on all of the files you
have which are "permanent" by using "chmod -w .*". This will take
write permission away from you and if you try to delete a file it will
prompt you with a message about "over-riding" the lack of write
permission. Second, make a copy of all of your dot files and other
scripts in another directory and on your hard disk.
Christopher J. Whaley
chris....@gtri.gatech.edu
Bill Walker
In article <telecom...@massis.lcs.mit.edu>, ptow...@massis.lcs.
mit.edu (TELECOM Digest Editor) wrote:
> Losing all the processing scripts in a clumsy session at the
> keyboard! What a way to wind up fifteen years of this Digest!
> And this weekend approaching does mark fifteen years of it and
> the start of year sixteen. Thanks to my heroes at lcs, I'll be
> around to annoy you, abuse you and otherwise amuse you for
> another year I guess.
Pat, here's a safety tip: change the file attributes on all your
valuable scripts (and other "permanent" files) and make them
read-only, so that you have to go change the file attributes before
you can delete them. That way, an inadvertant "rm *" won't blow them
away.
I'm not a Unix guy, but I know you can do this on DOS and VMS, so I've
gotta assume you can do it on Unix. I mean, really, if _MS-DOS_ can
do it, how hard can it be? :-)
Bill Walker, QUALCOMM, Inc., San Diego, CA USA
WWa...@qualcomm.com
[TELECOM Digest Editor's Note: All the archives files (except those
which rely on scripts to update them regularly) are set as you suggest
for read only. PAT]
Mark Brader
[TELECOM Digest Editor's Note: For the final item in this issue,
I saved the best for last. If this does not get your juices
stirred up and moving, I do not know what will. You might like
to save this article and put it up on a few bulletin boards around
your office. PAT]
Pat (ptow...@massis.lcs.mit.edu) writes:
> So I did 'unalias rm' then I did 'rm *' -- but the trouble is ...
(The audience leaps to its feat shouting "No!")
> I had ** forgotten to change directories to the one I wanted **.
Gee, you'd think he'd be nervous about typing *'s like that now. :-)
I'm reminded of the following article, which first appeared on Usenet
back in 1986. Familiarity with UNIX is assumed; that's what most of
Usenet ran on in those days.
--------------------------
Mario Wolczko wrote:
Have you ever left your terminal logged in, only to find when you came
back to it that a (supposed) friend had typed "rm -rf ~/*" and was
hovering over the keyboard with threats along the lines of "lend me a
fiver 'til Thursday, or I hit return"? Undoubtedly the person in
question would not have had the nerve to inflict such a trauma upon
you, and was doing it in jest. So you've probably never experienced the
worst of such disasters ...
It was a quiet Wednesday afternoon. Wednesday, 1st October, 15:15
BST, to be precise, when Peter, an office-mate of mine, leaned away
from his terminal and said to me, "Mario, I'm having a little trouble
sending mail." Knowing that msg was capable of confusing even the
most capable of people, I sauntered over to his terminal to see what
was wrong. A strange error message of the form (I forget the exact
details) "cannot access /foo/bar for userid 147" had been issued by
msg. My first thought was "Who's userid 147?; the sender of the
message, the destination, or what?" So I leant over to another
terminal, already logged in, and typed
grep 147 /etc/passwd
only to receive the response
/etc/passwd: No such file or directory.
Instantly, I guessed that something was amiss. This was confirmed
when in response to
ls /etc
I got
ls: not found.
I suggested to Peter that it would be a good idea not to try anything
for a while, and went off to find our system manager.
When I arrived at his office, his door was ajar, and within ten
seconds I realised what the problem was. James, our manager, was
sat down, head in hands, hands between knees, as one whose world has
just come to an end. Our newly-appointed system programmer, Neil, was
beside him, gazing listlessly at the screen of his terminal. And at
the top of the screen I spied the following lines:
# cd
# rm -rf *
Oh, shit, I thought. That would just about explain it.
I can't remember what happened in the succeeding minutes; my memory is
just a blur. I do remember trying ls (again), ps, who and maybe a few
other commands beside, all to no avail. The next thing I remember was
being at my terminal again (a multi-window graphics terminal), and
typing
cd /
echo *
I owe a debt of thanks to David Korn for making echo a built-in of his
shell; needless to say, /bin, together with /bin/echo, had been
deleted. What transpired in the next few minutes was that /dev, /etc
and /lib had also gone in their entirety; fortunately Neil had
interrupted rm while it was somewhere down below /news, and /tmp, /usr
and /users were all untouched.
Meanwhile James had made for our tape cupboard and had retrieved what
claimed to be a dump tape of the root filesystem, taken four weeks
earlier. The pressing question was, "How do we recover the contents
of the tape?". Not only had we lost /etc/restore, but all of the
device entries for the tape deck had vanished. And where does mknod
live? You guessed it, /etc. How about recovery across Ethernet of
any of this from another VAX? Well, /bin/tar had gone, and
thoughtfully the Berkeley people had put rcp in /bin in the 4.3
distribution. What's more, none of the Ether stuff wanted to know
without /etc/hosts at least. We found a version of cpio in
/usr/local, but that was unlikely to do us any good without a tape
deck.
Alternatively, we could get the boot tape out and rebuild the root
filesystem, but neither James nor Neil had done that before, and we
weren't sure that the first thing to happen would be that the whole
disk would be re-formatted, losing all our user files. (We take dumps
of the user files every Thursday; by Murphy's Law this had to happen
on a Wednesday). Another solution might be to borrow a disk from
another VAX, boot off that, and tidy up later, but that would have
entailed calling the DEC engineer out, at the very least. We had a
number of users in the final throes of writing up PhD theses and the
loss of a maybe a weeks' work (not to mention the machine down time)
was unthinkable.
So, what to do? The next idea was to write a program to make a device
descriptor for the tape deck, but we all know where cc, as and ld
live. Or maybe make skeletal entries for /etc/passwd, /etc/hosts and
so on, so that /usr/bin/ftp would work. By sheer luck, I had a
gnuemacs still running in one of my windows, which we could use to
create passwd, etc., but the first step was to create a directory to
put them in. Of course /bin/mkdir had gone, and so had /bin/mv, so we
couldn't rename /tmp to /etc. However, this looked like a reasonable
line of attack.
By now we had been joined by Alasdair, our resident UNIX guru, and as
luck would have it, someone who knows VAX assembler. So our plan
became this: write a program in assembler which would either rename
/tmp to /etc, or make /etc, assemble it on another VAX, uuencode it,
type in the uuencoded file using my gnu, uudecode it (some bright
spark had thought to put uudecode in /usr/bin), run it, and hey
presto, it would all be plain sailing from there. By yet another
miracle of good fortune, the terminal from which the damage had been
done was still su'd to root (su is in /bin, remember?), so at least we
stood a chance of all this working.
Off we set on our merry way, and within only an hour we had managed to
concoct the dozen or so lines of assembler to create /etc. The
stripped binary was only 76 bytes long, so we converted it to hex
(slightly more readable than the output of uuencode), and typed it in
using my editor. If any of you ever have the same problem, here's the
hex for future reference:
070100002c000000000000000000000000000000000000000000000000000000
0000dd8fff010000dd8f27000000fb02ef07000000fb01ef070000000000bc8f
8800040000bc012f65746300
I had a handy program around (doesn't everybody?) for converting ASCII
hex to binary, and the output of /usr/bin/sum tallied with our
original binary. But hang on---how do you set execute permission
without /bin/chmod? A few seconds thought (which as usual, lasted a
couple of minutes) suggested that we write the binary on top of an
already existing binary, owned by me...problem solved.
So along we trotted to the terminal with the root login, carefully
remembered to set the umask to 0 (so that I could create files in it
using my gnu), and ran the binary. So now we had a /etc, writable by
all. From there it was but a few easy steps to creating passwd,
hosts, services, protocols, (etc), and then ftp was willing to play
ball. Then we recovered the contents of /bin across the ether (it's
amazing how much you come to miss ls after just a few, short hours),
and selected files from /etc. The key file was /etc/rrestore, with
which we recovered /dev from the dump tape, and the rest is history.
Now, you're asking yourself (as I am), what's the moral of this story?
Well, for one thing, you must always remember the immortal words,
DON'T PANIC. Our initial reaction was to reboot the machine and try
everything as single user, but it's unlikely it would have come up
without /etc/init and /bin/sh. Rational thought saved us from this
one.
The next thing to remember is that UNIX tools really can be put to
unusual purposes. Even without my gnuemacs, we could have survived by
using, say, /usr/bin/grep as a substitute for /bin/cat.
And the final thing is, it's amazing how much of the system you can
delete without it falling apart completely. Apart from the fact that
nobody could login (/bin/login?), and most of the useful commands
had gone, everything else seemed normal. Of course, some things can't
stand life without say /etc/termcap, or /dev/kmem, or /etc/utmp, but
by and large it all hangs together.
I shall leave you with this question: if you were placed in the same
situation, and had the presence of mind that always comes with
hindsight, could you have got out of it in a simpler or easier way?
Answers on a postage stamp to:
Mario Wolczko
Dept. of Computer Science ARPA: miw%uk.ac.m...@cs.ucl.ac.uk
The University USENET: mcvax!ukc!man.cs.ux!miw
Manchester M13 9PL JANET: m...@uk.ac.man.cs.ux
U.K. 061-273 7121 x 5699
------------------------------------------
When I reposted the item in 1993, by the way, Hugh Grierson noted in a
followup that in addition to booting off tape or diskette, one might
have a spare "miniroot" partition on the hard disk, normally unmounted.
If you can boot off that partition, you're all set; if not, if you can
copy files you at least have a place to copy them from.
Mark Brader, m...@sq.com | "True excitement lies in doing
SoftQuad Inc., Toronto | 'sdb /unix /dev/kmem'" -- Pontus Hedman
My text in this article is in the public domain.
[TELECOM Digest Editor's Note: There! I told you you would love it!
Thanks very much for a nice close to this issue, Mark. PAT]
Marc SCHAEFER
> Now, I try to be smart with potentially disasterous commands like
> 'rm' and I personally have 'rm' aliased to 'rm -i' meaning to not
> erase a file without asking for confirmation. The problem is, if
That's very dangerous, because you get used to answering yes
anyway.
> you have a whole directory full of garbage files to get rid of
> then if you go to that directory and do 'rm *' it will stop over
> and over again, asking about each file. The command 'rm -f' will
> NOT overrride 'rm -i' on this machine at least, although 'rm -f'
> will work in a script running in the background with its own shell
> regardless of what arguments I happen to have attached to 'rm'
> for my use in the foreground.
Do a \rm which overrides any alias you did. Or type unalias rm
> did 'unalias rm' then I did 'rm *' -- but the trouble is I had
> ** forgotten to change directories to the one I wanted **.
Go to your friendly system administrator and ask for backups :-)
> "In the future we will not use tape backups to restore
> individual files. We will use them only for restoration
> after a disk failure or other major system problem. I have
> 400 other users to support here besides you."
There are automated backup software, like Networker for UNIX, allowing
any user to backup their own files and restore their files.
BTW: I suggest you take the backup tape with you.
Nevin Liber
I do an accidental rm about once a year. Welcome to Unix. :-(
The current trick I use (which saved me last month) is to do a
mkdir ./-i
chmod 000 ./-i
in every directory I create.
Now, when you do an
rm -rf *
it either (depending on your shell and settings) expands to
rm -rf -i ...
(since "-" is relatively early in sorting order) and the "-i" usually
overrides the "-f" since it appears later on the command line, or it
expands to
rm -rf -i/ ...
which is an error, since "-/" is not a command line option to "rm".
Note: I found that aliasing rm to "rm -i" doesn't work for me, since I
end up ignoring the messages if I am prompted for them each and every
time.
Nevin ":-)" Liber ne...@CS.Arizona.EDU (520) 293-2799
http://www.cs.arizona.edu/people/nevin/
js...@eddie.mit.edu
Here's how to avoid this in the future:
% mkdir .backup
Copy all the files you want to save into that directory. Put the
dot before it so you don't accidentally delete it.
Then make .backup2, which should contain a copy of all incoming mail
in .backup2/inbox. Clean this out every so often.
This way you should not need the backups.
Marc SCHAEFER
> suggestions. First, change the permissions on all of the files you
> have which are "permanent" by using "chmod -w .*". This will take
> write permission away from you and if you try to delete a file it will
> prompt you with a message about "over-riding" the lack of write
Far better is to remove the w from the directory: UNIX deletion
depends on the right to modify the containing directory.
First example: overridable-delete (dangerous if you have the
habit of answering y to whatever you see)
<mips> schaefer:/usr/var/tmp2/t1> ls
<mips> schaefer:/usr/var/tmp2/t1> touch a
<mips> schaefer:/usr/var/tmp2/t1> chmod 444 a
<mips> schaefer:/usr/var/tmp2/t1> rm a
rm: a: 444 mode. Remove (y/n)? y
Better security with:
<mips> schaefer:/usr/var/tmp2/t1> touch a
<mips> schaefer:/usr/var/tmp2/t1> chmod -w .
<mips> schaefer:/usr/var/tmp2/t1> rm a
UX:rm: ERROR: a not removed: Permission denied.
<mips> schaefer:/usr/var/tmp2/t1>
Some UNIX systems like HP/UX offer a temporary trashcan where your
deleted files go. This is also easy to implement with a small sh
script. Then you can add a purge command, executed manually, or on too
old files in this temporary trashcan directory.
The SunOS filemanager for example implements a manual trashcan.
> permission. Second, make a copy of all of your dot files and other
> scripts in another directory and on your hard disk.
I would say start by doing
cd important_place
tar cvf - . | gzip -9 > $HOME/backup.tar.gz
Can be extracted with
gzip -d < $HOME/backup.tar.gz | tar oxvf -
If you don't have gzip, use compress without the -9 argument. Some
operating systems use the z option of tar instead, but this one
should work.