My sysadmin toolbox: second helping
marc...@gmail.com
When I wrote last month's my sysadmin toolbox column, I knew that
Linux.com readers would probably have a few suggestions. I was
surprised, however, by the sheer number of responses we got from
readers with suggestions for other tools. With all those good
suggestions, it seemed like a good idea to compile a list of the most
popular reader-suggested tools and utilities to cover some of the
programs that didn't make the first column.
netcat
By far, the most popular suggestion from readers was to replace Telnet
with netcat. Netcat was so popular, in fact, that I decided to write up
an introductory CLI Magic column on netcat for readers who might not be
familiar with the joys of netcat.
Netcat can be used to test services in the same way that Telnet can be
used to test services, but is much more flexible, with additional
features you won't find with Telnet.
tcpdump
Readers also suggested tcpdump as an application that one should have
on hand when doing systems administration. As the name suggests,
tcpdump will print out (dump) packets passing through a network
interface, allowing an admin to eyeball network traffic more closely.
You can use the application to dump packets to standard out and watch
them fly by, or save them to a file to review later, or both.
The tcpdump utility is handy for a lot of troubleshooting and testing.
For instance, it lets you see what kind of traffic an application is
sending from your system -- and, sometimes more importantly, where it's
sending it to.
But tcpdump wouldn't be terribly useful if you had to look through all
traffic transmitted on a network interface, when you're only concerned
about a small subset of traffic. Using tcpdump, it's possible to get
very specific about what traffic is captured and presented. For
example, it's possible to tell tcpdump to display only traffic to one
host, or of one protocol, or packets that are transmitted over a
specific port -- or to exclude a specific host, port, or protocol and
display everything else. Combinations of those elements are also
possible, of course.
The bottom line is that tcpdump is invaluable in many situations.
Ethereal
While tcpdump is extremely useful at capturing packets for examination,
its presentation can leave something to be desired. Admins who need to
deal with this kind of data on a regular basis should look into the
Ethereal network protocol analyzer.
Ethereal can read tcpdump's capture files, as well as a slew of other
formats used by other popular tools, such as Sun's snoop and atmsnoop,
Microsoft's Network Monitor, Novell LANalyzer, HP-UX nettl, and many
others.
Unlike most of the tools that I've touched on so far, Ethereal is a GUI
application. It presents protocol information in an easy-to-read
fashion that makes analyzing captured traffic easy. According to the
Ethereal site, it "dissects" 724 protocols, including HTTP, AIM, IMAP,
NetBIOS, MySQL, and many, many more.
Ethereal also offers the ability to follow a specific TCP stream, can
create I/O graphs, provide traffic summaries and statistics by
protocol, and a lot more. If you're doing any serious work that
requires analyzing network traffic, you want Ethereal handy.
nmap
Need to scan your network, find out whether a specific IP address is in
use, or want to find out what services are running on a host? Nmap is a
must-have. As one anonymous reader pointed out, a quick nmap hostname
can show an admin what services are running on a host.
The nmap utility can also help determine whether a firewall is
stateful, or merely blocking incoming packets. It can usually detect a
remote host's operating system, and even the remote host's uptime. In
short, nmap can reveal a wealth of information about remote systems,
which can be extremely useful when you're examining your servers to see
what potential attackers might be able to learn about your systems.
MultiTail
Most admins are already familiar with using tail -f logfile to watch
system, application, and error logs when they're troubleshooting.
However, the tail utility only follows one file at a time. If you need
to watch two or more logfiles at the same time, which is fairly common,
the MultiTail utility by Folkert van Heusden is an excellent tool to
have handy.
Basically, MultiTail handles two or more files simultaneously, and
presents them in a split-screen view that makes it convenient to watch
two, three, or more logfiles at the same time. This has come in
particularly handy for me when troubleshooting issues with Web sites
and mail issues. Technically, one can replicate the MultiTail
experience by using GNU Screen to create a split screen environment and
using tail -f logfile multiple times -- but why? MultiTail makes it
simple.
find
Several readers suggested the find utility as an essential tool, and I
have to agree. As a rule, I don't need find to actually find files, but
I use it pretty often to perform operations on multiple files at the
same time.
As an example, if I wanted to find all files that have not been
modified for more than 60 days, I'd run find -mtime +60 to get the
list. If I wanted to do something with those files -- probably delete
them -- all that I would need to do is run find -mtime +60 -exec rm {}
\;. (I recommend running the find command without the -exec command the
first time, however, if you're going to delete or modify files, so that
there are no surprises.)
I've also used find to change ownership on multiple files. Let's say
you have a directory tree with a lot of files that are owned by user
bert, and you want to change the ownership to ernie. Brute force chown
-R ernie * isn't the answer, since some of the files may not be owned
by bert, and you don't want to touch them. But find -user bert -exec
chown ernie {} \; does the trick nicely, without modifying any files
that aren't owned by bert.
Find is, of course, also exceedingly handy for finding files. If you've
misplaced a file or two, you can track them down by file name, date,
file type (symlink, regular file, directory, etc.), user, group,
permissions, and so forth.
If you're new to the find utility, be sure to check out Joe Barr's
excellent CLI Magic column on GNU find.
xargs
The xargs utility is used to take output from other programs to build a
command line. For example, you might have a text file with a list of
files that you want to delete using rm. The rm command doesn't read
from a file, and won't accept output from cat directly -- so you can't
just cat a list of file names at rm and expect it to do the right
thing. However, xargs allows you to work around this limitation by
using cat to feed the list of file names to xargs, which can then send
the file names to the rm command, like this: cat filename | xargs rm.
That, of course, is a very simplistic demo of xargs' capabilities.
Another use for xargs is to limit the number of processes that are
executed at one time. This can be useful, for example, when deleting a
large number of files when the rm command complains that there are too
many arguments.
I don't use xargs more than once or twice a week, on average, but it is
extremely useful to have around when I need it.
AWK and Sed
A couple of readers deemed Sed and AWK as "must have" tools for systems
administration. Sed is a "stream editor," and mightily handy for
editing text on the fly. There are plenty of times when admins need to
process a stream of text, either when it comes directly from another
application, or when you're making changes to a large number of files
at the same time.
The AWK programming language is also well-liked by systems admins. The
implementation you'll find on Linux (and the one I use) is the GNU
Project's implementation, Gawk. Like Sed, AWK is the cat's meow for
manipulating batches of text files or extracting data from text files.
bash
Another popular suggestion is the bash shell itself. Again, I heartily
agree that bash is extraordinarily useful. I didn't count bash
originally, because I wasn't really thinking of it as a separate tool
-- it's part of the environment, like the basic utilities that one
expects to have on almost any *nix operating system: cp, mv, ls, ps,
top, and so forth.
But, the more I thought about it, I think bash does deserve some
recognition as a tool in its own right. If I'm using an OS that doesn't
have bash installed by default, say FreeBSD, installing bash is one of
the first things I do.
CVS
Most people think of Concurrent Versions System (CVS) as a tool for
developers rather than system administrators. However, there are a lot
of reasons why system administrators find CVS useful as well -- and I'm
not just talking about pulling down code from CVS repositories.
For example, many systems administrators check configuration files into
CVS to be able to provision new machines quickly, or to roll back to
known-good configurations if a modification munges things up. Some
admins are even known to use revision control to store their home
directory.
Of course, this isn't specific to CVS. Any of the popular revision
control systems, like Subversion or monotone, can serve the same
purpose.
Share your toolbox
Thanks again to all the readers who wrote in or posted comments with
great suggestions of tools and utilities for system administrators.
Even with the expanded list, I'm sure there are plenty of other tools
and utilities that we've yet to cover, and we'd also like to hear about
how they're being used. Tell us about your favorite tools and
utilities, and how you use them, and if we publish your sysadmin
toolbox article, we'll pay you $100.
marc...@gmail.com
Vim
Since I spend a lot of my time working with text files, either when I'm
writing and editing or when I'm mucking with configuration files and
shell scripts, I've become very attached to my editor of choice -- Vim.
Over the years, I've tried a lot of other editors, but none of them has
been sufficient to coax me away from Vim. Part of the reason for that
is the fact that I no longer have to think about using Vi-style
keybindings, and adjusting to anything else would seriously hinder my
productivity.
But there are also a lot of features in Vim that I use extensively --
split window modes, encryption, macros and text expansion, syntax
highlighting, and many other features that help shave time off of
repetitive tasks or generally make life a little easier. And, despite
having used Vim for several years, I'm still discovering new features
or tricks that make me a little more productive.
GNU Screen
The first time I read about GNU Screen, I thought it was a bit of a
silly program. Why not just open a bunch of xterms, instead of mucking
about with all the complex keybindings used by Screen to switch between
its windows? Sure, it might be useful if you only work from the
console, but how many folks really do that anymore?
Then I actually spent a little bit of time with Screen, and I found out
what a valuable utility it really is. Instead of worrying about dozens
of xterms, I could have a single terminal window with multiple Screen
windows that I could switch between easily and quickly.
The other thing I have found to be extraordinarily useful about Screen
is that it can be used as a kind of VNC for the console. Screen allows
a console session to be "detached" from the console or xterm, so that
the job is still running but not displayed on that console any longer.
You can then resume the screen session from the same console or
terminal emulator, or you can resume the session from another console
entirely.
Let's say you're at work, and you start a long-running process that
you'll need to monitor at periodic intervals, but you don't want to
have to stay at work for the next few hours to watch it. Simply fire up
a Screen process, start your job, then head home and reattach to the
Screen process and see the status of that job.
Screen even allows you to monitor a window for its status, so that
activity or a period of inactivity will send a message to the current
Screen window. This way you can write an email message while waiting
for that long-running job to finish -- and be notified as soon as it's
finished, rather than having to switch back and forth periodically.
CDargs
CDargs provides a browser and bookmarks for navigating directories at
the shell. I recently covered it in detail.
This utility makes it easy to navigate the directory structure without
doing a lot of typing, or doing silly things like creating symlinks to
directories deeper in the directory structure, or having to define
shell variables.
CDargs has two basic modes of operation. One is the shell browser that
allows you to navigate using a text menu; the other uses the shell
builtin cdb and your bookmarks. For example, instead of having to type
cd /var/www/website/cgi-bin, you can create a bookmark for the
directory called "cg" (or whatever you prefer) and use cdb cg. This
utility has saved me many a keystroke over the years.
GNU Wget
GNU Wget is a non-interactive download tool that can grab files via
HTTP, HTTPS, and FTP.
Want to retrieve a long list of files, like the CD ISOs for the SUSE
10.0 release? Just feed Wget a text file with the URLs, and let it run
in the background while you work on other tasks. Wget is also useful
for grabbing MP3 albums from sites such as Magnatune.
Wget works well in shell scripts where you need to retrieve files
without any user intervention. It supports cookies, authentication,
proxies, and many other features that make it ideal for scripting and
even for testing Web sites.
Domain Internet Groper (DIG)
If you find yourself spending time troubleshooting DNS issues, you'll
find DIG useful. The DIG utility is actually part of the BIND
distribution, but it can be (and often is) installed separately from
BIND -- usually as part of a DNSutils package for most distros.
With DIG, you can perform DNS lookups and query nameservers directly in
order to troubleshoot DNS. Many people would be surprised by how many
problems in a Web services environment can be traced back to problems
with DNS, so it's a good idea to save yourself time and trouble by
testing the obvious problems -- such as DNS failures -- first.
pwgen
Another program that I use frequently is pwgen, a utility to create
random passwords. If you need to create user accounts with random
passwords, this is a quick and easy way to come up with reasonably
secure passwords.
Pwgen creates passwords that are supposed to be easy to memorize, and I
have found that to be true, most of the time. This is a major bonus
when you need to remember a large number of passwords over time, which
is a pretty common condition for admins and for users who work in
environments that require a new password at regular intervals. (I could
explain why I think mandatory password aging is a very bad idea, but
that's a topic for another day.)
If you use pwgen interactively without passing it any arguments or
options, it will generate a screenful of passwords that you can choose
from. If you use pwgen in a script or pipe the output to another
command, it will simply generate one password -- which makes it useful
if you need to incorporate it into a script to generate new user
accounts.
By default, pwgen creates passwords that are eight characters long,
with at least one capital letter and one numeral. The passwords are
pronounceable, sometimes with a little imagination, but not taken from
the dictionary -- so there's very little likelihood that a dictionary
attack would work against any password generated by pwgen.
abcde
In the past, I did most of my CD-to-MP3 conversion with Grip, but since
I was turned on to abcde, I've started using it more and more.
Basically, abcde (like Grip) is a front-end for cdparanoia, lame, and
other utilities to convert an audio CD to your favorite digital audio
format. (Assuming your favorite digital audio format happens to be WAV,
MP3, Ogg/Vorbis, Ogg/Speex, Flac, or another supported format....)
Once you configure abcde, all you need to do is pop the CD into the
drive, run abcde, and let it rip.
Sure, this doesn't sound like a productivity tool, but it's hard to
spend eight (or more) hours at a computer without a little music.
Checkinstall
I prefer to install software from native packages meant for my
distribution, from the vendor or project that publishes the distro, as
much as possible. This has a number of advantages, not the least of
which is that I don't have to recompile the software myself whenever
there's a security update or bug fix for the software.
However, there are a lot of instances where this just isn't possible.
Maybe the software isn't available for the distro that I'm using, or I
need to use a newer version that's not available from the vendor or
project. In those cases, I still like to build a native package so that
it's easier to manage than just installing directly from source code.
For those cases, I use CheckInstall to simplify creating a package.
CheckInstall lets me create an RPM, Debian package, or Slackware
package almost as easily as compiling software from source. Instead of
running ./configure; make; make install I just run ./configure; make;
checkinstall.
I've been using CheckInstall for several years, and I've never run into
a problem with any of the packages it creates. It's also handy for
rolling packages for limited distribution -- for example, when I want
to install a piece of software on three or four machines without
needing to compile it separately on each system.
Telnet
Okay, why on Earth would anyone still want to use Telnet? Rather than
using Telnet for connecting to a remote shell, I use Telnet to test
services.
For example, if I want to test a mail server, I can simply run telnet
servername 25 to connect to the remote server and test sending a
message via SMTP. This works well with a number of protocols: POP3,
IMAP, HTTP, and even HTTPS and IMAPS if you have a Telnet client with
SSL support. If you "speak" the protocol you're testing, you can find
out a lot using Telnet to "talk" directly to a service.
What I don't use Telnet for is to log into remote systems for work at
the shell. If I need to log into a remote system, which is pretty
often, I turn to OpenSSH instead.
rsync
Another tool I use frequently is rsync, a utility that provides
incremental file transfer, meaning that rsync doesn't need to send the
entire file every time. It can also update entire directory structures,
omit specific types of files (such as .bak files), and generally
provides an excellent way to synchronize files from one host to
another.
I use three different workstations on a regular basis, and the best way
for me to keep important files synchronized is to use rsync to sync up
the files. I run a short script every hour that synchronizes my
configuration files and my personal "bin" directory with my scripts,
and another script that copies my writing, MP3s, and other data every
night. Instead of backing up to optical media or a tape drive, I just
back up my data on multiple machines with rsync.
Summary
So there you have it -- my top 10 in order of general usage. There are
plenty of other useful tools for admins out there, and a number of
other utilities that I use regularly. Maybe some of them are on your
top 10 list. Let us know about your most valuable utilities (there need
not be 10 of them), and if we publish your work, we'll pay you $100.