comp.unix.shell FAQ - Answers to Frequently Asked Questions
Joe Halpin
Joe Halpin
Posting-Frequency: monthly
Version: $Id: cus-faq.html,v 1.6 2004/05/02 20:56:41 jhalpin Exp jhalpin $
Maintainer: Joe Halpin
This FAQ list contains the answers to some Frequently Asked Questions
often seen in comp.unix.shell. It spells "unix" in lower case letters
to avoid arguments about whether or not Linux, FreeBSD, etc are
unix. That's not the point of this FAQ, and I'm ignoring the issue.
This document as a whole is Copyright (c) 2003 Joe Halpin. It may be
copied freely. Exceptions are noted in individual answers.
Suggestions, complaints, et al, should be sent to the maintainer at
j....@comcast.net or posted to comp.unix.shell
There are two levels of questions about shells.
One is the use of the shell itself as an interface to the operating
system. For example, "how do I run a program in the background, and go
on with other things?". Or "how do I setup environmental variables
when I log in?".
The other level is how to write shell scripts. This often involves
having the shell execute unix utilities to perform part of the work
the shell script needs to accomplish, and requires knowledge of these
utilities, which isn't nominally in the scope of shell
programming. However, unless the question involves something other
than standard unix utilities, it should be included in this FAQ.
Standard unix utilities are defined by either POSIX or the Single Unix
Specification. These are now joined and are normally abbreviated as
"POSIX/SUS". This specification can be found at
http://www.opengroup.org/onlinepubs/007904975/toc.htm
The man pages found on that web page define standard behavior for any
given utility (or the shell itself). However, you should also check
the man page on your system for any utility or shell you need to
use. There isn't always a perfect correspondence between the standard
and a particular implementation (in fact, I'm not sure there's any
case in which they perfectly correspond).
There is also an Austin Group FAQ, which describes the standardization
effort in more detail ls
at
http://www.opengroup.org/austin/faq.html
Other good web sites that provide information about shells and shell
programming (including OS utilities) include:
http://www.shelldorado.com/
http://cfaj.freeshell.org/shell/
http://www.faqs.org/faqs/by-newsgroup/comp/comp.unix.shell.html
This FAQ is available at
http://home.comcast.net/~j.p.h/
http://www.newsville.com/cgi-bin/getfaq?file=comp.unix.shell/comp.unix.shell_FAQ_-_Answers_to_Frequently_Asked_Questions
---------------------------
The predictable legal stuff
---------------------------
The answers given in this FAQ list are provided with the best
intentions, but they may not be accurate for any particular
shell/os. They may be completely wrong for any shell/os. If you don't
test the answers, that's a bug in your procedures.
There are no guarantees for the answers or recommendations given in
this document. In fact, I don't even claim to have tested any or all
of them myself. Many of the answers here have been contributed by one
or more regular participants in the newsgroup, who I believe to be
competent (certainly more competent than I am), but THERE ARE NO
GUARANTEES.
Did I really need to make that all uppercase? Hopefully not, but there
are a lot of lawyers around with too much time on their hands, so I
want to make it clear that THERE ARE NO GUARANTEES
about the accuracy of answers in this FAQ list. This is, hopefully, an
aid to people trying to learn shell programming, but it is NOT a
supported product. You have to figure out for yourself whether or not
the answers here work for what you're trying to do.
Under no circumstances will the maintainer of this FAQ list, or any
contributors to it, be held liable for any mistakes in this
document. If the answers work for you, well and good. If not, please
tell me and I'll modify them appropriately so that this will be more
useful.
If you don't agree to that, don't read any farther than this. Reading
beyond this point indicates your agreement.
If you do test the answers and find a problem, please send email to
the maintainer (see above), so it can be corrected, or (preferably)
post a question to the newsgroup so it can be discussed and corrected
if there's a problem.
A number of people have contributed to this FAQ, knowingly or
unknowingly. Some of the answers were taken from previous postings in
the group, and other people contributed questions and answers
directly to the maintainer, which you are welcome to do as well.
Among the contributors is Heiner Steven, who also provided the
momentum to get this FAQ list started. He maintains a web site about
shell programming that has a lot of good stuff in it.
======================================================================
CONTENTS:
0 COPYING
0a. Glossary
Google
POSIX/SUS ("the standard")
UUOC
dotfile
portable
race condition
shebang
shells
top-posting
0b. Notes about using echo
1. How can I send e-mails with attached files?
2. How can I generate random numbers in shell scripts?
3. How can I automatically transfer files using FTP with error checking?
4. How can I remove whitespace characters within file names?
5. How can I automate a telnet session?
6. How do I do date arithmetic?
7. Why did someone tell me to RTFM?
9. How do I create a lock file?
10. How can I convert DOS text files to unix, and vice versa?
11. How can a shell prompt be set up to change the title of xterm?
12. How do I batch an FTP download/upload?
13. How do I get the exit code of cmd1 in cmd1|cmd2
14. Why do I get "script.sh: not found"
15. Why doesn't echo do what I want?
16. How do I loop through files with spaces in their name?
17. how do I change my login shell?
18. When should I use a shell instead of perl/python/ruby/tcl...
19. Why shouldn't I use csh?
20. How do I reverse a file?
21. How do I remove last n lines?
22. how do I get file size, or file modification time?
23. How do I get a process id given a process name? Or, how do I find out if a process is still running, given a process ID?
24. How do I get a script to update my current environment?
25. how do I rename *.foo to *.bar?
26. How do I use shell variables in awk scripts
27. How do I input the user with a timeout?
28. How do I get one character input from the user?
29. why isn't my .profile read?
30. why do I get "[5" not found in "[$1 -eq 2]"?
31. How do I exactly display the content of $var (with a \n appended).
32. How do I exactly display the content of $var (without a \n appended).
33. How do I split a pathname into the directory and file?
34. How do I make an alias take an argument?
35. How do I deal with a file whose name begins with a weird character
36. Why do I lose the value of global variables that are set in a loop
Appendix A: Some example scripts
Appendix B: References. These correspond with numbers in square
brackets (e.g. [1]) which may appear in the text.
======================================================================
ANSWERS
0 COPYING
Some contributors may copyright their submissions and license them
differently than this document.
[1] Chris F.A. Johnson. Examples marked with COPYING[1] were
contributed by Chris F.A. Johnson. He has copyrighted these
examples, and licensed them under the GNU General Public
License (GPL). Copying them directly into another script will
cause that script to also come under the GPL. For details see
http://www.fsf.org/licenses/licenses.html
0a.Glossary
-------------------------------
Google
Google is one of the search engines on the Internet. It took
over dejanews some years ago, and now is the standard reference
when directing someone to a past thread one some topic. This is
a very good place to start when researching a question about
shell programming (and just about anything else).
http://groups.google.com/advanced_group_search
-------------------------------
POSIX/SUS ("the standard")
POSIX (Portable Operating System Interface) and SUS (Single Unix
Specification) have been joined into one standard. This is what
people usually mean when they refer to "the standard" in
discussions about unix. When people in this group refer to the
POSIX shell, they are talking about the shell prescribed by this
specification. You can find this standard at
http://www.opengroup.org/onlinepubs/007904975/toc.htm
-------------------------------
UUOC
This is short for "Useless use of cat". It's used to point out
that some example script has used cat when it could have used
redirection instead. It's more efficient to redirect input than
it is to spawn a process to run cat. For example
$ cat file | tr -d 'xyz'
runs two processes, one for cat and one for tr. This is less
efficient than
$ tr -d 'xyz' < file
In general, "cat file | somecommand" can be more efficiently
replaced by "somecommand < file"
or (especially for multi-file input)
$ somecommand file [file ...]
but check the man page for "somecommand" to find out if it will
accept this syntax.
For more details about this, as well as other things like it, see
http://rhols66.adsl.netsonic.fi/era/unix/award.html
-------------------------------
dotfile
This refers to a file which starts with '.' (a dot). These files
are not shown in directory listings without the -a (or -A in
newer versions of ls - check the man page on your system) option
to ls. Often they are configuration files, subdirectories used
by applications to store configuration files, NFS swap files, et
al.
-------------------------------
portable
The word "portable" means different things to different people,
in different situations, which is to say, there isn't one
definition of "portable".
At one extreme, a portable script is one which will work under
any shell, on any operating system. At this end of the spectrum,
there is no such thing as a portable shell script (some
operating systems don't even have shells). If we confine the
operating system to unix (which would make sense since this is
comp.unix.shell), the only truly portable scripts are those
which make no use of built-in shell facilities or syntax, but
which only call external utilities. For example
echo Hello World
would probably qualify. However, that doesn't do anyone much
good.
Given that there are probably few (if any) scripts which have to
meet such a standard, a more frequent use of the word "portable"
indicates the degree to which a script will run under different
shells and/or different operating environments.
For example, if you're writing an installation script for an
application, and the platforms on which that application runs
are defined, then the problem is pretty well bounded. The choice
of shell is one which is available on all required platforms,
and the syntax to be used is the smallest subset of all the
variants of that shell on the target platforms.
The degree to which your shell script needs to be portable has
to be determined by you, or the requirements you've been given
for the script.
-------------------------------
race condition
This is a situation in which two entities (processes, threads,
etc) are trying to access a shared resource, or perform the same
action, and the result depends on the order of execution of the
two entities.
-------------------------------
shebang
This is the first line of a shell script, which indicates to the
operating system which interpreter (shell) it should invoke to
interpret the script. It has the form
#!/path/to/shell [ argument ]
where /path/to/shell might be /bin/sh, /usr/local/bin/bash, etc.
This line is only interpreted by the operating system. That is,
if a shell script (test.sh) is executable and run from the
command line by typing its name and giving the script as an
argument, as in
$ sh test.sh
then sh interprets test.sh. For sh, the shebang line is simply a
comment, and is ignored.
-------------------------------
shells
A number of shells are discussed in this group, including
sh
csh
pdksh
ksh88
ksh93
tcsh
zsh
rc
es
bash
ash
dash
These (and more) are names of shells which are referenced in the
group. A comparison of some of these is available at
http://www.faqs.org/faqs/unix-faq/shell/shell-differences/
However, it does not make specific the differences between
ksh88, ksh93 and pdksh, which are not entirely compatible.
-------------------------------
top-posting
A. top posting
Q. What's the most irritating way to respond on usenet?
Please see the following:
http://catb.org/~esr/jargon/html/T/top-post.html
http://www.uwasa.fi/~ts/http/quote.html
http://members.fortunecity.com/nnqweb/
http://www.guckes.net/mail/editing.html
======================================================================
0b. Notes about using echo
This isn't really a FAQ, but discussions about using echo come up
often enough that it seems reasonable to have something about it in
the FAQ list.
The echo command is not consistent in the handling of its arguments
from implementation to implementation. Sometimes a string with
backslash quoted characters will be interpreted in one way, and
sometimes another.
Also, if the string being echoed wasn't built into the script
itself, then it could have shell metacharacters in it, which could
confuse things. In cases where external input is used to build a
string to be echoed the string typically should be quoted.
For example
s="a string with\na newline and\ta tab"
Following are some results with various shells:
------
bash:
$ echo "$s"
a string with\na newline and\ta tab
$ echo -e "$s"
a string with
a newline and a tab
-------
pdksh:
$ echo "$s"
a string with
a newline and a tab
$ echo -e "$s"
$ echo "$s"
a string with
a newline and a tab
--------
ksh88:
$ echo "$s"
a string with
a newline and a tab
$ echo -e "$s"
-e a string with
a newline and a tab
-------
ksh93:
$ echo "$s"
a string with\na newline and\ta tab
$ echo -e "$s"
-e a string with\na newline and\ta tab
Note that ksh93 makes the handling of arguments system dependent
when they contain '\', and/or the first argument begins with '-'.
http://www.cs.princeton.edu/~jlk/kornshell/doc/man93.html
POSIX does not allow the -e option. It also makes the result of
using -n or any string with '\' in it implementation-defined.
However, on XSI-conforming systems, it disallows options, and
defines the use of backslash-quoted characters.
In general, the behavior of echo is system and/or shell dependent
if its arguments contain a backslash, or its first argument is -n
or -e.
The biggest problem with echo is when using it to output strings
that the script got externally (e.g. user input, or reading from a
file). These strings may have '\' characters in them for
example. In this case, results may not be what you expect.
print is available in some shells, although printf(1) is perhaps
more portable. Additionally, a here document will give predictable
results in that it will not expand escape sequences.
cat <<EOF
$s
EOF
produces
a string with\na newline and\ta tab
So consider not using echo unless you are sure what will happen,
given the shell you're using.
======================================================================
1. How can I send e-mails with attached files?
a. Use uuencode
This is the simplest way to do this. For example
$ uuencode surfing.jpeg surfing.jpeg | mail som...@some.where
To send regular text as well
$ (cat mailtext; uuencode surfing.jpeg surfing.jpeg) |
mail som...@some.where
b. Use MIME
$ metasend -b -t som...@some.where -s "Hear our son!" \
-m audio/basic -f crying.au
These examples are adapted from
http://www.shelldorado.com/articles/mailattachments.html which
goes into much more detail about this.
c. Use pine (with a patch) or mutt
======================================================================
2. How can I generate random numbers in shell scripts?
This depends on the shell, and the facilities available from the
OS.
a. Some shells have a variable called RANDOM, which evaluates to a
different value every time you dereference it. If your shell has
this variable,
$ number=$RANDOM will produce a random number.
b. Some systems have a /dev/urandom device, which generates a
stream of bits. This can be accessed using the dd(1) utility. An
example of this (from a more extensive discussion of different
techniques at http://www.shelldorado.com/scripts/cmds/rand)
n=`dd if=/dev/urandom bs=1 count=4 2>/dev/null | od -t u4 | \
awk 'NR==1 {print $2}'`
also:
od -vAn -N4 -tu4 < /dev/urandom
c. Use a utility such as awk(1), which has random number generation
included. This approach is the most portable between shells and
operating systems.
awk 'BEGIN {srand();print rand()}'
However, note that if you call this line more than once within
the same second, you'll get the same number you did the previous
time.
======================================================================
3. How can I automatically transfer files using FTP with error
checking?
First, there are tools to do that: curl, wget, lftp, ncftp. But,
they are generally not part of the base system (you need to
install them).
zsh (version 4 and above) provides a FTP facility, see "info -f
zsh -n 'zsh/zftp Module'"
#! /usr/bin/zsh
zftp open host user passwd || exit
zftp get /remote/file > /local/file; r=$?
zftp close && exit r
With your system "ftp" command, two ways:
1- using "ftp -n". Without the -n option, ftp expects user
interaction to enter the password, so you'd need to use
"expect". With "-n", you provide the user and password as any
other FTP command.
#! /bin/sh
ftp -n << EOF
open ftp.domain.org
user anonymous ${LOGNAME:-`who am i`}@
binary
get /remote/file /local/file
bye
EOF
The error checking can't be made correctly (if "open" fails, the
"user" command will be still sent even if it shouldn't).
2- using ~/.netrc
If you put:
<<
machine ftp.domain.org
login mylogin
password mypasswd
macdef init
binary
get /remote/file /local/file
bye
>>
(with the trailing empty line) in your ~/.netrc (ensure it's not
world readable) and then run "ftp ftp.domain.org", ftp will find
the matching "machine" entry in your ~/.netrc and use the
parameters provided there to make the ftp transaction.
Those work at least on Linux, FreeBSD, Solaris, HPUX
======================================================================
4. How can I remove whitespace characters within file names?
File names in unix can contain all kinds of whitespace characters,
not just spaces. The following examples only work with spaces,
adjust accordingly.
a. Use the substitution capabilities of awk, sed, et al.
f=`printf '%s\n' "$filename" | sed 's/ /_/g'`
f=`printf '%s\n' "$filename" | awk '{gsub(" ","_");print $0}'`
f=`printf '%s\n' "$filename" | tr ' ' _`
Add characters to the tr command line as needed (see the man
page for tr to find out the available escape sequences).
Additionally (although not exactly a one-liner)
f=`tr ' ' _ <<EOF
$filename
EOF
`
See section 0a "Notes about using echo" for why echo is not used
here.
b. Use the substitution capabilities of the shell if it has
them. Check the man page for your shell (probably under a
section named something like "Parameter expansion") to see. For
example:
f=${filename// /_}
With zsh:
autoload -U zmv
zmv '* *' '$f:gs/ /_/'
======================================================================
5. How can I automate a telnet session?
This is outside the realm of shell programing, per se. You need
a more special purpose scripting language such as expect. See
http://expect.nist.gov/
Perl scripts can also do this with the Telnet module from CPAN.
======================================================================
6. How do I do date arithmetic?
This depends on exactly what you have in mind.
a. Finding yesterday's date
The GNU version of date has some nice features in this
respect. For example
To find yesterday's date
$ date --date yesterday
To find tomorrow's date
$ date --date tomorrow
See the man page for GNU date for other options. It can also
provide dates more than one day in the past/future.
The FreeBSD version of date also provides extensions that can
do things like this.
$ date
Wed Oct 22 13:48:29 CDT 2003
$ date -v-1d
Tue Oct 21 13:45:16 CDT 2003
Playing with the TZ variable isn't a reliable method. If you
need to do something like this, but don't have GNU or FreeBSD
date available, see section g. "Arbitrary date arithmetic".
b. Determining relative ages of files
If you want to determine whether or not one file is older than
another, you can (with bash, pdksh, ksh93) do
$ [[ file1 -ot file2 ]] && echo file1 is older
or you can use find to search a directory tree for files that
are newer/older than some file:
$ find . -name '*.c' -newer test.c
c. Finding elapsed time
If you want to find elapsed time, perhaps because you want to
know when some operation has timed out, some shells (bash, ksh
zsh [,??]) have a SECONDS variable which tell how many seconds
have elapsed since the invocation of the shell, or since the
last time it was set.
ksh93 has a floating point SECONDS which is locale dependent.
zsh 4.1 one can be made floating point with: float SECONDS
zsh 4.1 also has $EPOCHSECONDS for seconds since 1970-1-1 0:0:0
UTC (see zsh/datetime module).
d. Determining leap year
A leap year in the Gregorian calendar is defined as a year which
is evenly divisible by 4, however, if it's also evenly divisible
by 100 then it's not a leap year unless it's also evenly
divisible by 400. It gets worse than that, actually, but this
is as far as I go :-).
In the Julain calendar which was used before in Europe, only the
years divisible by 4 where leap years.
The standard "cal" utility performed the switch between Julian
and Gregorian calendar in september 1752 (see cal 9 1752) which
corresponds to the date used in England. The Gregorian calendar
(created by Pope Gregory III) was first used in 1582 in many
other countries.
One possibility for a ksh function to do this (after 1600 AD/CE)
is
isleap()
{
y=$1
four=$(( $y % 4 ))
hundred=$(( $y % 100 ))
fourhundred=$(( $y % 400 ))
if [ $four -eq 0 ];then
if [ $hundred -eq 0 ];then
if [ $fourhundred -eq 0 ];then
echo leap year
else
echo not a leap year
fi
else
echo leap year
fi
else
echo not a leap year
fi
}
Or, valid with any date with the same calendar switch day as
POSIX cal's (POSIX syntax):
is_leap_year() # args: year
# NB: year before year 1 is year -1, not 0.
{
[ "$1" -lt 0 ] && set -- "$(($1 + 1))"
[ "$(($1 % 4))" -eq 0 ] && {
[ "$(($1 % 100))" -ne 0 ] || [ "$(($1 % 400))" -eq 0 ] \
|| [ "$1" -le 1752 ]
}
}
Or in any Bourne shell (see COPYING[1]):
is_leap_year() { ## USAGE: is_leap_year [year]
isl_year=${1:-`date +%Y`}
case $isl_year in
*0[48] |\
*[2468][048] |\
*[13579][26] |\
*[13579][26]0|\
*[2468][048]00 |\
*[13579][26]00 ) _IS_LEAP_YEAR=1
return 0 ;;
*) _IS_LEAP_YEAR=0
return 1 ;;
esac
}
On FreeBSD, use the -f option to date(1) to pass in the
(supposed) February 29 in the current year and then print it the
day of the month again to see if there really is such a date
(note that you need -j as well as -f, otherwise date(1) thinks
you want to set the clock):
if [ $(date -jf%Y%m%d $(date +%Y0229) +%d) = 29 ]; then
echo Leap year!
fi
e. Determining the last day of a month.
There are a number of possibilities for doing this which have
been mentioned in the group. The following is a sampling:
In any Bourne-type shell (in conjunction with is_leap_year() as
given above, when month is February) (see COPYING[1]):
days_in_month() { ## USAGE: days_in_month [month [year]]
if [ -n "$1" ]
then
dim_m=$1
dim_y=$2
else
eval `date "+dim_m=%m dim_y=%Y"`
fi
case $dim_m in
9|09|4|04|6|06|11) _DAYS_IN_MONTH=30 ;;
1|01|3|03|5|05|7|07|8|08|10|12) _DAYS_IN_MONTH=31 ;;
2|02) is_leap_year ${dim_y:-`date +%Y`} &&
_DAYS_IN_MONTH=29 || _DAYS_IN_MONTH=28 ;;
esac
[ ${SILENT_FUNCS:-0} -eq 1 ] || echo $_DAYS_IN_MONTH
}
With GNU date:
year=2003
month=9
date -d "$year/$month/1 +1 month -1 day" +%d
With FreeBSD date use the -v-1d option to date(1) to get the day
before the first day of the next month:
$ MONTH=12
$ date -v-1d -jf%Y-%m-%d $(date +%Y-$(((MONTH+1)%12))-01) +%d
31
In the shell using cal (But beware of implementations of cal
which print more than one month at a time):
month=9 ; year=2003 # adjust
##
for lday in `cal $month $year` ; do : ; done
echo $lday
## or
set -- `cal $month $year` ; eval lday=\${$#}
echo $lday
In ksh, bash and zsh:
: $(cal)
days_in_month=$_
In zsh:
days_in_month=${$(cal)[-1]}
f. Determining the day of the week for a given date.
This algorithm is known as Zeller's congruence. An explanation
of it is available from the Dictionary of Algorithms and Data
Structures web page at NIST:
Also, a fuller explanation is available at
http://www.merlyn.demon.co.uk/zeller-c.htm#ZC
An example in C, with a short explanation, is given at
http://wwwcdf.pd.infn.it/MLO/Calendars/Notes.html#zeller
A shell (ksh93) implementation of a homework assignment (given
for illustration only - don't turn this in as yours - you might
be sorry if it's wrong :-)
dayofweek()
{
# Implementation of a homework assignment given at
# http://carbon.cudenver.edu/~traup/fa02/lec/hw3.html
#
# call with day: 1 - 31
# month: March = 1, Jan & Feb are months 11 and
# 12 of the previous year.
# year: The year of the century
# c: The previous century
#
# For example, for July 4, 1989,
# m = 5, d = 4, y = 89, and c = 19,
# while for January 25, 1989,
# m = 11, d = 25, y = 88, and c = 19.
#
# The output is the day of the week with Sunday = 0,
# Monday = 1, etc.
d=$1
m=$2
y=$3
c=$4
A=$(( ($m * 13 - 1) / 5 ))
B=$(( $y / 4 ))
C=$(( $c / 4 ))
D=$(( $A + $B + $C + $d + $y - ($c * 2) ))
echo $(( $D % 7 ))
}
On FreeBSD, use the -f option to date(1) to pass in the date of
interest and +%A to print the day of the week:
$ date -jf%Y-%m-%d 2000-01-01 +%A
Saturday
(Use +%u or +%w if you want the weekday as a number. See the
strftime(3) manpage for details.)
g. Arbitrary date arithmetic
To do arbitrary date calculations is more complicated. One
possibility is to call an external utility, or a program in
another scripting language, which has this built in. For
example, perl has wrappers for the unix time functions built in,
so it can provide some relief in this regard. C programs can
also be easily written to do date arithmetic (see the examples
section). One thing to keep in mind, however, is that unix time
functions are, strictly speaking, limited to the range of time
between January 1 1970 at midnight, and January 19, 2038 at
3:14:07. C/Perl programs which calculate dates outside this
range might work, or they might not, that would depend on the
implementation.
To do arbitrary date arithmetic in the shell itself is also
possible. An article provided on the web by SysAdmin magazine
describes one way to do this.
http://www.samag.com/documents/s=8284/sam0307b/0307b.htm
Another possibility is given in the examples section, from
http://groups.google.com/groups?selm=n6d6zalnpk.fsf%40ogion.it.jyu.fi
On FreeBSD, the -f and -v options to date(1) cover most things
you might want to do, with the caveat that only dates within the
range mentioned above are defined. Dates outside that range are
not guaranteed to work.
Also, zsh 4.1 hash the zsh/datetime module that provides the
$EPOCHSECONDS and the strftime function.
h. Getting the number of seconds since the epoch
- GNU date has the %s format option which returns the epoch
time.
- More portably, use awk.
awk 'BEGIN {srand(); printf("%d\n", srand())}'
This works because srand() sets its seed value with the
current epoch time if not given an argument. It also returns
the previous seed value, so the second call gives the epoch
time.
Note that this doesn't work with older versions of awk. This
requires a version supporting the POSIX spec for srand(). For
example, on Solaris this will not work with /usr/bin/awk, but
will with nawk or /usr/xpg4/bin/awk.
Depending on scheduling, when the call is actually executed,
etc, this might be off by a second.
- Another way is to use perl if you have it.
perl -le 'print time'
- Also, zsh 4.1 hash the zsh/datetime module that provides the
$EPOCHSECONDS and the strftime function.
======================================================================
7. Why did someone tell me to RTFM?
Because you didn't :-)
RTFM is part of Usenet lingo, and means "Read The F-ing Manual".
Generally people say this when someone asks a question that is
asked so often, and is answered plainly in some relevant man page,
that they're tired of seeing it asked.
http://catb.org/~esr/jargon/html/R/RTFM.html
So RTFM, and the FAQs first before asking. Also, if you're new to
the group, search Google Groups
http://groups.google.com/advanced_group_search
before asking questions. And please don't post your homework
questions to the group unless you've tried to figure them out, and
have some specific questions. People will generally be happy to
help you with your homework if you post what you've got and ask
specific questions.
======================================================================
9. How do I create a lock file?
Very carefully :-)
The scheduler can stop one process in the middle of a non-atomic
operation, and run another one, which wants to perform the same
operation. The second one, having a full timeslice, might finish
the operation. When control returns to the first process, confusion
will reign.
The trick is to do something atomic, so that this won't
happen. There are a couple ways to do this. One is to create a
directory instead of a file, the other is to create a symbolic
link. Both operations are defined to be atomic by POSIX/SUS, by
virtue of the fact that they both require invocation of the
corresponding system calls, which are atomic.
Beware of trying to create ANY kind of lock file on an NFS
partition. NFS pretty much eliminates anything like atomicity. If
you're going to create a lock file, make sure you're doing it on a
local partition, such as /tmp.
Netscape/Mozilla uses the symbolic link method for its lockfile (in
spite of the fact that it creates it in the user's home directory,
which may be NFS mounted). When it starts up it creates a file
named for the IP address of the machine it's running on, and the
pid of the creating process. Then it tries to create a symbolic
link named "lock", which points to that file. If this symlink
already exists, link(2) will return an error. In a script this
would work something like
touch /tmp/xxx
ln -s /tmp/xxx /tmp/lockfile 2>/dev/null
ret=$?
rm /tmp/xxx
if [ $ret -ne 0 ];then
echo lockfile already exists
exit 1
else
echo success
fi
If you have procmail installed, another possibility is the
lockfile(1) command that comes with it.
======================================================================
10. How can I convert DOS text files to unix, and vice versa?
Unix text files consist of lines delimited by an LF ("line-feed")
character (ASCII 10). DOS uses the two characters CR LF ("carriage
return", "line feed"; ASCII 13, 10) for the same purpose.
To convert a DOS text into unix text format, the CR characters
(control-M) at the end of a line have to be removed. To create a
DOS text file, the CR character should be added.
A couple ways to remove CR characters:
sed 's/^M$//' dos.txt > unix.txt
tr -d '\r' < dosfile > unixfile
To add them:
sed 's/$/^M/' unix.txt > dos.txt
Note that "^M" in this case is an embedded control character, (CR,
ASCII 13). Many shells allow embedding control characters by
entering ^V first (control-V), resulting in the sequence
^V^M
for entering "^M".
However, zsh, bash or ksh93 allow for:
sed $'s/$/\r/'
There is one special case to be considered: DOS text files
sometimes contain an explicit end-of-file character ^Z (ASCII 26,
or octal 32), which has no correspondent character for unix text
files, where the end-of-file condition is determined
implicitly. To remove that as well as the CR characters:
tr -d '\r\032' < dosfile > unixfile
Note that sed does not understand that notation, but awk does, and
one simple way to do the opposite conversion is
$ awk '{printf "%s\r\n" $0}END{printf "%c", 26}' unixfile > dosfile
This assume a not-quite-ancient awk, in practice anything
but Solaris /bin/awk (use nawk or /usr/xpg4/bin/awk in Solaris).
Finally, your system may come with utilities named something like
dos2unix and unix2dos, or d2u dos2unix fromdos non-standard
utilities, and GNU recode: recode /CRLF
======================================================================
11. How can a shell prompt be set up to change the title of xterm?
Gives escape sequences for xterm. For example, to change the name
of the current window to "XXX" (in bash), do
$ echo -en "\033]2;XXX\007"
or, more portably:
$ printf '%b' '\e]2;XXX\a'
See also "Why doesn't echo do what I want?"
======================================================================
12. How do I batch a FTP download/upload?
The best way to handle this is with ncftpput and ncftpget which
are part of the ncftp program. ncftpput -u username -p password
somewebsite.com /pics *jpg The above usage of the username and
password is not recomend though as it will be seen by anyone using
"ps" while the script is running. ncftp has a way to handle that
as well. Just create a file with the information in the following
formate:
host somewebsite.com
user username
pass password
Then just use the -f option on the ncftp program:
ncftpput -f /home/username/somefile somewebsite.com /pics *jpg
ncftp can be found at http://freshmeat.net/projects/ncftp/
If you want to do this interactively, there's no need to keep the
password in a file. For example, if you're building a program on
one machine, but testing it on another, and you have to keep
ftp'ing the files, you can cut down on typing by doing something
like
#!/bin/ksh
ftp -n $remote_host <<EOF
user <username>
cd <path to build directory>
prompt
bin
mget "$@"
bye
EOF
The ftp program will automatically ask you for the password, then
do the rest for you.
======================================================================
13. How do I get the exit code of cmd1 in cmd1|cmd2
First, note that cmd1 exit code could be non-zero and still don't
mean an error. This happens for instance in
cmd | head -1
you might observe a 141 (or 269 with ksh93) exit status of cmd1,
but it's because cmd was interrupted by a SIGPIPE signal when
"head -1" terminated after having read one line.
To know the exit status of the elements of a pipeline
cmd1 | cmd2 | cmd3
a. with zsh:
The exit codes are provided in the pipestatus special array.
cmd1 exit code is in $pipestatus[1], cmd3 exit code in
$pipestatus[3], so that $? is always the same as
$pipestatus[-1].
b. with bash:
The exit codes are provided in the PIPESTATUS special array.
cmd1 exit code is in ${PIPESTATUS[0]}, cmd3 exit code in
${PIPESTATUS[2]}, so that $? is always the same as
${PIPESTATUS: -1}.
c. with any other Bourne like shells
You need to use a trick to pass the exit codes to the main
shell. You can do it using a pipe(2). Instead of running
"cmd1", you run "cmd1; echo $?" and make sure $? makes it way
to the shell.
exec 3>&1
eval `
# now, inside the `...`, fd4 goes to the pipe
# whose other end is read and passed to eval;
# fd1 is the normal standard output preserved
# the line before with exec 3>&1
exec 4>&1 >&3 3>&-
{
cmd1 4>&-; echo "ec1=$?;" >&4
} | {
cmd2 4>&-; echo "ec2=$?;" >&4
} | cmd3
echo "ec3=$?;" >&4
`
d. with a POSIX shell
You can use this function to make it easier:
run() {
j=1
while eval "\${pipestatus_$j+:} false"; do
unset pipestatus_$j
j=$(($j+1))
done
j=1 com= k=1 l=
for a; do
if [ "x$a" = 'x|' ]; then
com="$com { $l "'3>&-
echo "pipestatus_'$j'=$?" >&3
} 4>&- |'
j=$(($j+1)) l=
else
l="$l \"\$$k\""
fi
k=$(($k+1))
done
com="$com $l"' 3>&- >&4 4>&-
echo "pipestatus_'$j'=$?"'
exec 4>&1
eval "$(exec 3>&1; eval "$com")"
exec 4>&-
j=1
while eval "\${pipestatus_$j+:} false"; do
eval "[ \$pipestatus_$j -eq 0 ]" || return 1
j=$(($j+1))
done
return 0
}
use it as:
run cmd1 \| cmd2 \| cmd3
exit codes are in $pipestatus_1, $pipestatus_2, $pipestatus_3
======================================================================
14. Why do I get "script.sh: not found"
a. While script starts with "#!/bin/sh" (^M issue)
That's the kind of error that occurs when you transfer a file
by FTP from a MS Windows machine. On those systems, the line
separator is the CRLF sequence, while on unix the line
separator is LF alone, CR being just another ordinary character
(the problem is that it is an invisible one on your terminal
(where it actually moves the cursor to the beginning of the
line) or in most text editors or pagers).
So, if a MSDOS line is "#!/bin/sh", when on a Unix system, it
becomes "#!/bin/sh<CR>" (other names for <CR> are \r, \015, ^M,
<Ctrl-M>).
So, if you run the file as a script, the system will look in
/bin for an interpreter named "sh<CR>", and report it doesn't
exist.
$ sed 'l;d;q' < script.sh
#!/bin/sh\r$
shows you the problem ($ marks the end of line, \r is the CR
character).
b. PATH issue
Sometimes a shell is installed someplace other than /bin or
/usr/bin. For example, a shell which was not part of the OS
installation might be installed into /usr/local/bin. If the
script was written on a machine which had ksh located in
/usr/bin, but was run on a machine where ksh was located in
/usr/local/bin, the shebang line would not resolve correctly.
This is unlikely to occur when using sh. However, if the shell
is bash, zsh, et al, it might be installed in different places
on different machines.
One way around this is to use the env command in the shebang
line. So instead of
#!/bin/sh
use
#!/usr/bin/env sh
Of course, env might itself live in some other directory than
/usr/bin, but it's not likely.
======================================================================
15. Why doesn't echo do what I want?
See also section 0a "Notes about using echo"
The echo command is not consistent from shell to shell. For
example, some shells (bash, pdksh [,?]) use the following
arguments
-n suppress newline at the end of argument list
-e interpret backslash-escaped characters
-E disable interpretation of backslash-escaped characters, even
on systems where interpretation is the default.
However, pdksh also allows using \c to disable a newline at the
end of the argument list.
POSIX only allows \c to be used to suppress newlines, and doesn't
accept any of the above arguments.
ksh88 and ksh93 leave the interpretation of backslash-escaped
characters up to the implementation.
[descriptions of behavior of other shells welcome]
In short, you have to know how echo works in any environment you
choose to use it in, and its use can therefore be problemmatic. If
available, print(1) or printf(1) would be better.
======================================================================
16. How do I loop through files with spaces in their name?
So, you're going to loop through a list of files? How is this list
stored? If it's stored as text, there probably was already an
assumption about the characters allowed in a filename. Every
character except '\0' (NUL) is allowed in a file path on Unix. So
the only way to store a list of file names in a file is to
separate them by a '\0' character (if you don't use a quoting
mechanism as for xargs input).
Unfortunately most shells (except zsh) and most standard unix text
utilities (except GNU ones) can't cope with "\0"
characters. Moreover, many tools, like "ls", "find", "grep -l"
output a \n separated list of files. So, if you want to
postprocess this output, the simpler is to assume that the
filenames don't contain newline characters (but beware that once
you make that assumption, you can't pretend anymore your code is
reliable (and thus can't be exploited)).
So, if you've got a newline separated list of files in a
list.txt file, Here are two ways to process it:
1-
while IFS= read -r file <&3; do
something with "$file" # be sure to quote "$file"
done 3< list.txt
(if your read doesn't have the "-r" option, either make another
assumption that filenames don't contain backslashes, or use:
exec 3<&0
sed 's/\\/&&/g' < list.txt |
while IFS= read file; do
something with "$file" <&3 3<&-
done
)
2-
IFS="
" # set the internal field separator to the newline character
# instead of the default "<space><tab><NL>".
set -f # disable filename generation (or make the assumption that
# filenames don't contain *, [ or ? characters (maybe more
# depending on your shell)).
for file in $(cat < list.txt); do
something with "$file" # it's less a problem if you forget to
# quote $file here.
done
Now, beware that there are things you can do before building
this list.txt. There are other ways to store filenames. For
instance, you have the positional parameters.
with:
set -- ./*.txt
you have the list of txt files in the current directory, and no
problem with weird characters. Looping through them is just a
matter of:
for file
do something with "$file"
done
You can also escape the separator. For instance, with
find . -exec sh -c 'printf %s\\n "$1" | sed -n '"':1
\$!{N;b1
}
s/|/|p/g;s/\n/|n/g;p'" '{}' '{}' \;
instead of
find . -print
you have the same list of files except that the \n in filenames
are changed to "|n" and the "|" to "|p". So that you're sure
there's one filename per line and you have to convert back "|n"
to "\n" and "|p" to "|" before referring to the file.
======================================================================
17. how do I change my login shell?
See http://www.faqs.org/faqs/unix-faq/shell/shell-differences
Unless you have a very good reason to do so, do not change root's
default login shell. By "default login shell" is meant the shell
recorded in /etc/passwd. Note that "I login as root but don't like
the default shell" isn't a good reason.
The default shell for root is one which will work in single user
mode, when only the root partition is mounted. This is one of the
contexts root works in, and the default shell must accommodate
this. So if you change it to a dynamically linked shell which
depends on libraries that are not in the root partition, you're
asking for trouble.
The safest way of changing root's shell is to login as root and
then
# SHELL=/preferred/shell; export SHELL
# exec <your preferred shell with login flag>
e.g.
# SHELL=/usr/bin/ksh; export SHELL
# exec $SHELL -l
Another possibility is to add something to root's .profile or
.login which checks to see if the preferred shell is runnable, and
then execs it. This is more complicated and has more pitfalls than
simply typing "exec <shell>" when you login though. For example,
one of the libraries that the desired shell relies on might have
been mangled, etc. One suggestion that has been made is
if [ -x /usr/bin/ksh ]; then
SHELL=/usr/bin/ksh; export SHELL
ENV=/root/.kshrc; export ENV
/usr/bin/ksh -l && exit
fi
A safer way is to try to run a command with the preferred shell
before you try to exec it. This will lessen the possibility that
the shell or one of the libraries it depends on has been
corrupted, or that one of the libraries it depends on is not in
the available mounted partitions.
if [ -x /usr/bin/ksh ]; then
/usr/bin/ksh -c echo >/dev/null 2>&1
if [ $? -eq 0 ];then
SHELL=/usr/bin/ksh; export SHELL
ENV=/root/.kshrc; export ENV
/usr/bin/ksh -l && exit
fi
fi
Another common approach is to create another user with UID 0. For
example, FreeBSD systems commonly create an account named toor,
which can be setup however you like. This bypasses the
controversy.
======================================================================
18. When should I use a shell instead of perl/python/ruby/tcl...
a. Portability
In many cases it can't be assumed that perl/python/etc are
installed on the target machine. Many customer sites do not
allow installation of such things. In cases like this, writing
a shell script is more likely to be successful. In the extreme,
writing a pure Bourne shell script is most likely to succeed.
b. Maintainability
If the script is one which serves some important purpose, and
will need to be maintained after you get promoted, it's more
likely that a maintainer can be found for a shell script than
for other scripting languages (especially less used ones such
as ruby, rexx, etc).
c. Policy
Sometimes you're just told what to use :-)
======================================================================
19. Why shouldn't I use csh?
http://www.grymoire.com/Unix/CshTop10.txt
http://www.grymoire.com/Unix/Csh.html#uh-0
http://www.faqs.org/faqs/unix-faq/shell/csh-whynot/
======================================================================
20. How do I reverse a file?
Non-standard commands to do so are GNU tac and "tail -r". sed
'1!G;h;$!d' is subject to sed limitation on the size of its hold
space and is generally slow.
The awk equivalent would be:
awk '{l[n++]=$0}END{while(n--)print l[n]}'
It stores the whole file in memory.
The best approach in terms of efficiency portability and resource
cosumption seems to be:
cat -n | sort -rn | cut -f2-
"cat -n" is not POSIX but appears to be fairly
portable. Alternatives are "grep -n '^'", "awk '{print NR,$0}'".
Also, nl can be used as
nl -ba -d'
'
i.e. NL as the delimiter.
You may also be able to use nl with the -p option, which is POSIX,
and widely available, going back to AT&T SysV/386 R3.2 from 1988
or thereabouts.
======================================================================
21. how do I remove the last n lines?
First we need to tell the code how many lines we want to cut
from the bottom of a file.
X=10
Then We can do this:
head -n $(( $(wc -l < file ) - $X )) file >$$ \
&& cat $$ >file && rm $$
The break down:
1) $(wc -l < file)
Find out how many lines are in the file. Need to use
redirection so wc won't print the file name.
2) $(( $lines_in_file - $X ))
Take the output from step one and do some math to find out
how many lines we want to have when all is said and done.
3) head -$lines_when_said_and_done file
extracts all but the unwanted lines from the file,
and >$$ puts those lines into a temp file that has
the name of the pid of the current shell.
4) && cat $$ > file
if everything has worked so far then cat the temp file into
the original file. This is better than mv or cp because it
insures that the permissions of the temp file do not
override with the perms of the original file.
5) && rm $$
Remove the temp file.
AWK solutions:
awk 'NR<=(count-12)' count="`awk 'END{print NR}' file`" file
awk 'NR>n{print a[NR%n]} {a[NR%n]=$0}' n=12 file
awk 'BEGIN{n=12} NR>n{print a[NR%n]} {a[NR%n]=$0}' file
Whenever a line is read, the line that came 12 lines ago is
printed, and then overwritten with the newly read line, using an
rolling array indexed 0..11.
See also question 26. for information about setting awk
variables on the command line.
$SHELL/sed/mv solutions:
L=`wc -l <file`
DL=`expr $L - 11`
sed "$DL,\$d" file
L=`wc -l <file`
DL=`expr $L - 12`
sed "${DL}q" file
sed "`expr \`wc -l <file\` - 12`q" file
sed -n -e :a -e '1,12{N;ba' -e '}' -e 'P;N;D' file
The last solution is basically same algorithm as the rolling
array awk solutions, and shares with them the advantage that
the file is only read once - they will even work in a pipe.
PERL solution:
perl -ne' print shift @x if @x == 12; push @x, $_ ' file
Using GNU dd:
ls -l file.txt | {
IFS=" "
read z z z z sz z
last=`tail -10 file.txt | wc -c`
dd bs=1 seek=`expr $sz - $last` if=/dev/null of=file.txt
}
This is different than other solutions in that, rather than
creating a new, shorter file it overwrites the trailing lines
in the original file with nulls.
======================================================================
22. how do I get file size, or file modification time?
If your system has stat(1), use it. On Linux, for example:
filesize=$(stat -c %s -- filename)
or use cut, awk, etc on the output.
Probably the most portable solution is to use wc
filesize=`wc -c < "$file"`
ls may be able to tell you what you want to know. From the man
page for ls we learn about "ls -l" the file mode, the number of
links to the file, the owner name, the group name, the size of the
file (in bytes), the timestamp, and the filename. For the file
size in human readable formate use the "-h" option.
For example:
$ ls -l timeTravel.html
-rw-rw-r-- 1 user user 20624 Jun 19 2002 timeTravel1.html
so to get the file size:
$ set -- `ls -l timeTravel1.html`
$ echo $5
20624
Note that ls doesn't always give the date in the same
format. Check the man page for ls on your system if that
matters. If you're interested in the file modification time.
Another possibility is to use GNU ls, which has a -T option giving
complete time information for the file, including month, day,
hour, minute, second and year.
See also GNU find (-printf), GNU stat, GNU date (-r) and zsh stat
(+mtime).
On FreeBSD 4, you can use the -lT option to ls(1) to get the full
modification time and the -f option to date(1) to parse it, for
example:
$ FILE=/etc/motd
$ date -jf'%b %d %T %Y' +%Y-%m-%dT%T \
$(ls -lT $FILE|tr -s ' ' \\t|cut -f6-9)
2003-09-09T16:04:06
Adjust syntax as needed if your shell is FreeBSD sh
======================================================================
23. How do I get a process id given a process name? Or, how do I find
out if a process is still running, given a process ID?
There isn't a reliable way to to this portably in the shell. Some
systems reuse process ids much like file descriptors. That is,
they use the lowest numbered pid which is not currently in use
when starting a new process. That means that the pid you're
looking for is there, but might not refer to the process you think
it does.
The usual approach is to parse the output of ps, but that involves
a race condition, since the pid you find that way may not refer to
the same process when you actually do something with that
pid. There's no good way around that in a shell script though, so
be advised that you might be stepping into a trap.
One suggestion is to use pgrep if on Solaris, and 'ps h -o pid -C
$STRING' if not, and your ps supports that syntax, but neither of
those are perfect or ubiquitous.
The normal solution when writing C programs is to create a pid
file, and then lock it with fcntl(2). Then, if another program
wants to know if that program is really running, it can attempt to
gain a lock on the file. If the lock attempt fails, then it knows
the file is still running.
We don't have options in the shell like that, unless we can supply
a C program which can try the lock for the script. Even so, the
race condition described above still exists.
======================================================================
24. How do I get a script to update my current environment?
Processes in unix cannot update the environment of the process
that spawned them. Consequently you cannot run another process
normally and expect it to do that, since it will be a child of the
running process. There are a couple ways it can be done though.
a. source the script
This means that you use whatever syntax your shell has to read
the desired script into the current environment.
In Bourne derived shells (sh/ksh/bash/POSIX/etc) the syntax
would be
$ . script
In csh type shells this would be
$ source script
b. use eval
The eval command constructs a command by evaluating and then
executing a set of arguments. If those arguments evaluate to a
shell variable assignment, the current environment will be
updated. For example
--- exportFoo
#!/bin/ksh
echo export FOO=bar
If you run this like
eval "`exportFoo`"
the value of FOO will be set to 'bar' in the calling
shell. Note that the quotes are recommended as they will
preserve any whitespace that may be present in the variables
being set.
However, be aware that eval'ing a script written in another
shell could turn out to be the wrong thing to do. For example,
eval'ing this from a ksh script
#!/bin/csh
echo setenv FOO bar
Would not do what you expect. It would produce an error,
because ksh doesn't have a setenv command.
======================================================================
25. How do I rename *.foo to *.bar?
Naive examples in ksh/bash (which may or may not work many times)
$ ls *.foo | while read f;do mv "$f" "${f%.*}".bar
More generically
$ ls *.foo | while read f;do mv "$f" `basename "$f" .foo`.bar
However, these examples contain a potentially unnecessary use of
ls (ie, if the number of files is small enough to not overflow the
command line buffer), and will fail if any file names contain a
newline, or if there are leading or trailing spaces. An
alternative is:
for file in *.foo
do
mv -- "$file" "`basename -- \"$file\" .foo`.bar"
done
Also, tests for existence of files should also be incorporated,
e.g.:
for file in *.foo
do
newfile=`basename "$file" .foo`.bar
[ -f "$file" ] || continue
[ -f "$newfile" ] && continue ## or deal with it another way
mv "$file" `basename "$file" .foo`.bar
done
In some linux distributions you may be able to use the rename
command
$ rename .foo .bar *
If not (Debian, for one, comes with a perl version of rename that
won't work with that command line) try
$ rename 's/.foo/.bar/' *.foo
More options, and much more discussion about this, is available
from http://www.faqs.org/faqs/unix-faq/faq/part2/section-6.html
Note that for file specifications which don't match existing
files, the shell usually responds with something like "ls: *.foo:
No such file or directory", which will mess up your processing of
file names. One possibility is
#! /bin/sh
set x [*].foo ./*.foo
case "$2$3" in
"[*].foo./*.foo") ;;
*)
shift 2
for file
do
repl=`basename "$file" .foo`.bar
mv "$file" "$repl"
done;;
esac
Except that contrary to (zsh) mmv or zmv it doesn't check for
file overwriting and fails for filenames with NLs before the "."
and doesn't handle dotfiles.
======================================================================
26. How do I use shell variables in awk scripts
Depending on the version of awk being used, either use the -v
command line option,
$ awk -v var=xxx '{print $0,var}'
or add the variable after the command, as in
$ awk '{print $0,var}' var=xxx file
Note that using the latter syntax var will not be available in the
BEGIN section.
See the man page for awk on your system to see which is
applicable.
======================================================================
27. How do I get input from the user with a timeout?
In bash or ksh93 you can use the read built-in with the "-t"
option.
In zsh, use the zsh/zselect module.
You can also use your terminal capability to do that.
{
s=$(stty -g)
stty -icanon min 0 time 100
var=$(head -n 1)
stty "$s"
}
For a 10 second timeout (reset at each key press).
======================================================================
28. How do I get one character input from the user?
In bash this can be done with the "-n" option to read.
In ksh93 it's read -N
In zsh it's read -k
More portably:
OLDSTTY=$(stty -g) # save our terminal settings
stty cbreak # enable independent processing of each input character
ONECHAR=$(dd bs=1 count=1 2>/dev/null) # read one byte from standard in
stty $OLDSTTY # restore the terminal settings
Use the `something` format if your shell doesn't understand
$(something). This reads from standard input, which may or may not
be desirable. If you want to read from the terminal regardless of
where standard input is, add "if=$(tty)" to the dd command.
======================================================================
29. why isn't my .profile read?
~/.profile is only read for login shells. In short if you don't
see a login prompt then your ~/.profile isn't being read. You
can fix this by either porting all of the things in your
~/.profile to /etc/profile or your shells rc script such as
~/.bashrc or ~/.zshrc.
You may have to set the ENV variable in your login shell to get
the .*rc shell read. See the man page for your shell to understand
how it works.
======================================================================
30. why do I get "[5" not found in "[$1 -eq 2]"?
Because you didn't RTFM :-)
"[" is an alias for the "test" command. As such, it's called by a
script like any other command (this applies even if test is
builtin). Since the command line uses spaces to separate a command
from its arguments, you have to put a space between '[' and its
argument. So:
$ [ -f xxx ] isn't the same as
$ [-f xxx ]
In the latter case, the shell will think that "[-f" is the
command, not "[" with arguments "-f xxx ]
======================================================================
31. How do I exactly display the content of $var (with a \n appended).
A: on POSIX systems or with shells with builtin printf (bash2,
ksh93, zsh4.1, dash...)
printf '%s\n' "$var"
(except for memory/environment full errors, should be expected
to work at least if $var is not longer than LINE_MAX (supposed
to be at least _POSIX2_LINE_MAX == 2048), no hardcoded limits in
zsh/ksh/bash/dash builtins)
ksh, zsh:
print -r -- "$var"
zsh:
echo -E - "$var"
Other bourne like shells:
cat << EOF
$var
EOF
(creates a temporary file and forks a process)
expr "x$var" : 'x\(.*\)'
(limited to 126 characters with some exprs, may return a
non-null exit code).
With ash:
(unset a; ${a?$var}) 2>&1
======================================================================
32. How do I exactly display the content of $var (without a \n
appended).
printf %s "$var" # posix
print -rn -- "$var" # zsh/ksh
echo -nE - "$var" # zsh
awk 'NR>1{print ""}{printf("%s",$0)}' << EOF
$var
EOF
======================================================================
33. How do I split a pathname into the directory and file?
The most portable way of doing this is to use the external
commands dirname(1) and basename(1), as in
pathname='/path/to/some/file'
dir=`dirname $pathname`
file=`basename $pathname`
However, since this executes an external command, it's slower than
using shell builtins (if your shell has them). For ksh, bash, and
POSIX shells the following will do the same thing more
efficiently:
pathname=/path/to/some/file
file=${pathname##*/}
To get the directory using the shell builtin, you should first
ensure that the path has a '/' in it.
case $pathname in
*/*) dir=${pathname%/*};;
*) dir=''
esac
======================================================================
34. How do I make an alias take an argument?
In Bourne-derived shells aliases cannot take arguments, so if you
need to be able to do that, define a shell function rather than
an alias.
Aliases are often used to reduce the need to type long command
lines:
alias prog='/opt/bin/prog -x -y -z --long-option'
Or to make a command default to using certain parameters:
alias ls='ls -F'
Shell functions must be used when arguments are needed. For
example, this will move one or more files to a Trash directory:
trash() { mv -- "$@" ~/.Trash; }
======================================================================
35. How do I deal with a file whose name begins with a weird character
Do something to hide the weird character from the command being
used. Assuming that command is rm, try things like
rm ./-foo
rm -- -foo
rm -i -- * (and then decide what you want to delete interactively)
If the weird character is not printable, the last option may be
your best bet. Another possibility in that case is to pipe the
output of ls into od -c and figure out what the weird character
is. Then use sed to isolate it and nuke it. However, the rm -i
approach is probably much safer.
For more particulars, see the rm man page for your system.
======================================================================
36. Why do I lose the value of global variables that are set in a loop.
Given the following program
#!/bin/sh
x="this is the initial value of x"
cat dataFile | while read line;do
x="$line"
done
echo x = $x
You may get the following for output
x = this is the initial value of x
This is because in the Bourne shell redirected control structures
run in a subshell, so the value of x only gets changed in the
subshell, and is lost when the loop ends.
In other shells the same result may be seen because of the way
pipelines are handled. In shells other than ksh (not pdksh) and
zsh elements of a pipeline are run in subshells. In ksh and zsh,
the last element of the pipeline is run in the current shell.
An alternative for non-Bourne shells is to use redirection
instead of the pipeline
#!/bin/sh
x="this is the initial value of x"
while read line;do
x="$line"
done < dataFile
echo x = $x
With a Bourne shell you need to reassign file descriptors, so no
pipline or redirection in the loop is involved.
exec 3<&0 # save stdin
exec < file
while read line; do
x=$line
done
exec 0<&3 # restore stdin
Note that putting #!/bin/sh at the top of a script doesn't
guarantee you're using the Bourne shell. Some systems link /bin/sh
to some other shell. Check your system documentation to find out
what shell you're really getting in this case.
======================================================================
Appendix A: Examples
Web sites:
-------------------------------------------
Heiner Steven's Shelldorado site has quite a few example scripts,
tutorials, and links to other such places.
Arbitrary Date Arithmetic
-------------------------------------------
From: Tapani Tarvainen (t...@it.jyu.fi)
Subject: Re: yesterday's date under the shell
View: Complete Thread (8 articles)
Original Format
Newsgroups: comp.unix.shell
Date: 2002-02-12 07:45:05 PST
"Jean-No?l" <Je...@freckles.de> writes:
> To determine the yesterday date i do it so:
> TZ=PST+24 date +%d
> it work well but my question is:
> does this work on all systems and all shells
> or should i do it otherwise ???
No, it does not work on all systems at all times.
In some it will work practically always,
on others never, on most it works sometimes
and sometimes not. I would recommend against it.
Unfortunately there is no short and sweet portable
solution. If you have or can install Gnu date it
will do it cleanly, otherwise you can find a number
of solutions posted in this group in the past.
For a general solution you could try the following,
which should work with POSIXy shells (I've only tested
it with HP's which is essentially ksh88, though):
#! /usr/bin/sh
# Date calculations using POSIX shell
# Tapani Tarvainen July 1998, February 2001 (POSIXified)
# This code is in the public domain.
# Julian Day Number from calendar date
date2julian() # day month year
{
day=$1; month=$2; year=$3
tmpmonth=$((12 * year + month - 3))
tmpyear=$((tmpmonth / 12))
print $(( (734 * tmpmonth + 15) / 24 - 2 * tmpyear + \
tmpyear/4 - tmpyear/100 + tmpyear/400 + day + 1721119 ))
}
# Calendar date from Julian Day Number
julian2date() # julianday
{
tmpday=$(($1 - 1721119))
centuries=$(( (4 * tmpday - 1) / 146097))
tmpday=$((tmpday + centuries - centuries/4))
year=$(( (4 * tmpday - 1) / 1461))
tmpday=$((tmpday - (1461 * year) / 4))
month=$(( (10 * tmpday - 5) / 306))
day=$((tmpday - (306 * month + 5) / 10))
month=$((month + 2))
year=$((year + month/12))
month=$((month % 12 + 1))
print $day $month $year
}
# Day of week, Monday=1...Sunday=7
dow() # day month year
{
print $(( $(date2julian $1 $2 $3) % 7 + 1))
}
##################### The End ########################
Those allow rather arbitrary date computations.
For example, yesterday's date can be computed like this:
julian2date $(( $(date2julian $(date +"%d %m %Y") ) - 1 ))
--
Tapani Tarvainen
======================================================================
Appendix B: References
Heiner Steven
> The FAQ has been posted here, and is also available at
> http://home.comcast.net/~j.p.h/
I have a suggestion: what do you think about having
a Wiki FAQ?
Heiner
--
___ _
/ __| |_ _____ _____ _ _ Heiner STEVEN <heiner...@nexgo.de>
\__ \ _/ -_) V / -_) ' \ Shell Script Programmers: visit
|___/\__\___|\_/\___|_||_| http://www.shelldorado.com/
j...@invalid.address
> Joe Halpin wrote:
>
> > The FAQ has been posted here, and is also available at
> > http://home.comcast.net/~j.p.h/
>
> I have a suggestion: what do you think about having
> a Wiki FAQ?
I'd be interested in that if others here would use it. In fact, I've
been wanting to setup a wiki for our group at work, but haven't had
the time to figure it all out yet. Are you volunteering to help? :-)
Joe
--
We can't all be heroes because someone has to sit on the curb and
clap as they go by.
- Will Rogers