Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
No Fair! (for Unix Geeks)
2 views
Skip to first unread message
Ted L.
May 20, 2013, 6:30:44 PM5/20/13
to
I've been trying to teach myself shell programming just for the fun of
it, seeing as this Mac is based on Unix. It was going pretty well, but
every now and then I'd have something that would gave strange results
and obscure errors -- I double-checked the syntax, placement of
semicolons and quotes, where lines ended etc -- still same sorts of
errors. I finally figured it out, and it's not obvious from any quick
intro's. The terminal window supports autocomplete-on-tab! (It attempts
to complete what you've typed based on what you've typed so far before
you hit tab -- like the first part of a command or file name -- and if
there are multiple completions possible it displays a list of choices.)
Nobody tol' me that. So, I'd write my script in a text editor (I like
TextWrangler for this purpose) and then cut and paste pieces of the
script into the terminal window to test them out -- some excerpts would
work fine, some wouldn't. For a long while I was sure I had the syntax
of "if" and "do" statements wrong! Turns out that of course with my
experience with other languages I put tabs in to format the scripts
nicely -- not knowing that those tabs would be interpreted by the
terminal handler as autocomplete, usually by trying to list all the
files in my home directory.
--
Ted L.
Benedictus qui venit in nomine Domini.
it, seeing as this Mac is based on Unix. It was going pretty well, but
every now and then I'd have something that would gave strange results
and obscure errors -- I double-checked the syntax, placement of
semicolons and quotes, where lines ended etc -- still same sorts of
errors. I finally figured it out, and it's not obvious from any quick
intro's. The terminal window supports autocomplete-on-tab! (It attempts
to complete what you've typed based on what you've typed so far before
you hit tab -- like the first part of a command or file name -- and if
there are multiple completions possible it displays a list of choices.)
Nobody tol' me that. So, I'd write my script in a text editor (I like
TextWrangler for this purpose) and then cut and paste pieces of the
script into the terminal window to test them out -- some excerpts would
work fine, some wouldn't. For a long while I was sure I had the syntax
of "if" and "do" statements wrong! Turns out that of course with my
experience with other languages I put tabs in to format the scripts
nicely -- not knowing that those tabs would be interpreted by the
terminal handler as autocomplete, usually by trying to list all the
files in my home directory.
--
Ted L.
Benedictus qui venit in nomine Domini.
Mark Warner
May 20, 2013, 7:02:32 PM5/20/13
to
--
Mark Warner
MEPIS Linux
Registered Linux User #415318
...lose .inhibitions when replying
Fred Exley
May 20, 2013, 7:06:01 PM5/20/13
to
tip! Sounds like you're getting there. I use Text Wrangler too, mainly
to strip off all the non-printing characters. The nice thing about
learning Unix stuff is, like learning any other foreign language, it
isn't going to change much in the future.
Hank
May 23, 2013, 11:57:40 AM5/23/13
to
In article <kne7v1$2eg$1...@dont-email.me>, Ted L. <ted...@yahoo.com> wrote:
>I've been trying to teach myself shell programming just for the fun of
>it, seeing as this Mac is based on Unix. It was going pretty well, but
>every now and then I'd have something that would gave strange results
>and obscure errors -- I double-checked the syntax, placement of
>semicolons and quotes, where lines ended etc -- still same sorts of
>errors. I finally figured it out, and it's not obvious from any quick
>intro's. The terminal window supports autocomplete-on-tab! (It attempts
>to complete what you've typed based on what you've typed so far before
>you hit tab -- like the first part of a command or file name -- and if
>there are multiple completions possible it displays a list of choices.)
Ted, the Unix utilities are quite handy. You can pick and choose
>I've been trying to teach myself shell programming just for the fun of
>it, seeing as this Mac is based on Unix. It was going pretty well, but
>every now and then I'd have something that would gave strange results
>and obscure errors -- I double-checked the syntax, placement of
>semicolons and quotes, where lines ended etc -- still same sorts of
>errors. I finally figured it out, and it's not obvious from any quick
>intro's. The terminal window supports autocomplete-on-tab! (It attempts
>to complete what you've typed based on what you've typed so far before
>you hit tab -- like the first part of a command or file name -- and if
>there are multiple completions possible it displays a list of choices.)
among shells. The core shell in AT&T Unix is /bin/sh, the Bourne
shell. That is not available in Linux, OpenBSD, or FreeBSD; what you
get is the GNU "bourne again shell," bash, aliased to the /bin/sh
name. The best alternative shell for scripting on these systems is
the Korn shell, ksh. You may need to download it if it isn't already
on the Mac setup. ksh93 has been in the public domain for years.
I think you're saying "terminal window" to mean "shell."
For documentation, O'Reilly in Sebastapol, Cal. is your friend. There
are Nutshell guides for all sorts of things that go well beyond what
the man pages on the system will tell you. For script preparation,
the vi editor is the one to use. O'Reilly has a good tutorial guide,
"Learning the VI Editor," as I recall. You'll curse vi every inch of
the way the first week or two you try to use it, and end up using it
for nearly everything after you get familiar with it. It's really two
editors in one package, ex and vi, a line editor and a screen editor.
What you'll get on the Mac setup is a vi wannabe, vim, but it behaves
the same, and O'Reilly has good documentation for it.
One tip: when writing a shell script, invoke the shell you're
scripting for in the first line.
#!/sbin/sh
exactly like that invokes the Bourne shell on Solaris (SVR4 Unix) in a
script.
Hank
Ted L.
May 23, 2013, 12:08:40 PM5/23/13
to
On 5/23/13 10:57 AM, Hank wrote:
> Ted, the Unix utilities are quite handy. You can pick and choose
> among shells. The core shell in AT&T Unix is /bin/sh, the Bourne
> shell. That is not available in Linux, OpenBSD, or FreeBSD; what you
> get is the GNU "bourne again shell," bash, aliased to the /bin/sh
> name. The best alternative shell for scripting on these systems is
> the Korn shell, ksh. You may need to download it if it isn't already
> on the Mac setup. ksh93 has been in the public domain for years.
> I think you're saying "terminal window" to mean "shell."
>
> For documentation, O'Reilly in Sebastapol, Cal. is your friend. There
> are Nutshell guides for all sorts of things that go well beyond what
> the man pages on the system will tell you. For script preparation,
> the vi editor is the one to use. O'Reilly has a good tutorial guide,
> "Learning the VI Editor," as I recall. You'll curse vi every inch of
> the way the first week or two you try to use it, and end up using it
> for nearly everything after you get familiar with it. It's really two
> editors in one package, ex and vi, a line editor and a screen editor.
> What you'll get on the Mac setup is a vi wannabe, vim, but it behaves
> the same, and O'Reilly has good documentation for it.
>
> One tip: when writing a shell script, invoke the shell you're
> scripting for in the first line.
> #!/sbin/sh
> exactly like that invokes the Bourne shell on Solaris (SVR4 Unix) in a
> script.
Thanks. I find TextWrangler a very good editor for shell scripts.
> Ted, the Unix utilities are quite handy. You can pick and choose
> among shells. The core shell in AT&T Unix is /bin/sh, the Bourne
> shell. That is not available in Linux, OpenBSD, or FreeBSD; what you
> get is the GNU "bourne again shell," bash, aliased to the /bin/sh
> name. The best alternative shell for scripting on these systems is
> the Korn shell, ksh. You may need to download it if it isn't already
> on the Mac setup. ksh93 has been in the public domain for years.
> I think you're saying "terminal window" to mean "shell."
>
> For documentation, O'Reilly in Sebastapol, Cal. is your friend. There
> are Nutshell guides for all sorts of things that go well beyond what
> the man pages on the system will tell you. For script preparation,
> the vi editor is the one to use. O'Reilly has a good tutorial guide,
> "Learning the VI Editor," as I recall. You'll curse vi every inch of
> the way the first week or two you try to use it, and end up using it
> for nearly everything after you get familiar with it. It's really two
> editors in one package, ex and vi, a line editor and a screen editor.
> What you'll get on the Mac setup is a vi wannabe, vim, but it behaves
> the same, and O'Reilly has good documentation for it.
>
> One tip: when writing a shell script, invoke the shell you're
> scripting for in the first line.
> #!/sbin/sh
> exactly like that invokes the Bourne shell on Solaris (SVR4 Unix) in a
> script.
Seeing as bash seems to be the standard for OS X, I use that one. (So
far I haven't found I do anything that's peculiar to a specific
interpreter.) My main problem in the few things I've tried is knowing
what I can use as an if or a while condition, but I've always found
alternatives to what I've sometimes first tried that didn't work.
Hank
May 23, 2013, 2:08:19 PM5/23/13
to
In general, you can type your shell script stuff directly to try out
the logic, before "canning" it into a script flat file with an editor.
Also, in sh/ksh, "set -s" will make the script echo each command in
the form it will be executed. Use "set +s" to turn it off. Also, set
the execute bit on the script file so you can invoke it directly.
vi has some very handy stuff that you can turn on/off to assist in
programming. Search/replace (several forms), show matching (), {},
etc. (very common typing errors in scripts), autoindent and autowrap,
come immediately to mind. And, as with shells, you can set up a
startup file to initialize optional settings (exrc is the name).
Unix was designed as an "open" system using flat files to control
everything. Nowadays, a lot of people think of point-and-click
"wizards," but the full range of things (and when all else fails, like
wizards often do) knowing where the flat file is and what to put in it
gets results.
Hank
Ted L.
May 23, 2013, 8:36:02 PM5/23/13
to
On 5/23/13 1:08 PM, Hank wrote:
> I can't comment on TexWrangler, as I use vi for just about everything.
>
> In general, you can type your shell script stuff directly to try out
> the logic, before "canning" it into a script flat file with an editor.
> Also, in sh/ksh, "set -s" will make the script echo each command in
> the form it will be executed. Use "set +s" to turn it off. Also, set
> the execute bit on the script file so you can invoke it directly.
>
> vi has some very handy stuff that you can turn on/off to assist in
> programming. Search/replace (several forms), show matching (), {},
> etc. (very common typing errors in scripts), autoindent and autowrap,
> come immediately to mind. And, as with shells, you can set up a
> startup file to initialize optional settings (exrc is the name).
I believe TextWrangler does all that, although I haven't explored all
> I can't comment on TexWrangler, as I use vi for just about everything.
>
> In general, you can type your shell script stuff directly to try out
> the logic, before "canning" it into a script flat file with an editor.
> Also, in sh/ksh, "set -s" will make the script echo each command in
> the form it will be executed. Use "set +s" to turn it off. Also, set
> the execute bit on the script file so you can invoke it directly.
>
> vi has some very handy stuff that you can turn on/off to assist in
> programming. Search/replace (several forms), show matching (), {},
> etc. (very common typing errors in scripts), autoindent and autowrap,
> come immediately to mind. And, as with shells, you can set up a
> startup file to initialize optional settings (exrc is the name).
the bells and whistles. I did notice that it shows you when you've
matched parentheses, brackets, or braces. If you're working on a Mac, I
suggest you give it a try. The other utility I really like is Platypus
-- it packages scripts (of many more kinds than I know anything about --
shell, Perl, Applescript and several others) into fully functional
applications. Very useful.
> Unix was designed as an "open" system using flat files to control
> everything. Nowadays, a lot of people think of point-and-click
> "wizards," but the full range of things (and when all else fails, like
> wizards often do) knowing where the flat file is and what to put in it
> gets results.
Oh, I agree -- being comfortable with the low-level stuff is quite handy
> everything. Nowadays, a lot of people think of point-and-click
> "wizards," but the full range of things (and when all else fails, like
> wizards often do) knowing where the flat file is and what to put in it
> gets results.
and at times just plain necessary. I do wish that Applescript (the
high-level scripting language) were easier to learn and use -- from what
I've seen it is quite powerful but I've only used a few little bits of
it. There probably are some good tutorials for it -- the basic
language manual itself ain't.
Fred Exley
May 23, 2013, 9:17:37 PM5/23/13
to
transactions. This Korn shell script has been running there every 5
minutes, 24/7 for years and years now. Good example of how to do some
pretty fancy stuff:
#
#
#This script loads all of the staging tables in the Oracle staging
tablespace with the data downloaded
#from the mainframe via Connect Direct. It also loads the verification
file associated with the
#table being loaded into the LOAD_VERIFY_TRIGGER_TABLE. The script is
initiated with Connect Direct, which
#passes arguments to the script via the command line. The two arguments
passed to this script are the
#staging table name, table($1), and the platform, plat($2). These
arguments(variables) are passed to the
#script as positional variables. This script only loads the
verification file if a .log file was created
#and a .bad file was not created when SQL*Loader was initiated to load
the data file. This script also
#archives the data & verification files to the /stage/archive directory
after successful loads. The script
#copies the load_verify_trigger_table control file to the
/ldrscripts/temp directory with a meaningful name
#so that all .log and .bad files will have different names. All
messages resulting from this script are
#written to the daily_load.log file.
#
export ORACLE_HOME=/u01/app/oracle/product/8.0.4
export ORACLE_SID=spsi1d
export ORACLE_TERM=vt100
export PATH=$ORACLE_HOME/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/bin:
#
#Move the table name and platform supplied as command line variables
through Connect Direct into
#user defined variables within the script
#
table=$1
plat=$2
#
#Get the julian date to use as a file extension when the data file and
verification file are
#archived after successful loads
#
jdate=`date '+%j'`
timestamp=`date '+%H%M'`
#
#Get the date in standard date format to use as the beginning of each
message sent to daily_load.log
#
ndate=`date`
#
#Ensure that the data file and verification file exist on the Unix box
prior to initiating the shell
#processing and SQL*Loader. If either file does not exist, send error
messages to the daily_load.log and exit
#the script. This is an if-then as opposed to an if-then-else test,
which is why the [-ne] test is used.
#
test -f ${table}_${plat}.dat
if [ $? -ne 0 ]
then
echo "${ndate} -- Error-Data file ${table}_${plat}.dat was NOT
downloaded correctly" >>daily_load.log
echo "${ndate} -- Error-Data file ${table}_${plat}.dat was NOT
downloaded correctly~${table}_${plat}" >>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
fi
test -f ${table}_ver_${plat}.dat
if [ $? -ne 0 ]
then
echo "${ndate} -- Error-Verification file ${table}_ver_${plat}.dat
was NOT downloaded correctly" >>daily_load.log
echo "${ndate} -- Error-Verification file ${table}_ver_${plat}.dat
was NOT downloaded correctly~LOAD_VERIFY_TRIGGER_TABLE"
>>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
fi
#
#Remove all .log and .bad files for this table that are left from the
last script execution.
#
if test -f ${table}_${plat}.log
then
rm ${table}_${plat}.log
fi
if test -f ${table}_ver_${plat}.log
then
rm ${table}_ver_${plat}.log
fi
if test -f ldrscripts/${table}_${plat}.bad
then
rm ldrscripts/${table}_${plat}.bad
fi
if test -f ldrscripts/temp/${table}_ver_${plat}.bad
then
rm ldrscripts/temp/${table}_ver_${plat}.bad
fi
#
# If the merchant reference table is being downloaded, performance a diff
# command to produce a file of changes from the previous download file.
#
if test $table = "merchant_ref"
then
if ls archive/${table}_${plat}.*.* > /dev/null 2>&1
then
#
# Get the most recent merchant reference download and performance diff.
#
ls -t archive/${table}_${plat}.*.* | read line
prevfile=`echo $line`
diff ${prevfile} ${table}_${plat}.dat | grep "> " | sed s/^'> '//
>${table}_${plat}_diff.dat
sqlldr stage/that_stage_thing
control=ldrscripts/${table}_${plat}.ctl data=${table}_${plat}_diff.dat
log=${table}_${plat}.log bad=${table}_${plat}.bad
# mv -f ${table}_${plat}_diff.dat
archive/${table}_${plat}_diff.${timestamp}.${jdate}
else
#
# If a previous download doesn't exist, load current file.
#
sqlldr stage/that_stage_thing
control=ldrscripts/${table}_${plat}.ctl data=${table}_${plat}.dat
fi
else
#
#Load the data file into the staging tablespace with its corresponding
control file using SQL*Loader
#
sqlldr stage/that_stage_thing
control=ldrscripts/${table}_${plat}.ctl data=${table}_${plat}.dat
fi
#
#If the .log file exists, SQL*Loader executed and ATTEMPTED to load the
data, else send an error
#to the daily_load.log for the data file
#
if test -f ${table}_${plat}.log
then
#
#If the .bad file exist, one or more of the records did not load, which
is processed in this
#script as an error written to daily_load.log. If it does not exist,
this script ASSUMES that
#the data loaded, sends a message indicating this to the daily_load.log
and executes the
#verification file load.
#
if test -f ldrscripts/${table}_${plat}.bad
then
echo "${ndate} -- Error loading the data file
${table}_${plat}.dat" >>daily_load.log
echo "${ndate} -- Error loading the data file
${table}_${plat}.dat~${table}_${plat}" >>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
else
echo "${ndate} -- Data file ${table}_${plat}.dat was loaded
successfully" >>daily_load.log
echo "${ndate} -- Data file ${table}_${plat}.dat was loaded
successfully~${table}_${plat}" >>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
#
#Prior to loading the verification file, the static verif. table .ctl file
#is copied/renamed to the temp directory for use in loading the
verification file. This is done so that
#the .log and .bad files created from the load will have a meaningful
name that relates to the .ctl file.
#
cp -f ldrscripts/load_verify_trigger_table.ctl
ldrscripts/temp/${table}_ver_${plat}.ctl
sqlldr stage/that_stage_thing
control=ldrscripts/temp/${table}_ver_${plat}.ctl
data=${table}_ver_${plat}.dat
#
#If the .log file exists, SQL*Loader executed and ATTEMPTED to load the
verif. record, else send
#an error to the daily_load.log
#
if test -f ${table}_ver_${plat}.log
then
#
#If the .bad file exist, one or more of the records did not load, which
is processed in this
#script as an error written to daily_load.log. If it does not exist,
this script ASSUMES that
#the data loaded and sends a message indicating this to the daily_load.log
#
if test -f ldrscripts/temp/${table}_ver_${plat}.bad
then
echo "${ndate} -- Error loading the verification file
${table}_ver_${plat}.dat" >>daily_load.log
echo "${ndate} -- Error loading the verification file
${table}_ver_${plat}.dat~LOAD_VERIFY_TRIGGER_TABLE"
>>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
else
echo "${ndate} -- Verification file
${table}_ver_${plat}.dat was loaded successfully" >>daily_load.log
echo "${ndate} -- Verification file
${table}_ver_${plat}.dat was loaded
successfully~LOAD_VERIFY_TRIGGER_TABLE" >>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
#
#The data and verification files are archived after a successful load
and the temporary .ctl file created
#specifically for this table's verification file is removed
#
mv -f ${table}_${plat}.dat
archive/${table}_${plat}.${timestamp}.${jdate}
mv -f ${table}_ver_${plat}.dat
archive/${table}_ver_${plat}.${timestamp}.${jdate}
rm ldrscripts/temp/${table}_ver_${plat}.ctl
fi
else
echo "${ndate} -- Error loading the verification file
${table}_ver_${plat}.dat" >>daily_load.log
echo "${ndate} -- Error loading the verification file
${table}_ver_${plat}.dat~LOAD_VERIFY_TRIGGER_TABLE"
>>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
fi
fi
else
echo "${ndate} -- Error loading the data file ${table}_${plat}.dat"
>>daily_load.log
echo "${ndate} -- Error loading the data file
${table}_${plat}.dat~${table}_${plat}" >>daily_load_${table}_${plat}.log
sqlldr stage/that_stage_thing
control=ldrscripts/sqlload_messages.ctl data=daily_load_${table}_${plat}.log
rm daily_load_${table}_${plat}.log
exit
fi
Ted L.
May 23, 2013, 10:26:44 PM5/23/13
to
<long shell script deleted>
Thanks Fred. I'll give it a study.
Thanks Fred. I'll give it a study.
0 new messages