lex / yacc grammar for ksh and/or POSIX sh
Kevin Collins
does anyone have lex and yacc grammar files for ksh and/or POSIX
sh? I am trying to build a tool for "indexing" shell scripts/functions
and what other commands, scripts and functions they call.
It seems the best bet I have is to write my own "shell interpreter"
which I can "run" each script with to capture any commands that should
be executed.
I've started this on my own already, but as I am a complete newbie to
lex/yacc, it is a painstaking process :)
Any help would be appreciated - especially if anyone can point me to a
tool that already does some or all of what I want to do.
Thanks,
Kevin
Icarus Sparry
One of the reasons that Plan9 abandoned the traditional shell was that
almost no one knew exactly what it did. In particular it didn't use
YACC/LEX. Its replacement 'rc' is built using YACC.
I am not clear exactly what you are expecting the index to actually do.
Perhaps your best bet would be to take an existing shell source and add
extra print statements to it.
You also should look at the '-n' and '-x' arguments to sh/ksh.
What do you want the following to put into the index
if : ; then echo true
else halt_and_catch_fire
fi
If you want it to work out, in the general case, that halt_and_catch_fire
is never called, then you had better be prepared for a very long wait.
This is a variant on the 'halting problem'. Other things are also
troublesome, for instance if the output of 'date' indicates that it is
Friday the 13th, then you define a function called 'ls' which attempts to
start a nuclear war. Later in the script you have the line 'ls'.
OK, these are extreme examples, but it does point out that this is not an
obvious trivial task.
Markus Gyger
> It seems the best bet I have is to write my own "shell interpreter"
> which I can "run" each script with to capture any commands that should
> be executed.
ksh93 comes with a command shcomp that creates a binary format file
of a shell script, maybe that would be easier to parse. AFAIK, it's
not documented (other than "shcomp --man") so you would need to get
the ast-ksh source from http://www.research.att.com/sw/download/
(see in src/cmd/ksh93 the files DESIGN, README and sh/shcomp.c).
OTOH, you might want to look at the flat database output of ksh93's
-R option that can be used with command cql (also part of ast-open).
Markus
Kevin Collins
Good points (I am well versed in -n, -x and -v) - but I am not
expecting to execute any code, only parse. I want to simply collect
the names of all commands executed by a particular script. So, as an
example, this script:
#!/usr/bin/ksh
function Dummy
{
fdate="$(/usr/local/utils/bin/format_date)"
/usr/local/bin/im_a_dummy $fdate
}
eval ./test_cmd \$stuff$(Dummy)
if grep dumb ./more_dumb_stuff
then
cat <<EOF
dumb
dumber
$(cat ./more_dumb_stuff)
EOF
fi
/home/dummy/another_dumb_cmd
would produce the following list:
/usr/local/utils/bin/format_date
/usr/local/bin/im_a_dummy
./test_cmd
Dummy()
grep
cat
cat <-- intentionally there a 2nd time
/home/dummy/another_dumb_cmd
Of course, this script is nonsensical, but it illustrates many of the
possible places you would find a "command" being executed.
The point being that when I need to update "/usr/local/bin/im_a_dummy
", I can (at a glance) get an idea of how many other potential scripts
are affected by generating a report with a "dependency tree".
Now, I can write a Perl script to do all the various parsing logic (in
fact, I had already started that), but I thought it might be simpler
to use lex/yacc since they are designed explicitly for doing this sort
of parsing.
Thanks,
Kevin
b...@iname.com
> ...
> Any help would be appreciated - especially if anyone can point me to a
> tool that already does some or all of what I want to do.
I've already done the work for you, although written in m4, sed, and awk,
and several thousand lines of "line-noise" code, it's a misbegotten
dinosaur of a script!
I'd send it to you, but it's definitely in alpha development status.
Better, I think, is for you to investigate other "xref"-like tools, some
specifically made for ksh(1). They are not currently available, but
for the email addresses of their developers, to whom you may inquire
if you are so motivated.
ciao, WebCiao, kia ("k-shell information abstraction database")
# taken off the free project download site -- email directly
http://www.research.att.com/~chen/
thesmallworlds - ksh edition
# (not currently accessible)
http://www.thesmallworlds.com/products/unixshellscriptsworld.html
You may be satisfied with the undocumented option to ksh93, -R, which
produced a datafile in ciao-format, which is essentially a graphical,
configurable engine to display cxref(1)-like info.
FYI, there _is_ a yacc grammar of ksh(1) in back of [at least] the first
edition of Bolsky and Korn's "The Kornshell Command and Programming Language".
However, this is only moderately valuable once you realize each semantic
unit of shell code requires _three_ quite different parsing passes (variable
list enumeration, redirection, and command line parsing).
=Brian
Kevin Collins
Brian,
thanks immensely for the information. I was beginning to think I
was the only person ever to find this type of thing usefull :)
I would be interested in seeing what you have done so far and I will
definitely check the references you mentioned. I'll email you directly
as well.
Thanks,
Kevin