ksh syntax tree and script call analysis
TD
written mainly in ksh. There are multiple scripts and multiple
directories. The main part of the analysis is calling between
scripts. There is also a significant use of autoload. I'm thinking a
tool which generates a syntax tree would do the job. Are there other
tools which could do the job? I've seen Coco and I'm checking that
out. Are there any other syntax tree generators?
TIA
TD
bsh
"TD" <thinkdi...@mailinator.com> wrote:
> I'm trying to do some analysis on the front end to an application
> written mainly in ksh.
"Some analysis"? Do you mean toolsets that explicate code coverage,
time/efficiency/latency profiling, data-entity/callgraph
visualization, code cross-reference, or what? Your subject
line mentions syntax tree generation and call analysis, but....
> I'm thinking a tool which generates a syntax tree would do the job.
> Are there other tools which could do the job?
Same quibble as above, adding the necessary comment that the
subject and generation of syntax trees are perhaps a broader
subject that you initially think.... DAGs? closures? Intermediate
code representation of 3- or 4-tuples? Pseudo-code or machine
specific?
A stack machine or canonical von Neumann? Interpreter, compiler,
virtual machine, or what?
> There are multiple scripts and multiple directories. The main part
> of the analysis is calling between scripts. There is also a
> significant use of autoload.
Good, good....
> I've seen Coco and I'm checking that out. Are there any other
> syntax tree generators?
Do you mean the "Coco/R" LL(k) compiler generator at:
http://www.ssw.uni-linz.ac.at/Coco/ ?
I would have said that you are engaging on a survey of
existing tools and solutions, were it not that you seek
to use such a sophisticated tool as the above.
Can your requirement be as simple as is satisfied by the
BNF description of ksh(1) in an appendix of B&K, or:
"scriptdeps.sh": recursively determine program dependencies
http://www.shelldorado.com/scripts/cmds/scriptdeps.txt
In general, k/sh(1) shell scripts are not amenable to parsing
with [LA]LR and LL grammars, insofar as their are _three_
separate parsing passes, each with its own lexical rules.
(Yes, I know that bash(1) is written in lex/yacc; I guess
anything is possible).
I have a more detailed, but obsolete, discussion of this
at:
As mentioned in the above reference, it is most ironic
that I've already written a purpose-built k/sh parser
(in m4/sed/oawk!), but it is currently inaccessible. I
even use it for my scripts to do call analysis in another
utility script. Too bad....
It mentions "ciao" (and its GUI, "Acacia") which utilizes
data visualization upon an arbitrary topological sort file;
a module for ksh(1) is included. They are currently
unavailable but may be had for an email to the author.
"ciao.c": graph visualization: Not currently accessible
http://www.research.att.com/~chen/
"Yih-Farn Robin CHEN" <ch...@research.att.com>
"Acacia.c": graph visualization: GUI frontend for ciao(1)
http://www.research.att.com/sw/tools/Acacia/
http://www.program-transformation.org/Transform/AcaCia/
CHEN, Yih-Farn Robin, Glenn S. Fowler, Eleftherios Koutsofios, Ryan S.
Wallach.
"Ciao: A Graphical Navigator for Software and Document Repositories".
1995. Proceedings of the International Conference on Software
Maintenance. ICSM archive.
ISBN:0-8186-7141-6. <http://citeseer.ist.psu.edu/chen95ciao.html>.
"AT&T Information Visualization Research Group":
http://public.research.att.com/areas/visualization/papers_videos/subpage.php?page=papers/author.html&title=Papers%20by%20Author
Now, If I understand your intention, the practical and
ideal solution is built in to ksh(1)! With the compile-
time option "KIA" (Kornshell Information Abstraction)
asserted for ksh93(1) (which is freely available in
source form) the creation of a relational database for
commands, variables and functions defined and referenced
by a script. The ksh93(1) option "-I <filename>" (or
"-R" <filename> -- documentation varies) causes the
database to be generated in <filename>, which can be
queried by the tool cql, and/or (presumably) viewed by
the graphical tools Acacia/ciao, which produces GraphViz-
like transition network graphics.
"cql.c":
http://www.research.att.com/~gsf/man/man1/cql.html
http://public.research.att.com/~gsf/cql/cql.html
"cdb.c":
http://www.research.att.com/~gsf/man/man1/cdb.html
If you go this route, I'd like to know your experiences.
For years I have investigated available resources
specifically for programmers of sh(1) and ksh(1) --
I have many more resources and ideas not here. You may
email me for inquiries outside of this thread. Good luck!
=Brian
TD
the info. I had a quick look at the showdependencies script but
unfortunately it relies on the scripts being documented with requires
and that is not the case with my system. It's an older system with no-
one who knows the system to explain it's structure. I was thinking
there ought to be something within ksh, as you described, which could
help. I might check that out. I have taken a quick and dirty
approach for now where I did a grep for each line mentioning the
basename of the file within any other file thus generating a list of
files and names which that file refers to. I'm now trying to generate
a dependency diagram from that list.
TD
bsh
> Thanks, Brian, some of that was a bit mind blowing. But thanks for
> the info. I had a quick look at the showdependencies script but
> unfortunately it relies on the scripts being documented with requires
> and that is not the case with my system. It's an older system with no-
> one who knows the system to explain it's structure. I was thinking
> there ought to be something within ksh, as you described, which could
> help. I might check that out.
You mean: "ksh93 -I depfile myscript" ? It is the preferable solution.
I hope you at least investigate it. (Ksh93 is freely downloadable
through
kornshell.com).
It would be a pity if you couldn't use it, because the AT&T AST
library
(including ksh, cdb, ciao, ...) is a 1000 man-years of software tools
primarily devised for reverse engineering of complex legacy software.
> I have taken a quick and dirty
> approach for now where I did a grep for each line mentioning the
> basename of the file within any other file thus generating a list of
> files and names which that file refers to. I'm now trying to generate
> a dependency diagram from that list.
This is a practical goal. If the above doesn't pan out,
GraphViz (or even better, WebDot) is a free and viable solution,
although you will have to generate the dependency table yourself.
Here is an example (with source, so cheat!):
GraphViz/WebDot example: "Module Dependencies":
http://www.graphviz.org/Gallery/undirected/softmaint.html
http://www.graphviz.org/webdot/
"WebDot.tcl": GraphVix GUI
http://www.graphviz.org/webdot/demo.html # look at "jsort"
I was thinking that "scriptran" just might be a java executable,
and therefore runnable on non-Solaris systems (if' that what
you have -- you didn't mention); it is at least worth investigating:
"scriptran": "Bourne shell script analysis tool"
http://soldc.sun.com/tools/linux/L3.html
http://developers.sun.com/solaris/downloads/lincat/lincat_3.html
=Brian