multi-line print from awk with shell

[Ed Morton]

The script below will let you write an awk script that prints multi-line
blocks of text. For example:

$ epawk -v awkvar="Hello" 'BEGIN{
print "-----------"
_print "
a multi-line block
of text just
to say "awkvar"
is kinda overkill,
I suppose!
"
print "-----------"
}'
-----------
a multi-line block
of text just
to say Hello
is kinda overkill,
I suppose!
-----------

Any suggestions on how it could be improved (e.g. made more robust)?

I've been asking specific questions about various parts of it over the
past couple of weeks here at comp.unix.shell and at comp.lang.awk, so I
think I need to post the whole script at this point to get more input -
thanks to everyone who helped me get it this far!

Ed.

##########################################################
# Extended Print AWK
#
# Allows printing of pre-formatted blocks of multi-line text in awk scripts.
#
# Before invoking the tool, do the following IN ORDER:
#
# 1) Start each block of pre-formatted text in your script with
# _print "
# on it's own line and end it with
# "
# on it's own line. Spaces are allowed before and/or after each symbol.
#
# 2) Within each block of pre-formatted text only:
# a) Put a backslash character (\) before every backslash.
# b) Put a backslash character before every double quote (").
# c) Enclose awk variables and and field references ($1, $2, etc.)
# in double quotes (without leading backslashes).
#
# 3) If the script is specified on the command line instead of via
# "-f script" then replace all single quote characters (') in or out
# of the pre-formatted blocks with their ANSI octal escape sequence
(\047).
# This is normal and is required because command-line awk scripts cannot
# contain single quote characters as those delimit the script. Use octal
# \047 instead of hex \x27, see
http://awk.freeshell.org/PrintASingleQuote.
#
# Then just use it like you would gawk with the small caveat that only
# "-W <option>", not "--<option>", is supported for long options so you
# can use "-W re-interval" but not "--re-interval" for example.
#
# See the bottom of this file for usage examples.
##########################################################

toolName="$(basename "$0")"

expand_prints() {
gawk '
/^[[:blank:]]*_print[[:blank:]]*["][[:blank:]]*$/ {
pfx="print \""; sfx="\""; next
}
pfx && /^[[:blank:]]*["][[:blank:]]*$/ {
pfx=""; sfx=""; next
}
{ print pfx $0 sfx }
' "$@"
}

unset awkArgs
unset scriptFiles
while getopts "v:F:W:f:" arg
do
case $arg in
f ) scriptFiles+=( "$OPTARG" ) ;;
[vFW] ) awkArgs+=( "-$arg" "$OPTARG" ) ;;
* ) exit 1 ;;
esac
done
shift $(( OPTIND - 1 ))

if [ -z "${scriptFiles[*]}" -a "$#" -gt "0" ]
then
# The script cannot contain literal 's because in cases like this:
# 'BEGIN{ ...abc'def... }'
# the args parsed here (and later again by gawk) would be:
# $1 = BEGIN{ ...abc
# $2 = def... }
# Replace 's with \047 if you need them.
scriptText="$1"
shift
fi

# Remaining symbols in "$@" must be data file names and/or variable
# assignments that do not use the "-v name=value" syntax.

if [ -n "${scriptFiles[*]}" ]
then
gawk "${awkArgs[@]}" "$(expand_prints "${scriptFiles[@]}")" "$@"

elif [ -n "$scriptText" ]
then
gawk "${awkArgs[@]}" "$(printf '%s\n' "$scriptText" |
expand_prints)" "$@"
else
printf '%s: ERROR: no awk script specified.\n' "$toolName" >&2
exit 1
fi

exit
##########################################################
USAGE EXAMPLES:

$ cat data.txt
abc def"ghi
$
#######
$ cat script.awk
{
awkVar="bar"

print "----------------"

_print "
literal backslash: \\

quoted text: \"text\"

single quote as ANSI sequence: \047

literal single quote (ONLY works when script is in a file): '

awk variable: "awkVar"

awk space-separated field: "$2"
"

print "----------------"
}
$
$ epawk -f script.awk data.txt
----------------
literal backslash: \

quoted text: "text"

single quote as ANSI sequence: '

literal single quote (ONLY works when script is in a file): '

awk variable: bar

awk space-separated field: def"ghi
----------------
$
#######
$ epawk -F\" '{
_print "
ANSI-tick-surrounded quote-separated field 2 (will work): \047"$2"\047
"
}' data.txt
ANSI-tick-surrounded quote-separated field 2 (will work): 'ghi'
$
#######
$ epawk -F\" '{
_print "
Literal-tick-surrounded quote-separated field 2 (will not work): '"$2"'
"
}' data.txt
Literal-tick-surrounded quote-separated field 2 (will not work):
$
#######


Posted using www.webuse.net

[Kaz Kylheku]

On 2012-01-23, Ed Morton <morto...@gmail.com> wrote:
> -----------
> a multi-line block
> of text just
> to say Hello
> is kinda overkill,
> I suppose!
> -----------
>
> Any suggestions on how it could be improved (e.g. made more robust)?

Common Lisp has multi-line literals also, and in the Lisp format function,
there is a useful feature. Format specifiers are introduced by the character ~
which is analogous to the % in printf. If a ~ is followed by a newline, it
means that the following whitespace is eaten. ~% means output a newline.

(format t "multi~%~
line~%~
output~%")

You could have a similar feature in _print, so that the indentation of
your multi-liners does not end up in the output.

Your epawk preprocessor is a good idea. You might want to formalize it
a little and make it extensible.

The preprocessed construct might stand out a little more in some way rather
than hiding them into the language. (IMHO item, obviously.) We can't tell that
_print is a preprocessing gadget, because it's a an ordinary identifier.
Which would be fine if you're prepared to integrate this stuff deeply into the
language, to the level that you actually parse it, but if the preprocessing is
something that "shallow (not aware of syntax) it's better if it is easily
recognizeable ("here is an invocation of the preprocessor, beginning here and
obviously terminating here.")

(I've been recently thinking of making a preprocessor for shell scripts which
will provide a cleaned-up language. For instance, in the preprocessed language,
a $VAR expansion would not be subject to word splitting or globbing. The
preprocessor can turn that into "$VAR". That's just one of the many things it
could do.)

One idea I have is this: the preprocessor should check cache the results of
preprocessing, analogous to a compiler producing an object file or executable.

Thus, don't call the processed scripts .awk but, say, .epawk. The epawk
compiler can check, when given file.epawk, whether a file.awk already exists.
If file.awk already exists with a newer timestamp, then just run file.awk,
otherwise preprocess file.epawk to file.awk, and run it.

You should be able to write #! scripts using #!/usr/bin/epawk (with the above
compilation caching).

Don't forget to turn epawk into a .epawk script which uses _print internally
so that epawk is needed to compile epawk! :)

[Ed Morton]

Kaz Kylheku <k...@kylheku.com> wrote:

> On 2012-01-23, Ed Morton <morto...@gmail.com> wrote:
> > -----------
> > a multi-line block
> > of text just
> > to say Hello
> > is kinda overkill,
> > I suppose!
> > -----------
> >
> > Any suggestions on how it could be improved (e.g. made more robust)?
>
> Common Lisp has multi-line literals also, and in the Lisp format function,
> there is a useful feature. Format specifiers are introduced by the
character ~
> which is analogous to the % in printf. If a ~ is followed by a newline, it
> means that the following whitespace is eaten. ~% means output a newline.
>
> (format t "multi~%~
> line~%~
> output~%")
>
> You could have a similar feature in _print, so that the indentation of
> your multi-liners does not end up in the output.

I could see that being useful. Like the "-" in here docs:

$ cat <<!
foo
bar
!
foo
bar

$ cat <<-!
foo
bar
!
foo
bar

You've got me thinking I should maybe change epawk so instead of the
syntax for printing multi-line blocks being:

_print "
stuff-to-print with leading spaces
"

it should be:

cat <<TEXT
stuff-to-print with leading tabs
TEXT

and:

cat <<-TEXT
stuff-to-print without leading tabs
TEXT

so users can specify whatever TEXT they like just like in a here doc. I
need to think about that a bit more.

>
> Your epawk preprocessor is a good idea. You might want to formalize it
> a little and make it extensible.
>
> The preprocessed construct might stand out a little more in some way rather
> than hiding them into the language. (IMHO item, obviously.) We can't
tell that
> _print is a preprocessing gadget, because it's a an ordinary identifier.
> Which would be fine if you're prepared to integrate this stuff deeply
into the
> language, to the level that you actually parse it, but if the
preprocessing is
> something that "shallow (not aware of syntax) it's better if it is easily
> recognizeable ("here is an invocation of the preprocessor, beginning
here and
> obviously terminating here.")

Yeah, at one point I had it delimited by "HERE_{" and "}_HERE" and various
other strings that are not valid awk (C) symbols but eventually I liked
the simplicity and brevity of _print "...".

>
> (I've been recently thinking of making a preprocessor for shell scripts
which
> will provide a cleaned-up language. For instance, in the preprocessed
language,
> a $VAR expansion would not be subject to word splitting or globbing. The
> preprocessor can turn that into "$VAR". That's just one of the many
things it
> could do.)
>
> One idea I have is this: the preprocessor should check cache the results of
> preprocessing, analogous to a compiler producing an object file or
executable.
>
> Thus, don't call the processed scripts .awk but, say, .epawk. The epawk
> compiler can check, when given file.epawk, whether a file.awk already
exists.
> If file.awk already exists with a newer timestamp, then just run file.awk,
> otherwise preprocess file.epawk to file.awk, and run it.
>
> You should be able to write #! scripts using #!/usr/bin/epawk (with the
above
> compilation caching).
>
> Don't forget to turn epawk into a .epawk script which uses _print internally
> so that epawk is needed to compile epawk! :)

Not sure about all of that. I'm trying to avoid temp files and don't
really see the benefit other than speed of execution which I don't see as
being an issue as the speed of parsing the script should be negligible.

Ed.


Posted using www.webuse.net

[Kaz Kylheku]

For now; give it a year or two. :)

[Ed Morton]

I decided I would implement something like that but I used "print" instead
of "cat". You now delimit your blocks with:

print <<!
your text
!

You can replace "!" with any symbol you like and if you put a "-"
immediately after the "<<" then it will strip all leading tabs, like shell
here-documents.

I added a tweak that's not available in shell here-documents though - if you
specify "+" instead of "-" then the tool will only strip the number of
leading tabs that are common across all of your lines in that block. That's
so you can indent the block to look good within your script AND indent some
lines of the pre-formatted text so it looks good for output. Here's the
difference between the 3 specifications:

##################
$ cat script1.awk
BEGIN {

print << HERE

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

HERE

}
$ epawk -f script1.awk

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

##################
$ cat script2.awk
BEGIN {

print <<- HERE

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

HERE

}
$ epawk -f script2.awk

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

##################
$ cat script3.awk
BEGIN {

print <<+ HERE

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

HERE

}
$ epawk -f script3.awk

main(int argc)
{
if (argc) {
doStuff();
}

return;
}

##################

The only difference in the input files above is the char that follows "<<".
All leading white space is tabs.

The resulting script is below if anyone cares, but it's gotten kinda lengthy!

Ed.

##########################################################
# Extended Print AWK
#
# Allows printing of pre-formatted blocks of multi-line text in awk scripts.
#
# Before invoking the tool, do the following IN ORDER:
#
# 1) Start each block of pre-formatted text in your script with

# print << TERMINATOR

# on it's own line and end it with

# TERMINATOR
# on it's own line. TERMINATOR can be any sequence of characters you like.
# Spaces are allowed before and/or after "print" and TERMINATOR but are
# not required. If << is followed by -, e.g.:
# print <<- TERMINATOR
# then all leading tabs are removed from the block of pre-formatted
# text (just like shell here documents), if it's followed by + instead, e.g.:
# print <<+ TERMINATOR
# then however many leading tabs are common across all non-blank lines
# in the current pre-formatted block are removed.
# By default no leading tabs are removed.

#
# 2) Within each block of pre-formatted text only:
# a) Put a backslash character (\) before every backslash.
# b) Put a backslash character before every double quote (").
# c) Enclose awk variables and and field references ($1, $2, etc.)
# in double quotes (without leading backslashes).
#
# 3) If the script is specified on the command line instead of via
# "-f script" then replace all single quote characters (') in or out
# of the pre-formatted blocks with their ANSI octal escape sequence (\047)

# or the sequence '\'' (tick backslash tick tick). This is normal and is
# required because command-line awk scripts cannot contain single quote
# characters as those delimit the script. Do not use hex \x27, see
# http://awk.freeshell.org/PrintASingleQuote.

#
# Then just use it like you would gawk with the small caveat that only
# "-W <option>", not "--<option>", is supported for long options so you
# can use "-W re-interval" but not "--re-interval" for example.
#

# To just see the post-processed script and not execute it, call this
# script with the "-X" option.
#

##########################################################

toolName="$(basename "$0")"

expand_prints() {

gawk '

!inBlock && sub(/^[[:blank:]]*print[[:blank:]]*<</,"") {

if ( sub(/^[-]/,"") ) { skipType = "-" }
else if ( sub(/^[+]/,"") ) { skipType = "+" }
else { skipType = "" }

terminator = $0
gsub(/(^[[:blank:]]+|[[:blank:]]+$)/,"",terminator)

startBlock()

next
}

inBlock {

stripped=$0
gsub(/(^[[:blank:]]+|[[:blank:]]+$)/,"",stripped)

if ( stripped"" == terminator"" ) {
endBlock()
}
else {
updBlock()
}

next
}

{ print }

function startBlock() { inBlock=1; numLines=0 }

function updBlock() { block[++numLines] = $0 }

function endBlock( i,numSkip,indent) {

if (skipType == "") {
# do not skip any leading tabs
indent = ""
}
else if (skipType == "+") {

# skip however many leading tabs are common across
# all non-blank lines in the current pre-formatted block

for (i=1;i<=numLines;i++) {

if (block[i] ~ /[^[:blank:]]/) {

match(block[i],/^[\t]+/)

if ( (numSkip == "") || (numSkip > RLENGTH) ) {
numSkip = RLENGTH
}
}
}

for (i=1;i<=numSkip;i++) {
indent = indent "\t"
}
}

for (i=1;i<=numLines;i++) {
sub(indent,"",block[i])
print "print \"" block[i] "\""
}

inBlock=0

}

' "$@"

}

unset awkArgs
unset scriptFiles

expandOnly=0
while getopts "v:F:W:f:X" arg

do
case $arg in
f ) scriptFiles+=( "$OPTARG" ) ;;
[vFW] ) awkArgs+=( "-$arg" "$OPTARG" ) ;;

X ) expandOnly=1 ;;

* ) exit 1 ;;
esac
done
shift $(( OPTIND - 1 ))

if [ -z "${scriptFiles[*]}" -a "$#" -gt "0" ]
then
# The script cannot contain literal 's because in cases like this:
# 'BEGIN{ ...abc'def... }'
# the args parsed here (and later again by gawk) would be:
# $1 = BEGIN{ ...abc
# $2 = def... }

# Replace 's with \047 or '\'' if you need them:
# 'BEGIN{ ...abc\047def... }'
# 'BEGIN{ ...abc'\''def... }'

scriptText="$1"
shift
fi

# Remaining symbols in "$@" must be data file names and/or variable
# assignments that do not use the "-v name=value" syntax.

if [ -n "${scriptFiles[*]}" ]
then

if (( expandOnly == 1 ))
then
expand_prints "${scriptFiles[@]}"
else

gawk "${awkArgs[@]}" "$(expand_prints "${scriptFiles[@]}")" "$@"

fi

elif [ -n "$scriptText" ]
then
if (( expandOnly == 1 ))
then
printf '%s\n' "$scriptText" | expand_prints
else

gawk "${awkArgs[@]}" "$(printf '%s\n' "$scriptText" |
expand_prints)" "$@"

fi

else
printf '%s: ERROR: no awk script specified.\n' "$toolName" >&2
exit 1
fi

Posted using www.webuse.net

[Dave Gibson]

Ed Morton <morto...@gmail.com> wrote:

> The resulting script is below if anyone cares, but it's gotten
> kinda lengthy!

It's about to get a little longer.

> unset awkArgs
> unset scriptFiles

Explicitly declare awkArgs and scriptFiles to be arrays by adding
either this (not within a function -- typeset creates locals):

typeset -a awkArgs scriptFiles

or these (may be used anywhere):

awkArgs=( $awkArgs )
scriptFiles=( $scriptFiles )

This is for the benefit of zsh which expands an uninitialised
double-quoted array to a null string rather than completely
removing the parameter:

$ zsh -c 'n() { echo $# ; } ; n "${a[@]}"'
1
$ zsh -c 'n() { echo $# ; } ; a=( $a ) ; n "${a[@]}"'
0

Initialising the arrays with known-empty parameters appears to be
necessary for ksh93 which sets a[0] to "()" when given a=().

> while getopts "v:F:W:f:X" arg
> do
> case $arg in
> f ) scriptFiles+=( "$OPTARG" ) ;;
> [vFW] ) awkArgs+=( "-$arg" "$OPTARG" ) ;;
> X ) expandOnly=1 ;;
> * ) exit 1 ;;
> esac
> done

> gawk "${awkArgs[@]}" "$(expand_prints "${scriptFiles[@]}")" "$@"

^^^^^^^^^^^^^^^
Here's where it all goes wrong. If the script isn't given any pass-
through-to-awk arguments (-v, -F, -W) awkArgs won't be populated and
zsh will invoke gawk with an empty script string.