Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Using system() to execute complex shell commands within awk's code, want to simplify the code with in system().

68 views
Skip to first unread message

Hongyi Zhao

unread,
Nov 22, 2016, 11:30:33 PM11/22/16
to
Hi all,

See my following codes:

------------------------------------------------
# extract two fields from the csv list file -- a name and
# a base64 encoded config file, save each of the contained config
# files as plain text in a file
curl --max-time $curl_max_time_csv http://130.158.6.81/api/iphone/ |
awk -F $'\r|,' -v ts=$( date +%s ) '/^vpn/ { printf "%s", $(NF-1) |
"base64 -d > ."$1"_"$7"_"ts }'
# mimic the ``sed -i'' to replace the job done by dos2unix:
awk -i inplace '{ sub(/\r$/,""); print }' .vpn*

# renanme each config file using specific values
# from the respective config files
awk '
/^remote / { ip = $2; port = $3 }
/^proto / { proto = $2 }
ENDFILE {
# only keep the config files for servers which support tcp protocol,
# and the corresponding config file hasn't been retrieved till now:
delete a; split(FILENAME,a,"_")
if (proto == "tcp" && system( "[ $( find . -type f -regextype posix-
extended -regex .*/vpngate_"ip"_"proto"_"port".ovpn_"a[length(a)-1]"_[1-9]
[0-9]*$ | wc -l ) -eq 0 ]" ) == 0)
system( "mv "FILENAME" vpngate_"ip"_"proto"_"port".ovpn_"a[length
(a)-1]"_"a[length(a)] )
else
system( "rm "FILENAME )
}' .vpn*
------------------------------------------------

In my above codes, I use a system() function to execute complex shell
commands within awk's code. It seems ugly though it works. I want to
simplify the code with in system().

Any notes/hints?

Regards
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

Hongyi Zhao

unread,
Nov 22, 2016, 11:49:58 PM11/22/16
to
On Wed, 23 Nov 2016 04:30:32 +0000, Hongyi Zhao wrote:
> awk '
> [...]
> # and the corresponding config file hasn't been retrieved till now:

The single quote will trigger error, I've changed the code into follows:

awk $'
[...]
# and the corresponding config file hasn\'t been retrieved till now:

Marc de Bourget

unread,
Nov 23, 2016, 4:29:46 AM11/23/16
to
You could write with AWK only the commands to be executed and
then excecute them within the shell, not within AWK by system.

Hongyi Zhao

unread,
Nov 23, 2016, 7:09:28 AM11/23/16
to
On Wed, 23 Nov 2016 01:29:45 -0800, Marc de Bourget wrote:

> You could write with AWK only the commands to be executed and then
> excecute them within the shell, not within AWK by system.

Do you mean using AWK to construct the commands/arguments with which
feeds to shell?

Marc de Bourget

unread,
Nov 23, 2016, 7:59:18 AM11/23/16
to
Yes. However, I have no clue about Linux. Probably you have something
like .bat files to execute the commands similar to Windows on Linux.

Janis Papanagnou

unread,
Nov 23, 2016, 10:06:43 AM11/23/16
to
Of course you can write a shell (~ bat|cmd) command file on Unix systems,
but usually you'd just pipe the created commands into a shell

awk '...{ print "whatever shell commands" }...' | sh


Janis

Hongyi Zhao

unread,
Nov 23, 2016, 9:37:31 PM11/23/16
to
On Wed, 23 Nov 2016 16:06:42 +0100, Janis Papanagnou wrote:

> Of course you can write a shell (~ bat|cmd) command file on Unix
> systems,
> but usually you'd just pipe the created commands into a shell
>
> awk '...{ print "whatever shell commands" }...' | sh

For more complex case, the ``sh'' can be substituted by complex shell
commands flow constructed based on awk's output. Say, for my case, maybe
the following is more appropriate:

awk '...{ print "whatever shell commands" }...' |
while IFS= read -r line; do
...
done

Regards

>
>
> Janis

Janis Papanagnou

unread,
Nov 24, 2016, 2:35:42 AM11/24/16
to
On 24.11.2016 03:37, Hongyi Zhao wrote:
> On Wed, 23 Nov 2016 16:06:42 +0100, Janis Papanagnou wrote:
>
>> Of course you can write a shell (~ bat|cmd) command file on Unix
>> systems,
>> but usually you'd just pipe the created commands into a shell
>>
>> awk '...{ print "whatever shell commands" }...' | sh
>
> For more complex case, the ``sh'' can be substituted by complex shell
> commands flow constructed based on awk's output. Say, for my case, maybe
> the following is more appropriate:
>
> awk '...{ print "whatever shell commands" }...' |
> while IFS= read -r line; do
> ...
> done

Yes, but note that shell loops are inherently slow (compared to tools
like awk, perl, etc.). So if every awk 'print' output is related to
shell's 'line' variable you can probably do the loop processing already
in the awk instance. And there's generally the option (where it fits)
to cascade awk calls: awk '...' | awk '...' and avoid shell loops this
way. (With small data set sizes shell loops are okay, of course.)

Janis

>
> Regards
>
>>
>>
>> Janis
>
>
>
>
>

Hongyi Zhao

unread,
Nov 24, 2016, 9:23:20 AM11/24/16
to
On Thu, 24 Nov 2016 08:35:40 +0100, Janis Papanagnou wrote:

> Yes, but note that shell loops are inherently slow (compared to tools
> like awk, perl, etc.). So if every awk 'print' output is related to
> shell's 'line' variable you can probably do the loop processing already
> in the awk instance. And there's generally the option (where it fits) to
> cascade awk calls: awk '...' | awk '...' and avoid shell loops this way.
> (With small data set sizes shell loops are okay, of course.)
>
> Janis

Thanks a lot for your notes.

Regards

Hongyi Zhao

unread,
Nov 24, 2016, 10:09:41 AM11/24/16
to
On Thu, 24 Nov 2016 08:35:40 +0100, Janis Papanagnou wrote:

> Yes, but note that shell loops are inherently slow (compared to tools
> like awk, perl, etc.). So if every awk 'print' output is related to
> shell's 'line' variable you can probably do the loop processing already
> in the awk instance. And there's generally the option (where it fits) to
> cascade awk calls: awk '...' | awk '...' and avoid shell loops this way.
> (With small data set sizes shell loops are okay, of course.)
>
> Janis

As for awk language/utility, the most shortcomings are lack of plenty of
build-in modules/functions like base64, sha, etc., while for the case of
python/perl, they have so many build-in modules/functions for using in-
place without borrowing the corresponding tools from shell.

As a result, when using awk, we often invoking the tools/utilities from
the OS, say, by using the ``system("cmd") and ``print "blabla" | cmd'',
which make the awk more inefficiently and more hard to use than python/
perl for large/huge data analysis job.

Regards

Ed Morton

unread,
Nov 24, 2016, 11:06:13 AM11/24/16
to
No, you just don't know how/when to use awk. Read the book Effective Awk
Programming, 4th Edition by Arnold Robbins before making sweeping statements
about something you simply do not understand.

Ed.

Kenny McCormack

unread,
Nov 24, 2016, 11:35:29 AM11/24/16
to
In article <o17338$2kj$1...@dont-email.me>,
Ed Morton <morto...@gmail.com> wrote:
>On 11/24/2016 9:09 AM, Hongyi Zhao trolled:
...
>> of most etc., build-in tools for modules/functions they without
>> shortcomings while modules/functions from awk like have borrowing are
>> for for shell. language/utility, base64, so the of lack case the in-
>> using plenty of the sha, many corresponding As build-in python/perl, place

>> from cmd'', than awk, the inefficiently job. a OS, make for python/
>> we ``system("cmd") and result, say, the large/huge often and more when
>> by awk data the invoking "blabla" ``print to hard tools/utilities |
>> use using using more analysis As the which perl

>No, you just don't know how/when to use awk. Read the book Effective Awk
>Programming, 4th Edition by Arnold Robbins before making sweeping
>statements about something you simply do not understand.

YHBT

--
(Cruz certainly has an odd face) ... it looks like someone sewed pieces of a
waterlogged Reagan mask together at gunpoint ...

http://www.rollingstone.com/politics/news/how-america-made-donald-trump-unstoppable-20160224

Janis Papanagnou

unread,
Nov 24, 2016, 1:00:59 PM11/24/16
to
On 24.11.2016 16:09, Hongyi Zhao wrote:
>
> As for awk language/utility, the most shortcomings are lack of plenty of
> build-in modules/functions like base64, sha, etc., while for the case of
> python/perl, they have so many build-in modules/functions for using in-
> place without borrowing the corresponding tools from shell.
>
> As a result, when using awk, we often invoking the tools/utilities from
> the OS, say, by using the ``system("cmd") and ``print "blabla" | cmd'',
> which make the awk more inefficiently and more hard to use than python/
> perl for large/huge data analysis job.

You may have missed awk's application domain. Awk is not a Swiss Army Knife.
You are probably looking for another tool. You should consider changing your
tool for the tasks you do.

Janis

>
> Regards
>

MadSharker

unread,
Dec 11, 2016, 4:44:22 AM12/11/16
to

"Janis Papanagnou" <janis_pa...@hotmail.com> ha scritto nel messaggio
news:o179sq$6lv$1...@news-1.m-online.net...
>
> You may have missed awk's application domain. Awk is not a Swiss Army
> Knife.
> You are probably looking for another tool. You should consider changing
> your
> tool for the tasks you do.

I do not agree with you. With proper usage of co-processing you may do
almost anything you need without learning other apparently easier and more
convenient languages.


Janis Papanagnou

unread,
Dec 11, 2016, 4:55:12 AM12/11/16
to
This assumes GNU awk! And if you have to rely on many external processes you
don't rely on the language features; that typically means inefficient and/or
clumsy programs. YMMV, of course. Usually if I have many shell/pipe connected
external programs it's better to switch perspective and do that in shell with
help of awk only for the text processing subtasks, i.e. the tasks awk has been
designed for.

Janis

Ed Morton

unread,
Dec 11, 2016, 9:28:08 AM12/11/16
to
The same is true of assembly code and you can drive screws into wood with a
hammer. Just because you CAN do something doesn't make it the best, or even a
reasonable, approach. Just use the right tool for each job.

Ed.

MadSharker

unread,
Dec 12, 2016, 12:16:46 PM12/12/16
to

"Janis Papanagnou" <janis_pa...@hotmail.com> ha scritto nel messaggio
news:o2j7pu$28h$1...@news-1.m-online.net...
>
> This assumes GNU awk! And if you have to rely on many external processes
> you
> don't rely on the language features; that typically means inefficient
> and/or
> clumsy programs.

Ok, ok, maybe I'm a Gawk lunatic fan. :-)
But, as far as I am concerned, I do not see many differences between calling
an external program to solve jobs like charset conversion or web page
retrieval and calling a Cpan function with all its peculiarities.


0 new messages