Using shell variable in getline's redirction block.

[Hongyi Zhao]

Hi all,

In my script, I using the following snippet:

ovpn_file_list_dir=$topdir/ovpn_file_list.d
awk 'BEGIN{FS="/";while ((getline < "./ovpn_file_list.d/all-ovpn-
file.list" ) > 0) key[$NF]} !($NF in key )' FS="/" <( find ./current/ -
type f -regextype posix-extended -regex ".*/vpngate_([0-9]{1,3}\.){3}[0-9]
{1,3}_tcp_[0-9]{1,5}[.]ovpn$" | sort -t / -k3V | awk -F'/' '!a[$NF]++')
>> $ovpn_file_list_dir/all-ovpn-file.list

If I want to using shell variable after the getline's redirect symbol:

i.e., change the following:

getline < "./ovpn_file_list.d/all-ovpn-file.list"

into something like this:

getline < "var-pointed-to-all-ovpn-file.list"

How can I achieve this goal?

Regards
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

[Hongyi Zhao]

On Sun, 24 Jan 2016 04:31:37 +0000, Hongyi Zhao wrote:

> ovpn_file_list_dir=$topdir/ovpn_file_list.d awk 'BEGIN{FS="/";while
> ((getline < "./ovpn_file_list.d/all-ovpn- file.list" ) > 0) key[$NF]}
> !($NF in key )' FS="/" <( find ./current/ - type f -regextype
> posix-extended -regex ".*/vpngate_([0-9]{1,3}\.){3}[0-9]
> {1,3}_tcp_[0-9]{1,5}[.]ovpn$" | sort -t / -k3V | awk -F'/'
> '!a[$NF]++')
>>> $ovpn_file_list_dir/all-ovpn-file.list

I've tried the following method, but failed to work:

awk -v awkvar=$ovpn_file_list_dir 'BEGIN{FS="/";while ((getline < awkvar

) > 0) key[$NF]} !($NF in key )' FS="/" <( find ./current/ -type f -
regextype posix-extended -regex ".*/vpngate_([0-9]{1,3}\.){3}[0-9]{1,3}
_tcp_[0-9]{1,5}[.]ovpn$" | sort -t / -k3V | awk -F'/' '!a[$NF]++') >>
$ovpn_file_list_dir/all-ovpn-file.list

[Janis Papanagnou]

On 24.01.2016 05:31, Hongyi Zhao wrote:
> Hi all,
>
> In my script, I using the following snippet:
>
> ovpn_file_list_dir=$topdir/ovpn_file_list.d
> awk 'BEGIN{FS="/";while ((getline < "./ovpn_file_list.d/all-ovpn-
> file.list" ) > 0) key[$NF]} !($NF in key )' FS="/" <( find ./current/ -
> type f -regextype posix-extended -regex ".*/vpngate_([0-9]{1,3}\.){3}[0-9]
> {1,3}_tcp_[0-9]{1,5}[.]ovpn$" | sort -t / -k3V | awk -F'/' '!a[$NF]++')
>>> $ovpn_file_list_dir/all-ovpn-file.list

Are you really expecting that folks analyse that (off-topic, unformatted)
shell trash?

>
> If I want to using shell variable after the getline's redirect symbol:
>
> i.e., change the following:
>
> getline < "./ovpn_file_list.d/all-ovpn-file.list"
>
> into something like this:
>
> getline < "var-pointed-to-all-ovpn-file.list"

Then use a variable instead of a string constant literal, and write
something like

getline < var

Basics like that can be found in any awk introduction; get a book and read
it, as vou've been suggested so many times.

Janis

[Kaz Kylheku]

On 2016-01-24, Hongyi Zhao <hongy...@gmail.com> wrote:
> If I want to using shell variable after the getline's redirect symbol:
>
> i.e., change the following:
>
> getline < "./ovpn_file_list.d/all-ovpn-file.list"
>
> into something like this:
>
> getline < "var-pointed-to-all-ovpn-file.list"
>
> How can I achieve this goal?

I'm surprised that after all this time you don't know how to pass
a piece of data from the shell into Awk. This topic has been rehashed
numerous times.

The most transparent way is to pass the variable via the environment:

$ MY_VARIABLE=whatever awk 'BEGIN { print ENVIRON["MY_VARIABLE"] }'
whatever

Other than that, there are hacky approaches where you have to be clever,
depending on the content of the variable. There is the "intermediately
hacky" method using Awk's -v option:

$ awk -v foo=whatever 'BEGIN { print foo }'
whatever

Here, "whatever" is considered Awk syntax; it cannot be arbitrary
characters. In the Awk script, foo is installed an Awk variable.

Most hacky way: use shell single quotes and interpolation:

$ awk "BEGIN { print \"$TERM\" }"
xterm

If the variable comes as user input from the Internet, a malicious user
can inject code into your Awk script.

$ MALICIOUS="\"} { print \$1 + \$2 } END { print \""

$awk "BEGIN { print \"$MALICIOUS\" }"

1 2
3
3 4
7

[Hongyi Zhao]

On Sun, 24 Jan 2016 06:15:08 +0100, Janis Papanagnou wrote:

> Then use a variable instead of a string constant literal, and write
> something like
>
> getline < var
>
> Basics like that can be found in any awk introduction; get a book and
> read it, as vou've been suggested so many times.

Thanks a lot, I've got it.

Regards
>
> Janis

[Ed Morton]

You asked and got an answer to this on stackoverflow
(http://stackoverflow.com/questions/34972278/using-a-shell-variable-in-getlines-redirection-block).

The questions you post on usenet are like polluting a river, please don't start
polluting other rivers (e.g. stackoverflow) with the same effluent. For some
reason people keep giving you answers on usenet - be happy with that and keep
dumping your sewage in just this one river, there's no need for you to start
polluting others.

Ed.

[Ed Morton]

On 1/23/2016 11:59 PM, Kaz Kylheku wrote:
> On 2016-01-24, Hongyi Zhao <hongy...@gmail.com> wrote:
>> If I want to using shell variable after the getline's redirect symbol:
>>
>> i.e., change the following:
>>
>> getline < "./ovpn_file_list.d/all-ovpn-file.list"
>>
>> into something like this:
>>
>> getline < "var-pointed-to-all-ovpn-file.list"
>>
>> How can I achieve this goal?
>
> I'm surprised that after all this time you don't know how to pass
> a piece of data from the shell into Awk. This topic has been rehashed
> numerous times.
>
> The most transparent way is to pass the variable via the environment:
>
> $ MY_VARIABLE=whatever awk 'BEGIN { print ENVIRON["MY_VARIABLE"] }'
> whatever
>
> Other than that, there are hacky approaches where you have to be clever,
> depending on the content of the variable. There is the "intermediately
> hacky" method using Awk's -v option:
>
> $ awk -v foo=whatever 'BEGIN { print foo }'
> whatever
>
> Here, "whatever" is considered Awk syntax; it cannot be arbitrary
> characters.

That statement is incorrect, "whatever" is just a numeric-string stored in a
variable and can be arbitrary characters. Not all characters are treated
literally, though - a backslash followed by a character is treated as the
expansion of that escape sequence, so "\t" becomes a literal tab char and if you
want the literal string "\t" you need to escape the backslash "\\t".

You may have been thinking of the mush of shell variables expanding inside awk
below - the contents of those shell variables DO become part of the awk program
and so must conform to awk syntax.

Ed.

[Kaz Kylheku]

On 2016-01-24, Ed Morton <morto...@gmail.com> wrote:
> On 1/23/2016 11:59 PM, Kaz Kylheku wrote:
>> On 2016-01-24, Hongyi Zhao <hongy...@gmail.com> wrote:
>>> If I want to using shell variable after the getline's redirect symbol:
>>>
>>> i.e., change the following:
>>>
>>> getline < "./ovpn_file_list.d/all-ovpn-file.list"
>>>
>>> into something like this:
>>>
>>> getline < "var-pointed-to-all-ovpn-file.list"
>>>
>>> How can I achieve this goal?
>>
>> I'm surprised that after all this time you don't know how to pass
>> a piece of data from the shell into Awk. This topic has been rehashed
>> numerous times.
>>
>> The most transparent way is to pass the variable via the environment:
>>
>> $ MY_VARIABLE=whatever awk 'BEGIN { print ENVIRON["MY_VARIABLE"] }'
>> whatever
>>
>> Other than that, there are hacky approaches where you have to be clever,
>> depending on the content of the variable. There is the "intermediately
>> hacky" method using Awk's -v option:
>>
>> $ awk -v foo=whatever 'BEGIN { print foo }'
>> whatever
>>
>> Here, "whatever" is considered Awk syntax; it cannot be arbitrary
>> characters.
>
> That statement is incorrect, "whatever" is just a numeric-string stored in a
> variable and can be arbitrary characters.

Contradicted immediately below:

> literally, though - a backslash followed by a character is treated as the
> expansion of that escape sequence, so "\t" becomes a literal tab char and if you
> want the literal string "\t" you need to escape the backslash "\\t".

\ and t are two arbitrary characters. Their conversion to TAB is due to
Awk syntax. It's not the whole syntax, of course, just some of it,
namely the syntax of a STRING token. You can't put a "pattern action"
phrase into a -v, obviously.

This syntactic processing doesn't happen under environment passing:

$ foo='\t' awk 'BEGIN { print ENVIRON["foo"] }'
\t
$ awk -vfoo='\t' 'BEGIN { print foo }'

POSIX and implementations seem to be at odds with each other here in
some ways. Both mawk and gawk (and gawk --posix) reproduce the quotes
in this example:

$ awk -v foo='"bar"' 'BEGIN { print foo }' # gawk or mawk
"bar"

But POSIX says, very clearly:

"The characters following the equal sign will be interpreted as if
they appeared in the awk program preceded and followed by a
double-quote (") character, as a STRING token (see Grammar ), except
that if the last character is an unescaped backslash, it will be
interpreted as a literal backslash rather than as the first character
of the sequence \". The variable will be assigned the value of that
STRING token."

According to this, we should get an empty string, because "bar" is
put between quotes, resulting in ""bar"", and when a STRING token
is extracted from that, we get en empty one, with the bar"" remaining
untokenized. Or else we should get a syntax error, because the input
didn't correspond to a STRING token in its entirety.

About STRING, the specification says:

A string constant will be terminated by the first unescaped occurrence
of the character after the one that begins the string constant.

Maybe this is undefined behavior? In C, an undefined behavior situation
exists when the token pasting ## operator produces something that isn't
a lexically a single, valid preprocessor token. This is vaguely similar:
the material from -v is pasted up into a string literal, but the result
isn't a single string literal token.

By the way it is also written that "[A] newline character will not occur
within a string constant." That's a very significant way of not
supporting "arbitrary characters". Gawk (with or without --posix) and
Mawk again *do* support a newline rather than diagnosing a bad string
literal syntax, which is better behavior of course, but out of spec:

$ awk -v foo='hey
there' 'BEGIN { print foo }'
hey
there

[Ed Morton]

Sigh. If you believe that escape sequences being expanded means that you have to
worry about awk syntax when assigning variables then feel free to believe that,
even though you pointed out yourself that you can't include a newline in an awk
string constant but you can in a "-v" variable assignment which shows that that
assertion is false. You can include newlines in the var init because the awk
variable is bring inited from a shell string constant, not an awk string
constant, and so is unrelated to awk syntax rules. The rest of the discussion
below is a mixture of obvious statements, irrelevant quotes and mixing up shell
string constants with awk string constants.

Ed.

[Kaz Kylheku]

\t is an example of Awk syntax.

> free to believe that, > even though you pointed out yourself that you
> can't include a newline in an awk string constant but you can in a
> "-v" variable assignment which shows that that

You missed that this is not according to POSIX though, just according to
the behavior of two implementations I tried. POSIX says that there must
be no newline.

> assertion is false.

What assertion does that show is false?

> You can include newlines in the var init because the awk
> variable is bring inited from a shell string constant, not an awk string

You can include newlines in the var init because you're using an awk
implementation which flouts that area of POSIX. Full stop; that is the
reason.

Yes, I used the shell syntax correctly to demonstrate this, which is
irrelevant:

$ awk -v foo='abc
def' ...

I could show it with C syntax also:

execlp("awk", "awk", "-v", "foo=abc\ndef", (char *) NULL);

Here the newline is represented in C as \n. The point is
that the awk interpreter is getting an argument string with
a newline character in it.

Posix says that var=init is treated like this. First, if init
ends in a backslash, then this init\ is converted to init\\
so that the trailing backslash is always literal.
Next, init is wrapped in double quotes, to make "init".
Then, the result is processed as a STRING token.

STRING tokens do not allow an embedded newline, nor unescaped double
quotes.

If you can do foo='ab"c' or use a newline, that's an implementation
behavior which deviates from POSIX.

Anyway, even with these behaviors, it's not transparent.

> constant, and so is unrelated to awk syntax rules. The rest of the
> discussion below is a mixture of obvious statements, irrelevant quotes
> and mixing up shell string constants with awk string constants.

A mixup between awk string literals and shell string literals would
require a depth of stupidity rivaling your own.

[Kenny McCormack]

In article <201601241...@kylheku.com>,
Kaz Kylheku <609-57...@kylheku.com> wrote:
...

>A mixup between awk string literals and shell string literals would
>require a depth of stupidity rivaling your own.

Ed makes no claims to being even close to being the brightest bulb in the pack.

--
> No, I haven't, that's why I'm asking questions. If you won't help me,
> why don't you just go find your lost manhood elsewhere.

CLC in a nutshell.

[Kaz Kylheku]

On 2016-01-25, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <201601241...@kylheku.com>,
> Kaz Kylheku <609-57...@kylheku.com> wrote:
> ...
>>A mixup between awk string literals and shell string literals would
>>require a depth of stupidity rivaling your own.
>
> Ed makes no claims to being even close to being the brightest bulb in
> the pack.

Ah, that likely points to excessively high resistance of the filament,
Watts being V^2/R.

[Ed Morton]

Another valuable contribution to the community from the guy who chose to be
named after a South Park character and a typically topical/useful response from
the guy who answers most shell/awk questions with a suggestion to use his own
wacky home-grown tool with its user community of 1 instead.

Thanks kids, I hope your moms continue to tell you every day that everyone else
is just jealous.

Ed.

[Kaz Kylheku]

On 2016-01-25, Ed Morton <morto...@gmail.com> wrote:
> On 1/25/2016 1:48 AM, Kaz Kylheku wrote:
>> On 2016-01-25, Kenny McCormack <gaz...@shell.xmission.com> wrote:
>>> In article <201601241...@kylheku.com>,
>>> Kaz Kylheku <609-57...@kylheku.com> wrote:
>>> ...
>>>> A mixup between awk string literals and shell string literals would
>>>> require a depth of stupidity rivaling your own.
>>>
>>> Ed makes no claims to being even close to being the brightest bulb in
>>> the pack.
>>
>> Ah, that likely points to excessively high resistance of the filament,
>> Watts being V^2/R.
>
> Another valuable contribution to the community from the guy who chose to be
> named after a South Park character and a typically topical/useful response from
> the guy who answers most shell/awk questions with a suggestion to use his own
> wacky home-grown tool with its user community of 1 instead.

Oh I see, so *that's* where I mixed up awk and shell literals.
I'm wiping the egg from my face now. How can I ever live this down ...

[Kenny McCormack]

In article <20160125...@kylheku.com>,

Kaz Kylheku <609-57...@kylheku.com> wrote:
...

>Oh I see, so *that's* where I mixed up awk and shell literals.
>I'm wiping the egg from my face now. How can I ever live this down ...

I don't think I have any problem picking sides in this fight.

I doubt anyone else does or will either.

--
"I have a simple philosophy. Fill what's empty. Empty what's full. And
scratch where it itches."

Alice Roosevelt Longworth

[Kaz Kylheku]

On 2016-01-25, Kenny McCormack <gaz...@shell.xmission.com> wrote:

> In article <20160125...@kylheku.com>,
> Kaz Kylheku <609-57...@kylheku.com> wrote:
> ...
>>Oh I see, so *that's* where I mixed up awk and shell literals.
>>I'm wiping the egg from my face now. How can I ever live this down ...
>
> I don't think I have any problem picking sides in this fight.

What makes him so sure you're named after Southpark character?

That one is spelled McCormick.

If we do a Google "Kenneth McCormack", there are plenty of hits.

One people search site finds over 40 records across the US alone.

At the University of British Columbia -- in the CS department! --
there was a student called Bill Gates who went on to become a staff
member. You'd be walking through the hallway, and come across
a "Bill Gates" office door. It's just an extremely common given name,
paired with a not uncommon surname.

"Now, the fact of it is I just ain't gunna read any paper written by a
feller who names himself after the Microsoft man, or attend any of his
lectures, so help me God!"