How do you insert a newline using AWK?

[Chris Roberts]

Using AWK, I am unable to insert a newline "\n"
Nothing I try seems to work here.
Not sure why it makes it all into a single line like this.
Does anyone know of a simple way to fix this?
And/Or an explanation?

#ls -l
total 276
-rwxrwxrwx 1 crzzy1 crzzy1 211456 Sep 1 2020 NppToolBucket.dll
-rwxrwxrwx 1 crzzy1 crzzy1 212 Sep 3 2020 backup
-rwxrwxrwx 1 crzzy1 crzzy1 0 May 18 15:03 badip
-rwxrwxrwx 1 crzzy1 crzzy1 1399 May 11 11:53 ccc
-rwxrwxrwx 1 crzzy1 crzzy1 1622 Nov 2 2020 ccclab
-rwxrwxrwx 1 crzzy1 crzzy1 2217 May 7 13:09 cccp

#a=`ls -l | awk '{print $8"\n"}'
#echo $a
2020 2020 15:03 11:53 2020 13:09


Thanks
crzzy1

[Lew Pitcher]

On Tue, 18 May 2021 12:33:27 -0700, Chris Roberts wrote:

> Using AWK, I am unable to insert a newline "\n"
> Nothing I try seems to work here.
> Not sure why it makes it all into a single line like this.
> Does anyone know of a simple way to fix this?
> And/Or an explanation?
>
> #ls -l
> total 276
> -rwxrwxrwx 1 crzzy1 crzzy1 211456 Sep 1 2020 NppToolBucket.dll
> -rwxrwxrwx 1 crzzy1 crzzy1 212 Sep 3 2020 backup
> -rwxrwxrwx 1 crzzy1 crzzy1 0 May 18 15:03 badip
> -rwxrwxrwx 1 crzzy1 crzzy1 1399 May 11 11:53 ccc
> -rwxrwxrwx 1 crzzy1 crzzy1 1622 Nov 2 2020 ccclab
> -rwxrwxrwx 1 crzzy1 crzzy1 2217 May 7 13:09 cccp
>
> #a=`ls -l | awk '{print $8"\n"}'

Here's a hint: try testing in parts

1) You know what output
ls -l
will give you (see above), so find out what output

ls -l | awk '{print $8"\n"}'

will give you. I bet that it /won't/ be the output
that you expect.

2) try to find out what the backtick operator does
to output. You know that
ls -l
outputs lines, so what does
a=`ls -l`
echo $a
give you? Hint: bash(1) "Command Substitution"

> #echo $a
> 2020 2020 15:03 11:53 2020 13:09

That result is not unexpected, given your use of backticks and echo

>
> Thanks
> crzzy1

HTH
--
Lew Pitcher
"In Skills, We Trust"

[Helmut Waitzmann]

Chris Roberts <thec...@gmail.com>:

Try

echo "$a"

rather than

echo $a

>Not sure why it makes it all into a single line like this.
>

The unquoted shell parameter (here: variable) expansion is the
culprit. 

>Does anyone know of a simple way to fix this?
>

Use quoted parameter expansion. 


>And/Or an explanation?
>

Unquoted parameter expansions suffer from field splitting
(<https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_05>). 

As a rule of thumb:  In the shell's command line, always use quoted
rather than unquoted parameter expansions (unless you want those
strange effects with unquoted parameter expansions to happen). 

[Janis Papanagnou]

On 18.05.2021 21:33, Chris Roberts wrote:
> Using AWK, I am unable to insert a newline "\n"
> Nothing I try seems to work here.
> Not sure why it makes it all into a single line like this.
> Does anyone know of a simple way to fix this?
> And/Or an explanation?

The command substitution `...` or the equivalent but better $(...)
does that; it connects all the parts in one line, separated by the
field separator, and leading/trailing whitespace truncated.

I suppose you don't want another newline (awk's print does that
already for you) but want to have these data unmodified, one entry
per line. If so you can do that by disabling the IFS variable for
the command

IFS= a=`ls -l | awk '{print $8}'`

IFS= a=$(ls -l | awk '{print $8}')

To suppress the empty line from the ls header do

IFS= a=$(ls -l | awk 'NR>1 {print $8}')


Janis

[Helmut Waitzmann]

Janis Papanagnou <janis_pa...@hotmail.com>:

>On 18.05.2021 21:33, Chris Roberts wrote:

>> Using AWK, I am unable to insert a newline "\n"
>> Nothing I try seems to work here.
>> Not sure why it makes it all into a single line like this.
>> Does anyone know of a simple way to fix this?
>> And/Or an explanation?
>
>The command substitution `...` or the equivalent but better $(...)
>does that; it connects all the parts in one line, separated by the
>field separator, and leading/trailing whitespace truncated.

I don't think so.  Try


( a=$( printf '%s\n' 1 2 3 ) && echo "$a" )

>I suppose you don't want another newline (awk's print does that
>already for you) but want to have these data unmodified, one entry
>per line. If so you can do that by disabling the IFS variable for
>the command

I don't think so either.  Try


( IFS= a=$( printf '%s\n' 1 2 3 ) && echo $a )

[Janis Papanagnou]

On 19.05.2021 11:48, Helmut Waitzmann wrote:
> Janis Papanagnou <janis_pa...@hotmail.com>:
>> On 18.05.2021 21:33, Chris Roberts wrote:
>
>>> Using AWK, I am unable to insert a newline "\n"
>>> Nothing I try seems to work here.
>>> Not sure why it makes it all into a single line like this.
>>> Does anyone know of a simple way to fix this?
>>> And/Or an explanation?
>>
>> The command substitution `...` or the equivalent but better $(...)
>> does that; it connects all the parts in one line, separated by the
>> field separator, and leading/trailing whitespace truncated.
>
>
> I don't think so. Try
>
> ( a=$( printf '%s\n' 1 2 3 ) && echo "$a" )

You are right. The assignment is an internal command and needs
no quoting. The effect is visible with an unquoted variable or
if the command is unquoted like echo $( printf '%s\n' 1 2 3 )
where a quoted command echo "$( printf '%s\n' 1 2 3 )" works
fine as well.

Setting IFS= also works fine it seems...

>> I suppose you don't want another newline (awk's print does that
>> already for you) but want to have these data unmodified, one entry per
>> line. If so you can do that by disabling the IFS variable for the command
>
>
> I don't think so either. Try
>
> ( IFS= a=$( printf '%s\n' 1 2 3 ) && echo $a )

$ ( a=$( printf '%s\n' 1 2 3 ) && echo $a )
1 2 3

$ ( IFS= a=$( printf '%s\n' 1 2 3 ) && echo $a )
1
2
3

Please explain your thought.

Janis

[Barry Margolin]

In article <s82jpo$d7u$1...@news-1.m-online.net>,

Janis Papanagnou <janis_pa...@hotmail.com> wrote:

> On 18.05.2021 21:33, Chris Roberts wrote:
> > Using AWK, I am unable to insert a newline "\n"
> > Nothing I try seems to work here.
> > Not sure why it makes it all into a single line like this.
> > Does anyone know of a simple way to fix this?
> > And/Or an explanation?
>
> The command substitution `...` or the equivalent but better $(...)
> does that; it connects all the parts in one line, separated by the
> field separator, and leading/trailing whitespace truncated.

You're right that it's command substitution doing it, but you have some
details wrong.

It just removes any trailing newlines. Not leading whitespace, and not
any other trailing whitespace.

However,

echo $a

performs word splitting, so all leading and trailing whitespaces are
discarded, and embedded sequences of whitespace are merged into a single
word delimiter. This is why variables should generally be quoted:

echo "$a"

--
Barry Margolin, bar...@alum.mit.edu
Arlington, MA
*** PLEASE post questions in newsgroups, not directly to me ***

[Helmut Waitzmann]

Janis Papanagnou <janis_pa...@hotmail.com>:

>On 19.05.2021 11:48, Helmut Waitzmann wrote:
>> Janis Papanagnou <janis_pa...@hotmail.com>:

>>> The command substitution `...` or the equivalent but better $(...)
>>> does that; it connects all the parts in one line, separated by the
>>> field separator, and leading/trailing whitespace truncated.
>>
>>
>> I don't think so. Try
>>
>>
>> ( a=$( printf '%s\n' 1 2 3 ) && echo "$a" )
>
>You are right. The assignment is an internal command and needs
>no quoting. The effect is visible with an unquoted variable or
>if the command is unquoted like echo $( printf '%s\n' 1 2 3 )
>where a quoted command echo "$( printf '%s\n' 1 2 3 )" works
>fine as well.
>
>Setting IFS= also works fine it seems...
>
>
>>> I suppose you don't want another newline (awk's print does that
>>> already for you) but want to have these data unmodified, one
>>> entry per line. If so you can do that by disabling the IFS
>>> variable for the command
>>
>>
>> I don't think so either. Try
>>
>>
>> ( IFS= a=$( printf '%s\n' 1 2 3 ) && echo $a )
>
>
>$ ( a=$( printf '%s\n' 1 2 3 ) && echo $a )
>1 2 3
>
>$ ( IFS= a=$( printf '%s\n' 1 2 3 ) && echo $a )
>1
>2
>3
>
>Please explain your thought.
>

I'm sorry, because I showed the wrong command.  I meant to write


(
set_or_unset_var()
{
# Set or unset a variable the name of which is given as
# the first positional parameter. The second parameter
# if given will be used to set the variable. If there
# is no second parameter then the variable will be unset.

: "${1:?'missing variable name'}" &&
eval "$1"'=${2-}' &&
${2+:} eval 'unset -v -- '"$1"
} &&

# Save the variable 'IFS' into the variable 'saved_IFS':
#
set_or_unset_var saved_IFS ${IFS+"${IFS}"} &&

# Then empty the variable 'IFS' and assign two and a
# half line to the variable 'a':
#
IFS= &&

a=$( printf '%s\n' 1 2 3 ) &&

# Restore the original value of the variable 'IFS'
#
set_or_unset_var IFS ${saved_IFS+"$saved_IFS"} &&

echo $a
)


It will output


1 2 3

even though the variable 'IFS' was empty at the time of the
assignment to the variable 'a'.  The expansion of the variable 'a'
will suffer from field splitting when 'IFS' has been reset.

[Ed Morton]

Copy/paste your 2-line script into http://shellcheck.net and it'll tell
you what the problems are with it.

Ed.

[Janis Papanagnou]

Isn't the shell-checker's suggestion:
"Use find instead of ls to better handle non-alphanumeric filenames."
rather strange (to say the least)?

(Shouldn't a shell-checker focus on the shell and not on used external
or compound commands? That it obviously cannot semantically grasp.)

Janis

>
> Ed.
>

[Lew Pitcher]

Bench-check the script, and /learn/ what the problems are

* the awk print prints the requisite data, plus an extra blank
line because the OP included a "/n" in the print stream. Had
the OP coded that step as
awk '{print $8}'
each datum would print on it's own line, without additional blanks

* when using the backtick operator, trailing newlines are deleted but
embedded newlines aren't (although "they may be removed during word
splitting"), so

a=`ls -l | awk '{print $8"\n"}'

fills shell variable <<a>> with the doublespaced results, minus the
trailing newline(s)

* in the naked
echo $a
the shell variable <<a>> becomes a candidate for word splitting ("The
shell scans the results of parameter expansion, command substitution,
and arithmetic expansion that did not occur within double quotes for
word splitting"). the shell uses the value of the <<IFS>> shell variable
(or space, tab, and newline, if <<IFS>> is not set) to perform splitting,
and, thus, replaces the newlines with a single whitespace value. As this
echo did not doublequote it's argument, word splitting occurred, replacing
all embedded newline, tab and space sequences with single whitespace elements.

[Ed Morton]

No, that's generally good advice, see https://mywiki.wooledge.org/ParsingLs.

The tool is just telling you about _potential_ issues, though, once you
understand the issue you can change the code as the tool suggests or add
a comment disabling the warning or just ignore it as you like.

That isn't all that shellcheck reports though.

Given the original 2-line script:

-----

a=`ls -l | awk '{print $8"\n"}'

echo $a
-----

shellcheck outputs:

-----
Line 1:

a=`ls -l | awk '{print $8"\n"}'

^-- SC1009: The mentioned syntax error was in this variable assignment.
^-- SC1073: Couldn't parse this backtick expansion. Fix to allow more
checks.

Line 2:
echo $a
^-- SC1072: Fix any mentioned problems and try again.
-----

When you fix that to get:

-----
a=`ls -l | awk '{print $8"\n"}'`
echo $a
-----

shellcheck outputs:

-----
Line 1:
a=`ls -l | awk '{print $8"\n"}'`
^-- SC2148: Tips depend on target shell and yours is unknown. Add a
shebang or a 'shell' directive.
^-- SC2006: Use $(...) notation instead of legacy backticked `...`.
^-- SC2012: Use find instead of ls to better handle non-alphanumeric
filenames.

Did you mean: (apply this, apply all SC2006)
a=$(ls -l | awk '{print $8"\n"}')

Line 2:
echo $a
^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
echo "$a"
-----

When you add a shebang to fix the first message:

-----
#!/usr/bin/env bash
a=`ls -l | awk '{print $8"\n"}'`
echo $a
-----

shellcheck outputs:

-----
Line 2:
a=`ls -l | awk '{print $8"\n"}'`
^-- SC2006: Use $(...) notation instead of legacy backticked `...`.
^-- SC2012: Use find instead of ls to better handle non-alphanumeric
filenames.

Did you mean: (apply this, apply all SC2006)
a=$(ls -l | awk '{print $8"\n"}')

Line 3:
echo $a
^-- SC2086: Double quote to prevent globbing and word splitting.

Did you mean: (apply this, apply all SC2086)
echo "$a"
-----

Note that at this point shellcheck is telling you of 3 issues in the
script and of course that last one is the key to the problem the OP is
specifically asking about.

Ed.

[Ed Morton]

One of the nice things about running shellcheck is it gives you a link
to click on where you can find out more about each issue so not only do
we not have to identify the issues in the OPs code, we don't have to
explain them either. Just run the tool.... Not saying it'll
uncover/explain everything but it goes a long way for most common issues
and then we can help with any issues remaining after the code has been
cleaned up to the point of making shellcheck happy.

Ed.

[Spiros Bousbouras]

On Thu, 20 May 2021 11:34:49 -0500
Ed Morton <morto...@gmail.com> wrote:
> On 5/20/2021 10:44 AM, Janis Papanagnou wrote:
> > On 20.05.2021 17:10, Ed Morton wrote:

[...]

> >> Copy/paste your 2-line script into http://shellcheck.net and it'll tell
> >> you what the problems are with it.

[...]

> The tool is just telling you about _potential_ issues, though, once you
> understand the issue you can change the code as the tool suggests or add
> a comment disabling the warning or just ignore it as you like.

Pretty much anything can count as a potential issue. To some people using the
shell at all as opposed to a "proper" language (like Perl or Python) is a
potential issue.

> That isn't all that shellcheck reports though.
>
> Given the original 2-line script:
>
> -----
> a=`ls -l | awk '{print $8"\n"}'
> echo $a
> -----
>
> shellcheck outputs:
>
> -----
> Line 1:
> a=`ls -l | awk '{print $8"\n"}'
> ^-- SC1009: The mentioned syntax error was in this variable assignment.
> ^-- SC1073: Couldn't parse this backtick expansion. Fix to allow more
> checks.

This is useful. But an even better advice is that people asking programming
questions should paste in their post the code which caused the question
instead of retyping it. I'm guessing that the opening poster didn't do this.

> Line 2:
> echo $a
> ^-- SC1072: Fix any mentioned problems and try again.
> -----
>
> When you fix that to get:
>
> -----
> a=`ls -l | awk '{print $8"\n"}'`
> echo $a
> -----
>
> shellcheck outputs:
>
> -----
> Line 1:
> a=`ls -l | awk '{print $8"\n"}'`
> ^-- SC2148: Tips depend on target shell and yours is unknown. Add a
> shebang or a 'shell' directive.
> ^-- SC2006: Use $(...) notation instead of legacy backticked `...`.
> ^-- SC2012: Use find instead of ls to better handle non-alphanumeric
> filenames.

Both subjective preferences. In particular , "legacy" is a loaded term with
a negative connotation. If the code had $( ... ) one could equally well
say "Use backticks for wider portability" ! "Wider portability" is also
a loaded term but with a positive connotation.

By my count the site tells you about

1. A syntax error
2. Use find instead of ls
3. Use $( ... ) instead of ` ... `
4. Double quotes prevent globbing.
5. Double quotes prevent word splitting

It also wants to know what shell the user is using which is irrelevant. 2 and
3 are subjective (and irrelevant) , 1 and 4 are correct but irrelevant and
only 5 is key to the confusion of the opening poster. So I don't think that
shellcheck.net gave a good signal to noise ratio.

--
Any sufficiently advanced bug is indistinguishable from a feature.
Rich Kulawiec

[Spiros Bousbouras]

On Thu, 20 May 2021 11:40:03 -0500
Ed Morton <morto...@gmail.com> wrote:

> One of the nice things about running shellcheck is it gives you a link
> to click on where you can find out more about each issue so not only do
> we not have to identify the issues in the OPs code, we don't have to
> explain them either.

Noone has to identify or explain anything or indeed even read anything. Perhaps
people *enjoy* identifying the issues and explaining concepts.

> Just run the tool.... Not saying it'll
> uncover/explain everything but it goes a long way for most common issues
> and then we can help with any issues remaining after the code has been
> cleaned up to the point of making shellcheck happy.

If http://shellcheck.net provided a way to only flag objective issues (like
syntax errors) as opposed to subjective preferences of the author of the site
then it might have been useful on some occasions but , from the example you
posted in <s8633a$bqp$1...@dont-email.me> , it seems to me just as likely to
send the user on a wild goose chase (or several).

--
Statistics are like a bikini. What they reveal is suggestive, but
what they conceal is vital.
Aaron Levenstein

[Kenny McCormack]

In article <yTadvYnm...@bongo-ra.co>,
Spiros Bousbouras <spi...@gmail.com> wrote:
...

>Pretty much anything can count as a potential issue. To some people using the

>shell at all as opposed to a "proper" language (like C or assembler) is a
>potential issue.

FTFY

--
The scent of awk programmers is a lot more attractive to women than
the scent of perl programmers.

(Mike Brennan, quoted in the "GAWK" manual)

[Ed Morton]

On 5/20/2021 3:51 PM, Spiros Bousbouras wrote:
> On Thu, 20 May 2021 11:34:49 -0500
> Ed Morton <morto...@gmail.com> wrote:
>> On 5/20/2021 10:44 AM, Janis Papanagnou wrote:
>>> On 20.05.2021 17:10, Ed Morton wrote:
>
> [...]
>
>>>> Copy/paste your 2-line script into http://shellcheck.net and it'll tell
>>>> you what the problems are with it.
>
> [...]
>
>> The tool is just telling you about _potential_ issues, though, once you
>> understand the issue you can change the code as the tool suggests or add
>> a comment disabling the warning or just ignore it as you like.
>
> Pretty much anything can count as a potential issue.

shellcheck does a good job at identifying the issues you really should
think about. It's a great resource, especially for beginners.

<snip>

>
> By my count the site tells you about
>
> 1. A syntax error
> 2. Use find instead of ls
> 3. Use $( ... ) instead of ` ... `
> 4. Double quotes prevent globbing.
> 5. Double quotes prevent word splitting
>
> It also wants to know what shell the user is using which is irrelevant. 2 and
> 3 are subjective (and irrelevant) , 1 and 4 are correct but irrelevant and
> only 5 is key to the confusion of the opening poster. So I don't think that
> shellcheck.net gave a good signal to noise ratio.

I assume by "irrelevant" you mean "aren't causing this specific
problem". I strongly disagree that that categorization makes them
irrelevant - everything on that list regularly causes issues in scripts,
especially for beginners, and should be addressed in anyone's code
before they ask anyone else to help them debug it so we can rule out
those common issues as being the source of the current problem and not
waste time explaining them all for the OPs future, if not current,
benefit when the tool can do it.

Ed.

[Janis Papanagnou]

I consider this spellcheck's message bad advice because it's extremely
misleading. It is speaking of "filenames" problems where here only the
date field is extracted. There's a lot better advice (IMO) to point out
that you might provide ls with options to show date in a standardized
(non-varying) format or to avoid ls and awk to use stat directly on the
files or whatever. What I mean is; a "shell" checker tool shall not make
(most likely wrong or misleading) suggestions about the whole semantical
context, but it shall rather focus on the _shell code_ issues. But don't
get me wrong; it's generally nice to have such a tool.

> The tool is just telling you about _potential_ issues, though, once you
> understand the issue you can change the code as the tool suggests or add
> a comment disabling the warning or just ignore it as you like.

The approach of potential issues, I think, is doomed to fail if I see
suggestions like the one I commented on.

Janis

[Ed Morton]

I'm struggling to accept that you seem to be arguing that "don't parse
the output of ls" is bad advice. It's one of the most common mistakes
that newcomers make. I'd be very disappointed in any kind of shell
script checking tool that DIDN'T produce a warning when it encounters
code that's doing it. I do think that a link to
https://mywiki.wooledge.org/ParsingLs or some other lengthier
description of all problems with parsing `ls` would be an improvement in
the full error description that shellcheck provides but I'm just happy
it provides something/anything to give the user a heads up that there's
an issue with that code.

Ed.

[Janis Papanagnou]

On 21.05.2021 14:37, Ed Morton wrote:
>
> I'm struggling to accept that you seem to be arguing that "don't parse
> the output of ls" is bad advice.

I think (and said) that this advice is misleading here, and that the
tool would serve better - specifically for "newcomers" as you write -
to abstain from giving something like "rule of thumbs for newbies" in
contexts where it doesn't apply, and that the tool should focus on
the primary domain it covers (the shell). I want to add that I might
come to another conclusion if the tool would be able to distinguish
unsafe from save usages; but the tool is (in this respect) obviously
far too primitive. It's just reacting (more or less) on keywords in
simple code pattern contexts (here a pipe). In other syntax contexts
the tool is much more sophisticated; e.g. it's *not* suggesting to
"quote every variable expansion" (as wiseacres often do) - the tool
knows where it's necessary and where not.

Janis

[Kenny McCormack]

In article <s89cc8$bmj$1...@news-1.m-online.net>,
Janis Papanagnou <janis_pa...@hotmail.com> wrote:
...

>In other syntax contexts
>the tool is much more sophisticated; e.g. it's *not* suggesting to
>"quote every variable expansion" (as wiseacres often do) - the tool
>knows where it's necessary and where not.

Really? I thought it *did* complain about any unquoted variables
references.

It also complains about things like:

gawk '{ print $1 }'

because it thinks you need to use double quotes in order for variable
expansion to occur. I.e., it doesn't understand that $1 is an AWK
variable, not a shell variable.

It seems to me that it could be smarter about "command line programs" in
languages such as AWK.

--
It's possible that leasing office space to a Starbucks is a greater liability
in today's GOP than is hitting your mother on the head with a hammer.

[Janis Papanagnou]

On 22.05.2021 01:36, Kenny McCormack wrote:
> In article <s89cc8$bmj$1...@news-1.m-online.net>,
> Janis Papanagnou <janis_pa...@hotmail.com> wrote:
> ...
>> In other syntax contexts
>> the tool is much more sophisticated; e.g. it's *not* suggesting to
>> "quote every variable expansion" (as wiseacres often do) - the tool
>> knows where it's necessary and where not.
>
> Really? I thought it *did* complain about any unquoted variables
> references.

It recognizes that

b="1 2" ; case $b in ... esac

and

b=1 ; echo $b

are both okay, but that

b="1 2" ; echo $b

isn't. - Quite sophisticated, I'd say.

OTOH, it doesn't detect that this works correctly

b="1 2" ; IFS= ; echo $b

> It also complains about things like:
>
> gawk '{ print $1 }'
>
> because it thinks you need to use double quotes in order for variable
> expansion to occur. I.e., it doesn't understand that $1 is an AWK
> variable, not a shell variable.

Hmm, okay. - I'd have expected that such embedded scripts would be
properly analyzed (especially if the given single-quoting prevents
any variable expansion inside; it could be considered a black-box).

But to be honest, I have just tried a hand full of code patterns to
see how clever or stupid it is. I suppose there are a couple things
where we'd be surprised (false positives and false negatives, so to
say).

>
> It seems to me that it could be smarter about "command line programs" in
> languages such as AWK.

Indeed.

Janis

[Ed Morton]

On 5/21/2021 7:20 PM, Janis Papanagnou wrote:
> On 22.05.2021 01:36, Kenny McCormack wrote:

<snip>

>> It also complains about things like:
>>
>> gawk '{ print $1 }'

Not it doesn't.

#!/usr/bin/env bash
gawk '{ print $1 }'

$ shellcheck myscript
No issues detected!

Ed.

[Janis Papanagnou]

On 22.05.2021 02:20, Janis Papanagnou wrote:
> On 22.05.2021 01:36, Kenny McCormack wrote:
>> In article <s89cc8$bmj$1...@news-1.m-online.net>,
>> Janis Papanagnou <janis_pa...@hotmail.com> wrote:
>> ...
>>> In other syntax contexts
>>> the tool is much more sophisticated; e.g. it's *not* suggesting to
>>> "quote every variable expansion" (as wiseacres often do) - the tool
>>> knows where it's necessary and where not.
>>
>> Really? I thought it *did* complain about any unquoted variables
>> references.
>
> It recognizes that
>
> b="1 2" ; case $b in ... esac
>
> and
>
> b=1 ; echo $b
>
> are both okay, but that
>
> b="1 2" ; echo $b
>
> isn't. - Quite sophisticated, I'd say.
>
> OTOH, it doesn't detect that this works correctly
>
> b="1 2" ; IFS= ; echo $b
>

The first impression often has to be corrected after having a closer
look into the issue. From the next two sample codes the first one is
considered okay, but not the second one.

if [ "$1" -eq 0 ]
then b="1 2"
else b=1
fi
echo $b

if [ "$1" -ne 0 ]
then b=1
else b="1 2"
fi
echo $b

We can see that the tool uses only a primitive heuristic analysis.

Janis

[Janis Papanagnou]

On 22.05.2021 07:59, Ed Morton wrote:
> On 5/21/2021 7:20 PM, Janis Papanagnou wrote:
>> On 22.05.2021 01:36, Kenny McCormack wrote:
> <snip>
>>> It also complains about things like:
>>>
>>> gawk '{ print $1 }'
>
> Not it doesn't.

I haven't verified Kenny's example, but I've just tried the statement

gawk "{ print $1 }"

which also gives no warning.

Janis

[Janis Papanagnou]

On 20.05.2021 17:44, Janis Papanagnou wrote:
> On 20.05.2021 17:10, Ed Morton wrote:
>> On 5/18/2021 2:33 PM, Chris Roberts wrote:
>>> Using AWK, I am unable to insert a newline "\n"
>>> Nothing I try seems to work here.
>>> Not sure why it makes it all into a single line like this.
>>> Does anyone know of a simple way to fix this?
>>> And/Or an explanation?
>>>
>>> #ls -l
>>> total 276
>>> -rwxrwxrwx 1 crzzy1 crzzy1 211456 Sep 1 2020 NppToolBucket.dll
>>> -rwxrwxrwx 1 crzzy1 crzzy1 212 Sep 3 2020 backup
>>> -rwxrwxrwx 1 crzzy1 crzzy1 0 May 18 15:03 badip
>>> -rwxrwxrwx 1 crzzy1 crzzy1 1399 May 11 11:53 ccc
>>> -rwxrwxrwx 1 crzzy1 crzzy1 1622 Nov 2 2020 ccclab
>>> -rwxrwxrwx 1 crzzy1 crzzy1 2217 May 7 13:09 cccp
>>>
>>> #a=`ls -l | awk '{print $8"\n"}'
>>> #echo $a
>>> 2020 2020 15:03 11:53 2020 13:09
>>
>> Copy/paste your 2-line script into http://shellcheck.net and it'll tell
>> you what the problems are with it.
>
> Isn't the shell-checker's suggestion:
> "Use find instead of ls to better handle non-alphanumeric filenames."
> rather strange (to say the least)?

Another interesting test case is replacing the 'ls' by a 'ps' command;
the tool doesn't see any problems with parsing 'ps' output.

But suggesting 'find', in the first place, is already quite arguable;
find produces other output. You have to define e.g. -maxdepth, then
the list of files needs additional post-processing, one might want to
get rid of the '.' directory, the spurious './' file path prefixes
needs handling, and last but not least is 'ls' output sorted, but not
'find's, and the shell coder (novice or else) might be just wondering.

[Kenny McCormack]

In article <s8ac15$kfg$1...@news-1.m-online.net>,
Janis Papanagnou <janis_pa...@hotmail.com> wrote:

>On 22.05.2021 07:59, some weirdo wrote:
>> On 5/21/2021 7:20 PM, Janis Papanagnou wrote:
>>> On 22.05.2021 01:36, Kenny McCormack wrote:
>> <snip>
>>>> It also complains about things like:
>>>>
>>>> gawk '{ print $1 }'
>>
>> Not it doesn't.
>
>I haven't verified Kenny's example, but I've just tried the statement
>
> gawk "{ print $1 }"
>
>which also gives no warning.

It turns out that shellcheck does, indeed, "know" about "awk" (and "gawk")
as being "special". So, Janis is correct - that invoked as "gawk", it
doesn't complain. However, as it happens, in my coding environment,
whenever I invoke AWK, it is as "gawk4" and, not surprisingly, shellcheck
doesn't recognize that as being a flavor of AWK.

Which is a long-winded way of saying that if you try the above example
using, say, "gawk4", instead of just plain "gawk", it (shellcheck) *will*
flag it.

So, what it boils down to is: Should a tool like shellcheck "know" about
the semantics of the things we generally call "shell tools" (things like
AWK, ls, find, Perl, etc) ? On the one hand, aesthetically, the answer to
this should be "no", because it just feels like a "slippery slope". This
seems to be Janis's position (his original objection to the tool telling
you to use "find" instead of "ls"). Intuitively, the tool *should* be
agnostic w.r.t. choice of tools. I think we can all agree on this -
intuitively.

However, on the other hand, we can all recognize that the usefulness of the
tool is greatly diminished if it doesn't have such knowledge. This seems
to be the position held by that other poster. And, in fact, this is the
position I originally espoused, w.r.t. AWK - that shellcheck *should*
understand about AWK command line scripts and realize that they are
generally single (not double) quoted for a good reason.

Which it, in fact, does. Provided you invoke AWK via a recognized command
name (a set of which "gawk4" is, alas, not a member).

P.S. All comments by in regards to the shellcheck tool are based on my
experience with the normal, downloadable version - which I have used on
some of my own scripts. I have no knowledge or experience with any
"online" version. I assume, however, that they are basically the same.

--
Men rarely (if ever) manage to dream up a God superior to themselves.
Most Gods have the manners and morals of a spoiled child.