Broken pipe error and an AWK question

krzys-iek

unread,

May 1, 2004, 3:31:55 PM5/1/04

to

Hi!

I have written this small AWK/Bash script, It was my homework, and it works.
But I wonder if it is possible to make it more simple - here I call AWK
three times...

This script reads passwd file and display only that people we want it to (by
GROUP ID).
It also sorts them - we can choose a sorting rule (by name, last name or
login).

Sample use: ./script 2000 1 - it will sort people with GID=2000 by login
names and display a neat table.

If i run ./script |grep
I get a broken pipe error :(? How can I get rid of this?

If someone could rewrite this script so AWK is called only once, I would
appreciate it :)
I am new to AWK and I couldn't do it better :|
Thank you!

First AWK outputs only lines containing our GID, second AWK outputs
login:name:surname (see the example passwd on my web page), then we sort
lines by sort app, and the last AWK, finaly we display a nice table with
wanted data. Many pipes, 3 AWK, it doesn't look great, does it? :>

----------------------

cat /etc/passwd | awk -F : '$4 ~ /'$1'/' | awk -F "[:,]" '{
printf"%s%s%s\n",$1,":",$5; }' | \
awk -F "[: ]" '{ printf"%s:%s:%s\n",$1,$2,$3}' | \
sort -b --field-separator=: --key=$2 | awk 'BEGIN { FS="[:]";
print "Displaying users with GID: '$1':";
print " ------------------------------------------------------";
printf"%s\n","| # | login | name | surname |";
print " ------------------------------------------------------";}
{printf "| %3s %2s %10s %2s %11s %2s %15s %2s \n", ile+1, "|", $1, "|", $2,
"|", $3, "|"; ile+=1;}
END {print " ------------------------------------------------------";
printf"Together we have %s users.\n\n",ile}'

------------------

if you want a sample passwd file to test script on it - here it is
www.krionix.net/awk
regards
k.p

Ed Morton

unread,

May 1, 2004, 7:28:49 PM5/1/04

to

krzys-iek wrote:
> Hi!
>
> I have written this small AWK/Bash script, It was my homework, and it works.
> But I wonder if it is possible to make it more simple - here I call AWK
> three times...
>
> This script reads passwd file and display only that people we want it to (by
> GROUP ID).
> It also sorts them - we can choose a sorting rule (by name, last name or
> login).
>
> Sample use: ./script 2000 1 - it will sort people with GID=2000 by login
> names and display a neat table.
>
> If i run ./script |grep
> I get a broken pipe error :(? How can I get rid of this?

I dont' see any reason for you to get that from the script you posted.

> If someone could rewrite this script so AWK is called only once, I would
> appreciate it :)
> I am new to AWK and I couldn't do it better :|
> Thank you!
>
> First AWK outputs only lines containing our GID, second AWK outputs
> login:name:surname (see the example passwd on my web page), then we sort
> lines by sort app, and the last AWK, finaly we display a nice table with
> wanted data. Many pipes, 3 AWK, it doesn't look great, does it? :>
>
> ----------------------
>
> cat /etc/passwd | awk -F : '$4 ~ /'$1'/'

Useless Use Of Cat (UUOC) and you're mixing shell and awk variables
dangerously:

awk -v grp="$1" -F":" '$4 ~ grp' /etc/passwd

| awk -F "[:,]" '{
> printf"%s%s%s\n",$1,":",$5; }' | \

No need for the semicolon, nor for the "\" at the end of the line.

> awk -F "[: ]" '{ printf"%s:%s:%s\n",$1,$2,$3}' | \

You could've accomplished the above with "split" on your original line,
e.g.:

awk -v grp="$1" -F":" '$4 ~ grp {
split($0,arr,"[:,]")
printf"%s:%s:%s\n",arr[1],arr[2],arr[3]}
}' /etc/passwd

> sort -b --field-separator=: --key=$2 | awk 'BEGIN { FS="[:]";

Why change now from awk -F":"? Also, no need for the trailing semicolons
after each line. You COULD call "sort" from system() and use "getline"
to read the results if you like. I don't see anything particularly
horrible about doing it the way you have, except that you now have to
separate the header stuff you want printed from everything else,
potentially forcing an extra awk invocation.

> print "Displaying users with GID: '$1':";

Use an awk variable.

Not much point using the %s in the above case:

> print " ------------------------------------------------------";}

You could have done all of the above with a cat "here document" before
calling awk, then the following line could be part of the awk script.

> {printf "| %3s %2s %10s %2s %11s %2s %15s %2s \n", ile+1, "|", $1, "|", $2,
> "|", $3, "|"; ile+=1;}

"ile" is a count of input lines - NR provides that for you.

Doing formating on the "|"s seems like overkill - just make them part of
the format string:

{printf "| %3s | %10s | %11s | %15s | \n",NR,$1,$2,$3}

> END {print " ------------------------------------------------------";
> printf"Together we have %s users.\n\n",ile}'
>
> ------------------
>
> if you want a sample passwd file to test script on it - here it is
> www.krionix.net/awk

No, I'm not going to click on some random link in a news posting. If you
want to provide a sample input file, then post it here.

So, if you included all the above, you'd have something like this:

cat <<!
Displaying users with GID: $1:
------------------------------------------------------
| # | login | name | surname |
------------------------------------------------------
!
awk -v grp="$1" -F":" '$4 ~ grp {split($0,arr,"[:,]")
printf"%s:%s:%s\n",arr[1],arr[2],arr[3]}
}' /etc/passwd |
sort -b --field-separator=: --key=$2 |
awk -F":" '{printf "| %3s | %10s | %11s | %15s | \n",NR,$1,$2,$3}
END {print "------------------------------------------------------"
printf "Together we have %s users.\n\n",NR}'
}'

which only invokes awk twice instead of 4 times (though it does
introduce a "cat"). If you just want to invoke awk once - either do the
sorting using awk arrays, or look up "system" and "getline" in the man
pages, or see if you can't sort the passwd file before calling awk.

Ed.

Patrick TJ McPhee

unread,

May 1, 2004, 10:41:02 PM5/1/04

to

In article <c70u06$b0o$1...@atlantis.news.tpi.pl>,
krzys-iek <krzys-iek@-----wp.pl> wrote:

% I have written this small AWK/Bash script, It was my homework, and it works.
% But I wonder if it is possible to make it more simple - here I call AWK
% three times...

Here are a few comments:

% ... | awk -F : '$4 ~ /'$1'/' | ...

Don't interpolate shell variables into awk programs. This would be
easier to read as

awk -F : -v gid="$1" '$4 ~ gid'

note that it's not correct. You want to use equality here rather than
a regular expression match. For instance, try your script with 1 as
the group id to search for.

Any script like this

awk 'some test' | awk '{ do_something() }'

can be rewritten to invoke awk only once

awk 'some test { do_something() }'

In this case, you're changing the field separator, but not for
any good reason. What you want to do is extract a value from $5
of a colon-delimited file, so use split() on $5.

% printf"%s%s%s\n",$1,":",$5; }' | \

This is much more confusing-looking than it needs to be. Why not

printf"%s:%s\n",$1,$5; }' | \

? The same goes for

% {printf "| %3s %2s %10s %2s %11s %2s %15s %2s \n", ile+1, "|", $1, "|", $2,
% "|", $3, "|"; ile+=1;}

wouldn't it be easier to read if it were

{printf "| %3s | %10s | %11s | %15s | \n", ile+1, $1, $2, $3; ile+=1;}

?

You could do this with one invocation of awk if you pipe the output
of sort into awk. This will sort more than you need, but you're not
dealing with a lot of data -- I mean, how many millions of users do
you have on your system? -- so it's probably a case where making the
script simpler is better than trying to make it more efficient.

--

Patrick TJ McPhee
East York Canada
pt...@interlog.com

krzys-iek

unread,

May 2, 2004, 5:58:21 AM5/2/04

to

Ed Morton <mor...@lsupcaemnt.com> napisal nam:

>
> I dont' see any reason for you to get that from the script you posted.
>

But there is a BUG, look (I use slackware 9.1):

root@linux:~/test# ./go 1000 1 | grep
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
./go: line 6: 13194 Broken pipe cat <<!

./go: line 11: 13195 Done awk -v grp="$1" -F":" '$4 ~ grp
{split($0,arr,"[:,]")
printf"%s:%s:%s\n",arr[1],arr[2],arr[3]}' passwd
13196 | sort -b --field-separator=: --key=$2
13197 Broken pipe | awk -F":" '{printf "| %3s | %10s |

%11s | %15s | \n",NR,$1,$2,$3}
END {print " ------------------------------------------------------"
printf "Together we have %s users.\n\n",NR}'

root@linux:~/test#

Strange... ;|

> Useless Use Of Cat (UUOC) and you're mixing shell and awk variables
> dangerously:
>
> awk -v grp="$1" -F":" '$4 ~ grp' /etc/passwd
>

yeah, I tried ARGV[] before but got "could not open file" error :)
Your solution is better.

> No need for the semicolon, nor for the "\" at the end of the line.
>

OK, I will save few bytes of HDD space :)

> awk -v grp="$1" -F":" '$4 ~ grp {
> split($0,arr,"[:,]")
> printf"%s:%s:%s\n",arr[1],arr[2],arr[3]}
> }' /etc/passwd
>

I see, It works but...
-----------------------------------------------

awk -v grp="$1" -F":" '$4==grp {split($0,arr,"[: ,]")
printf"%s:%s:%s\n",arr[1],arr[5],arr[6]}' passwd |

sort -b --field-separator=: --key=$2 |
awk -F":" '{printf "| %3s | %10s | %11s | %15s | \n",NR,$1,$2,$3}
END {print " ------------------------------------------------------"
printf "Together we have %s users.\n\n",NR}'

----------------------------------------------------
I have fixed few mistakes in your script me so now it works.

> want to provide a sample input file, then post it here.
>

OK, my input file, try this script with that:

kz:x:600:1000:Katarzyna Zadarnowska:/export/prac/kz:/bin/tcsh
matrix:x:608:600:Ilona Radziewicz:/export/koledzy/matrix:/bin/tcsh
mika:x:615:1000:Mika:/export/prac/mika:/bin/bash
mucha:x:603:1000:Robert Muszynski,C3 p309,,:/export/prac/mucha:/bin/tcsh
pawel:x:504:1000:Pawel Rogalinski:/export/prac/pawel:/bin/bash
psak:x:2704:600:Pawel Sakowicz:/export/koledzy/psak:/bin/bash
sq6elt:x:606:1000:Pawel Jarosz:/export/prac/sq6elt:/bin/tcsh
kreczmer:x:607:1000:Bogdan Kreczmer:/export/prac/kreczmer:/bin/bash
wheart:x:503:100:Wojciech Penar:/export/prac/wheart:/bin/bash
arent:x:609:1000::/export/prac/arent:/bin/bash
jjakubia:x:610:1000::/export/prac/jjakubia:/bin/bash
martin:x:611:1000:Marcin Smereka:/export/prac/martin:/bin/bash
witold:x:616:1000:Witold Paluszynski:/export/prac/witold:/bin/tcsh
harlan:x:617:600:Wojciech Nowak:/export/koledzy/harlan:/bin/bash
mxhubert:x:618:600:Hubert Maksymiec:/export/koledzy/mxhubert:/bin/bash
rafit:x:619:1000:Rafal Tomczak:/export/prac/rafit:/bin/bash
gdziubin:x:620:1000:Grzegorz Dziubinski:/export/prac/gdziubin:/bin/bash
darek:x:621:1000:Dariusz Caban:/export/prac/darek:/bin/bash
chorazy:x:622:1000:Artur Chorazyczewski:/export/prac/chorazy:/bin/bash
hmac:x:623:1000:Henryk Maciejewski:/export/prac/hmac:/bin/bash
kangur:x:624:1000:P Michalik:/export/prac/kangur:/bin/bash
serwik:x:625:1000:S Serwik:/export/prac/serwik:/bin/bash
tyciu:x:627:1000:Mateusz Tykierko:/export/prac/tyciu:/bin/bash
asniezyk:x:626:1000:Andrzej Sniezyk:/export/prac/asniezyk:/bin/bash
lbardzin:x:4501:3000:Lukasz
Bardzinski,100570:/export/stud/lbardzin:/bin/bash
pbednars:x:4502:3000:Pawel Bednarski,100580:/export/stud/pbednars:/bin/bash
abienkie:x:4503:3000:Anna Bienkiewicz,100594:/export/stud/abienkie:/bin/bash
pdrozdek:x:4504:3000:Pawel Drozdek,100704:/export/stud/pdrozdek:/bin/bash
tfurtak:x:4505:3000:Tomasz Furtak,100741:/export/stud/tfurtak:/bin/bash
rgrzesko:x:4506:3000:Remigiusz
Grzeskowiak,104944:/export/stud/rgrzesko:/bin/bash
jgulmant:x:4507:3000:Jacek
Gulmantowicz,100798:/export/stud/jgulmant:/bin/bash
gkosciol:x:4508:3000:Grzegorz
Kosciolek,118629:/export/stud/gkosciol:/bin/bash
dkubik:x:4509:3000:Daniel Kubik,100975:/export/stud/dkubik:/bin/bash
ikulaga:x:4510:3000:Ireneusz Kulaga,100983:/export/stud/ikulaga:/bin/bash
jlipinsk:x:4511:3000:Jerzy Lipinski,86527:/export/stud/jlipinsk:/bin/bash

And the script takes login path (home path) as the surname (but it is
empty!)
Take a look:

Displaying users with GID: 1000:

------------------------------------------------------
| # | login | name | surname |
------------------------------------------------------

Could you please fix it?

Regards
k.

krzys-iek

unread,

May 2, 2004, 6:03:30 AM5/2/04

to

Patrick TJ McPhee <pt...@interlog.com> napisal nam:

> awk -F : -v gid="$1" '$4 ~ gid'
>

true, I had problems with that :)

> note that it's not correct. You want to use equality here rather than
> a regular expression match. For instance, try your script with 1 as
> the group id to search for.

Precious comment! Thx, now I use == .

> awk 'some test { do_something() }'

OK

> wouldn't it be easier to read if it were
>
> {printf "| %3s | %10s | %11s | %15s | \n", ile+1, $1, $2, $3;
> ile+=1;}
>
> ?

It would :)

> of sort into awk. This will sort more than you need, but you're not
> dealing with a lot of data -- I mean, how many millions of users do
> you have on your system? -- so it's probably a case where making the
> script simpler is better than trying to make it more efficient.

hmm only 1000-2000 users.

thank you
regards
k.

krzys-iek

unread,

May 2, 2004, 8:05:48 AM5/2/04

to

awk -v grp="$1" -F":" '$4==grp {split($0,arr,"[:]") split(arr[5],ar,"[ ,]")
printf"%s:%s:%s\n",arr[1],ar[1],ar[2]}' passwd |

sort -b --field-separator=: --key=$2 |
awk -F":" '{printf "| %3s | %10s | %11s | %15s | \n",NR,$1,$2,$3}
END {print " ------------------------------------------------------"
printf "Together we have %s users.\n\n",NR}'

OK, I did it, I think so....
AWK is a very nice tool!

Ed Morton

unread,

May 2, 2004, 9:20:10 AM5/2/04

to

krzys-iek wrote:

I just noticed looking back at your original that you said you got the
broken pipe error when calling your script with no arguments. I suspect
that's because of how you were jumping between awk and shell to expand
"$1", but it could be because your "sort" is expecting an argument of
"$2" too. Has that problem gone away now? In any case, if your script
requires 2 arguments, you should enforce it with this at the top of the
script:

if [ $# != 2 ]
then
echo "`basename $0`: ERROR: 2 arguments expected, $# provided." >&2
exit 1
fi

You could also output a usage/help statment in that case. You could also
investigate how to use "getopts" for general argument parsing. For any
more general UNIX help, post follow-ups to comp.unix.shell.

Ed.

krzys-iek

unread,

May 2, 2004, 9:59:21 AM5/2/04

to

Ed Morton <mor...@lsupcaemnt.com> napisal nam:

> I just noticed looking back at your original that you said you got the
> broken pipe error when calling your script with no arguments.

Upsss my mistake, of course I call script with proper arguments, and I check
with bash if they are ok ( abc instead of 1 .. etc)

So broken pipe error if I call:
./script 2000 1 | grep

> You could also output a usage/help statment in that case. You could

I have it, I didn't mention it here (to make my post and trouble clear).

> also investigate how to use "getopts" for general argument parsing.
> For any
> more general UNIX help, post follow-ups to comp.unix.shell.
>

I will post there, AWK seems to work OK.
I am not a good plumber and there is a leak somewhere :>