Ignoring comments and blank lines in a data file
Mark Hobley
blank lines as follows:
# This is a comment ignore this line
# Ignore the blank line below
# The following network address is commented out and should be ignored
# 79.34.64.0/18
102.32.192.0/18
109.6.0.0/16 # This is a comment and should be ignored
85.23.76.19
I plan to read these files using a loop in a shell script as follows:
cat list1 list2 list3 |
while read netaddr
do
# I need to strip out comments and spaces here
if "$netaddr" ne ""; then
firewallout $netaddr
fi
done
What is the simplest way to cut out the hash symbols and any text
beyond? I am thinking of using some sort of arrangement using cut as
follows:
cut -f 1 -d '#'
This would strip the comments, but I need to also get rid of blank lines
and spaces.
Please advise.
Mark.
--
Mark Hobley,
393 Quinton Road West,
Quinton, BIRMINGHAM.
B32 1QE.
Dan Stromberg
I often use this when I don't need line numbers from the input file, and
want to remove comments and blank lines:
sed -e 's/#.*$//' -e '/^[[:space:]]*$/d' < inputfile > outputfile
If you need line numbers, you could probably still use this in
combination with an egrep -n '.|^$'
Bit Twister
> blank lines as follows:
>
> # This is a comment ignore this line
> # Ignore the blank line below
>
> 25.0.0.0/8
> 51.0.0.0/8
Then there is the case of a line with spaces or tabs but no netaddr. :(
> I plan to read these files using a loop in a shell script as follows:
Might be instructive for you to do a
man test
and for some light reading, http://tldp.org/LDP/abs/html/index.html
cat list1 list2 list3 |
while read line
do
set -- $line
netaddr=$1
if [ -n "$netaddr" ] ; then
if [ "${netaddr:0:1}" != "#" ] ; then
firewallout $netaddr
fi
fi
done
Glenn Jackman
> I have a set of data files containing a list of network addresses and
> blank lines as follows:
>
> I plan to read these files using a loop in a shell script as follows:
>
> cat list1 list2 list3 |
> while read netaddr
awk '1 {sub(/#.*/,"")} /^ *$/ {next} 1' list1 list2 list3 |
while read netaddr
# ...
--
Glenn Jackman
Write a wise saying and your name will live forever. -- Anonymous
pk
> I have a set of data files containing a list of network addresses and
> blank lines as follows:
>
> # This is a comment ignore this line
> # Ignore the blank line below
>
> 25.0.0.0/8
> 51.0.0.0/8
>
> # The following network address is commented out and should be ignored
> # 79.34.64.0/18
> 102.32.192.0/18
> 109.6.0.0/16 # This is a comment and should be ignored
> 85.23.76.19
>
> I plan to read these files using a loop in a shell script as follows:
>
> cat list1 list2 list3 |
> while read netaddr
> do
> # I need to strip out comments and spaces here
> if "$netaddr" ne ""; then
> firewallout $netaddr
> fi
> done
>
> What is the simplest way to cut out the hash symbols and any text
> beyond?
You can filter out lines that do NOT contain an uncommented IP address:
sed '/^\([0-9]\{1,3\}\.\)\{3\}[0-9]\{1,3\}/!d' file | \
while read -r netaddr dummy; do
...
done
assuming noncommented IP address are the first thing in a line. Of course,
you can improve the regexp to match more closely what you want.
--
All the commands are tested with bash and GNU tools, so they may use
nonstandard features. I try to mention when something is nonstandard (if
I'm aware of that), but I may miss something. Corrections are welcome.
Dave B
> I have a set of data files containing a list of network addresses and
> blank lines as follows:
> What is the simplest way to cut out the hash symbols and any text
> beyond? I am thinking of using some sort of arrangement using cut as
> follows:
>
> cut -f 1 -d '#'
>
> This would strip the comments, but I need to also get rid of blank lines
> and spaces.
You can filter the input through sed:
sed -n 's/#.*//;/^[[:space:]]*$/!p' file | while IFS= read -r netaddr; do ....
Note that your IP addresses use different formats, ie some have the /nn
(prefix length) and some don't.
--
echo 0|sed 's909=oO#3u)o19;s0#0ooo)].O0;s()(0bu}=(;s#}#.1m"?0^2{#;
s)")9v2@3%"9$);so%op]t(p$e#!o;sz(z^+.z;su+ur!z"au;sxzxd?_{h)cx;:b;
s/\(\(.\).\)\(\(..\)*\)\(\(.\).\)\(\(..\)*#.*\6.*\2.*\)/\5\3\1\7/;
tb'|awk '{while((i+=2)<=length($1)-18)a=a substr($1,i,1);print a}'
Dave B
> sed -n 's/#.*//;/^[[:space:]]*$/!p' file | while IFS= read -r netaddr; do ....
Correction: you should also remove spaces before comments, just in case, ie
sed -n 's/[[:space:]]*#.*//;/^[[:space:]]*$/!p' file ...
mop2
Just using shell (bash) resources:
cat list1 list2 list3 |while read netaddr
do netaddr=${netaddr%%\#*};netaddr=${netaddr// /}
[ "$netaddr" ]&&firewallout $netaddr
done
Chris F.A. Johnson
> I have a set of data files containing a list of network addresses and
> blank lines as follows:
>
> # This is a comment ignore this line
> # Ignore the blank line below
>
> 25.0.0.0/8
> 51.0.0.0/8
>
> # The following network address is commented out and should be ignored
> # 79.34.64.0/18
> 102.32.192.0/18
> 109.6.0.0/16 # This is a comment and should be ignored
> 85.23.76.19
>
> I plan to read these files using a loop in a shell script as follows:
>
> cat list1 list2 list3 |
> while read netaddr
> do
> # I need to strip out comments and spaces here
> if "$netaddr" ne ""; then
> firewallout $netaddr
> fi
> done
cat list[123] |
while read netaddr junk
do
case $netaddr in
\#* | "" ) ;;
*) firewallout "$netaddr";;
esac
done
--
Chris F.A. Johnson, author <http://cfaj.freeshell.org/shell/>
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence
shargaur
================================================================================================
Easy Way
cat list1 list2 list3 | grep -v
'#' # Strip out the comment lines
here
while read netaddr
do
# strip out spaces here
Chris F.A. Johnson
> Easy Way
Wrong way.
> cat list1 list2 list3 | grep -v '#' # Strip out the comment lines here
That also strips out the lines with comments after the netaddr.
Also, you left out a pipe.
> while read netaddr
> do
> # strip out spaces here
> if "$netaddr" ne ' ' ; then
Syntax error and incorrect operator. That should be:
if [ "$netaddr" != ' ' ] ; then
Or should it be:
if [ "$netaddr" != '' ] ; then
> firewallout $netaddr
> fi
> done
Michael Paoli
Hobley) wrote:
> I have a set of data files containing a list of network addresses and
> blank lines as follows:
>
> # This is a comment ignore this line
> # Ignore the blank line below
>
> cat list1 list2 list3 |
> while read netaddr
> do
> # I need to strip out comments and spaces here
> What is the simplest way to cut out the hash symbols and any text
> beyond? I am thinking of using some sort of arrangement using cut as
> follows:
> cut -f 1 -d '#'
> This would strip the comments, but I need to also get rid of blank lines
> and spaces.
> Please advise.
cat datafile(s) |
grep '^[ ]*[^ #]' |
... (process the data)
The whitespace within each [] above is one space character and one tab
character.
The grep then just matches (filters to) stuff that starts with zero or
more blanks and/or
tabs followed by something other than blank, tab, or # on that same
line.
If it doesn't match that, grep won't pass it through.
Mark Hobley
> The whitespace within each [] above is one space character and one tab
> character.
That is interesting. Does literal character representation not work
within square brackets? for example [ \t]
Geoff Clare
> Michael Paoli <micha...@yahoo.com> wrote:
>> The whitespace within each [] above is one space character and one tab
>> character.
>
> That is interesting. Does literal character representation not work
> within square brackets? for example [ \t]
Sometimes it does, sometimes it doesn't. For POSIX conforming
utilities, the general rule is that backslash loses its special
meaning inside a bracket expression, but there are two specific
exceptions: awk and lex, which do backslash processing before
looking for bracket expressions.
Furthermore, some implementations of POSIX utilities have a
default non-conforming mode in which backslash is recognised
inside bracket expressions. One such is GNU sed:
$ printf 'a\tb\nt\n' | sed -n '/[\t]/p'
a b
$ printf 'a\tb\nt\n' | POSIXLY_CORRECT=1 sed -n '/[\t]/p'
t
--
Geoff Clare <net...@gclare.org.uk>
shar...@gmail.com
i just provided the logic...he can make it better
hpt
> On Thu, 17 Jul 2008 00:59:07 -0300, Mark Hobley <markhob...@hotpop.donottypethisbit.com> wrote:
> > I have a set of data files containing a list of network addresses and
> > blank lines as follows:
>
> > # Ignore the blank line below
>
> > 25.0.0.0/8
> > 51.0.0.0/8
>
> > # The following network address is commented out and should be ignored
> > # 79.34.64.0/18
> > 102.32.192.0/18
> > 85.23.76.19
>
> > I plan to read these files using a loop in a shell script as follows:
>
> > cat list1 list2 list3 |
> > while read netaddr
> > do
> > # I need to strip out comments and spaces here
> > if "$netaddr" ne ""; then
> > firewallout $netaddr
> > fi
> > done
>
> > What is the simplest way to cut out the hash symbols and any text
> > beyond? I am thinking of using some sort of arrangement using cut as
> > follows:
>
> > cut -f 1 -d '#'
>
> > This would strip the comments, but I need to also get rid of blank lines
> > and spaces.
>
> > Please advise.
>
> > Mark.
>
> Just using shell (bash) resources:
>
> cat list1 list2 list3 |while read netaddr
> do netaddr=${netaddr%%\#*};netaddr=${netaddr// /}
> [ "$netaddr" ]&&firewallout $netaddr
> done
It seems this is the better method I think. And It seems that the
"IFS" has big effect on this problem ... The default IFS is "\n\t ".
So the "netaddr=${netaddr// /}" seems doesn't be needed...
William James
Removes comments and ignores blank lines, including
those containing spaces or tabs.
awk '{sub(/[ \t]*#.*/,"")} NF' file
hpt
> > Just using shell (bash) resources:
>
> > cat list1 list2 list3 |while read netaddr
> > do netaddr=${netaddr%%\#*};netaddr=${netaddr// /}
> > [ "$netaddr" ]&&firewallout $netaddr
> > done
>
> It seems this is the better method I think. And It seems that the
> "IFS" has big effect on this problem ... The default IFS is "\n\t ".
> So the "netaddr=${netaddr// /}" seems doesn't be needed...
No, "netaddr=${netaddr// /}" shouldn't be omitted. It is trying to
remove the space between the contents of the line and the "#". There
should be only one space. Because the "read" will reduce the space to
one base on IFS="\n\t ". If there are only spaces and tabs on a line,
it will be null after "read" reads it.
And there should be a extra line : netadd=${netaddr//<tab>/}, there
the "<tab>" is a tab character. This line be used to remove the
potential tabs between the contents and the "#".