sed multiple -e pattern match problem

[chengiz]

There's a file called file.txt with one line:
foo.c bar.h baz.h

On running the following:
sed -e 's/ /\n/g' -e '/\.h/!d' file.txt
(ie. replace spaces by newlines, then remove all lines that do not
contain .h)

I expected the output to be:
bar.h
baz.h

But I get
foo.c
bar.h
baz.h

If I pipe to another sed:
sed -e 's/ /\n/g' file.txt | sed -e '/\.h/!d'
it works.

Seems like the second -e in the first version is still working on the
original line. Is it possible to do what I want in the same sed
instance? Thanks.

[Ben Bacarisse]

chengiz <che...@my-deja.com> writes:

> There's a file called file.txt with one line:
> foo.c bar.h baz.h
>
> On running the following:
> sed -e 's/ /\n/g' -e '/\.h/!d' file.txt
> (ie. replace spaces by newlines, then remove all lines that do not
> contain .h)
>
> I expected the output to be:
> bar.h
> baz.h
>
> But I get
> foo.c
> bar.h
> baz.h
>
> If I pipe to another sed:
> sed -e 's/ /\n/g' file.txt | sed -e '/\.h/!d'
> it works.
>
> Seems like the second -e in the first version is still working on the
> original line.

Yes, that's essentially how sed works.

> Is it possible to do what I want in the same sed
> instance? Thanks.

Well, you could delete the non .h file names:

sed -e 's/ *[^ ]*\.[^h] *//g' -e 's/ */\n/g'

(and I am sure that could be simpler, but it's always dodgy simplifying
when the problem is not 100% understood).

--
Ben.

[Dave Gibson]

chengiz <che...@my-deja.com> wrote:
> There's a file called file.txt with one line:
> foo.c bar.h baz.h
>
> On running the following:
> sed -e 's/ /\n/g' -e '/\.h/!d' file.txt
> (ie. replace spaces by newlines, then remove all lines that do not
> contain .h)
>
> I expected the output to be:
> bar.h
> baz.h
>
> But I get
> foo.c
> bar.h
> baz.h
>
> If I pipe to another sed:
> sed -e 's/ /\n/g' file.txt | sed -e '/\.h/!d'
> it works.
>
> Seems like the second -e in the first version is still working on the
> original line.

Once a line is loaded into the pattern space the edit commands operate
on that space -- embedded newlines and all.

> Is it possible to do what I want in the same sed
> instance? Thanks.

sed -e 's/ /\n/g ; :a ; h ; s/\n.*// ; /\.h$/p ; g ; D ; t a' file.txt

s/ /\n/g # replace spaces with newlines
:a # label "a" -- a branch target
h # copy pattern space to hold space
s/\n.*// # delete from first embedded newline to end of pattern space
/\.h$/p # print if pattern space ends in ".h"
g # copy hold space to pattern space
D # delete up to first embedded newline in pattern space
t a # branch to label "a" if most recent s/// succeeded

[Ed Morton]

chengiz <che...@my-deja.com> wrote:

> There's a file called file.txt with one line:
> foo.c bar.h baz.h
>
> On running the following:
> sed -e 's/ /\n/g' -e '/\.h/!d' file.txt
> (ie. replace spaces by newlines, then remove all lines that do not
> contain .h)

sed is an excellent tool for simple substitutions on a single line. For
anything else, use awk:

$ cat file
foo.c bar.h baz.h

$ awk -v RS=" " '/\.h/' file
bar.h
baz.h

Regards,

Ed.

Posted using www.webuse.net

[Lem Novantotto]

Ed Morton ha scritto:

> For anything else, use awk:

So "when the going gets awk, the awk's (awk, nawk, gawk) get going" eh? ;)

Yep, I know I should learn it some day. :-p
--
Bye, Lem
Ceterum censeo ISLAM esse delendum
_________________________________________________________________
Non sprecare i cicli idle della tua CPU, né quelli della tua GPU.
http://www.worldcommunitygrid.org/index.jsp
http://www.rnaworld.de/rnaworld/ http://home.edges-grid.eu/home/
http://www.gpugrid.net/

[Ed Morton]

Lem Novantotto <Le...@Hotmail.com> wrote:

> Ed Morton ha scritto:
>
> > For anything else, use awk:
>
> So "when the going gets awk, the awk's (awk, nawk, gawk) get going" eh? ;)
>
> Yep, I know I should learn it some day. :-p

Yeah, it took me some convincing as I'd seen some horrible messes written in
old, broken awk (/usr/bin/awk on Solaris) and could never get the darn thing
to just BEHAVE in any sensible way but once I discovered the newer awks
(nawk, gawk, /usr/xpg4/bin/awk, etc.) the clouds parted and harps started
playing....

I've been using sed for 30 years and still use it fairly frequently, but I
think the only commands I use in it are "s" and "g". Once you start using
any other sed commands, awk's just simpler to write and more robust and even
for the simple stuff it's often more robust, e.g.:

$ echo "abc-def" | sed 's/-/\n/'
abcndef

$ echo "abc-def" | awk 'sub(/-/,"\n")'
abc
def

$ x="foo
> bar"

$ echo "abc-def" | sed "s/-/$x/"
sed: -e expression #1, char 7: Unterminated `s' command
$ echo "abc-def" | sed 's/-/'"$x"'/'
sed: -e expression #1, char 7: Unterminated `s' command

$ echo "abc-def" | awk -v x="$x" 'sub(/-/,x)'
abcfoo
bardef

I think there's some combination of quotes that'll let sed understand the
embedded newline but it's just not worth trying to remember it.

[Lem Novantotto]

Ed Morton ha scritto: [...]

Just for the sake of my curiosity:

> $ echo "abc-def" | sed 's/-/\n/'
> abcndef

On my system (linux with bash) this one works as expected:

$ echo "abc-def" | sed 's/-/\n/'
abc

def

However I've been told that, with sed, if you want to search a newline
'\n' is good, but if you want to substitute a newline, you'd better use
'\[ENTER]'.
Maybe this works on your system too:

$ echo "abc-def" | sed 's/-/\

> /'

abc
def

> $ x="foo
>> bar"
>
> $ echo "abc-def" | sed "s/-/$x/"
> sed: -e expression #1, char 7: Unterminated `s' command
> $ echo "abc-def" | sed 's/-/'"$x"'/'
> sed: -e expression #1, char 7: Unterminated `s' command

I've been able to make it work, on my system, only using '\n':

$ x="foo\nbar"

$ echo "abc-def" | sed "s/-/$x/"

abcfoo
bardef

which, by the way, seems in contradiction with what I've just written
about substitutions - and shouldn't work on your system.
I don't think there's a way to do it quoting the sed command: to
succesfully use a real newline, you should work in single quotes, but
single quotes prevent variable expansion, so...

You're right: surely awk behaviour looks more straightforward. :)

[Ed Morton]

Lem Novantotto <Le...@Hotmail.com> wrote:

> Ed Morton ha scritto: [...]
>
> Just for the sake of my curiosity:
>
> > $ echo "abc-def" | sed 's/-/\n/'
> > abcndef
>
> On my system (linux with bash) this one works as expected:
>
> $ echo "abc-def" | sed 's/-/\n/'
> abc
> def

Yeah, I think it depends on which sed you have. The above worked when I tried it
with GNU sed.

> However I've been told that, with sed, if you want to search a newline
> '\n' is good,

I don't think you can [easily] search for a newline with sed since sed is line
oriented. For example if I want to change the string:

def
ghi

to the string

foo

then I'd just tell awk not to treat newlines as the end of a record and then use
the sub() command just like for any other substitution:

$ cat file
abc
def
ghi
klm

$ awk -v RS="" 'sub(/def\nghi/,"foo")' file
abc
foo
klm

but how would you do that in sed?

> but if you want to substitute a newline, you'd better use '\[ENTER]'.

Yes, that's what I hear too.

<snip>

> You're right: surely awk behaviour looks more straightforward. :)

Testify! :-).

Ed.


Posted using www.webuse.net

[Bill Marcum]

On 09/14/2012 01:58 PM, Ed Morton wrote:
> $ echo "abc-def" | sed "s/-/$x/"
> sed: -e expression #1, char 7: Unterminated `s' command
> $ echo "abc-def" | sed 's/-/'"$x"'/'
> sed: -e expression #1, char 7: Unterminated `s' command
>
> $ echo "abc-def" | awk -v x="$x" 'sub(/-/,x)'
> abcfoo
> bardef
>
> I think there's some combination of quotes that'll let sed understand the
> embedded newline but it's just not worth trying to remember it.
>

I think the variable would need to contain a backslash, for example:
x='\
'

[Lem Novantotto]

Ed Morton ha scritto:

> $ awk -v RS="" 'sub(/def\nghi/,"foo")' file
>abc
>foo
>klm
>
> but how would you do that in sed?

Well, in a spot like this one you could use something like:

$ sed '/^def$/{N;{/^def\nghi$/s/def\nghi/foo/}}' file

or, as we've just said:

$ sed '/^def$/{N;{/^def\nghi$/s/def\
> ghi/foo/}}' file

Ok, it's a mess, I admit. But since I'm far away from being a sed guru,
at least it probably could be better built. :)

[Lem Novantotto]

Bill Marcum ha scritto:

> I think the variable would need to contain a backslash, for example:
> x='\
> '

Well put!

$ x='foo\
> bar'

$ echo "abc-def" | sed "s/-/$x/"

abcfoo
bardef

[Geoff Clare]

Ed Morton wrote:

> $ cat file
> abc
> def
> ghi
> klm
>
> $ awk -v RS="" 'sub(/def\nghi/,"foo")' file
> abc
> foo
> klm
>
> but how would you do that in sed?

sed '$!N;s/def\nghi/foo/;P;D'

--
Geoff Clare <net...@gclare.org.uk>

[Lem Novantotto]

Geoff Clare ha scritto:

> sed '$!N;s/def\nghi/foo/;P;D'

QED. Brilliant. :)

Could you explain why did you use that "$!"?

Doesn't

sed 'N;s/def\nghi/foo/;P;D' file

work properly? TIA.

[pk]

On 17 Sep 2012 13:11:03 GMT, Lem Novantotto <Le...@Hotmail.com> wrote:

> Geoff Clare ha scritto:
>
> > sed '$!N;s/def\nghi/foo/;P;D'
>
> QED. Brilliant. :)
>
> Could you explain why did you use that "$!"?
>
> Doesn't
>
> sed 'N;s/def\nghi/foo/;P;D' file
>
> work properly? TIA.

"If no next line of input is available, the N command verb shall branch to
the end of the script and quit without starting a new cycle or copying the
pattern space to standard output."

[Ed Morton]

Lem Novantotto <Le...@Hotmail.com> wrote:

> Geoff Clare ha scritto:
>
> > sed '$!N;s/def\nghi/foo/;P;D'
>
> QED.

Yes, that exactly demonstrates my point. Compare the cryptic combination of
single characters, punctuation marks, etc. above to simply:

awk -v RS="" 'sub(/def\nghi/,"foo")' file

As I said, sed is an excellent tool for simple substitutions on a single line
but for anything else, just use awk (or perl or...).

Regards,

Ed,

Posted using www.webuse.net

[Lem Novantotto]

pk ha scritto:

> "If no next line of input is available, the N command verb shall branch
> to the end of the script and quit without starting a new cycle or
> copying the pattern space to standard output."

This would have been relevant if we had run sed with "-n" option.
But we didn't.

See:

$ sed -n 'N;s/^def\nghi$/foo/;P;D' file # no good
abc
foo

$ sed -n '$!N;s/^def\nghi$/foo/;P;D' file # good
abc
foo
klm

$ sed 'N;s/^def\nghi$/foo/;P;D' file # good
abc
foo
klm

$ sed '$!N;s/^def\nghi$/foo/;P;D' file # good
abc
foo
klm

[pk]

On 17 Sep 2012 14:29:15 GMT, Lem Novantotto <Le...@Hotmail.com> wrote:

> pk ha scritto:
>
> > "If no next line of input is available, the N command verb shall branch
> > to the end of the script and quit without starting a new cycle or
> > copying the pattern space to standard output."
>
> This would have been relevant if we had run sed with "-n" option.

Read it again.

[Lem Novantotto]

Ed Morton ha scritto:

> As I said, sed is an excellent tool for simple substitutions on a single
> line but for anything else, just use awk (or perl or..

Ed, I agree with you. I'm not joking. :)
I was just saying that Geoff's sed solution is better that mine.

I can't be 100% by your side just because I really don't know awk. I know
awk can do really complicated things, and I've had some examples, from
time to time, in this newsgroup too.

All this is just fun for me: so sometimes I try to go with sed, which I
know a bit, even if I strongly suspect awk would be probably better.

[Lem Novantotto]

pk ha scritto:

> Read it again.

Could you please try yourself?

[pk]

On 17 Sep 2012 14:45:34 GMT, Lem Novantotto <Le...@Hotmail.com> wrote:

> pk ha scritto:
>
> > Read it again.
>
> Could you please try yourself?

Try with non-gnu sed.

[pk]

or even gnu sed with --posix

[Lem Novantotto]

pk ha scritto:

>> Try with non-gnu sed.
>
> or even gnu sed with --posix

Thanks. :)

[Janis Papanagnou]

On 17.09.2012 16:44, Lem Novantotto wrote:
> Ed Morton ha scritto:
>
>> As I said, sed is an excellent tool for simple substitutions on a single
>> line but for anything else, just use awk (or perl or..
>
> Ed, I agree with you. I'm not joking. :)
> I was just saying that Geoff's sed solution is better that mine.
>
> I can't be 100% by your side just because I really don't know awk. I know
> awk can do really complicated things, and I've had some examples, from
> time to time, in this newsgroup too.

The point with awk is not so much that you can do complicated things,
rather that with sed code gets complicated quickly if you want to do
something non-trivial.

Janis