Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Remove the repeated words but the first appearance of the word from a file.

1 view
Skip to first unread message

Hongyi Zhao

unread,
Nov 20, 2009, 5:33:07 AM11/20/09
to
Hi all,

I want to remove all of the follow-up repeated words but the first
appearance of the word from a file, i.e., for the following minimal
example,

bla1 bla2 bla3 bla2 bla4
bla5 bla6 bla5

We should obtain the following result:

bla1 bla2 bla3 bla4
bla5 bla6

Any hints on this issue?

Best regards.
--
.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

pk

unread,
Nov 20, 2009, 6:17:10 AM11/20/09
to
Hongyi Zhao wrote:

> Hi all,
>
> I want to remove all of the follow-up repeated words but the first
> appearance of the word from a file, i.e., for the following minimal
> example,
>
> bla1 bla2 bla3 bla2 bla4
> bla5 bla6 bla5
>
> We should obtain the following result:
>
> bla1 bla2 bla3 bla4
> bla5 bla6

Assuming "words" are space-separated (which might or might not be the case),
you can try

awk '{
t=s=""
for(i=1;i<=NF;i++){
if(!($i in a)){
a[$i]
t=t s $i
s=" "
}
}
if(t"")print t
}'

this seems to work with your sample; may not work with your real data.

Ed Morton

unread,
Nov 20, 2009, 8:10:05 AM11/20/09
to
Hongyi Zhao wrote:
> Hi all,
>
> I want to remove all of the follow-up repeated words but the first
> appearance of the word from a file, i.e., for the following minimal
> example,
>
> bla1 bla2 bla3 bla2 bla4
> bla5 bla6 bla5
>
> We should obtain the following result:
>
> bla1 bla2 bla3 bla4
> bla5 bla6
>
> Any hints on this issue?
>
> Best regards.

awk '{s=""; for (i=1;i<=NF;i++) if (!a[$i]++) {printf "%s%s",s,$i; s=OFS} print
""}' file

Ed.

Hongyi Zhao

unread,
Nov 20, 2009, 8:53:26 AM11/20/09
to
On Fri, 20 Nov 2009 07:10:05 -0600, Ed Morton <morto...@gmail.com>
wrote:

>awk '{s=""; for (i=1;i<=NF;i++) if (!a[$i]++) {printf "%s%s",s,$i; s=OFS} print
>""}' file

A more complicated case is: the file include non-ascii multibyte
characters, e.g., Chinese. In this case, these words are not
separated by space, in fact, the Chinese characters are directly
adjacent to each other when typesetting. The above awk code will fail
for dealling with these multibyte characters.

John W. Krahn

unread,
Nov 20, 2009, 10:43:43 AM11/20/09
to
Hongyi Zhao wrote:
>
> I want to remove all of the follow-up repeated words but the first
> appearance of the word from a file, i.e., for the following minimal
> example,
>
> bla1 bla2 bla3 bla2 bla4
> bla5 bla6 bla5
>
> We should obtain the following result:
>
> bla1 bla2 bla3 bla4
> bla5 bla6
>
> Any hints on this issue?

$ echo "bla1 bla2 bla3 bla2 bla4
bla5 bla6 bla5
" | perl -pe's/(\S+)/$x{$1}++?"":$1/eg'


bla1 bla2 bla3 bla4
bla5 bla6

John
--
The programmer is fighting against the two most
destructive forces in the universe: entropy and
human stupidity. -- Damian Conway

0 new messages