How to process duplicate entries in tab separated file?

[James Egan]

In the __DATA__ below, there are two duplicate entries which occur twice,
07020000279 and 05020004293, which are customer numbers. What I need to
do is find any duplicate customer numbers in a file, and print (write)
them.

-Thanks

#!/usr/bin/perl

use strict;
use warnings;

use vars qw(
@file
@fields
$fields
$line
$ctr
$custno
@customers
);

while ( <DATA> ) {
push(@file, $_);
}

my @matches = grep $line eq '07020000279', @file;

foreach $line (@file) {

print $line;

}

__DATA__
07020000279 Joe Smith 54 Abbey Road
05020033486 John Jones 98 New York Ave.
07020000279 George Washington 234 Washington Ave.
06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road
0302004472 Fred Jones 98 New York Ave.

[Ben Morrow]

Quoth James Egan <jega...@comcast.net>:

> In the __DATA__ below, there are two duplicate entries which occur twice,
> 07020000279 and 05020004293, which are customer numbers. What I need to
> do is find any duplicate customer numbers in a file, and print (write)
> them.

perldoc -q duplicate. The answer doesn't apply as it stands, but
modifying the concept to record duplicates instead of discarding them is
not difficult.

Ben

[ccc31807]

On Nov 1, 8:30 pm, James Egan <jegan...@comcast.net> wrote:
> In the __DATA__ below, there are two duplicate entries which occur twice,
> 07020000279 and 05020004293, which are customer numbers.  What I need to
> do is find any duplicate customer numbers in a file, and print (write)
> them.

Use a hash to store your data, with the customer number as the key to
the hash. The keys of a hash are guaranteed to be unique, therefore
you will never have a duplicate entry.

Then, when you are initializing the hash from the data, test each hash
key to see if it has already been initialized, and if it has, write
the record to an error file. Something like this (untested)

while (<DATA>)
{
chomp;
my ($id, $fn, $ln, $addy) = split;
print "$id is duplicate: $fn $ln, $addy\n" if $customers{$id};
$customers{$id} = "$fn, $ln, $addy" unless $customer{$key};
}

[Tad McClellan]

James Egan <jega...@comcast.net> wrote:

[snip]
> use strict;
[snip]
> use vars qw(
> @file

"use vars" is an anachronism. You should avoid its use with modern perls.

perldoc vars

... the functionality provided
by this pragma has been superseded by "our" declarations

But you do not appear to actually need package (global) variables, so
you should be using lexical variables instead:

my( @file ...

--
Tad McClellan
email: perl -le "print scalar reverse qq/moc.noitatibaher\100cmdat/"

[Ben Morrow]

Quoth Tad McClellan <ta...@seesig.invalid>:

> James Egan <jega...@comcast.net> wrote:
>
> [snip]
> > use strict;
> [snip]
> > use vars qw(
> > @file
>
>
> "use vars" is an anachronism. You should avoid its use with modern perls.
>
> perldoc vars
>
> ... the functionality provided
> by this pragma has been superseded by "our" declarations

It is worth being aware that the two are subtly different. This will
pass strict checks:

package A;
our $x;

package B;
print $x;

whereas this will not:

package A;
use vars qw/$x/;

package B;
print $x;

and in the first case the variable printed is $A::x rather than $B::x.

Ben

[Viking]

Don't use global variables when lexical variables would do.

#!/usr/bin/perl

use strict;
use warnings;

local $\ = qq{\n}; # auto-append newlines to every print

my %h;
while (<DATA>) {
chomp; next unless /\S/;
my $custno = (split)[0];
print $custno if exists $h{$custno};
$h{$custno}++;
}

[James Egan]

On Tue, 03 Nov 2009 08:31:40 -0800, Viking wrote:

Using your solution below, the only thing that is output is:

07020000279

What I'm trying to do is print each record, BUT if there are two or three
records with the same customer number, print those out consecutively.

#!/usr/bin/perl

use strict;
use warnings;

use vars qw(
@file
@fields
$fields
$line
$ctr
$custno
@customers
);

local $\ = qq{\n};

my %h;

while (<DATA>) {
chomp; next unless /\S/;
my $custno = (split)[0];
print $custno if exists $h{$custno};
$h{$custno}++;
}

[James Egan]

What I'm trying to do is IF there are duplicate customer numbers, then
print them consecutively. So the __DATA__ below printed out would look
like this, with duplicate customers printed together, and unique
customers printed as they are like:

07020000279 Joe Smith 54 Abbey Road

07020000279 George Washington 234 Washington Ave.

05020033486 John Jones 98 New York Ave.

06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road

03020004472 Fred Jones 98 New York Ave.

__DATA__
07020000279 Joe Smith 54 Abbey Road
05020033486 John Jones 98 New York Ave.
07020000279 George Washington 234 Washington Ave.
06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road

03020004472 Fred Jones 98 New York Ave.

[Ben Morrow]

Quoth James Egan <jega...@comcast.net>:

> On Mon, 02 Nov 2009 05:10:18 -0800, ccc31807 wrote:
>
> > On Nov 1, 8:30 pm, James Egan <jegan...@comcast.net> wrote:
> >> In the __DATA__ below, there are two duplicate entries which occur
> >> twice, 07020000279 and 05020004293, which are customer numbers.  What I
> >> need to do is find any duplicate customer numbers in a file, and print
> >> (write) them.
> >
> > Use a hash to store your data, with the customer number as the key to
> > the hash. The keys of a hash are guaranteed to be unique, therefore you
> > will never have a duplicate entry.
> >
> > Then, when you are initializing the hash from the data, test each hash
> > key to see if it has already been initialized, and if it has, write the
> > record to an error file. Something like this (untested)
> >
> > while (<DATA>)
> > {
> > chomp;
> > my ($id, $fn, $ln, $addy) = split;
> > print "$id is duplicate: $fn $ln, $addy\n" if $customers{$id};
> > $customers{$id} = "$fn, $ln, $addy" unless $customer{$key};
> > }
>
>
> What I'm trying to do is IF there are duplicate customer numbers, then
> print them consecutively. So the __DATA__ below printed out would look
> like this, with duplicate customers printed together, and unique
> customers printed as they are like:

So, what does your code currently look like, and where are you having
difficulties?

Ben

[ccc31807]

On Nov 3, 7:08 pm, James Egan <jegan...@comcast.net> wrote:
> What I'm trying to do is IF there are duplicate customer numbers, then
> print them consecutively.  So the __DATA__ below printed out would look
> like this, with duplicate customers printed together, and unique
> customers printed as they are like:

Then each hash key would contain a reference to an anonymous array.
You still get the keys as unique IDs, but the value, instead of being
a scalar, would be a reference to an anonymous array, which you would
push like this:
push @{$customers{$key}}, "data";

You would print them like this:

foreach my $key (sort keys %customers)
{
#print each unique identifier
print "$key\n";
#then print the values for that key
foreach my $ele (@{$customers{$key}})
{
print " -- $ele\n";
}
}

This really isn't difficult, but simply a data management problem. An
author named Cross (last name) wrote a book about ten years ago about
munging data with Perl. It's an old book, but cheap now, and worth the
$5.00 (or so) you will pay on half.com or amazon.com, and will
certainly introduce you to some techniques of doing stuff like this.

CC.

[Tad McClellan]

James Egan <jega...@comcast.net> wrote:

> use vars qw(

use vars is still an anachronism...

and global variables are still bad...

[James Egan]

On Wed, 04 Nov 2009 00:28:45 +0000, Ben Morrow wrote:

Here's some of the things I've tried suggested in this group:

#!/usr/bin/perl

use strict;
use warnings;

use vars qw(
@file
@fields
$fields
$line
$ctr
$custno
@customers

$customers
%customer
%customers
$key
);

while (<DATA>) {
chomp;
my ($id, $fn, $ln, $addy) = split;

$customers{$id} = "$fn, $ln, $addy" unless $customers{$key};

print "$id is duplicate: $fn $ln, $addy\n" if $customers{$id};
}

#local $\ = qq{\n};
#my %h;

#while (<DATA>) {
# chomp; next unless /\S/;
# my $custno = (split)[0];
# print $custno if exists $h{$custno};
# $h{$custno}++;
#}

__DATA__
07020000279 Joe Smith 54 Abbey Road
05020033486 John Jones 98 New York Ave.
07020000279 George Washington 234 Washington Ave.
06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road

0302004472 Fred Jones 98 New York Ave.

[John W. Krahn]

James Egan wrote:
> On Wed, 04 Nov 2009 00:28:45 +0000, Ben Morrow wrote:
>
>
> Here's some of the things I've tried suggested in this group:
>
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
>
> use vars qw(
> @file
> @fields
> $fields
> $line
> $ctr
> $custno
> @customers
> $customers
> %customer
> %customers
> $key
> );
>
>
> while (<DATA>) {
> chomp;
> my ($id, $fn, $ln, $addy) = split;

The address field contains whitespace so you are losing some data:

my ($id, $fn, $ln, $addy) = split ' ', $_, 4;

> $customers{$id} = "$fn, $ln, $addy" unless $customers{$key};

$key is always false so the test is superfluous:

$customers{$id} = "$fn, $ln, $addy";

You are overwriting the previous value of $customers{$id} every time $id
is a duplicate so you are losing some data.

> print "$id is duplicate: $fn $ln, $addy\n" if $customers{$id};

$customers{$id} is always true so the test is superfluous:

print "$id is duplicate: $fn $ln, $addy\n";

> }
>
>
> #local $\ = qq{\n};
> #my %h;
>
> #while (<DATA>) {
> # chomp; next unless /\S/;
> # my $custno = (split)[0];
> # print $custno if exists $h{$custno};
> # $h{$custno}++;
> #}
>
>
> __DATA__
> 07020000279 Joe Smith 54 Abbey Road
> 05020033486 John Jones 98 New York Ave.
> 07020000279 George Washington 234 Washington Ave.
> 06020004293 Fred Flintstone 123 Main St.
> 05020004293 Wilma Flintstone Bedrock Road
> 0302004472 Fred Jones 98 New York Ave.

John
--
The programmer is fighting against the two most
destructive forces in the universe: entropy and
human stupidity. -- Damian Conway

[shar...@hotmail.com]

On Nov 4, 8:52 am, James Egan <jegan...@comcast.net> wrote:
> On Wed, 04 Nov 2009 00:28:45 +0000, Ben Morrow wrote:
>
> Here's some of the things I've tried suggested in this group:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> use vars qw(
>   @file
>   @fields
>   $fields
>   $line
>   $ctr
>   $custno
>   @customers
>   $customers
>   %customer
>   %customers
>   $key
> );
>

You've not removed the use vars construct as was strongly suggested in
this group.
There really is no need to go for global variables in this case.

What you can try is this code (untested though!):

#!/usr/bin/perl
use strict;
use warnings;

local $\ = qq{\n};

my %h;
while (<DATA>) {
chomp; next unless /\S/;

my $custno = (split)[0];

if (!exists $h{$custno}) {
push @{$h{$custno}}, $., $_;
}
else {
push @{$h{$custno}}, $_;
}
}

print for
map { $_->[1] }
sort { $a->[0] <=> $b->[0] }
map { my @A=@{$h{$_}}; [shift(@A), join("\n",@A)] }
keys %h;

--Rakesh

[s...@netherlands.com]

On Mon, 02 Nov 2009 01:30:23 GMT, James Egan <jega...@comcast.net> wrote:

>In the __DATA__ below, there are two duplicate entries which occur twice,
>07020000279 and 05020004293, which are customer numbers. What I need to
>do is find any duplicate customer numbers in a file, and print (write)
>them.
>
>-Thanks
>

Late to the thread. No use going over all the pro's and cons.

I would say that your natural delimeter is tab, not space.
It is unnecessary to split out first/last name's, and its useless
to think that it can be predictable.

Validation, where is the validation and defaults?
Split is far too over-used in this group. It implies a perfect
world.

Add to the list one more method ..

-sln
---------

use strict;
use warnings;

open DATA, '<id.txt' or die "can't open id data file: $!";

my %customers;
my %dups;
my $file_pos = tell(DATA);

# Validate form, store position info
while (<DATA>) {
if (my ($id) = /^ ([\w]+) \t?[\w ]* \t?([\w ]*) /x)
{
if ( exists($customers{$id}) )
{ push @{$dups{$id}}, $file_pos; }
else
{ $customers{$id} = $file_pos; }
}
$file_pos = tell(DATA);
}

# Display multiple id data
for my $id (sort keys %dups)
{
print "\nMult_Id: $id\n";
for my $position ($customers{$id}, @{$dups{$id}}) {
seek (DATA, $position, 0);
$_ = <DATA>;
my ($name, $addy) = /^ [\w]+ \t?([\w ]*) \t?([\w ]*) /x;
print "\t'$name', '$addy'\n";
}
}

close DATA;

__END__

id.txt:

07020000279 Joe Smith 54 Abbey Road
05020033486 John Jones 98 New York Ave.
07020000279 George Washington 234 Washington Ave.
06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road
0302004472 Fred Jones 98 New York Ave.

05020004293 Wilma Flintstone Bedrock Road

05020004293 Bedrock Road

05020004293 Wilma Flintstone Bedrock Road

05020004293
05020004293 Bedrock Road

Output:

Mult_Id: 05020004293
'Wilma Flintstone', 'Bedrock Road'
'Wilma Flintstone', 'Bedrock Road'
'', 'Bedrock Road'
'Wilma Flintstone', 'Bedrock Road'
'', ''
'Bedrock Road', ''

Mult_Id: 07020000279
'Joe Smith', '54 Abbey Road'
'George Washington', '234 Washington Ave'

[James Egan]

On Wed, 04 Nov 2009 12:05:13 -0800, sln wrote:
> id.txt:
>
> 07020000279 Joe Smith 54 Abbey Road
> 05020033486 John Jones 98 New York Ave. 07020000279 George
Washington
> 234 Washington Ave. 06020004293 Fred Flintstone 123 Main St.
05020004293
> Wilma Flintstone Bedrock Road 0302004472 Fred Jones 98 New
York Ave.
>
> 05020004293 Wilma Flintstone Bedrock Road
05020004293 Bedrock Road
> 05020004293 Wilma Flintstone Bedrock Road 05020004293
> 05020004293 Bedrock Road
>
> Output:
>
> Mult_Id: 05020004293
> 'Wilma Flintstone', 'Bedrock Road'
> 'Wilma Flintstone', 'Bedrock Road'
> '', 'Bedrock Road'
> 'Wilma Flintstone', 'Bedrock Road'
> '', ''
> 'Bedrock Road', ''
>
> Mult_Id: 07020000279
> 'Joe Smith', '54 Abbey Road'
> 'George Washington', '234 Washington Ave'

If I change id.txt so it has two sets of two duplicate customer numbers
like:

07020000279 Joe Smith 54 Abbey Road
05020033486 John Jones 98 New York Ave.
07020000279 George Washington 234 Washington Ave.
06020004293 Fred Flintstone 123 Main St.
05020004293 Wilma Flintstone Bedrock Road

03020004472 Fred Jones 98 New York Ave.

It's only finding one of the dups, and the output is:

Mult_Id: 07020000279
'Joe Smith', '54 Abbey Road'
'George Washington', '234 Washington Ave'

Close though!

-Thanks

[James Egan]

On Wed, 04 Nov 2009 11:44:03 -0800, sharma__r wrote:
>
> print for
> map { $_->[1] }
> sort { $a->[0] <=> $b->[0] }
> map { my @A=@{$h{$_}}; [shift(@A), join("\n",@A)] } keys %h;
>
> --Rakesh

So far this is the only code that actually processes the records with
duplicate customer numbers together. But now I can't seem to format the
output. I was using the "write" function to output07020000279 formatted
data, but would be happy if I could print formatted output like:

Customer ID: 07020000279
Name: Joe Smith
Address: 54 Abbey Road

Is this possible?

-Thanks

[Jim Gibson]

In article <DuoIm.123944$Xw3....@en-nntp-04.dc1.easynews.com>, James
Egan <jega...@comcast.net> wrote:

Only if you can reliably identify the name and address fields. Your
sample data contains only two-word names. Can you also have one word
names ("Cher") or three-word names ("John C. Reilly")? Can you arrange
to put a unique delimiter character between your fields? If so, then
printing the data as shown above is straight-forward: just use the
print or printf statements.

--
Jim Gibson

[Ben Morrow]

Quoth James Egan <jega...@comcast.net>:

> On Wed, 04 Nov 2009 11:44:03 -0800, sharma__r wrote:
> >
> > print for
> > map { $_->[1] }
> > sort { $a->[0] <=> $b->[0] }
> > map { my @A=@{$h{$_}}; [shift(@A), join("\n",@A)] } keys %h;
>

> So far this is the only code that actually processes the records with
> duplicate customer numbers together.

So far you have not shown us any code you have written yourself. If you
want someone to write the code for you, you will need to hire a
programmer.

>But now I can't seem to format the
> output. I was using the "write" function to output07020000279 formatted
> data, but would be happy if I could print formatted output like:
>
> Customer ID: 07020000279
> Name: Joe Smith
> Address: 54 Abbey Road

perldoc -f print
The section in perlop that deals with double-quoted strings.

Ben

[s...@netherlands.com]

On Wed, 04 Nov 2009 23:49:42 GMT, James Egan <jega...@comcast.net> wrote:

>
>If I change id.txt so it has two sets of two duplicate customer numbers
>like:
>
>07020000279 Joe Smith 54 Abbey Road
>05020033486 John Jones 98 New York Ave.
>07020000279 George Washington 234 Washington Ave.
>06020004293 Fred Flintstone 123 Main St.
>05020004293 Wilma Flintstone Bedrock Road
>03020004472 Fred Jones 98 New York Ave.
>
>It's only finding one of the dups, and the output is:
>
>Mult_Id: 07020000279
> 'Joe Smith', '54 Abbey Road'
> 'George Washington', '234 Washington Ave'
>
>
>Close though!
>
>-Thanks

Thats the problem, its not written to be 'close'.
Hashes don't lie, eyesight sometimes does.

07020000279 is the only duplicate in your data shown above.

These:

06020004293 Fred Flintstone 123 Main St.

^

05020004293 Wilma Flintstone Bedrock Road

are not duplicates!

-sln