Checking Unique Code Generated using My PHP Script

[Pavan PHP]

I have written a php program to create unique alphanumeric strings of 8 alphabets.

To test and make sure that my program creates unique values always, I have written another php program to write all these unique alphanumeric string (5 million) in a seperate line by creating a text file and it worked.

Since file size of the text file is so huge that I can not open the text file in text editor because of RAM memory limitation so I am looking for any tool to open the text file to check whether these codes are really unique or duplicate.

I request you to help me in this regard.

Regards
Pavan PHP

[Robert Gonzalez]

Is this for your own use only, to be used on your own machine in isolation?

--
 
 

--

Robert Gonzalez

   

[Pavan PHP]

Hi Robert,

This is to test the output of script as unique code.

It's ok, if there is any program which I can install to check the uniqueness of codes or it's also ok, if there is any web based tool to check this.

At this moment, I use WAMP so I can set the maximum time to execute the script as well but RAM memory is the problem.

--
 
 

[vishal bhandare]

Hi,

on linux we can easily find unique using uniq command.many more command available on linux for ur purpose.

On window as there no such utility, a small program can be written to read single line and compare all below line for unique ness. Make sure your dont read all file in memory as it will lead to memory exhaust.

if u dont know how to read large file using php just google reading large file using php on google

Pavan PHP

--
 
 

--

[Hairunizam Mohamed Salleh]

Hi,

This is my opinion. In your second program, instead of printing those sting into a file, why don't you use database. By using SQL command, you can have count for each string faster.

Pavan PHP

--
 
 

--
Regards 
-Hairunizam Mohamed Salleh

[Robert Gonzalez]

This isn't necessarily the case, as you are going to be rifling through 5M records. The one thing you will have going for you is that you can filter your results to only return records where count is more than 1. Not necessarily performant, but it will give you what you want.

-- MySQL Query. You may have to alter this for your database server

SELECT uniq_string, COUNT(uniq_string) as my_count FROM uniq_strings GROUP BY uniq_string HAVING my_count > 1;

--
 
 

[Ovidiu Alexa]

COUNT, GROUP BY and HAVING .... hmmmm you could get away faster if you compare hashes (correct me if I'm WRONG)

SELECT s.uniq_string, 

COUNT(s.uniq_string) as my_count 

FROM (SELECT MD5(uniq_string) FROM uniq_strings) s

GROUP BY s.uniq_string 

HAVING my_count > 1;

--