Trying to somehow measure similarity of text files

[lbr...@gmail.com]

~
It is easy for files which are binary equal or can be somehow match
line by line with some diff-like code, but other than I couldn't
really find anything in java or otherwise
~
There is the dump3.sourceforge.net project, but I find it a little
two much and way too demanding to be run in a server. It appeared to
be concived with pictures as data format in mind
~
Do you know of any C, CPP or java project or a good well-out white
paper on this problematic?
~
Thanks
lbrtchx

[Roedy Green]

On Sun, 6 Jan 2008 21:59:29 -0800 (PST), lbr...@gmail.com wrote,
quoted or indirectly quoted someone who said :

> It is easy for files which are binary equal or can be somehow match
>line by line with some diff-like code, but other than I couldn't
>really find anything in java or otherwise

If you had a way of chunking the file. e.g. sentences in a text file,
newlines in a CSV file ,then you could compute a hashCode for each
"sentence".

You could then process your two files and create a list of hashcodes.
Then sort each list. Then compare counting matches. That gives you a
rough idea of how many sentences they have in common and how many are
unique to each. Compute a ratio of common/total unique sentences.

Ignore collisions (two sentences (either same of different) in same
file producing same hash code.

It is rude to ask questions in one group with followup to another.

I was thinking of some logic like this for creating delta files, that
could efficiently transmit changes to text files that have mainly been
reordered.
--
Roedy Green Canadian Mind Products
The Java Glossary
http://mindprod.com

[Daniel Pitts]

There are many different approaches to this problem that are specific
to the needs of your application...

For instance, if you are dealing with simple text files, and simply
want to know how many lines have changed, using diff makes sense.

If you care about other statistical properties, more sophisticated
approaches also exist. Keep in mind that diff itself uses a
sophisticated algorithm to find the differences between two (or more)
text files. If that is what you want to do, look at the source code
for diff (or use a system call and use diff as is)

[DavidGRAYsPeopleFINDER.com]

See if the files are the same number of bites
Right click on the file
....Select Properties
.......Compare the size in BITES
...........ignore the size on disk

--
David
http://DavidGRAYsPeopleFINDER.com
http://ADOPTIONsearching.com

<lbr...@gmail.com> wrote in message
news:0a4dba67-6b57-4286...@41g2000hsy.googlegroups.com...