Removing duplicate mail messages

[Manuel Carro]

Hello. I am using Gnus (No Gnus v0.11, currently) in Emacs 23.0.91
mainly to read mail (secondarily for a few rss feeds and newsgroups,
lile this one). I would like to remove duplicate mail messages which
either come to me from two different sources, or which I wronly imported
twice from previous mail folders (VM folders, mainly). I am using nnml
as backed, as I really want to have my mail locally stored and I am too
lazy to set up a local imap server + fetchmail.

So, in practice I would like to be able to:

* Check incoming messages (which I read using pop3) against already read
and stored messages, and remove the new messages whose contents [*]
match existing messages.

* Check my existing messages and remove duplicates, both inside a group
and, if possible, across groups.

[*] This would be a wish, as sometimes a message is duplicated because I
archive it locally when I send it, but I am as well in the recipient
list of the message. In this case the Id of the message is, I guess,
not meaningful, as I am using postfix as transport agent to send mail
and I think the message Id is assigned by postfix after it has left
Gnus.

I have seen (with "fdupes -r") that in fact messages are not stored
twice, but they appear twice in the summary window. I'd prefer them to
appear only once - for peace of mind, maily.

I have activated duplicate suppression (see below) but I do not know up
to which point it is working:

(setq gnus-suppress-duplicates t)
(setq gnus-save-duplicate-list t)
(setq gnus-duplicate-list-length 1000)
(setq gnus-duplicate-file "~/GNUS-Mail/SUPPRESSED-DUPLICATES")
(setq nnmail-treat-duplicates 'delete)

Maybe there is one way of having a trace of when duplicate messages are
removed and when they are not?

Also, I do not know how the suppressed duplicate list is exactly used:
my intuition is that it works as a "window" which keeps the N youngest
messages to check against, and from where old messages are flushed when
full. Is that right?

--
+------------------------------------------------------------------------------+
| Manuel Carro --- Facultad de Informática -- U. Politécnica de Madrid (UPM) |
| Campus de Montegancedo --- E-28660 Boadilla del Monte --- Spain |
| Phone: +34-913363747 --- FAX: +34-913363669 |

[Ted Zlatanov]

On Tue, 27 Oct 2009 10:45:19 +0100 Manuel Carro <mca...@fi.upm.es> wrote:

MC> * Check incoming messages (which I read using pop3) against already read
MC> and stored messages, and remove the new messages whose contents [*]
MC> match existing messages.

MC> * Check my existing messages and remove duplicates, both inside a group
MC> and, if possible, across groups.

Are the message IDs the same? You could use the Gnus registry to check
quickly if a message ID has been seen before.

Ted