Re: [courier-users] Duplicate files, single message
Sam Varshavchik
> I am occasionally seeing duplicated files in my courier-IMAP folders. Here's an example:
>
> $ ls -nls 1260094560.39840_0*
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,S
> $ diff 1260094560.39840_0.mail.covisp.net\:2\, 1260094560.39840_0.mail.covisp.net\:2\,S
> $
>
> looking at the content of the file it is the exact same email, but looking at my MUA the message only exists once.
>
> Any ideas why this happens? Can I prevent it?
>
> (the reason I want to prevent it is that I am scanning the maildir for messages that are not marked as 'seen' with a terminal 'S'. I'm not doing anything that would 'hold those messages, just doing a
>
> ls −1dA $HOME/Maildir/.*/{cur,new}/*{net,','} 2>/dev/null | awk -F/ '{print $5}'
>
> to count any unread messages)
What other scripts or software, instead of Courier-IMAP, do you use with
your mailbox?
Sam Varshavchik
> Nothing that moves the messages other than my MUA. I use procmail, but it writes the messages to .mailinglist-2009-12/ and does not access new or cur folders.
For procmail, or anyone else, it is impossible to actually accomplish the
task of delivering a message to a maildir unless the tmp and new
subdirectories are actually touched. Scribbling something into their parent
directory will have no effect.
Last time I checked, procmail really doesn't support maildirs. What its
documentation claims to be maildir support merely involves writing each
message to a unique file. This is insufficient, there's more to it than
that. There is a well defined process by which new messages get delivered to
a maildir.
The filenames you've shown do not look like filenames that would be created
by Courier or maildrop when delivering messages. Courier-IMAP itself does
not deliver mail. An IMAP server provides access to existing mail in a
mailbox, it does not deliver any mail.
> the mailserver is running postfix.
>
> The issue does not come up very often, so I haven't been able to track it to any particular event, and it's invisible from the MUA side since the duplicate files only show once in client.
The filenames you've shown appear to be formed by using the current
timestamp and the delivering process's PID. Quite a while ago, it became
apparent that modern servers that support modern of levels of mail traffic
are quite capable of recycling the same process id during the same
chronological second, as a result of rapid spawning of short-lived processes
that typically occur when delivering mail. As such, the same filename may
end up being generated for two discrete messages. The end results are
unpredictable, message corruption is likely.
Whatever you're actually using to deliver mail, you should update it to
something more current, that uses additional factors in composing the
individual message filenames, removing the chance of potential filename
collision.
Another possibility is filesystem corruption, of course. Take this partition
offline and fsck, as a safety measure.
LuKreme
> For procmail, or anyone else, it is impossible to actually accomplish the task of delivering a message to a maildir unless the tmp and new subdirectories are actually touched. Scribbling something into their parent directory will have no effect.
Procmail knows how to deliver properly to maildirs by specifying .maildir/ what I was trying to make clear was that I was not putting the messages directly into ./maildir/new as some people have a bad habit of doing.
> Last time I checked, procmail really doesn't support maildirs. What its documentation claims to be maildir support merely involves writing each message to a unique file. This is insufficient, there's more to it than that. There is a well defined process by which new messages get delivered to a maildir.
Procmail does support maildir, and the messages are delivered by procmail. They show up in new (moved there after being written to tmp. This is not where the problem occurs). When they get moved from new to cur (once they've been seen by an MUA) is when the duplication occurs. It is not occurring on delivery.
>From man procmailrc:
If the mailbox name ends in "/", then this directory
is presumed to be a maildir folder; i.e., procmail will deliver the
message to a file in a subdirectory named "tmp" and rename it to be
inside a subdirectory named "new".
> The filenames you've shown appear to be formed by using the current timestamp and the delivering process's PID. Quite a while ago, it became apparent that modern servers that support modern of levels of mail traffic are quite capable of recycling the same process id during the same chronological second, as a result of rapid spawning of short-lived processes that typically occur when delivering mail. As such, the same filename may end up being generated for two discrete messages. The end results are unpredictable, message corruption is likely.
My server is several orders of magnitude less busy than that. It will take it days, probably, to recycle a PID.
--
I WILL REMEMBER TO TAKE MY MEDICATION
Bart chalkboard Ep. 2F14
------------------------------------------------------------------------------
Join us December 9, 2009 for the Red Hat Virtual Experience,
a free event focused on virtualization and cloud computing.
Attend in-depth sessions from your desk. Your couch. Anywhere.
http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________
courier-users mailing list
courie...@lists.sourceforge.net
Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-users
Gordon Messmer
> I am occasionally seeing duplicated files in my courier-IMAP folders. Here's an example:
>
> $ ls -nls 1260094560.39840_0*
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,S
>
> looking at the content of the file it is the exact same email, but looking at my MUA the message only exists once.
>
If you're sure this didn't happen as a result of restoring from backup,
you should locate the mail logs for that message's ID and see if there's
any info there.
Sam Varshavchik
> There is one little point, when the message is initially written, the
> filename ends with '.net' but once it's in cur it is renamed to '.net:2,'
> or 'net:2,S'. I suspect that at some point there is where the duplication
> occurs, particularly if Courier duplicates the message to ,S and is then
> supposed to remove the original?
Courier does not duplicate the message. The message is always renamed.
> Oh, wait a minute, I just remembered that procmail has a TRAP on some mailing lists that is supposed to 'mark' a message as read.
>
> TRAP='mv "$LASTFOLDER" "${LASTFOLDER}:2,S"'
>
> So that is something else. While it is a mv command and it shouldn't ever leave the original file behind, I suppose it is possible something there is mucking up on rare enough occasions that I see these left-over dupes?
By itself, no. However, you will need to analyze the exact interaction
between this bit of code, and everything else you use to access your
mailbox.
Georg Lutz
> I am occasionally seeing duplicated files in my courier-IMAP folders. Here's an example:
>
> $ ls -nls 1260094560.39840_0*
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,S
I had exactly the same problem after restoring from a backup. At that
time I wrote a small script which fixed the issue. Perhaps it also helps
you:
http://git.georglutz.de/gitweb/?p=maildir_merge_duplicates.git;a=summary
Regards
Georg
--
Georg
------------------------------------------------------------------------------
Return on Information:
Google Enterprise Search pays you back
Get the facts.
http://p.sf.net/sfu/google-dev2dev
Matus UHLAR - fantomas
> I am occasionally seeing duplicated files in my courier-IMAP folders.
> Here's an example:
>
> $ ls -nls 1260094560.39840_0*
> 6 -rw------- 2 1004 1005 4780 Dec 6 03:16 1260094560.39840_0.mail.covisp.net:2,
2 hardlinks - aren't they hard links to the same file?
try "ls -nlsi"
--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe.
------------------------------------------------------------------------------
This SF.Net email is sponsored by the Verizon Developer Community
Take advantage of Verizon's best-in-class app development support
A streamlined, 14 day to market process makes app distribution fast and easy
Join now and get one step closer to millions of Verizon customers
http://p.sf.net/sfu/verizon-dev2dev