sporadic 'Permission denied' errors on a well configured system

[Gertjan Oude Lohuis [Byte]]

Hi!
We are experiencing some problems on our mailservers. Currently we are
using 3 mailservers, which are running Debian Sarge with Postfix
2.1.5-9. All machines are heavily used and our configuration is working
just fine. However, occasionally mail can't be delivered into a maildir,
and the following message pops up:

Dec 18 11:22:54 mail3 postfix/virtual[18799]: 2E0A37E9B:
to=<AAAAA@hostname>, relay=virtual, delay=1, status=deferred (maildir
delivery failed: create /var/maildir/AAAAA/tmp/1166437373.P18799.mail3:
Permission denied)
Dec 18 11:22:54 mail3 postfix/virtual[18799]: warning: maildir access
problem for UID/GID=118/118: create
/var/maildir/AAAAA/tmp/1166437373.P18799.mail3: Permission denied

AAAAA is the name of the maildir, hostname is our mailname. In the last
two weeks this error has occured 25 times, on a grand total of over two
million delivered messages. Those 25 times are spread over 7 different
mailboxes, of which one got 19 of those 25 errors. This mailbox is used
for our internal error-reporting and monitoring tools, and is by far the
most heavily used mailbox.

I already tried to re-create the maildir, but that didn't help. UID/GID
are certainly correct: more than 99% of the mails are being delivered
flawlessly, and when this error occurs the mail gets delivered in the
next try (or after a flush of the mailqueue).

/var/maildir is an NFS-partition which runs on JFS on our fileserver. It
seems these errors started to occur after a filesystem resize of the
JFS-partition on the fileserver two weeks ago. The resize went smoothly
and everything works fine now, but we fear that something internally
might have gone wrong which is causing these problems.

Is there anyone who recognizes this problem, and/or knows a solution. Or
someone who can give me some advice on debugging this? How are your
experiences with maildirs on an NFS-directory, and/or JFS as filesystem?

Regards, Gertjan

[Wietse Venema]

Gertjan Oude Lohuis [Byte]:
[ Charset ISO-8859-1 unsupported, converting... ]

> Hi!
> We are experiencing some problems on our mailservers. Currently we are
> using 3 mailservers, which are running Debian Sarge with Postfix
> 2.1.5-9. All machines are heavily used and our configuration is working
> just fine. However, occasionally mail can't be delivered into a maildir,
> and the following message pops up:
>
> Dec 18 11:22:54 mail3 postfix/virtual[18799]: 2E0A37E9B:
> to=<AAAAA@hostname>, relay=virtual, delay=1, status=deferred (maildir
> delivery failed: create /var/maildir/AAAAA/tmp/1166437373.P18799.mail3:
> Permission denied)
> Dec 18 11:22:54 mail3 postfix/virtual[18799]: warning: maildir access
> problem for UID/GID=118/118: create
> /var/maildir/AAAAA/tmp/1166437373.P18799.mail3: Permission denied

Keep in mind that Postfix is the messenger. It just asks the kernel
to create the specified file, and the kernel reports a problem of
"permission denied" (EACCES error code).

> /var/maildir is an NFS-partition which runs on JFS on our fileserver. It
> seems these errors started to occur after a filesystem resize of the
> JFS-partition on the fileserver two weeks ago. The resize went smoothly
> and everything works fine now, but we fear that something internally
> might have gone wrong which is causing these problems.

It could be NFS related.

Postfix asks the NFS client kernel to create a file that must not
already exist; if the operation succeeds on the NFS server, but
the reply to the NFS client is lost, the NFS client would retransmit
the request, and the NFS server could reply with an error.

I don't know if this can be an issue with NFS over TCP, but
duplicate request errors happen with NFS over UDP or under high
load when packets are be lost and the previous reply isn't found
in the NFS server's RPC reply cache.

Postfix has a number of workarounds for NFS-related errors, but
it can't be done when creating a file that must not already exist.

Check for: NFS over TCP or UDP
Check for: nfsstat output

Wietse