Dovecot (imap) Issues On FhGFS Mount

539 views
Skip to first unread message

David .

unread,
Jun 27, 2013, 2:22:02 AM6/27/13
to fhgfs...@googlegroups.com
G'day FhGFS Team,

    I testing FhGFS as I would like it to replace our classic ext4/nfs/samba file server.  Everything is going well, except when I set up Dovecot (imap) service using the FhGFS mount for my home directory, I get the following errors in the maillog:

mbox:
Jun 27 14:56:01 kwd-parallelfs-client-01 dovecot: imap(david): Error: open(/kusers/techies/david/Mail/.imap/INBOX/.temp.kwd-parallelfs-client-01.scem.uws.edu.au.27709.1a785396e340c87e) failed: Too many open files
Jun 27 14:56:01 kwd-parallelfs-client-01 dovecot: imap(david): Error: file_dotlock_open() failed with file /kusers/techies/david/Mail/.imap/INBOX/dovecot.index.log: Too many open files

Maildir:
Jun 27 15:01:10 kwd-parallelfs-client-01 dovecot: imap(david): Error: dotlock /kusers/techies/david/Maildir/Account_Removal_2012_Semester_1/dovecot-uidlist.lock was immediately recreated under us

    Dovecot works on an nfs mounted share.

    Any one had any luck, or got any ideas on what's going on?

    The client is as follows:

 dovecot-2.0.9-5.el6.x86_64

-bash-4.1$ dovecot -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-358.6.2.el6.x86_64 x86_64 CentOS release 6.4 (Final) 
auth_verbose = yes
listen = *
mail_location = mbox:~/Mail:INBOX=/var/spool/mail/%u
maildir_copy_with_hardlinks = no
mbox_lazy_writes = no
mbox_write_locks = fcntl
namespace {
  hidden = no
  inbox = yes
  list = yes
  location = 
  prefix = Mail/
  separator = 
  subscriptions = yes
  type = private
}
namespace {
  hidden = no
  inbox = no
  list = yes
  location = maildir:~/Maildir:LAYOUT=fs
  prefix = Maildir/
  separator = 
  subscriptions = yes
  type = private
}
passdb {
  driver = pam
}
protocols = imap
service imap-login {
  process_min_avail = 1
}
shutdown_clients = no
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_key = </etc/pki/dovecot/private/dovecot.pem
ssl_parameters_regenerate = 0
userdb {
  driver = passwd
}

    What I get in my mail folders are broken symlinks:

1 lrwxrwxrwx  1 david techies 145 Jun 27 14:59 dovecot.index.log -> ../../../../../kusers/techies/david/Maildir/Account_Removal_2012_Semester_1/.temp.kwd-parallelfs-client-01.scem.uws.edu.au.27792.5cd87ebe063e7bb2

Rather than, from the working dovecot service:

24 -rw-------.  1 david techies  18716 Jun 27 13:37 dovecot.index.log


Cheers,
David.

Sven Breuner

unread,
Jun 27, 2013, 3:05:51 AM6/27/13
to fhgfs...@googlegroups.com, da...@scem.uws.edu.au
Hi David,

we haven't tried Dovecot yet and I'm not sure whether the two problems
(too many open files and symlinks) are related.

So regarding the number of open files: Are you runnig the fhgfs server
daemons as user or with root privileges? I don't know how many files
Dovecot usually tries to open simultaneously, but user processes are
normally limited to about 1024 open files (and the fhgfs servers are
stateful, so they also need to keep the chunkfile file open for every
file that is open on a client). If the fhgfs servers have root
privileges, they automatically try to increase their limit to 50000.
The currently effective limit can be checked with "ulimit -a" and can be
changed via sysctl.
(Of course, there is also a chance that this is a bug in dovecut where
it forgets to close files in situations where something else is going
wrong.)

Regarding the dangling symlinks: Do you know if Dovecut tries to create
hardlinks? Because they are not supported by fhgfs yet (which is subject
to change with the upcoming releases).
As a workaround, the fhgfs client by default creates symlinks when an
application tries to create a hardlink - which works well for some apps,
but not so well for others.
Some applications also have their own workarounds built in when they see
that a file system does not support hardlinks, so you might want to try
changing the corresponding option in /etc/fhgfs/fhgfs-client.conf to
sysCreateHardlinksAsSymlinks=false, which will result in an appropriate
error code instead of a symlink when an application tries to create a
hardlink.

Best regards,
Sven

David Minard

unread,
Jun 30, 2013, 9:40:00 PM6/30/13
to fhgfs...@googlegroups.com
G'day Sven,

On 27/06/2013, at 5:05 PM, Sven Breuner wrote:

> Hi David,
>
> we haven't tried Dovecot yet and I'm not sure whether the two problems (too many open files and symlinks) are related.
>
> So regarding the number of open files: Are you runnig the fhgfs server daemons as user or with root privileges? I don't know how many files Dovecot usually tries to open simultaneously, but user processes are normally limited to about 1024 open files (and the fhgfs servers are stateful, so they also need to keep the chunkfile file open for every file that is open on a client). If the fhgfs servers have root privileges, they automatically try to increase their limit to 50000.
> The currently effective limit can be checked with "ulimit -a" and can be changed via sysctl.
> (Of course, there is also a chance that this is a bug in dovecut where it forgets to close files in situations where something else is going wrong.)
>

The FhGFS server services are running as root.

[root@kwd-parallelfs-01 ~]# ulimit -a
open files (-n) 1024

After changing the "sysCreateHardlinksAsSymlinks" option as suggested I'm not getting the "Too many open files" message. However....


> Regarding the dangling symlinks: Do you know if Dovecut tries to create hardlinks? Because they are not supported by fhgfs yet (which is subject to change with the upcoming releases).
> As a workaround, the fhgfs client by default creates symlinks when an application tries to create a hardlink - which works well for some apps, but not so well for others.
> Some applications also have their own workarounds built in when they see that a file system does not support hardlinks, so you might want to try changing the corresponding option in /etc/fhgfs/fhgfs-client.conf tosysCreateHardlinksAsSymlinks=false, which will result in an appropriate error code instead of a symlink when an application tries to create a hardlink.
>

Dovecot does by default use hardlinks for file copies. I have set it to not do this.

/etc/dovecot/conf.d/10-mail.conf:
maildir_copy_with_hardlinks = no

Now we're getting sets of messages like this:

Jun 27 17:57:20 kwd-parallelfs-client-01 dovecot: imap(david): Error: link(/kusers/techies/david/Maildir/Junk_To_Sort/Other/.temp.kwd-parallelfs-client-01.scem.uws.edu.au.1918.bc5446b8e71c817c, /kusers/techies/david/Maildir/Junk_To_Sort/Other/dovecot-uidlist.lock) failed: Operation not permitted
Jun 27 17:57:20 kwd-parallelfs-client-01 dovecot: imap(david): Error: file_dotlock_create(/kusers/techies/david/Maildir/Junk_To_Sort/Other/dovecot-uidlist) failed: Operation not permitted

It looks like Dovecot is trying to use hardlinks for other things......

Any e.t.a. on the hardlink update?

Cheers,
> --
> You received this message because you are subscribed to the Google Groups "fhgfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to fhgfs-user+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.

David Minard.
Ph: 0247 360 155
Fax: 0247 360 770

School of Computing, Engineering, and Mathematics
Building Y - Penrith Campus (Kingswood)
Locked bag 1797
Penrith South DC
NSW 1797

[Sometimes waking up just isn't worth the insult of the day to come.]


--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Sven Breuner

unread,
Jul 2, 2013, 3:46:39 AM7/2/13
to fhgfs...@googlegroups.com, David Minard
Hi David,

David Minard wrote on 07/01/2013 03:40 AM:
> Dovecot does by default use hardlinks for file copies. I have set it to not do this.
>
> /etc/dovecot/conf.d/10-mail.conf:
> maildir_copy_with_hardlinks = no
>
> Now we're getting sets of messages like this:
>
> Jun 27 17:57:20 kwd-parallelfs-client-01 dovecot: imap(david): Error: link(/kusers/techies/david/Maildir/Junk_To_Sort/Other/.temp.kwd-parallelfs-client-01.scem.uws.edu.au.1918.bc5446b8e71c817c, /kusers/techies/david/Maildir/Junk_To_Sort/Other/dovecot-uidlist.lock) failed: Operation not permitted
> Jun 27 17:57:20 kwd-parallelfs-client-01 dovecot: imap(david): Error: file_dotlock_create(/kusers/techies/david/Maildir/Junk_To_Sort/Other/dovecot-uidlist) failed: Operation not permitted
>
> It looks like Dovecot is trying to use hardlinks for other things......
>
> Any e.t.a. on the hardlink update?

from the messages above it seems like dovecot would only need to
hardlink files within the same directory. This is phase 1 of fhgfs
hardlink support, which is already completely implemented in our
corresponding internal feature branch.
Regarding eta: While fhgfs version 2012.10-r5 has now successfully made
its way through final testing and is scheduled for release later this
week, r6 release testing has already begun without hardlinks phase 1. So
hardlinks phase 1 is currently scheduled for r7, for which we will start
final release testing immediately after the release of r6.

Best regards,
Sven

David .

unread,
Nov 11, 2013, 9:06:46 PM11/11/13
to fhgfs...@googlegroups.com, David Minard, sven.b...@itwm.fraunhofer.de
G'day Guys,

I'd like to update you on how this is going now, plus a question.  I've got the most current release of FHGFS, and things are working a lot better.

I set up a few tests to see how Dovecot works with FHGFS now:

1)  used unfs3 to export my FHGFS directory onto our imap (dovecot) server.

The results were similar to before with lots of errors, and mailbox access locking up.

Nov  1 12:38:43 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/sent-mail/dovecot-uidlist.lock) failed: Stale file handle
Nov  1 12:38:43 mace-windu dovecot: imap(david): Error: file_dotlock_create(/mnt/kusers_fhgfs/techies/david/Maildir/sent-mail/dovecot-uidlist) failed: Stale file handle
Nov  1 12:39:18 mace-windu dovecot: imap(david): Error: fcntl(read-lock) locking failed for file /mnt/kusers_fhgfs/techies/david/Mail/.imap/INBOX/dovecot.index: Input/output error
Nov  1 12:40:25 mace-windu dovecot: imap(david): Error: fcntl(write-lock) locking failed for file /mnt/kusers_fhgfs/techies/david/Maildir/sent-mail/dovecot.index.log: Input/output error
Nov  1 12:40:25 mace-windu dovecot: imap(david): Error: mail_index_wait_lock_fd() failed with file /mnt/kusers_fhgfs/techies/david/Maildir/sent-mail/dovecot.index.log: Input/output error
Nov  1 12:41:31 mace-windu dovecot: imap(david): Error: fcntl(read-lock) locking failed for file /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot.index: Input/output error
Nov  1 12:42:37 mace-windu dovecot: imap(david): Error: fcntl(read-lock) locking failed for file /mnt/kusers_fhgfs/techies/david/Maildir/Sent Messages/dovecot.index: Input/output error


2) Kernel NFS mounted my FHGFS directory to the imap server.

The results were much better.  My mail client worked very well.  However, there were a few errors in the maillog.  These popped up when a mail box was changed by the mail client - eg: received a new mail, moved a mail from one mailbox to another.


Nov  6 10:53:52 mace-windu dovecot: imap(david): Error: Corrupted index cache file /mnt/kusers_fhgfs/techies/david/Maildir/Work_Bits/How To's/XCP/dovecot.index.cache: field type corrupted
Nov  6 10:54:08 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/Work_Bits/Samba/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:08 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/Work_Bits/Samba/dovecot-uidlist
Nov  6 10:54:08 mace-windu dovecot: imap(david): Error: Corrupted index cache file /mnt/kusers_fhgfs/techies/david/Maildir/Work_Bits/fhgfs/dovecot.index.cache: invalid record size
Nov  6 10:54:12 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:12 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Corrupted index cache file /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot.index.cache: invalid record size
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/SPAM/dovecot-uidlist
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Corrupted index cache file /mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/Tax_2012_2013/dovecot.index.cache: invalid record size
Nov  6 10:54:19 mace-windu dovecot: imap(david): Error: Corrupted index cache file /mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/Tax_2012_2013/dovecot.index.cache: invalid record size
Nov  6 10:54:22 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/People/David Minard/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:22 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/People/David Minard/dovecot-uidlist
Nov  6 10:54:23 mace-windu dovecot: imap(david): Error: fstat(/mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/People/David Minard/dovecot-uidlist.lock) failed: Bad file descriptor
Nov  6 10:54:23 mace-windu dovecot: imap(david): Error: Maildir uidlist dotlock overridden: /mnt/kusers_fhgfs/techies/david/Maildir/Non_Work_Bits/People/David Minard/dovecot-uidlist



3) I cloned our mail server, and ran up an FHGFS client on it.  Dovecot worked perfectly.

However, this brought up a question.  Is it possible to "root_squash" the FHGFS mount?  I noticed that root can access everything (rw) on the fhgfs client.  If we have to set up various servers as fhgfs clients, because NFS mounting fhgfs isn't fully working, I'd like to remove root privilege from these clients: shell server, web server, home directories etc....

Cheers,
David.

Frank Kautz

unread,
Nov 12, 2013, 8:45:47 AM11/12/13
to fhgfs...@googlegroups.com
Hello David,

Am 11/12/2013 03:06 AM, schrieb David .:
> G'day Guys,
>
> I'd like to update you on how this is going now, plus a question. I've
> got the most current release of FHGFS, and things are working a lot better.
>
> I set up a few tests to see how Dovecot works with FHGFS now:
>
> 1) used unfs3 to export my FHGFS directory onto our imap (dovecot) server.
>
> The results were similar to before with lots of errors, and mailbox
> access locking up.

unfs3 doesn't support file locking.

> 2) Kernel NFS mounted my FHGFS directory to the imap server.
>
> The results were much better. My mail client worked very well.
> However, there were a few errors in the maillog. These popped up when
> a mail box was changed by the mail client - eg: received a new mail,
> moved a mail from one mailbox to another.

Does it work on a NFS mount without fhgfs?

> 3) I cloned our mail server, and ran up an FHGFS client on it. Dovecot
> worked perfectly.
>
> However, this brought up a question. Is it possible to "root_squash"
> the FHGFS mount? I noticed that root can access everything (rw) on the
> fhgfs client. If we have to set up various servers as fhgfs clients,
> because NFS mounting fhgfs isn't fully working, I'd like to remove root
> privilege from these clients: shell server, web server, home directories
> etc....

In the moment fhgfs doesn't support root_squash.

kind regards,
Frank
frank_kautz.vcf

David Minard

unread,
Nov 12, 2013, 6:02:39 PM11/12/13
to fhgfs...@googlegroups.com

On 13/11/2013, at 12:45 AM, Frank Kautz wrote:

> Hello David,
>
> Am 11/12/2013 03:06 AM, schrieb David .:
>> G'day Guys,
>>
>> I'd like to update you on how this is going now, plus a question. I've
>> got the most current release of FHGFS, and things are working a lot better.
>>
>> I set up a few tests to see how Dovecot works with FHGFS now:
>>
>> 1) used unfs3 to export my FHGFS directory onto our imap (dovecot) server.
>>
>> The results were similar to before with lots of errors, and mailbox
>> access locking up.
>
> unfs3 doesn't support file locking.
>

Cool. I only tried this as a stop gap measure when kernel nfs was not working in FHGFS.

>> 2) Kernel NFS mounted my FHGFS directory to the imap server.
>>
>> The results were much better. My mail client worked very well.
>> However, there were a few errors in the maillog. These popped up when
>> a mail box was changed by the mail client - eg: received a new mail,
>> moved a mail from one mailbox to another.
>
> Does it work on a NFS mount without fhgfs?
>

Yes. We nfs mount all home directories, which are ext4, on the mail server. It all works perfectly.

>> 3) I cloned our mail server, and ran up an FHGFS client on it. Dovecot
>> worked perfectly.
>>
>> However, this brought up a question. Is it possible to "root_squash"
>> the FHGFS mount? I noticed that root can access everything (rw) on the
>> fhgfs client. If we have to set up various servers as fhgfs clients,
>> because NFS mounting fhgfs isn't fully working, I'd like to remove root
>> privilege from these clients: shell server, web server, home directories
>> etc....
>
> In the moment fhgfs doesn't support root_squash.

Any thoughts of adding this feature?
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>
> <frank_kautz.vcf>
Reply all
Reply to author
Forward
0 new messages