Extreme performance falling on ZFS by durable operation of a server

[Alexandr D. Sergeev]

Hi, all!

There are problems with a server on FreeBSD 7.2 CURRENT (6 Jul 2009), carrying out the task of mail server SMTP, IMAP.

Messages are stored in maildirs on file system ZFS (mirror of 2 SAS 3k drives).
Dovecot is used as IMAP server.

After reboot the system works perfectly, however, somewhere in a month, at the same load, arise periodic long delays at opening of new letters.

Judging by output iostat -x -w 1, during these moments the disk subsystem is strongly loaded.
%busy disks comes nearer to 100 %. Readings nearby 300 per sec prevail. and approx. 10MBytes per sec.

Somebody can advise, what diagnostics needed, for understanding of the reason of problems?

There is an opinion that a problem with ZFS.

Probably, it does not have enough memory. Because during the periods of occurrence of a problem, ARC is have less 300MB:

vmstat -m
...
solaris 1245732 274745K - 25872717949 16,32,64,128,256,512,1024,2048,4096
...

Or it is bug of ZFS implementation.

--
Best regards,
Alexandr D.Sergeev

[K. Macy]

Hi Alexandr,

- How much memory do you have?

- What are your ARC settings?

Thanks,
Kip

> _______________________________________________
> freebsd-p...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-performance
> To unsubscribe, send any mail to "freebsd-perform...@freebsd.org"
>

[Alexandr D. Sergeev]

On Thu, Apr 08, 2010 at 01:14:42PM -0700, K. Macy wrote:
> Hi Alexandr,

Hi Kip,

>
> - How much memory do you have?

4Gb.
Applications consume a lot of memory, is partially used swap (~1Gb, but without active i/o).

We have planned increase in memory to 16G

>
> - What are your ARC settings?

Settings is default, that recommended for amd64 kernel.
But I am not assured that it is optimal.
Thus sysctl variables are automatically sets to:
vfs.zfs.arc_max ~800Mb
vfs.zfs.arc_mim ~128Mb

Now the next reboot was required.
Together with which following settings have been made:

vm.kmem_size=2048M
vfs.zfs.arc_max=1536M
vfs.zfs.arc_min=1024M
vfs.zfs.vdev.cache.size=64M

Let's look, as it will affect.

However there is confidence that no problem at a size of ARC.
Even at 200Mb ARC, productivity should not fall so strongly.

[Alexandr D. Sergeev]

Whether there are any ideas about the reasons of such problems?

Whether transition to FreeBSD 8.0 RELEASE / STABLE can help?
ZFS in it is declared as "bug free"

[K. Macy]

8-STABLE is definitely better, but you should also set arc_min to at
least 384M-512M. I'm not sure that arc_min is even honored under 7.2.

-Kip

[Willem Jan Withagen]

On 12-4-2010 21:56, K. Macy wrote:
> 8-STABLE is definitely better, but you should also set arc_min to at
> least 384M-512M. I'm not sure that arc_min is even honored under 7.2.

Your mileage might vary....

I've been tempted to report similar problems running
8.0-stable march 23rd
on amd64 with 8Gb
running as fileserver (nfs/smb), imap/dovecot, rsync backup

But I get stalls (of several seconds) when trying to do something in the
shell first thing in the morning. After running some more service
responce gets a bit more back to normal.

Same holds for access over SMB and NFS with suffers from delays as well.

Seen more of these type of reports with unconclusive answers thusfar,
several of the suggestions did not improve the situation.
Haven't found the time to very seriously trace this, and report more
than just this.

--WjW