Hi there since now three weeks we have very poor performance on one of our kvm server with zfs data storage.
The server is a compute module in a intel modular server. The disks are in two storage pools with raid 5 with hotspare. A pool with SAS disks and a pool with SATA disks.
Here are some current values from different tools:
=== vmstat ==== procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 3 0 0 23175844 34508 15727028 0 0 333 96 50 36 4 3 89 4 0
Currently we don't know exactly what happened that the performance is so bad. We also stopped some virtual systems but the performance isn't better. One of the virtual systems is a terminal server for four peoples and the work on it is very slow and completly not possible.
Has someone a idea how we can optimize the performance? We have two other systems with the 80% same configuration. On another compute module on the save modular server the performance is a little bit better but not the best. On another modular server with a single compute module, which is the backup for all other systems. (zfs snapshots). the performance is on > 1000 MB/s.
1-a. What is the memory tuning (how much is there and how much is being used?)
1-b. is a long-running operation running (think of zfs send/recv or a scrub. See this in
zpool status -v
and/or
zpool history -i
2. Are you using dedup?
3. What is the general fs configuration (amount of mounts, presence of clones, did you enable compression, case-insensitivity, altered checksumming, quota, or copies=n policies?). The easiest way to answer all these questions at once is by doing
> Hi there since now three weeks we have very poor performance on one of > our kvm server with zfs data storage.
> The server is a compute module in a intel modular server. The disks > are in two storage pools with raid 5 with hotspare. A pool with SAS > disks and a pool with SATA disks.
> Here are some current values from different tools:
> === vmstat ====
> procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu-----
> r b swpd free buff cache si so bi bo in cs us > sy id wa st
> 3 0 0 23175844 34508 15727028 0 0 333 96 50 36
> 4 3 89 4 0
Guys, sorry for interfering with some way offtop IMO message (and my crappy English). But I can't stay calm.
zfs-fuse is not intended to use in high load production systems, as any other FUSE FS.
First of all, it don't have access to a low device level and consequently using some assumptions and tricks to work
correctly. Second, FUSE will never have at least same order perfomance values due to regular context switching.
Frankly speaking for me put zfs-fuse as a VPS storage is an recklessness.
If I vitally needs any zfs features, I would look at zfsonlinux (I heard rumors some bold men use recent versions
on a production) either real Solaris zfs as a central storage (using iSCSI over 10G/multipath).
On Mon, Jun 18, 2012 at 02:08:49AM -0700, Beleggrodion wrote:
> Hi there since now three weeks we have very poor performance on one of our kvm
> server with zfs data storage.
> The server is a compute module in a intel modular server. The disks are in two
> storage pools with raid 5 with hotspare. A pool with SAS disks and a pool with
> SATA disks.
> Here are some current values from different tools:
> === vmstat ====
> procs -----------memory---------- ---swap-- -----io---- --system--
> -----cpu-----
> r b swpd free buff cache si so bi bo in cs us sy id wa
> st
> 3 0 0 23175844 34508 15727028 0 0 333 96 50 36 4 3 89 > 4 0
> Guys, sorry for interfering with some way offtop IMO message (and my crappy English). But I can't stay calm.
Oh yes you can, but this is the internet, so you figured you might as well. And, you're welcome it :)
> zfs-fuse is not intended to use in high load production systems, as any other FUSE FS.
I'd agree about not recommending zfs-fuse for production loads, but the "high load" criterion seem to come from your imagination.
> First of all, it don't have access to a low device level and consequently using some assumptions and tricks to work
> correctly.
Please be specific. This resembles FUD
> Second, FUSE will never have at least same order perfomance values due to regular context switching.
Context switching is a real cost. In practice I'm not sure whether common load patterns would show it.
> Frankly speaking for me put zfs-fuse as a VPS storage is an recklessness.
> If I vitally needs any zfs features, I would look at zfsonlinux (I heard rumors some bold men use recent versions
> on a production) either real Solaris zfs as a central storage (using iSCSI over 10G/multipath).
I've use Solaris for my own fileserver. I now switched to zfsonlinux for it and it works fine. Only had glitches (hangs) with ftp shares. Might be gone with new version.
Zfs-fuse is still working nicely on some small home offices I admin (those are production systems yes) mainly because of the builtin redundancy, error checking on commodity hardware and the ability to do very efficient incremental offsite backups (using zfs send/receive).
3) I try to reproduce the exact configuration and home i don't forget something.
The mounts are the following:
Dateisystem Size Used Avail Use% Eingehängt auf
/dev/sda3 38G 19G 17G 53% /
tmpfs 36G 0 36G 0% /dev/shm
/dev/sda1 194M 107M 78M 58% /boot
/dev/sdb1 100G 49G 52G 49% /mnt/iso
sasDATA1_srv2_vm 2.2T 981G 1.2T 45% /sasDATA1_srv2_vm
sataDATA3_srv2_vm 306G 188G 118G 62% /sataDATA3_srv2_vm
I think there are no clones on the system at the moment and compression is complete disabled. We
don't use case-insensitivity and we only use the normal checksum process of zfs. quota's are disabled and
as i see there is no copies enabled.
And for your command, on our CentOS installation there is no output for your command, but think the follow
is the same output what you wanted:
zfs get all sasDATA1_srv2_vm
NAME PROPERTY VALUE SOURCE
sasDATA1_srv2_vm type filesystem -
sasDATA1_srv2_vm creation Sam Mär 24 9:48 2012 -
sasDATA1_srv2_vm used 1.23T -
sasDATA1_srv2_vm available 1.17T -
sasDATA1_srv2_vm referenced 980G -
sasDATA1_srv2_vm compressratio 1.00x -
sasDATA1_srv2_vm mounted yes -
sasDATA1_srv2_vm quota none default
sasDATA1_srv2_vm reservation none default
sasDATA1_srv2_vm recordsize 128K default
sasDATA1_srv2_vm mountpoint /sasDATA1_srv2_vm default
sasDATA1_srv2_vm sharenfs off default
sasDATA1_srv2_vm checksum on default
sasDATA1_srv2_vm compression off default
sasDATA1_srv2_vm atime on default
sasDATA1_srv2_vm devices on default
sasDATA1_srv2_vm exec on default
sasDATA1_srv2_vm setuid on default
sasDATA1_srv2_vm readonly off default
sasDATA1_srv2_vm zoned off default
sasDATA1_srv2_vm snapdir hidden default
sasDATA1_srv2_vm aclmode groupmask default
sasDATA1_srv2_vm aclinherit restricted default
sasDATA1_srv2_vm canmount on default
sasDATA1_srv2_vm xattr on default
sasDATA1_srv2_vm copies 1 default
sasDATA1_srv2_vm version 4 -
sasDATA1_srv2_vm utf8only off -
sasDATA1_srv2_vm normalization none -
sasDATA1_srv2_vm casesensitivity sensitive -
sasDATA1_srv2_vm vscan off default
sasDATA1_srv2_vm nbmand off default
sasDATA1_srv2_vm sharesmb off default
sasDATA1_srv2_vm refquota none default
sasDATA1_srv2_vm refreservation none default
sasDATA1_srv2_vm primarycache all default
sasDATA1_srv2_vm secondarycache all default
sasDATA1_srv2_vm usedbysnapshots 276G -
sasDATA1_srv2_vm usedbydataset 980G -
sasDATA1_srv2_vm usedbychildren 211M -
sasDATA1_srv2_vm usedbyrefreservation 0 -
sasDATA1_srv2_vm logbias latency default
sasDATA1_srv2_vm dedup off default
sasDATA1_srv2_vm mlslabel off -
Greetings
Beleggrodion
Am Montag, 18. Juni 2012 23:05:19 UTC+2 schrieb Seth Heeren:
> 1-a. What is the memory tuning (how much is there and how much is being > used?)
> 1-b. is a long-running operation running (think of zfs send/recv or a > scrub. See this in
> zpool status -v > and/or > zpool history -i
> 2. Are you using dedup?
> 3. What is the general fs configuration (amount of mounts, presence of > clones, did you enable compression, case-insensitivity, altered > checksumming, quota, or copies=n policies?). The easiest way to answer > all these questions at once is by doing
On Tue, Jun 19, 2012 at 01:51:26AM +0200, sgheeren wrote:
> On 19-06-12 01:35, Alexey Kurnosov wrote:
> >Guys, sorry for interfering with some way offtop IMO message (and my crappy English). But I can't stay calm.
> Oh yes you can, but this is the internet, so you figured you might
> as well. And, you're welcome it :)
Thanks for your approval:)
> >zfs-fuse is not intended to use in high load production systems, as any other FUSE FS.
> I'd agree about not recommending zfs-fuse for production loads, but
> the "high load" criterion seem to come from your imagination.
The hardware config surely not SOHO. And my imagination shows me that VPS with low disk loads is a very rare case.
Should I call to a psychiatrist?:)
> >First of all, it don't have access to a low device level and consequently using some assumptions and tricks to work
> >correctly.
> Please be specific. This resembles FUD
Actually I am an amateur here.
Read ahead, NCQ lenght, locks because of IO wait, any IO sheduling as well, correct buffer flushing, fully involvement in memory
management (not just humble brk syscall), and so far. All this stuff can be handle by FUSE? If so, very impressive.
> >Second, FUSE will never have at least same order perfomance values due to regular context switching.
> Context switching is a real cost. In practice I'm not sure whether
> common load patterns would show it.
It will show in that conditions (or already do). I am sure.
> >Frankly speaking for me put zfs-fuse as a VPS storage is an recklessness.
> >If I vitally needs any zfs features, I would look at zfsonlinux (I heard rumors some bold men use recent versions
> >on a production) either real Solaris zfs as a central storage (using iSCSI over 10G/multipath).
> I've use Solaris for my own fileserver. I now switched to zfsonlinux
> for it and it works fine. Only had glitches (hangs) with ftp shares.
> Might be gone with new version.
> Zfs-fuse is still working nicely on some small home offices I admin
> (those are production systems yes) mainly because of the builtin
> redundancy, error checking on commodity hardware and the ability to
> do very efficient incremental offsite backups (using zfs
> send/receive).
