Re: ZFS, compression, system load, pauses (livelocks?)
Wiktor Niesiobedzki
> The context of this post is file servers running FreeBSD 8 and ZFS with
> compressed file systems on low-end hardware, or actually high-end hardware
> on VMWare ESX 3.5 and 4, which kind of makes it low-end as far as storage is
> concerned. The servers are standby backup mirrors of production servers -
> thus many writes, few reads.
>
> Running this setup I notice two things:
>
> 1) load averages get very high, though the only usage these systems get is
> file system usage:
> 2) long pauses, in what looks like vfs.zfs.txg.timeout second intervals,
> which seemengly block everything, or at least the entire userland. These
> pauses are sometimes so long that file transfers fail, which must be
> avoided.
>
> Looking at the list of processes it looks like a large number of kernel and
> userland processes are woken up at once. From the kernel side there are
> regularily all g_* threads, but also unrelated threads like bufdaemon,
> softdepflush, etc. and from the userland - top, syslog, cron, etc. It is
> like ZFS livelocks everything else.
>
> Any ideas on the "pauses" issue?
>
Hi,
I've a bit striped your post. It's kind of "me too" message (more
details here: http://lists.freebsd.org/pipermail/freebsd-geom/2009-December/003810.html).
What I've figured out so far is, that lowering the kernel thread
priority (as pjd@ suggested) gives quite promising results (no
livelocks at all). Though my bottleneck were caused by GELI thread.
The pattern there is like this:
sched_prio(curthread, PRIBIO);
[...]
msleep(sc, &sc->sc_queue_mtx, PDROP | PRIBIO, "geli:w", 0);
I'm running right now with changed wersion - where I have:
msleep(sc, &sc->sc_queue_mtx, PDROP, "geli:w", 0);
So I don't change initial thread priority. It doesn't give such result
as using PUSER prio, but I fear, that using PUSER may cause livelocks
in some other cases.
This helps my case (geli encryption and periodic locks during ZFS
transaction commits) with some performance penalty, but I have similar
problems in other cases.
When I run:
# zfs scrub tank
Then "kernel" system process/thread consumes most of CPU (>95% in
system) and load rises to 20+ for the period of scrubbing. During
scrub my top screen looks like:
last pid: 87570; load averages: 8.26, 2.84, 1.68
199 processes: 3 running, 179 sleeping, 17 waiting
CPU: 2.4% user, 0.0% nice, 97.0% system, 0.6% interrupt, 0.0% idle
Mem: 66M Active, 6256K Inact, 1027M Wired, 104K Cache, 240K Buf, 839M Free
Swap: 4096M Total, 4096M Free
PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND
0 root 69 -8 0 0K 544K - 104:40 67.19% kernel
24 root 1 -8 - 0K 8K geli:w 9:56 5.66% g_eli[0] ad6
26 root 1 -8 - 0K 8K geli:w 9:50 5.47% g_eli[0] ad10
25 root 1 -8 - 0K 8K geli:w 9:53 5.37% g_eli[0] ad8
8 root 12 -8 - 0K 104K vgeom: 61:35 3.27% zfskern
3 root 1 -8 - 0K 8K - 3:22 0.68% g_up
11 root 17 -60 - 0K 136K WAIT 31:21 0.29% intr
Intresting thing, is that I have 17 processes waiting for CPU reported
(though only intr is the only process that is reported as in WAIT
state - at least for top40 processes).
I just wonder, whether this might be a scheduler related issue. I'm
thinking about giving a SCHED_4BSD a try.
Cheers,
Wiktor Niesiobędzki
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hacke...@freebsd.org"
Ivan Voras
I have tried before reducing priority of ZFS taskqueues but only to
PRIBIO, not below it - not much effect wrt "pauses".
Wiktor Niesiobedzki
> I have tried before reducing priority of ZFS taskqueues but only to
> PRIBIO, not below it - not much effect wrt "pauses".
I was testing with getting the thread as low priority as PUSER (with
original pjd@ patch) and it was actually performing better than the
current solution, though I found that quite risky.
Cheers,
Wiktor Niesiobędzki