Regarding regular zfs

[Joar Jegleim]

Hi FreeBSD !

I've already sent this one to ques...@freebsd.org, but realised this list
would be a better option.

So I've got this setup where we have a storage server delivering about
2 million jpeg's as a backend for a website ( it's ~1TB of data)
The storage server is running zfs and every 15 minutes it does a zfs
send to a 'slave', and our proxy will fail over to the slave if the
main storage server goes down .
I've got this script that initially zfs send's a whole zfs volume, and
for every send after that only sends the diff . So after the initial zfs
send, the diff's usually take less than a minute to send over.

I've had increasing problems on the 'slave', it seem to grind to a
halt for anything between 5-20 seconds after every zfs receive . Everything
on the server halts / hangs completely.

I've had a couple go's on trying to solve / figure out what's
happening without luck, and this 3rd time I've invested even more time
on the problem .

To sum it up:
-Server was initially on 8.2-RELEASE
-I've set some sysctl variables such as:

# 16GB arc_max ( server got 30GB of ram, but had a couple 'freeze'
situations, suspect zfs.arc ate too much memory)
vfs.zfs.arc_max=17179869184

# 8.2 default to 30 here, setting it to 5 which is default from 8.3 and
onwards
vfs.zfs.txg.timeout="5"

# Set TXG write limit to a lower threshold. This helps "level out"
# the throughput rate (see "zpool iostat"). A value of 256MB works well
# for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on
# disks which have 64 MB cache. <<BR>>
# NOTE: in <v28, this tunable is called 'vfs.zfs.txg.write_limit_override'.
#vfs.zfs.txg.write_limit_override=1073741824 # for 8.2
vfs.zfs.write_limit_override=1073741824 # for 8.3 and above

-I've implemented mbuffer for the zfs send / receive operations. With
mbuffer the sync went a lot faster, but still got the same symptoms
when the zfs receive is done, the hang / unresponsiveness returns for
5-20 seconds
-I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
V28), same symptoms
-I've upgraded to 9.1-RELEASE, still same symptoms

The period where the server is unresponsive after a zfs receive, I
suspected it would correlate with the amount of data being sent, but
even if there is only a couple MB's data the hang / unresponsiveness
is still substantial .

I suspect it may have something to do with the zfs volume being sent
is mount'ed on the slave, and I'm also doing the backups from the
slave, which means a lot of the time the backup server is rsyncing the
zfs volume being updated.
I've noticed that the unresponsiveness / hang situations occur while
the backupserver is rsync'ing from the zfs volume being updated, when
the backupserver is 'done' and nothing is working with files in the
zfs volume being updated i hardly notice any of the symptoms (mabye
just a minor lag for much less than a second, hardly noticeable) .

So my question(s) to the list would be:
In my setup have I taken the use case for zfs send / receive too far
(?) as in, it's not meant for this kind of syncing and this often, so
there's actually nothing 'wrong'.

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------
_______________________________________________
freeb...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "freebsd-fs-...@freebsd.org"

[Damien Fleuriot]

Quick and dirty reply, what's your pool usage % ?

>75-80% an performance takes a dive.

Let's just make sure you're not there yet.

[Peter Maloney]

On 2013-04-05 13:07, Damien Fleuriot wrote:
> -I've implemented mbuffer for the zfs send / receive operations. With
> mbuffer the sync went a lot faster, but still got the same symptoms
> when the zfs receive is done, the hang / unresponsiveness returns for
> 5-20 seconds
> -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
> V28), same symptoms
> -I've upgraded to 9.1-RELEASE, still same symptoms

> So my question(s) to the list would be:
> In my setup have I taken the use case for zfs send / receive too far
> (?) as in, it's not meant for this kind of syncing and this often, so
> there's actually nothing 'wrong'.

I do the same thing on an 8.3-STABLE system, with replication every 20
minutes (compared to your 15 minutes), and it has worked flawlessly for
over a year. Before that point, it was hanging often, until I realized
that all hangs were from when there was more than 1 writing "zfs"
command running at the same time (snapshot, send, destroy, rename, etc.
but not list, get, etc.). So now *all my scripts have a common lock
between them* (just a pid file like in /var/run; cured the hangs), and I
don't run manual zfs commands without stopping my cronjobs. If the hang
was caused by a destroy or smething during a send, I think it would
usually unhang when the send is done, do the destroy or whatever else
was blocking, then be unhung completely, smoothly working. In other
cases, I think it would be deadlocked.


NAME USED REFER USEDCHILD USEDDS USEDSNAP
AVAIL MOUNTPOINT
tank 38.5T 487G 37.4T 487G 635G
9.54T /tank
tank/backup 7.55T 1.01T 5.08T 1.01T 1.46T
9.54T /tank/backup
...

Sends are still quick with 38 T to send. The last replication run
started 2013-04-05 13:20:00 +0200 and finished 2013-04-05 13:22:18
+0200. I have 234 snapshots at the moment (one per 20 min today + one
daily for a few months).

[Joar Jegleim]

zpool usage is 9% :)

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------

> > -I've implemented mbuffer for the zfs send / receive operations. With
> > mbuffer the sync went a lot faster, but still got the same symptoms
> > when the zfs receive is done, the hang / unresponsiveness returns for
> > 5-20 seconds
> > -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
> > V28), same symptoms
> > -I've upgraded to 9.1-RELEASE, still same symptoms
> >

> > The period where the server is unresponsive after a zfs receive, I
> > suspected it would correlate with the amount of data being sent, but
> > even if there is only a couple MB's data the hang / unresponsiveness
> > is still substantial .
> >
> > I suspect it may have something to do with the zfs volume being sent
> > is mount'ed on the slave, and I'm also doing the backups from the
> > slave, which means a lot of the time the backup server is rsyncing the
> > zfs volume being updated.
> > I've noticed that the unresponsiveness / hang situations occur while
> > the backupserver is rsync'ing from the zfs volume being updated, when
> > the backupserver is 'done' and nothing is working with files in the
> > zfs volume being updated i hardly notice any of the symptoms (mabye
> > just a minor lag for much less than a second, hardly noticeable) .
> >

> > So my question(s) to the list would be:
> > In my setup have I taken the use case for zfs send / receive too far
> > (?) as in, it's not meant for this kind of syncing and this often, so
> > there's actually nothing 'wrong'.
> >

> > --
> > ----------------------
> > Joar Jegleim
> >
>
> Quick and dirty reply, what's your pool usage % ?
>
> >75-80% an performance takes a dive.
>
> Let's just make sure you're not there yet.
>

[Joar Jegleim]

You make some interesting points .
I don't _think_ the script 'causes more than 1 zfs write at a time, and I'm
sure 'nothing else' is doing that neither . But I'm gonna check that out
because it does sound like a logical explanation.
I'm wondering if the rsync from the receiving server (that is: the backup
server is doing rsync from the zfs receive server) could 'cause the same
problem, it's only reading though ...

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------

On 5 April 2013 13:36, Peter Maloney <peter....@brockmann-consult.de>wrote:

> On 2013-04-05 13:07, Damien Fleuriot wrote:
>

> -I've implemented mbuffer for the zfs send / receive operations. With
> mbuffer the sync went a lot faster, but still got the same symptoms
> when the zfs receive is done, the hang / unresponsiveness returns for
> 5-20 seconds
> -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
> V28), same symptoms
> -I've upgraded to 9.1-RELEASE, still same symptoms
>
>

> So my question(s) to the list would be:
> In my setup have I taken the use case for zfs send / receive too far
> (?) as in, it's not meant for this kind of syncing and this often, so
> there's actually nothing 'wrong'.
>
>

> I do the same thing on an 8.3-STABLE system, with replication every 20
> minutes (compared to your 15 minutes), and it has worked flawlessly for
> over a year. Before that point, it was hanging often, until I realized that
> all hangs were from when there was more than 1 writing "zfs" command
> running at the same time (snapshot, send, destroy, rename, etc. but not

> list, get, etc.). So now *all my scripts have a common lock between them*(just a pid file like in /var/run; cured the hangs), and I don't run manual

> zfs commands without stopping my cronjobs. If the hang was caused by a
> destroy or smething during a send, I think it would usually unhang when the
> send is done, do the destroy or whatever else was blocking, then be unhung
> completely, smoothly working. In other cases, I think it would be
> deadlocked.
>
>
> NAME USED REFER USEDCHILD USEDDS USEDSNAP
> AVAIL MOUNTPOINT
> tank 38.5T 487G 37.4T 487G 635G
> 9.54T /tank
> tank/backup 7.55T 1.01T 5.08T 1.01T 1.46T
> 9.54T /tank/backup
> ...
>
> Sends are still quick with 38 T to send. The last replication run started
> 2013-04-05 13:20:00 +0200 and finished 2013-04-05 13:22:18 +0200. I have
> 234 snapshots at the moment (one per 20 min today + one daily for a few
> months).
>
>

[Ronald Klop]

On Fri, 05 Apr 2013 15:02:12 +0200, Joar Jegleim <joar.j...@gmail.com>
wrote:

> You make some interesting points .
> I don't _think_ the script 'causes more than 1 zfs write at a time, and
> I'm
> sure 'nothing else' is doing that neither . But I'm gonna check that out
> because it does sound like a logical explanation.
> I'm wondering if the rsync from the receiving server (that is: the backup
> server is doing rsync from the zfs receive server) could 'cause the same
> problem, it's only reading though ...
>
>
>

Do you run the rsync from a snapshot or from the 'live' filesystem? The
live one changes during zfs receive. I don't know if that has anything to
do with your problem, but rsync from a snapshot gives a consistent backup
anyway.

BTW: It is probably more simple for you to test if the rsync is related to
the problem, than for other people to theorize about it here.

Ronald.

[Peter Maloney]

On 2013-04-05 16:07, Ronald Klop wrote:
> On Fri, 05 Apr 2013 15:02:12 +0200, Joar Jegleim
> <joar.j...@gmail.com> wrote:
>
>> You make some interesting points .
>> I don't _think_ the script 'causes more than 1 zfs write at a time,
>> and I'm
>> sure 'nothing else' is doing that neither . But I'm gonna check that out
>> because it does sound like a logical explanation.
>> I'm wondering if the rsync from the receiving server (that is: the
>> backup
>> server is doing rsync from the zfs receive server) could 'cause the same
>> problem, it's only reading though ...
>>
>>
>>
>
> Do you run the rsync from a snapshot or from the 'live' filesystem?
> The live one changes during zfs receive. I don't know if that has
> anything to do with your problem, but rsync from a snapshot gives a
> consistent backup anyway.
>
> BTW: It is probably more simple for you to test if the rsync is
> related to the problem, than for other people to theorize about it here.
>
> Ronald.

Also I don't believe using rsync either on the snapshot or the file
system (read or write) should be related in any way to the hang I
described. I let my cronjob rsync backups run wild without issues.

When I say zfs commands, I don't mean random other commands on the zfs
file system, but only the "zfs" command with a writing subcommand, such
as destroy, recv, or snapshot, which obviously need some locking, and
also send which locks some things, such as preventing the snapshot you
are sending from being removed while you send it.

Next time it hangs, just run something like: ps axl | grep zfs. If you
see 2 zfs commands running at once that aren't parent/child of
eachother, then you may have the same problem I described. If not (such
as if you see your send + rsync at the same time), then it is something
else.

[Adam McDougall]

On 04/05/13 06:17, Joar Jegleim wrote:
> Hi FreeBSD !
>
> I've already sent this one to ques...@freebsd.org, but realised this list
> would be a better option.
>
> So I've got this setup where we have a storage server delivering about
> 2 million jpeg's as a backend for a website ( it's ~1TB of data)
> The storage server is running zfs and every 15 minutes it does a zfs
> send to a 'slave', and our proxy will fail over to the slave if the
> main storage server goes down .
> I've got this script that initially zfs send's a whole zfs volume, and
> for every send after that only sends the diff . So after the initial zfs
> send, the diff's usually take less than a minute to send over.
>
> I've had increasing problems on the 'slave', it seem to grind to a
> halt for anything between 5-20 seconds after every zfs receive . Everything
> on the server halts / hangs completely.
>
> I've had a couple go's on trying to solve / figure out what's
> happening without luck, and this 3rd time I've invested even more time
> on the problem .
>
> To sum it up:
> -Server was initially on 8.2-RELEASE
> -I've set some sysctl variables such as:
>
> # 16GB arc_max ( server got 30GB of ram, but had a couple 'freeze'
> situations, suspect zfs.arc ate too much memory)
> vfs.zfs.arc_max=17179869184

What is vm.kmem_size? I suggest setting it to 2x physical ram (60G) but
higher won't hurt because recent-ish kernels will properly cap it at 2x.
This will allow more space in kmem to reduce ARC fragmentation in
memory which can cause stalls or panics. Also, I would set
vfs.zfs.arc_max almost as high as your physical memory (or not at all)
unless you need to reserve some for non-zfs usage. With ZFS from the
latest year or two, I generally don't adjust anything other than making
sure vm.kmem_size is sufficiently high on all systems, and some systems
I use arc_max. Rarely arc_min but in an attempt to encourage ARC use.
By avoiding ARC fragmentation, I generally feel more ARC is better than
less.

Also keep an eye on kstat.zfs.misc.arcstats.size when you run zfs recv
to see how it behaves.

>
> # 8.2 default to 30 here, setting it to 5 which is default from 8.3 and
> onwards
> vfs.zfs.txg.timeout="5"
>
> # Set TXG write limit to a lower threshold. This helps "level out"
> # the throughput rate (see "zpool iostat"). A value of 256MB works well
> # for systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on
> # disks which have 64 MB cache. <<BR>>
> # NOTE: in <v28, this tunable is called 'vfs.zfs.txg.write_limit_override'.
> #vfs.zfs.txg.write_limit_override=1073741824 # for 8.2
> vfs.zfs.write_limit_override=1073741824 # for 8.3 and above
>

> -I've implemented mbuffer for the zfs send / receive operations. With
> mbuffer the sync went a lot faster, but still got the same symptoms
> when the zfs receive is done, the hang / unresponsiveness returns for
> 5-20 seconds
> -I've upgraded to 8.3-RELEASE ( + zpool upgrade and zfs upgrade to
> V28), same symptoms
> -I've upgraded to 9.1-RELEASE, still same symptoms
>

I think it is good to be on 9.x to get ZFS fixes before 8.

[Peter Jeremy]

On 2013-Apr-05 12:17:27 +0200, Joar Jegleim <joar.j...@gmail.com> wrote:
>I've got this script that initially zfs send's a whole zfs volume, and
>for every send after that only sends the diff . So after the initial zfs
>send, the diff's usually take less than a minute to send over.

Are you deleting old snapshots after the newer snapshots have been sent?

>I've had increasing problems on the 'slave', it seem to grind to a
>halt for anything between 5-20 seconds after every zfs receive . Everything
>on the server halts / hangs completely.

Can you clarify which machine you mean by server in the last line above.
I presume you mean the slave machine running "zfs recv".

If you monitor the "server" with "vmstat -v 1", "gstat -a" and "zfs-mon -a"
(the latter is part of ports/sysutils/zfs-stats) during the "freeze",
what do you see? Are the disks saturated or idle? Are the "cache" or
"free" values close to zero?

># 16GB arc_max ( server got 30GB of ram, but had a couple 'freeze'
>situations, suspect zfs.arc ate too much memory)

There was a bug in interface between ZFS ARC and FreeBSD VM that resulted
in ARC starvation. This was fixed between 8.2 and 8.3/9.0.

>I suspect it may have something to do with the zfs volume being sent
>is mount'ed on the slave, and I'm also doing the backups from the
>slave, which means a lot of the time the backup server is rsyncing the
>zfs volume being updated.

Do you have atime enabled or disabled? What happens when you don't run
rsync at the same time?

Are you able to break into DDB?

>In my setup have I taken the use case for zfs send / receive too far
>(?) as in, it's not meant for this kind of syncing and this often, so
>there's actually nothing 'wrong'.

Apart from the rsync whilst receiving, everything sounds OK. It's
possible that the rsync whilst receiving is triggering a bug.

--
Peter Jeremy

[Joar Jegleim]

the rsync was running from the live system.
As I wrote earlier, the problem seem to only occur while the backup server
is rsync'ing from the slave (zfs receiving side), so I was actually trying
to figure out if this was to be expected (as in zfs sync, where the
receiving end get a diff and roll 'back' to version = latest snapshot from
'master') with a setup with +1TB data and +2million files .




On 5 April 2013 16:07, Ronald Klop <ronald-...@klop.yi.org> wrote:

> On Fri, 05 Apr 2013 15:02:12 +0200, Joar Jegleim <joar.j...@gmail.com>
> wrote:
>
> You make some interesting points .
>> I don't _think_ the script 'causes more than 1 zfs write at a time, and
>> I'm
>> sure 'nothing else' is doing that neither . But I'm gonna check that out
>> because it does sound like a logical explanation.
>> I'm wondering if the rsync from the receiving server (that is: the backup
>> server is doing rsync from the zfs receive server) could 'cause the same
>> problem, it's only reading though ...
>>
>>
>>
>>
> Do you run the rsync from a snapshot or from the 'live' filesystem? The
> live one changes during zfs receive. I don't know if that has anything to
> do with your problem, but rsync from a snapshot gives a consistent backup
> anyway.
>
> BTW: It is probably more simple for you to test if the rsync is related to
> the problem, than for other people to theorize about it here.
>
> Ronald.
>

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------

[Joar Jegleim]

[...]"Are you deleting old snapshots after the newer snapshots have been
sent?"[...]
yeah, the script deletes old snapshots. The slave will usually hold 2
snapshots ( 1 being the initial snapshot received via zfs send from master,
2nd being the latest snapshot received from master) .

[...]"Can you clarify which machine you mean by server in the last line

above.
I presume you mean the slave machine running "zfs recv".

If you monitor the "server" with "vmstat -v 1", "gstat -a" and "zfs-mon -a"
(the latter is part of ports/sysutils/zfs-stats) during the "freeze",
what do you see? Are the disks saturated or idle? Are the "cache" or

"free" values close to zero?" [...]
The last line "Everything
on the server halts / hangs completely." I'm talking about the 'slave' (the
receiving end)
I'll check how cache is doing, but as I wrote in my previous reply, the
'slave' server is completely unresponsive, nothing works at all for 5-15
seconds, when the server is responsive again (can ssh in and so on) I can't
seem to find anything in dmesg or any log hinting about anything at all
that went 'wrong' .

"There was a bug in interface between ZFS ARC and FreeBSD VM that resulted
in ARC starvation. This was fixed between 8.2 and 8.3/9.0."

ah, ok .

"Do you have atime enabled or disabled? What happens when you don't run
rsync at the same time?

Are you able to break into DDB?"

atime is disabled. When I don't run rsync the server seem ok, I've tried to
detect any hang (as in I ssh into the server and issue various commands
such as top, ls and so on) while not rsync'ing and there might have been a
really minor 'glitch' but it was hardly noticeable at all, and nothing
compared to those 5-15 seconds when the backup server is doing the rsync
(from the live volume, not a snapshot) .

I could try DDB, I'm gonna have to get back to you on that, I haven't
debug'ed FreeBSD kernel before and the system is in production, so I would
have to be cautious. I might be able to try out that during this week .

[...]Apart from the rsync whilst receiving, everything sounds OK. It's
possible that the rsync whilst receiving is triggering a bug.[...]
I sort of think so too, at least since the whole OS is unresponsive / hang
for anything from 5-15 seconds .

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------

On 5 April 2013 23:12, Peter Jeremy <pe...@rulingia.com> wrote:

> On 2013-Apr-05 12:17:27 +0200, Joar Jegleim <joar.j...@gmail.com>

[Peter Jeremy]

On 2013-Apr-08 10:29:52 +0200, Joar Jegleim <joar.j...@gmail.com> wrote:
>I'll check how cache is doing, but as I wrote in my previous reply, the
>'slave' server is completely unresponsive, nothing works at all for 5-15
>seconds, when the server is responsive again (can ssh in and so on) I can't
>seem to find anything in dmesg or any log hinting about anything at all
>that went 'wrong' .

If you have iostat/gstat/top/... running, does it hang (stop updating)
during this period? Is it pingable during the "hang"?

How about iostat/gstat/top/... running on the console?

Do you have compression or dedup enabled?

>I could try DDB, I'm gonna have to get back to you on that, I haven't
>debug'ed FreeBSD kernel before and the system is in production, so I would
>have to be cautious. I might be able to try out that during this week .

Do you have a test system that you can reproduce the problem on? The
reason I ask about DDB it that it would be useful to get a 'ps' whilst
the system is hung and it sounds like DDB is the only way to get that.

--
Peter Jeremy

[Joar Jegleim]

no compression or dedup .
I've found some hardware to setup a test system where I can debug this
thorough, but it's gonna take some time .

Appreciate all help, I'll get back to you when I've got a test environment
and have done some more tests.

--
----------------------
Joar Jegleim
Homepage: http://cosmicb.no
Linkedin: http://no.linkedin.com/in/joarjegleim
fb: http://www.facebook.com/joar.jegleim
AKA: CosmicB @Freenode

----------------------