Ubuntu PPA 0.6.9-7~1.gbp52ec03
Gavin Chappell
1) The binaries have all moved from /usr/sbin where they were with
0.6.9-6 to /sbin. This caused my monthly scrub script to fail, and
also my byobo windows which load up "zpool status" and "zpool iostat"
in the background. Not a problem, just an undocumented (I couldn't
find mention of this in the changelog) change which I thought was
worth mentioning.
2) Once I'd fixed this and kicked off my monthly scrub, I noticed that
my SSH session became very laggy (definitely system lag as I was on
the same LAN at the time and it only happened after I launched the
scrub). My system load is currently at 3.77 and I keep seeing spikes
of iowait time in top which I don't remember seeing before. I've
noticed that when I do "ps -ef | grep zfs-fuse" this new version has
fewer command line arguments than before. I can't remember what
options it ran with before but it at least ran as "zfs-fuse -a 1 -e 1"
which I assume correspond to the two timeout entries in zfsrc (which
are both 3600, the default)? I don't think I've ever edited the zfsrc
file (I started off using Schmod's PPA originally which didn't have
anything to tune), so I've been running with whatever you guys have
shipped all the way through. Has anything changed in this release
which could cause this behaviour? Once the scrub has finished in an
hour or so, I'm going to try re-installing 0.6.9-6 and scrubbing again
to see what happens then regarding the latency, but I'm interested to
know whether there's anything that changed in this release which could
cause this behaviour, maybe a change in the init scripts which mean
zfs-fuse is started differently with different options or something?
Cheers,
Gavin
sgheeren
I got an update to this version last night and have noticed two things:
Note that I took up the slack for ubuntu packaging with zero prior experience. It turns out that the debian/ dir in the repo was all but up to date. The updated version is 'fast-forwarded' to be more in synch with the official debian (universe) package. This involved a lengthy email conversation off-list with
- Mike Hommey
- Sebastien Delafond
over the course of the last few weeks.
Also note that the version id 0.6.9-7~1.gbp52ec03 indicates a snapshot release. I'm not sure how I can prevent automatic upgrades, but surely in debian packaging lingo a snapshot is supposed to be a somewhat less official intermediate package.
1) The binaries have all moved from /usr/sbin where they were with 0.6.9-6 to /sbin. This caused my monthly scrub script to fail, and also my byobo windows which load up "zpool status" and "zpool iostat" in the background. Not a problem, just an undocumented (I couldn't find mention of this in the changelog) change which I thought was worth mentioning.
Note that the end goal of this is that the debian universe and my ppa package get on the same page so the packaging effort can be combined. I hate that there are several 'incongruent' packages floating around
That is unexpected. Here's is a bullet list of changes related to config:2) Scrub [...] lag] [...] Has anything changed in this release which could cause this behaviour?
[A] DAEMON_OPTS have been deprecated[1]. This is why fewer command line options occur.
If you still had the vanilla /etc/default/zfs-fuse and /etc/init.d/zfs-fuse files, this means that the traditional 'DAEMON_OPTS=-a 1 -e 1' has been dropped.[B] some fuse caching has been disabled
You could still use it like before, but note it's deprecation (use this for regression test if you want)
As long as you use the proper /etc/zfs/zfsrc settings as well, this should be exactly what you want. /etc/zfs/zfsrc hasn't changed.
Please refer to 'man zfs-fuse' for the reasons behind the deprecation: [2]
due to high-prio bug http://zfs-fuse.net/issues/65 the behaviour of zfs-fuse has changed to be like using --disable-page-cache (which is a total misnomer afaict) at all times. We have (see bug report) tried to benchmark that no significant performance loss occurs, but I admit that I personally have not tried that with scrub.
Can you verify that the performance was better with the 0.6.9-6 package? If so, please file a bug. In that case I can give you a test branch that reverts only this change to see whether that - in isolation - restores performance. I could send you a deb pacakge if you work on i386
maybe a change in the init scripts which mean zfs-fuse is started differently with different options or something?
debian/zfs-fuse.init
debian/zfs-fuse.default
contrib/zfsrc
Hope this helps,
Seth
[1]
bash$ git log --grep=DAEMON_OPTS maint
note that 'testing' contains more work in this area:
bash$ git log --grep=DAEMON_OPTS maint..testing
[2]
REMARKS ON PRECEDENCE
Note that the parameters passed on the command line take precedence over those supplied through /etc/zfs/zfsrc.
BUGS/CAVEATS
The path to the configuration file (/etc/zfs/zfsrc) cannot at this time be configured.
Most existing packages suggest settings can be set at the top of their init script. These get frequently overridden by a (distribution specific) /etc/default/zfs-fuse file, if it exists. Be sure to look at these
places if you want your changes to options to take effect.
The /etc/zfs/zfsrc is going to be the recommended approach in the future. So, packagers, please refrain from passing commandline parameters within the initscript (except for --pid-file).
sgheeren
>
> Also note that the version id 0.6.9-7~1.gbp52ec03 indicates a snapshot
> release. I'm not sure how I can prevent automatic upgrades, but surely
> in debian packaging lingo a snapshot is supposed to be a somewhat less
> official intermediate package.
I think the package is pretty dignifiable right now. I wanted to
sollicit feedback from the debian devs first before deciding on a proper
version. It would be great if we could also get the version tag in
synch, so people could simply drop my PPA and switch to the debian repos
in the future.
$0.02
Gavin Chappell
> experience. It turns out that the debian/ dir in the repo was all but up to
> date. The updated version is 'fast-forwarded' to be more in synch with the
> official debian (universe) package. This involved a lengthy email
> conversation off-list with
> - Mike Hommey
> - Sebastien Delafond
> over the course of the last few weeks.
It wasn't a complaint, I've been generally very happy with zfs-fuse :)
> Also note that the version id 0.6.9-7~1.gbp52ec03 indicates a snapshot
> release. I'm not sure how I can prevent automatic upgrades, but surely in
> debian packaging lingo a snapshot is supposed to be a somewhat less official
> intermediate package.
I think the usual approach is to have two PPAs (a la team-xbmc,
transmission, etc). A lot of projects seem to have a daily build PPA
(build nightly from whatever state the "trunk" (is that the right
term?) is in, if it works it's published, if the build fails, it
isn't), which then feeds into a separate beta PPA somehow for people
who like testing but not broken stuff, and finally into another
separate stable PPA for people who don't like testing but just want
the latest versions.
> Note that the end goal of this is that the debian universe and my ppa
> package get on the same page so the packaging effort can be combined. I hate
> that there are several 'incongruent' packages floating around
I agree, it's made more confusing IMO now that zfs-fuse is in Ubuntu
universe as well as the PPAs, so people might use that version
thinking it's kept up to date when it isn't (to the best of my
knowledge). This could become a particular issue since it's gone into
universe for 10.04 which is a LTS release, people could be running
these systems for up to 5 years on servers by which time the universe
version of zfs-fuse (0.6.0) will be lacking severely in performance
and features.
> If you still had the vanilla /etc/default/zfs-fuse and /etc/init.d/zfs-fuse
> files, this means that the traditional 'DAEMON_OPTS=-a 1 -e 1' has been
> dropped.
I'm 99% sure I did have the defaults although I might have changed
default/zfs-fuse to "restart on upgrade" because I have a couple of
directories under /home (NOT the whole thing) on their own datasets. I
wouldn't have changed DAEMON_OPTS though, not knowing what it did...
> Can you verify that the performance was better with the 0.6.9-6 package? If
> so, please file a bug. In that case I can give you a test branch that
> reverts only this change to see whether that - in isolation - restores
> performance. I could send you a deb pacakge if you work on i386
I do indeed work on i386. The scrub which is currently running is
showing 39 minutes left, I'm at work at the moment so will keep an eye
on it among other things and when it's finished I'll apt purge 0.6.9-7
and download/dpkg install 0.6.9-6 manually (after making sure there's
nothing zfs related running). Then once I'm back to that version, I'll
start another scrub and see what happens to my system with regard to
load and iowait.
> You might want to recheck that you had no
> customizations (although I expect debian to warn before overwriting these on
> upgrade). The relevant upstream repo files are
>
> debian/zfs-fuse.init
> debian/zfs-fuse.default
> contrib/zfsrc
I believe Debian asks you what to do if it comes across a file which
is marked as a config file in the .deb and doesn't match the
distributed checksum (i.e. has been modified after install). If you
opt to keep your changes, your file stays in place and you also get a
*.dpkg-dist file which is the one from the package. If you opt to use
the maintainer's version then your file is moved to *.dpkg-old and the
maintainer's file is moved into place. I've run "find /etc -name
zfs-fuse*dpkg*" which should have showed results if I had either of
those files and there were no matches, so I can only assume that I
hadn't modified the files and they were just overwritten cleanly and
silently with new versions as expected when config files are updated.
sgheeren
> up to date when it isn't (to the best of my
> knowledge).
Gavin Chappell
for any *zfs-fuse* files in /etc, there were none) and started another
scrub.
There's still quite a lot of iowait, however I forgot to mention in my
first post that these are USB2 disks, so that's probably normal
behaviour. The system load is hovering around 3 so far, with -7 it
hovered around 3.7 and peaked over 4. The latency is much improved,
with -7 nothing would happen while I was typing a command into the CLI
and then suddenly it would catch up in a big chunk, rather than, say,
a consistent 1sec lag per character. With -6 I'm able to type in
"real-time" again even though the load is still mid-high and the
iowait is present. I think therefore that the high iowait and high
load are "red herrings" and have always been that way on this system,
and I've just never noticed because I've never had need to check until
-7 and the latency issue.
With -6, as you said, the "-a 1 -e 1" command line is present again,
being specified in the init script and therefore overriding the
default zfsrc.
Hopefully this all means something to you, if you want to provide a
test build then I'm happy to install it and try another scrub, however
each scrub takes ~10 hours on this system and I don't know if it's
possible to cancel a scrub so it may take a bit of back-and-forth...
sgheeren
> Hopefully this all means something to you, if you want to provide a
> test build then I'm happy to install it and try another scrub, however
> each scrub takes ~10 hours on this system and I don't know if it's
> possible to cancel a scrub
-s
Stop scrubbing.
> so it may take a bit of back-and-forth...
>
Well we have two things to go at then, in order of rising plausibility:
(a) reinstate the "-a 1 -e 1" options via /etc/default/zfsrc in the
0.6.9.-7 version (I would be flabberghasted if that made the difference)
(b) retest with the keep_cache hint reverted (issue #65)
Obviously, this will be a test-only version, as we cannot afford to
generally enable the keep_cache thing.
To be honest, I don't expect any relief from both analysis steps at the
moment.
Gavin Chappell
> On 07/01/2010 11:37 AM, Gavin Chappell wrote:
>> Hopefully this all means something to you, if you want to provide a
>> test build then I'm happy to install it and try another scrub, however
>> each scrub takes ~10 hours on this system and I don't know if it's
>> possible to cancel a scrub
> $ man zpool|grep -C3 Stop
>
> -s
>
> Stop scrubbing.
Thanks, I actually never thought of the manpage!
>> so it may take a bit of back-and-forth...
>>
> Well we have two things to go at then, in order of rising plausibility:
>
> (a) reinstate the "-a 1 -e 1" options via /etc/default/zfsrc in the
> 0.6.9.-7 version (I would be flabberghasted if that made the difference)
I've re-upgraded to -7 again, started a scrub to see if the latency
returns, which it did. I've added those options back into to
DAEMON_OPTS in the init script, restarted zfs-fuse and scrubbed again.
Still have the same problem.
sgheeren
(b) retest with the keep_cache hint reverted (issue #65) > > Obviously, this will be a test-only version, as we cannot afford to > generally enable the keep_cache thing.
I posted a branch for testing idea (b); here is my response to the bug
Well we have two things to go at then, in order of rising plausibility:
(a) reinstate the "-a 1 -e 1" options via /etc/default/zfsrc in the
0.6.9.-7 version (I would be flabberghasted if that made the difference)
Obviously, this will be a test-only version, as we cannot afford to
generally enable the keep_cache thing.
moment.
http://gitweb.zfs-fuse.net/?p=sehe;a=shortlog;h=refs/heads/issue68
or here
http://downloads.sehe.nl/zfs-fuse/issue68/
[1] it is more convenient if you do so yourself, since you can sign up first and be automatically subscribed to the bug
I hope you can edit yourself onto the CC list somehow. You _will_ have to register at the site though