Should we simply disallow ZFS on FreeBSD/i386?
Maxim Sobolev
> Perhaps the 7.0 release notes should include a note to the effect that
> ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
By watching this and other threads on ZFS and reading Sun's own design
papers I am getting strong impression that this should be even more
strong than strong NOT RECOMMENDED. Perhaps ZFS should BE DISALLOWED to
run on i386 at all (unless one does some manual source code tweaking or
something like this, and hence can ask no official support from the
project).
I believe that 95% of hardware today that realistically is capable of
running ZFS is also capable of running 64bit code, so that potential ZFS
users are far better off switching to FreeBSD/amd64 and help
testing/improving that architecture than fighting architectural
limitations of already dying i386. And we are as a project are better
off too, by spending out limited resources on something that has future.
From my own experience FreeBSD/amd64 is quite mature for running most
if not all of the server tasks today and ZFS is first and foremost a
server FS. The only place where FreeBSD/i386 beats FreeBSD/amd64 is
desktop, due to binary drivers and such, but ZFS is almost useless
there. So that by simply officially disallowing ZFS on FreeBSD/i386 we
could win by a great margin.
Just my CAD0.02.
-Maxim
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
Adam McDougall
Gary Corcoran wrote:
>> Perhaps the 7.0 release notes should include a note to the effect that
>> ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
By watching this and other threads on ZFS and reading Sun's own design
papers I am getting strong impression that this should be even more strong
than strong NOT RECOMMENDED. Perhaps ZFS should BE DISALLOWED to run on
i386 at all (unless one does some manual source code tweaking or something
like this, and hence can ask no official support from the project).
I feel like stating my opinion on this, noting that I am usually stubborn enough to
squeeze alot of value out of any rough product, while avoiding complaining about
continuing problems if I am not prepared to put in appropriate effort to solve them.
A summary of my opinion on this matter is that some i386 FreeBSD servers do have a
place running zfs in a useful role, but some dedication and patience from the
administrator is usually required, and the effort to tune at least kmem is nearly
required on ALL hardware platforms, not just i386. I think kmem shortages from zfs
are simply more touchy on i386, and with enough ram and slightly more tuning than
amd64 the kmem can most likely be tuned away, but this does not do anything for
other zfs problems such as zil deadlocks and other deadlocks. I think doing
something to prevent FreeBSD/i386 users from using zfs will just rule out a portion
of the people having problems, and admins who take a little time to tune zfs AND use
it more than just lightly may continue to have problems, and will just come back to
the lists.
I have zfs on at least 4 systems presently, each one tuned to where I no longer
receive kmem panics at least based on their expected system load. 2 of them
are i386 and I would be quite dismayed to upgrade RELENG_7 to find ZFS has
been disabled for me (although since I read the mailing lists I would expect
it and deal appropriately). It would be a tradeoff between breaking a limited
amount of existing setups versus somehow limiting the influx of new zfs users
who _may_ encounter zfs problems (of course you will only hear from the people
who complain).
The amount of kmem required for a particular workload on any one machine can vary
alot. Believe it or not, it is one of my AMD64 systems that I had to increase kmem
to 1.6G to prevent kmem panics (it does some heavy nightly rsyncs); versus just
having kmem set to 1G on a i386 system that constantly serves out files to the
internet with various rsyncs running through the day. I don't think its exactly
fair to punish all i386 users by disabling functionality, but I'm not the one that
has to support all the users so I can understand the caution. I'm certainly in
favor of more dire warnings where appropriate. Perhaps even a simple runtime
warning when creating or importing a zpool? I still think problems with running zfs
on i386 will be the tip of that iceberg.
Certainly light use of zfs on an i386 might never see a crash. But I think this
might be a new frontier for a FreeBSD release to include functionality that is
blatently labeled experimental and considered unstable yet will have wide appeal and
interest; almost anyone can use a filesystem on FreeBSD, its not like it is just an
unstable network driver affecting a small portion of users.
I believe that 95% of hardware today that realistically is capable of
running ZFS is also capable of running 64bit code, so that potential ZFS
users are far better off switching to FreeBSD/amd64 and help
testing/improving that architecture than fighting architectural limitations
of already dying i386. And we are as a project are better off too, by
spending out limited resources on something that has future.
One of my busiest zfs servers is a dual Xeon 2.0ghz (i386 only) with 2 gigs
of ram (and I plan to add more 'just cuz'). I don't have any spare amd64
systems I could replace it with at this time, yet I feel the hardware is
up to the job if configured as best I can. I have plenty of spares of this
server type since it is older yet not useless. It pushes out data constantly
to the internet, boots off of zfs, and I believe has never crashed from a
kmem panic thanks to the tuning I have set below, gathered from wiki, mailing
lists, and experience:
vfs.zfs.prefetch_disable="1"
vfs.zfs.arc_max="104857600"
vm.kmem_size_max="1G"
vfs.zfs.zil_disable="1"
I will not pretend it is perfectly stable (it has hung every couple of weeks)
but it is not due to a kmem panic. zil was disabled because it caused deadlocks.
I think what I face now is some other kind of deadlock, which I will try harder
to debug the next time it happens, although I have to balance time spent with
benefit. It has only happened twice since I disabled zil. This server is doing
something useful with zfs yet I can put up with occasional downtime without much
complaint from users. I am using it as a semi-production testbed to kick the tires
on zfs and use the experience to boost my zfs skills and hopefully help out the
whole FreeBSD community with my results when possible.
Erich Dollansky
Maxim Sobolev wrote:
> Gary Corcoran wrote:
>
> I believe that 95% of hardware today that realistically is capable of
I do not think so.
> running ZFS is also capable of running 64bit code, so that potential ZFS
All new hardware since Intel started supporting 64 bits on their
Pentiums is.
> limitations of already dying i386. And we are as a project are better
Let's see it much more practical. Are all features and all ports all the
time supported on all platforms?
I do not think so.
So, just make it a requirement for ZFS to run only on 64 bit upward.
It is not that FreeBSD does not have some kind o file system for older
machines.
Erich
Christian Walther
On 06/01/2008, Maxim Sobolev <sob...@freebsd.org> wrote:
> Gary Corcoran wrote:
> > Perhaps the 7.0 release notes should include a note to the effect that
> > ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
>
[...]
The only place where FreeBSD/i386 beats FreeBSD/amd64 is
> desktop, due to binary drivers and such, but ZFS is almost useless
> there.
I don't think so. My guess is that partitioning/slicing disks is a
pain for most of the users. How does one tell how big a filesystem
should be, even or especially on the desktop? I found myself having to
cope with full filesystems several times.
Using ZFS even on one disk just to get rid of fixed partition/slice
boundaries is a good thing.
Christian
Igor Mozolevsky
> I don't think so. My guess is that partitioning/slicing disks is a
> pain for most of the users. How does one tell how big a filesystem
> should be, even or especially on the desktop? I found myself having to
> cope with full filesystems several times.
That could be a good thing (think programs creating lots of files and
not cleaning up after themselves)...
> Using ZFS even on one disk just to get rid of fixed partition/slice
> boundaries is a good thing.
How is that different to creating one / slice of FFS?
Igor :-)
Christian Walther
On 07/01/2008, Igor Mozolevsky <ig...@hybrid-lab.co.uk> wrote:
> On 07/01/2008, Christian Walther <cpts...@gmail.com> wrote:
>
> > I don't think so. My guess is that partitioning/slicing disks is a
> > pain for most of the users. How does one tell how big a filesystem
> > should be, even or especially on the desktop? I found myself having to
> > cope with full filesystems several times.
>
> That could be a good thing (think programs creating lots of files and
> not cleaning up after themselves)...
>
> > Using ZFS even on one disk just to get rid of fixed partition/slice
> > boundaries is a good thing.
>
> How is that different to creating one / slice of FFS?
With ZFS there aren't fixed boundaries as there are with the
slice/partition theme. You can use reservation and quota to determine
how much free space is guaranteed for a ZFS and the maximum size a ZFS
is allowed to grow to.
If you feel that these boundaries/limitations aren't of any use
anymore, changing/removing them is just a matter of "zfs set...".
> Igor :-)
Christian
Joao Barros
to CURRENT early 2007 on i386 with 1GB of RAM I would definitly say
NO!
Put up warnings, banners and whatever you want but disabling it just
because some users had some panics or just haven't given up time to
tune their system (I'm all in favor of auto tunning here) just doesn't
seem reason enough for me to limit other people's choices.
I've listed it before but again for the record:
i386 Xeon, 1GB RAM
4x320GB RAIDZ with root on zfs
zil enabled, prefetching disabled to improve video play
Shared via NFS and Samba
cat /boot/loader.conf
zfs_load="YES"
vfs.root.mountfrom="zfs:r4x320"
vfs.zfs.prefetch_disable=1
That's it on my loader.conf and for months now I haven't seen a panic.
Why should I or anyone else happilly running ZFS on i386 be denied of
doing so?
On Jan 6, 2008 9:56 PM, Maxim Sobolev <sob...@freebsd.org> wrote:
> Gary Corcoran wrote:
> > Perhaps the 7.0 release notes should include a note to the effect that
> > ZFS is *strongly* NOT RECOMMENDED on 32-bit systems at this time, due
>
> By watching this and other threads on ZFS and reading Sun's own design
> papers I am getting strong impression that this should be even more
> strong than strong NOT RECOMMENDED. Perhaps ZFS should BE DISALLOWED to
> run on i386 at all (unless one does some manual source code tweaking or
> something like this, and hence can ask no official support from the
> project).
>
> I believe that 95% of hardware today that realistically is capable of
> running ZFS is also capable of running 64bit code, so that potential ZFS
> users are far better off switching to FreeBSD/amd64 and help
> testing/improving that architecture than fighting architectural
> limitations of already dying i386. And we are as a project are better
> off too, by spending out limited resources on something that has future.
>
> From my own experience FreeBSD/amd64 is quite mature for running most
> if not all of the server tasks today and ZFS is first and foremost a
> server FS. The only place where FreeBSD/i386 beats FreeBSD/amd64 is
> desktop, due to binary drivers and such, but ZFS is almost useless
> there. So that by simply officially disallowing ZFS on FreeBSD/i386 we
> could win by a great margin.
>
> Just my CAD0.02.
>
> -Maxim
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
>
--
Joao Barros
Bernd Walter
> Adam McDougall wrote:
> >A summary of my opinion on this matter is that some i386 FreeBSD servers
> >do have a place running zfs in a useful role, but some dedication and
> >patience from the administrator is usually required, and the effort to
> >tune at least kmem is nearly
>
> bicycle, but will it ever enjoy it?"
I enjoy my i386/ZFS Servers.
It is running with just 384MB RAM as the only instability it currently
has is because the / disk ist dying.
But even with this it has 71 days uptime.
And my backup server is also based on ZFS with just 196MB RAM.
This one isn't running stable, but it is stable for just doing the zfs
imports and restoring some files.
The only 64 bit machines I have at home are alpha and spac64, so no
option for ZFS right now.
I don't see a real difference between running a 2G i386 and a 2G amd64
server.
8GB amd64 boxes are still not very common.
Of course the i386 must be tuned to have enough kmem and KVA and of
course doing so reduces the application space, but it is still within
the hardware limitations and an NFS-Server doesn't need much application
address space anyway.
Considered that we seem to have a limitation on running amd64 with more
than 2G kmem there is even more to consider.
> The same is here - seemingly due to the ZFS design limitations and
> limitations of the FreeBSD kernel you can't get ZFS to run reliably out
> of the box on i386. Yes, you can probably do some tweaks here and there,
> to make it more of less stable given the workload, but that's not what
> most of the FreeBSD users expect from the file system. Unlike you, most
> of administrators won't even bother to read tweaking documentation
> explaining why ZFS is so tricky in i386, let alone doing actual
> trial-and-error to determine the right set of tunables. More likely at
> the first incident they would just dismiss FreeBSD/ZFS as a crap.
This is true however.
I'm OK with a big fat warning on i386 and/or a loader env to be set
to reenable this for persons who know (or think they do) what they are
doing.
If we say ZFS on i386 isn't supported than at least it shouldn't be
able to be configured without anyone hitting a special knob.
But in my opinion ZFS shouldn't overflow kmem storage at first.
This is not only an i386 problem as rasing kmem makes ZFS more hungry
by default, which is only good if kmem isn't used for something else.
I have an amd64 system with 4G RAM and kmem defaults to 419430400 Bytes.
On my 384MB i386 box I have (tuned of course) 335544320 Bytes kmem.
And with more RAM I could easily go over the default on the amd64 box.
Well RAM is too expensive since it is SDRAM, but there are many DDR
boxes without amd64 functionality out there, which allows adding
affordable memory in the nGB range.
I had to reduce ARC sizes to get the 384MB box stable - the OS version
is quite old and things have been modified for that in the meantime,
but considered that 2G i386 systems still panic with more kmem than
I have RAM it simply says that it wouldn't run out of the box on amd64
with the same applications either.
--
B.Walter http://www.bwct.de http://www.fizon.de
be...@bwct.de in...@bwct.de sup...@fizon.de
Peter Schuller
Content-Type: text/plain;
charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
> That's it on my loader.conf and for months now I haven't seen a panic.
> Why should I or anyone else happilly running ZFS on i386 be denied of
> doing so?
100% agreement.
If you want to go to extremes, require the user to put=20
zfs.zfs.run_on_32_bit_and_i_understand_i_am_an_idiot_and_this_is_not_recomm=
ended=3D1=20
in loader.conf, or else have the kernel panic by design on boot.
But don't make it totally impossible without patching the source, *please*.
Obviously the exception is if development for i386 stops such that it actua=
lly=20
does not work. But disallowing it for artificial reasons... please leave=20
things like that to proprietary hardware/software vendors trying to squeeze=
=20
money of out consumers, and leave it out of a free operation system.
=2D-=20
/ Peter Schuller
PGP userID: 0xE9758B7D or 'Peter Schuller <peter.s...@infidyne.com>'
Key retrieval: Send an E-Mail to getp...@scode.org
E-Mail: peter.s...@infidyne.com Web: http://www.scode.org
--nextPart1781470.KER9S0gyGh
Content-Type: application/pgp-signature; name=signature.asc
Content-Description: This is a digitally signed message part.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)
iD8DBQBHglw4DNor2+l1i30RAt57AJ96qALVP8jZr9LylTA5mBctiFisIQCdESGC
scII2NAPG1uDsdtSxa7kuVY=
=2l94
-----END PGP SIGNATURE-----
--nextPart1781470.KER9S0gyGh--
Alexander Kabaev
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: quoted-printable
On Mon, 7 Jan 2008 18:06:55 +0100
Peter Schuller <peter.s...@infidyne.com> wrote:
> > That's it on my loader.conf and for months now I haven't seen a
> > panic. Why should I or anyone else happilly running ZFS on i386 be
> > denied of doing so?
>=20
> 100% agreement.
>=20
> If you want to go to extremes, require the user to put=20
> zfs.zfs.run_on_32_bit_and_i_understand_i_am_an_idiot_and_this_is_not_reco=
mmended=3D1=20
> in loader.conf, or else have the kernel panic by design on boot.
>=20
> But don't make it totally impossible without patching the source,
> *please*.
>=20
> Obviously the exception is if development for i386 stops such that it
> actually does not work. But disallowing it for artificial reasons...
> please leave things like that to proprietary hardware/software
> vendors trying to squeeze money of out consumers, and leave it out of
> a free operation system.
>=20
Just a note to put another '100% agreement' sign up. We have plenty of
other FSes which are half-cooked and can easily hurt people, but nobody
suggests removing them. Why ZFS should be singled out, it is in a way
better shape than most of them.
--=20
Alexander Kabaev
--Sig_/Eyf8Bt6jEcC9WNhU3WZF3_c
Content-Type: application/pgp-signature; name=signature.asc
Content-Disposition: attachment; filename=signature.asc
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (FreeBSD)
iD8DBQFHgqvtQ6z1jMm+XZYRAs4uAJ9uWJyj1Efpv7MJ8NVcpRyScdM/DACg6+I+
QAfWxADjYpYVn017axq0ePk=
=rwrR
-----END PGP SIGNATURE-----
--Sig_/Eyf8Bt6jEcC9WNhU3WZF3_c--
David Taylor
>
> The amount of kmem required for a particular workload on any one machine can vary
> alot. Believe it or not, it is one of my AMD64 systems that I had to increase kmem
> to 1.6G to prevent kmem panics (it does some heavy nightly rsyncs); versus just
> having kmem set to 1G on a i386 system that constantly serves out files to the
> internet with various rsyncs running through the day.
Note that you're probably running into an integer overflow in arc.c
if vm.kmem_size is set to 1GB or higher on i386. As a result
kstat.zfs.misc.arcstats.size won't grow above
kstat.zfs.misc.arcstats.c_min...
I posted to freebsd-fs about it, but haven't heard anything from pjd, yet.
Sadly, after fixing that problem I started encountering the kmem_map
too small panics.
--
David Taylor
Oliver Fromme
> Maxim Sobolev wrote:
> > Gary Corcoran wrote:
> >
>
Actually I think it's less than 95%.
Of the seven machines I have at home, only one is 64bit
capable -- and that one happens to be a DEC-Alpha which
doesn't support ZFS.
Of the machines at our office room (dunno the count,
must be about a dozen) only one is amd64 capable --
and that one happens to be a workstation that needs
to run 32bit i386 because of X11 graphics support
(and I don't really need to use ZFS on it).
> > running ZFS is also capable of running 64bit code, so that potential ZFS
>
> All new hardware since Intel started supporting 64 bits on their
> Pentiums is.
Nope. There's still hardware produced today that's not
64bit-capable.
FWIW, my NFS server at home is an EPIA PD-10k board with
a VIA C3 processor (32bit only). I chose that one because
of the very low power consumption. It works perfectly
well for my purposes.
> So, just make it a requirement for ZFS to run only on 64 bit upward.
I would certainly vote against such nonsense.
However, I think it does make sense to print a warning
if an admin tries to use ZFS on an i386 machine. It
wouldn't hurt anyway.
It's quite normal that running certain software requires
some tuning so that software will work at all. Typical
examples are squid (uses a lot of sysv message queues)
and PostgreSQL (semaphores) -- they won't run without
tuning, except for trivial setups that don't really do
much. The ZFS tuning issues aren't much different.
Best regards
Oliver
--
Oliver Fromme, secnetix GmbH & Co. KG, Marktplatz 29, 85567 Grafing b. M.
Handelsregister: Registergericht Muenchen, HRA 74606, Geschäftsfuehrung:
secnetix Verwaltungsgesellsch. mbH, Handelsregister: Registergericht Mün-
chen, HRB 125758, Geschäftsführer: Maik Bachmann, Olaf Erb, Ralf Gebhart
FreeBSD-Dienstleistungen, -Produkte und mehr: http://www.secnetix.de/bsd
"That's what I love about GUIs: They make simple tasks easier,
and complex tasks impossible."
-- John William Chambless