Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Re: ZFS committed to the FreeBSD base.

33 views
Skip to first unread message

Kris Kennaway

unread,
Apr 5, 2007, 11:07:50 PM4/5/07
to
On Fri, Apr 06, 2007 at 04:57:00AM +0200, Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.
>
> Commit log:
>
> Please welcome ZFS - The last word in file systems.
>
> ZFS file system was ported from OpenSolaris operating system. The code
> in under CDDL license.
>
> I'd like to thank all SUN developers that created this great piece of
> software.
>
> Supported by: Wheel LTD (http://www.wheel.pl/)
> Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/)
> Supported by: Sentex (http://www.sentex.net/)
>
> Limitations.
>
> Currently ZFS is only compiled as kernel module and is only available
> for i386 architecture. Amd64 should be available very soon, the other
> archs will come later, as we implement needed atomic operations.
>
> Missing functionality.
>
> - We don't have iSCSI target daemon in the tree, so sharing ZVOLs via
> iSCSI is also not supported at this point. This should be fixed in
> the future, we may also add support for sharing ZVOLs over ggate.
> - There is no support for ACLs and extended attributes.
> - There is no support for booting off of ZFS file system.
>
> Other than that, ZFS should be fully-functional.
>
> Enjoy!

Give yourself a pat on the back :)

Kris

Sean Bryant

unread,
Apr 5, 2007, 11:21:08 PM4/5/07
to
Is it fully 128bit? From wikipedia, which is by no means an
authoritative source but I have no idea if this was ever an issue.
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"

Juha Saarinen

unread,
Apr 5, 2007, 11:42:51 PM4/5/07
to
On 4/6/07, Kris Kennaway <kr...@obsecurity.org> wrote:
> > Please welcome ZFS - The last word in file systems.

> Give yourself a pat on the back :)

Seconded.


--
Juha
http://www.geekzone.co.nz/juha

Eric Anderson

unread,
Apr 6, 2007, 1:22:14 AM4/6/07
to
On 04/05/07 21:57, Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.


Pawel - you're a madman! :)

I'm afraid of what your next project will be.

Thanks for the solid work (again..),
Eric

Alex Dupre

unread,
Apr 6, 2007, 3:26:34 AM4/6/07
to
Pawel Jakub Dawidek wrote:
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system.

Congratulations! You're great!

> - There is no support for booting off of ZFS file system.

Even booting kernel from a removable ufs media and then mounting a zfs
root via vfs.root.mountfrom?

--
Alex Dupre

Ivan Voras

unread,
Apr 6, 2007, 4:36:52 AM4/6/07
to
Sean Bryant wrote:

> Is it fully 128bit? From wikipedia, which is by no means an
> authoritative source but I have no idea if this was ever an issue.

It's 64-bit even in Solaris. The "128-bitness" is only in the storage
format, not for file system ops visible to applications.

(AFAIK).

signature.asc

Robert Watson

unread,
Apr 6, 2007, 5:28:34 AM4/6/07
to

On Fri, 6 Apr 2007, Alex Dupre wrote:

> Pawel Jakub Dawidek wrote:
>> I'm happy to inform that the ZFS file system is now part of the FreeBSD
>> operating system.
>
> Congratulations! You're great!
>
>> - There is no support for booting off of ZFS file system.
>
> Even booting kernel from a removable ufs media and then mounting a zfs root
> via vfs.root.mountfrom?

I believe the key issue here is that the boot loader doesn't yet support ZFS.
In 6.x and 7.x, the mechanism for mounting the root file system is identical
to all other file systems, so it should be possible to use any file system as
the root file system as long as you get can get the kernel up and running.
And, in the case of ZFS, the ZFS module loaded (since it currently must be a
module).

This is really exciting work and I'm very glad to see this in the tree!

Robert N M Watson
Computer Laboratory
University of Cambridge

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 6:40:04 AM4/6/07
to

That's correct. We are limited by POSIX, but the on-disk format is
128bit.

--
Pawel Jakub Dawidek http://www.wheel.pl
p...@FreeBSD.org http://www.FreeBSD.org
FreeBSD committer Am I Evil? Yes, I Am!

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 7:29:11 AM4/6/07
to
On Fri, Apr 06, 2007 at 05:54:37AM +0100, Ricardo Correia wrote:
> I'm interested in the cross-platform portability of ZFS pools, so I have
> one question: did you implement the Solaris ZFS whole-disk support
> (specifically, the creation and recognition of the EFI/GPT label)?
>
> Unfortunately some tools in Linux (parted and cfdisk) have trouble
> recognizing the EFI partition created by ZFS/Solaris..

I'm not yet setup to move disks between FreeBSD and Solaris, but my
first goal was to integrate it with FreeBSD's GEOM framework.

We support cache flushing operations on any GEOM provider (disk,
partition, slice, anything disk-like), so bascially currently I treat
everything as a whole disk (because I simply can), but don't do any
EFI/GPT labeling. I'll try to move data from Solaris' disk to FreeBSD
and see what happen.

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 7:55:57 AM4/6/07
to
On Fri, Apr 06, 2007 at 05:28:34AM -0400, Robert Watson wrote:
>
> On Fri, 6 Apr 2007, Alex Dupre wrote:
>
> >Pawel Jakub Dawidek wrote:
> >>I'm happy to inform that the ZFS file system is now part of the FreeBSD
> >>operating system.
> >
> >Congratulations! You're great!
> >
> >> - There is no support for booting off of ZFS file system.
> >
> >Even booting kernel from a removable ufs media and then mounting a zfs root via vfs.root.mountfrom?
>
> I believe the key issue here is that the boot loader doesn't yet support ZFS. In 6.x and 7.x, the mechanism for mounting the root file system is identical to all other file
> systems, so it should be possible to use any file system as the root file system as long as you get can get the kernel up and running. And, in the case of ZFS, the ZFS
> module loaded (since it currently must be a module).

You are right in general, but it isn't really true for ZFS currently.
There are two very small issues:

1. Prefered way to mount ZFS file system is via 'zfs mount' command, but
it can be mounted using old way as well, so this really shouldn't be an
issue.

2. ZFS kernel module read /etc/zfs/zpool.cache file on load by accessing
it via file system. We would need to change it to load this file via
loader. Shouldn't be hard.

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 8:34:47 AM4/6/07
to
On Fri, Apr 06, 2007 at 01:29:11PM +0200, Pawel Jakub Dawidek wrote:
> On Fri, Apr 06, 2007 at 05:54:37AM +0100, Ricardo Correia wrote:
> > I'm interested in the cross-platform portability of ZFS pools, so I have
> > one question: did you implement the Solaris ZFS whole-disk support
> > (specifically, the creation and recognition of the EFI/GPT label)?
> >
> > Unfortunately some tools in Linux (parted and cfdisk) have trouble
> > recognizing the EFI partition created by ZFS/Solaris..
>
> I'm not yet setup to move disks between FreeBSD and Solaris, but my
> first goal was to integrate it with FreeBSD's GEOM framework.
>
> We support cache flushing operations on any GEOM provider (disk,
> partition, slice, anything disk-like), so bascially currently I treat
> everything as a whole disk (because I simply can), but don't do any
> EFI/GPT labeling. I'll try to move data from Solaris' disk to FreeBSD
> and see what happen.

First try:

GEOM: ad6: corrupt or invalid GPT detected.
GEOM: ad6: GPT rejected -- may not be recoverable.

:)

Roman Divacky

unread,
Apr 6, 2007, 11:17:53 AM4/6/07
to
On Fri, Apr 06, 2007 at 04:57:00AM +0200, Pawel Jakub Dawidek wrote:
> Hi.

>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.
>
> Commit log:

>
> Please welcome ZFS - The last word in file systems.

this is incredibly great! thnx..

do you have any benchmark numbers? I saw some in your *con paper
but since then we got new sx locks and you did some performance
improvements as well..

I am just curious :)

thnx again!

roman

Sean Bryant

unread,
Apr 6, 2007, 12:14:07 PM4/6/07
to
Pawel Jakub Dawidek wrote:
> On Fri, Apr 06, 2007 at 10:36:52AM +0200, Ivan Voras wrote:
>
>> Sean Bryant wrote:
>>
>>
>>> Is it fully 128bit? From wikipedia, which is by no means an authoritative source but I have no idea if this was ever an issue.
>>>
>> It's 64-bit even in Solaris. The "128-bitness" is only in the storage format, not for file system ops visible to applications.
>>
>> (AFAIK).
>>
>
> That's correct. We are limited by POSIX, but the on-disk format is
> 128bit.
>
>
Thanks for the update,
I'll probably update that Wikipedia entry to reflect recent changes and
more correctly state the limitations.

Johan Hendriks

unread,
Apr 6, 2007, 2:37:34 PM4/6/07
to

Great stuff.

Does it also needs mentioning in /boot/defaults/loader.conf to load the zfs module.

regards,
Johan

Bruce M. Simpson

unread,
Apr 6, 2007, 5:09:56 PM4/6/07
to
This is most excellent work which is going to help everyone in a very
big way. Many thanks for working on this.

Ceri Davies

unread,
Apr 6, 2007, 5:07:06 PM4/6/07
to
On Thu, Apr 05, 2007 at 09:58:47PM -0700, Rich Teer wrote:
> > I'm happy to inform that the ZFS file system is now part of the FreeBSD
> > operating system. ZFS is available in the HEAD branch and will be
> > available in FreeBSD 7.0-RELEASE as an experimental feature.
>
> This is fantastic news! At the risk of raking over ye olde arguments,
> as the old saying goes: "Dual licensing? We don't need no stinkeen
> dual licensing!". :-)

Actually, you might want to run that statement by a certain John Birrell
(j...@FreeBSD.org) regarding the DTrace port and see what answer you get.

Ceri
--
That must be wonderful! I don't understand it at all.
-- Moliere

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 5:48:04 PM4/6/07
to
On Fri, Apr 06, 2007 at 09:26:34AM +0200, Alex Dupre wrote:
> Pawel Jakub Dawidek wrote:
> > I'm happy to inform that the ZFS file system is now part of the FreeBSD
> > operating system.
>
> Congratulations! You're great!
>
> > - There is no support for booting off of ZFS file system.
>
> Even booting kernel from a removable ufs media and then mounting a zfs
> root via vfs.root.mountfrom?

I just verified that this will be possible:

# mount
tank on / (zfs, local)
devfs on /dev (devfs, local)

but I need some time to implement it right.

Gabor Kovesdan

unread,
Apr 6, 2007, 5:52:07 PM4/6/07
to
Ceri Davies schrieb:

> On Thu, Apr 05, 2007 at 09:58:47PM -0700, Rich Teer wrote:
>
>>> I'm happy to inform that the ZFS file system is now part of the FreeBSD
>>> operating system. ZFS is available in the HEAD branch and will be
>>> available in FreeBSD 7.0-RELEASE as an experimental feature.
>>>
>> This is fantastic news! At the risk of raking over ye olde arguments,
>> as the old saying goes: "Dual licensing? We don't need no stinkeen
>> dual licensing!". :-)
>>
>
> Actually, you might want to run that statement by a certain John Birrell
> (j...@FreeBSD.org) regarding the DTrace port and see what answer you get.
>
>
jhb@ is John Baldwin, John Birrel is jb@! :)

Regards,
Gabor

Bruno Damour

unread,
Apr 6, 2007, 7:06:26 PM4/6/07
to
Thanks, fantasticly interesting !

> Currently ZFS is only compiled as kernel module and is only available
> for i386 architecture. Amd64 should be available very soon, the other
> archs will come later, as we implement needed atomic operations.
>
I'm waiting eagerly to amd64 version....

> Missing functionality.
>
> - There is no support for ACLs and extended attributes.
>
Is this planned ? Does that means I cannot use it as a basis for a
full-featured samba share ?

Thanks for your great work !!

Bruno DAMOUR

Pawel Jakub Dawidek

unread,
Apr 6, 2007, 8:57:05 PM4/6/07
to
On Sat, Apr 07, 2007 at 12:39:14AM +0200, Bruno Damour wrote:
> Thanks, fantasticly interesting !
> > Currently ZFS is only compiled as kernel module and is only available
> > for i386 architecture. Amd64 should be available very soon, the other
> > archs will come later, as we implement needed atomic operations.
> >
> I'm waiting eagerly to amd64 version....
>
> >Missing functionality.
> >
> > - There is no support for ACLs and extended attributes.
> >
> Is this planned ? Does that means I cannot use it as a basis for a full-featured samba share ?

It is planned, but it's not trivial. Does samba support NFSv4-style
ACLs?

Bernd Walter

unread,
Apr 6, 2007, 10:56:45 PM4/6/07
to
On Fri, Apr 06, 2007 at 04:57:00AM +0200, Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.

I got a kmem panic just by copying a recent ports.tgz (36M) onto a ZFS.
My sandbox just has 128MB RAM so kmem was set to ~40M.
After raising kmem to 80M it survived copying the file, but paniced
again while tar -xvzf the file into the same pool.
vfs.zfs.vdev.cache.size is unchanged at 10M.

--
B.Walter http://www.bwct.de http://www.fizon.de
be...@bwct.de in...@bwct.de sup...@fizon.de

Alex Dupre

unread,
Apr 7, 2007, 3:39:44 AM4/7/07
to
Pawel Jakub Dawidek wrote:
> I just verified that this will be possible:
>
> # mount
> tank on / (zfs, local)
> devfs on /dev (devfs, local)
>
> but I need some time to implement it right.

I waited months for current ZFS implementation, I can wait more for root
support, now that I know it'll be possible :-) Thanks again.

Randall Stewart

unread,
Apr 7, 2007, 6:39:22 AM4/7/07
to
Great work Pawel...

I see you posted a quick start ... I will have
to move my laptop to use this as its non-root fs's :-D

R

Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.
>

> Commit log:
>
> Please welcome ZFS - The last word in file systems.
>

> ZFS file system was ported from OpenSolaris operating system. The code
> in under CDDL license.
>
> I'd like to thank all SUN developers that created this great piece of
> software.
>
> Supported by: Wheel LTD (http://www.wheel.pl/)
> Supported by: The FreeBSD Foundation (http://www.freebsdfoundation.org/)
> Supported by: Sentex (http://www.sentex.net/)
>
> Limitations.
>

> Currently ZFS is only compiled as kernel module and is only available
> for i386 architecture. Amd64 should be available very soon, the other
> archs will come later, as we implement needed atomic operations.
>

> Missing functionality.
>
> - We don't have iSCSI target daemon in the tree, so sharing ZVOLs via
> iSCSI is also not supported at this point. This should be fixed in
> the future, we may also add support for sharing ZVOLs over ggate.

> - There is no support for ACLs and extended attributes.

> - There is no support for booting off of ZFS file system.
>

> Other than that, ZFS should be fully-functional.
>
> Enjoy!
>


--
Randall Stewart
NSSTG - Cisco Systems Inc.
803-345-0369 <or> 803-317-4952 (cell)

Jorn Argelo

unread,
Apr 7, 2007, 6:54:57 AM4/7/07
to
Rich Teer wrote:
>> I'm happy to inform that the ZFS file system is now part of the FreeBSD
>> operating system. ZFS is available in the HEAD branch and will be
>> available in FreeBSD 7.0-RELEASE as an experimental feature.
>>
>
> This is fantastic news! At the risk of raking over ye olde arguments,
> as the old saying goes: "Dual licensing? We don't need no stinkeen
> dual licensing!". :-)
>
>
First of all, thanks a lot for all the hard work of both the FreeBSD
developers as the ZFS developers. I can't wait to give it a go.

That leads me to one question though: Why is *BSD able to bring it into
the OS as where Linux has licensing problems with the CDDL? AFAIK Linux
users can only run it in userland mode and not in kernel mode because of
the licenses.

I don't really know the differences between all the licenses, so feel
free to correct me if I'm saying something stupid.

Thanks,

Jorn

Florent Thoumie

unread,
Apr 7, 2007, 8:15:00 AM4/7/07
to
Pawel Jakub Dawidek wrote:
> Hi.
>
> I'm happy to inform that the ZFS file system is now part of the FreeBSD
> operating system. ZFS is available in the HEAD branch and will be
> available in FreeBSD 7.0-RELEASE as an experimental feature.

Thanks for working on it Pawel!

We're now all waiting for 7.0-RELEASE :-)

--
Florent Thoumie
f...@FreeBSD.org
FreeBSD Committer

signature.asc

Pawel Jakub Dawidek

unread,
Apr 7, 2007, 9:13:53 AM4/7/07
to
On Sat, Apr 07, 2007 at 04:56:45AM +0200, Bernd Walter wrote:
> On Fri, Apr 06, 2007 at 04:57:00AM +0200, Pawel Jakub Dawidek wrote:
> > Hi.
> >
> > I'm happy to inform that the ZFS file system is now part of the FreeBSD
> > operating system. ZFS is available in the HEAD branch and will be
> > available in FreeBSD 7.0-RELEASE as an experimental feature.
>
> I got a kmem panic just by copying a recent ports.tgz (36M) onto a ZFS.
> My sandbox just has 128MB RAM so kmem was set to ~40M.
> After raising kmem to 80M it survived copying the file, but paniced
> again while tar -xvzf the file into the same pool.
> vfs.zfs.vdev.cache.size is unchanged at 10M.

128MB RAM of suggested minimum in ZFS requirements, but it may be not
enough... Minimum of ARC is set to 1/8 of all memory or 64MB (whichever
is more). Could you locate these lines in
sys/contrib/opensolaris/uts/common/fs/zfs/arc.c file:

/* set min cache to 1/32 of all memory, or 64MB, whichever is more */
arc_c_min = MAX(arc_c / 4, 64<<20);

Change 64 to eg. 32, recompile and retest?

Wilko Bulte

unread,
Apr 7, 2007, 10:17:37 AM4/7/07
to
On Sat, Apr 07, 2007 at 12:54:57PM +0200, Jorn Argelo wrote..

> Rich Teer wrote:
> >>I'm happy to inform that the ZFS file system is now part of the FreeBSD
> >>operating system. ZFS is available in the HEAD branch and will be
> >>available in FreeBSD 7.0-RELEASE as an experimental feature.
> >>
> >
> >This is fantastic news! At the risk of raking over ye olde arguments,
> >as the old saying goes: "Dual licensing? We don't need no stinkeen
> >dual licensing!". :-)
> >
> >
> First of all, thanks a lot for all the hard work of both the FreeBSD
> developers as the ZFS developers. I can't wait to give it a go.
>
> That leads me to one question though: Why is *BSD able to bring it into
> the OS as where Linux has licensing problems with the CDDL? AFAIK Linux
> users can only run it in userland mode and not in kernel mode because of
> the licenses.

My guess(!) is that they do not want non-GPL-ed code in the standard kernel.

--
Wilko Bulte wi...@FreeBSD.org

Florian C. Smeets

unread,
Apr 7, 2007, 9:59:02 AM4/7/07
to
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pawel Jakub Dawidek wrote:
> On Sat, Apr 07, 2007 at 04:56:45AM +0200, Bernd Walter wrote:
>> My sandbox just has 128MB RAM so kmem was set to ~40M.
>> After raising kmem to 80M it survived copying the file, but paniced
>> again while tar -xvzf the file into the same pool.
>> vfs.zfs.vdev.cache.size is unchanged at 10M.
>
> 128MB RAM of suggested minimum in ZFS requirements, but it may be not
> enough... Minimum of ARC is set to 1/8 of all memory or 64MB (whichever
> is more). Could you locate these lines in
> sys/contrib/opensolaris/uts/common/fs/zfs/arc.c file:
>
> /* set min cache to 1/32 of all memory, or 64MB, whichever is more */
> arc_c_min = MAX(arc_c / 4, 64<<20);
>
> Change 64 to eg. 32, recompile and retest?
>

Hi Pawel,

i had the same problems like Bernd while trying to copy the src tree to
a ZFS volume. I have 384MB RAM but i got the same "kmem_map: too small"
panic. I compiled my kernel like you proposed and now i am able to copy
anything to the volume without panic :-)

Regards,
Florian

P.S. Thanks for all the work on ZFS!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (Darwin)

iD8DBQFGF6OmA+1tjUZ1YScRAnSMAJ4y27u0nGu9L4RgDBclxKh5q6Z/RgCgjbi7
1Ri2CZfH8YKqj8Bdmx7bedM=
=PUsh
-----END PGP SIGNATURE-----

Bernd Walter

unread,
Apr 7, 2007, 2:03:19 PM4/7/07
to
On Sat, Apr 07, 2007 at 06:58:00PM +0200, Bernd Walter wrote:

> On Sat, Apr 07, 2007 at 03:59:02PM +0200, Florian C. Smeets wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Pawel Jakub Dawidek wrote:
> > > On Sat, Apr 07, 2007 at 04:56:45AM +0200, Bernd Walter wrote:
> > >> My sandbox just has 128MB RAM so kmem was set to ~40M.
> > >> After raising kmem to 80M it survived copying the file, but paniced
> > >> again while tar -xvzf the file into the same pool.
> > >> vfs.zfs.vdev.cache.size is unchanged at 10M.
> > >
> > > 128MB RAM of suggested minimum in ZFS requirements, but it may be not
> > > enough... Minimum of ARC is set to 1/8 of all memory or 64MB (whichever
> > > is more). Could you locate these lines in
> > > sys/contrib/opensolaris/uts/common/fs/zfs/arc.c file:
> > >
> > > /* set min cache to 1/32 of all memory, or 64MB, whichever is more */
> > > arc_c_min = MAX(arc_c / 4, 64<<20);
> > >
> > > Change 64 to eg. 32, recompile and retest?
> > >
> >
> > Hi Pawel,
> >
> > i had the same problems like Bernd while trying to copy the src tree to
> > a ZFS volume. I have 384MB RAM but i got the same "kmem_map: too small"
> > panic. I compiled my kernel like you proposed and now i am able to copy
> > anything to the volume without panic :-)
>
> I had increased RAM to 384 and still had a panic with default kmem
> (IIRC around 100M) and even increasing kmem to 160M did help a long
> time, but still produced the panic after a while.
> I don't think 64M applies here as the real limit.

Now with 240M kmem it looks good, but I'm still unshure:
kstat.zfs.misc.arcstats.c_min: 67108864
kstat.zfs.misc.arcstats.c_max: 188743680
kstat.zfs.misc.arcstats.size: 87653376
c_max seemed to be increasing with kmem, but I did compare it with a
remebered value.
Should be good with:
vm.kmem_size: 251658240
But top shows wired memory which is roughly twice the size of
arcstats.size, so I'm still worried about kmem exhaustion if ARC runs
up to c_max.
Since the c_min/c_max values also influence the available RAM for other
purposes as well, can we have it at least a loader.conf tuneable?

Otherwise - the reboot after the panics where impressive.
No long fsck times or noticed data corruption - even with NFS clients.
All in all it is a great job.

Pawel Jakub Dawidek

unread,
Apr 7, 2007, 3:15:17 PM4/7/07
to
On Sat, Apr 07, 2007 at 08:03:19PM +0200, Bernd Walter wrote:
> Now with 240M kmem it looks good, but I'm still unshure:
> kstat.zfs.misc.arcstats.c_min: 67108864
> kstat.zfs.misc.arcstats.c_max: 188743680
> kstat.zfs.misc.arcstats.size: 87653376
> c_max seemed to be increasing with kmem, but I did compare it with a
> remebered value.
> Should be good with:
> vm.kmem_size: 251658240
> But top shows wired memory which is roughly twice the size of
> arcstats.size, so I'm still worried about kmem exhaustion if ARC runs
> up to c_max.
> Since the c_min/c_max values also influence the available RAM for other
> purposes as well, can we have it at least a loader.conf tuneable?

Just committed a change. You can tune max and min ARC size via
vfs.zfs.arc_max and vfs.zfs.arc_min tunnables.

Dag-Erling Smørgrav

unread,
Apr 7, 2007, 3:43:59 PM4/7/07
to
Pawel Jakub Dawidek <p...@FreeBSD.org> writes:
> Limitations.
>
> Currently ZFS is only compiled as kernel module and is only available
> for i386 architecture. Amd64 should be available very soon, the other
> archs will come later, as we implement needed atomic operations.

ZFS is now also available on pc98 and amd64.

DES
--
Dag-Erling Smørgrav - d...@des.no

Bernd Walter

unread,
Apr 7, 2007, 4:34:12 PM4/7/07
to
On Sat, Apr 07, 2007 at 09:43:59PM +0200, Dag-Erling Smørgrav wrote:
> Pawel Jakub Dawidek <p...@FreeBSD.org> writes:
> > Limitations.
> >
> > Currently ZFS is only compiled as kernel module and is only available
> > for i386 architecture. Amd64 should be available very soon, the other
> > archs will come later, as we implement needed atomic operations.
>
> ZFS is now also available on pc98 and amd64.

Great to read - is it just atomic.S missing for the remaining
architectures?

Dag-Erling Smørgrav

unread,
Apr 7, 2007, 5:16:12 PM4/7/07
to
Bernd Walter <ti...@cicely12.cicely.de> writes:
> On Sat, Apr 07, 2007 at 09:43:59PM +0200, Dag-Erling Smørgrav wrote:
> > ZFS is now also available on pc98 and amd64.
> Great to read - is it just atomic.S missing for the remaining
> architectures?

Yes. Ideally, ZFS would use FreeBSD's atomic operations instead of
its own. I believe that the reason it doesn't is (at least in part)
that we don't have 64-bit atomic operations for i386. I have
unfinished patches for cleaning up the atomic operations on all
platforms; I'll dust them off and see what I can do.

DES
--
Dag-Erling Smørgrav - d...@des.no

Bernd Walter

unread,
Apr 7, 2007, 5:24:14 PM4/7/07
to
On Sat, Apr 07, 2007 at 09:15:17PM +0200, Pawel Jakub Dawidek wrote:
> On Sat, Apr 07, 2007 at 08:03:19PM +0200, Bernd Walter wrote:
> > Now with 240M kmem it looks good, but I'm still unshure:
> > kstat.zfs.misc.arcstats.c_min: 67108864
> > kstat.zfs.misc.arcstats.c_max: 188743680
> > kstat.zfs.misc.arcstats.size: 87653376
> > c_max seemed to be increasing with kmem, but I did compare it with a
> > remebered value.
> > Should be good with:
> > vm.kmem_size: 251658240
> > But top shows wired memory which is roughly twice the size of
> > arcstats.size, so I'm still worried about kmem exhaustion if ARC runs
> > up to c_max.
> > Since the c_min/c_max values also influence the available RAM for other
> > purposes as well, can we have it at least a loader.conf tuneable?
>
> Just committed a change. You can tune max and min ARC size via
> vfs.zfs.arc_max and vfs.zfs.arc_min tunnables.

Thanks - I'd set c_max to 80M now and will see what happens, since
I had such a panic again with 240M kmem.

I'm a bit confused about the calculation as such.
Lets asume a 4G i386 system.
arg_c = 512M
c_min = 512M
c_max = 3G
But isn't this KVA space, of which we usually can't have 3G on i386
without limiting userland to 1G?
Even 512M KVA sounds very much on a i386, since 4G systems usually
have more use for limited KVA.

Bruno Damour

unread,
Apr 8, 2007, 2:03:11 AM4/8/07
to
hello,

After csup, buildworld fails for me in libumem.
Is this due to zfs import ?
Or my config ?

Thanks for any clue, i'm dying to try your brand new zfs on amd64 !!

Bruno

FreeBSD vil1.ruomad.net 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Fri Mar 23
07:33:56 CET 2007 ro...@vil1.ruomad.net:/usr/obj/usr/src/sys/VIL1 amd64

make buildworld:

===> cddl/lib/libumem (all)
cc -O2 -fno-strict-aliasing -pipe -march=nocona
-I/usr/src/cddl/lib/libumem/../../../compat/opensolaris/lib/libumem
-D_SOLARIS_C_SOURCE -c /usr/src/cddl/lib/libumem/umem.c
/usr/src/cddl/lib/libumem/umem.c:197: error: redefinition of 'nofail_cb'
/usr/src/cddl/lib/libumem/umem.c:30: error: previous definition of
'nofail_cb' was here
/usr/src/cddl/lib/libumem/umem.c:199: error: redefinition of `struct
umem_cache'
/usr/src/cddl/lib/libumem/umem.c:210: error: redefinition of 'umem_alloc'
/usr/src/cddl/lib/libumem/umem.c:43: error: previous definition of
'umem_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:233: error: redefinition of 'umem_zalloc'
/usr/src/cddl/lib/libumem/umem.c:66: error: previous definition of
'umem_zalloc' was here
/usr/src/cddl/lib/libumem/umem.c:256: error: redefinition of 'umem_free'
/usr/src/cddl/lib/libumem/umem.c:89: error: previous definition of
'umem_free' was here
/usr/src/cddl/lib/libumem/umem.c:264: error: redefinition of
'umem_nofail_callback'
/usr/src/cddl/lib/libumem/umem.c:97: error: previous definition of
'umem_nofail_callback' was here
/usr/src/cddl/lib/libumem/umem.c:272: error: redefinition of
'umem_cache_create'
/usr/src/cddl/lib/libumem/umem.c:105: error: previous definition of
'umem_cache_create' was here
/usr/src/cddl/lib/libumem/umem.c:291: error: redefinition of
'umem_cache_alloc'
/usr/src/cddl/lib/libumem/umem.c:124: error: previous definition of
'umem_cache_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:321: error: redefinition of
'umem_cache_free'
/usr/src/cddl/lib/libumem/umem.c:154: error: previous definition of
'umem_cache_free' was here
/usr/src/cddl/lib/libumem/umem.c:332: error: redefinition of
'umem_cache_destroy'
/usr/src/cddl/lib/libumem/umem.c:165: error: previous definition of
'umem_cache_destroy' was here
/usr/src/cddl/lib/libumem/umem.c:364: error: redefinition of 'nofail_cb'
/usr/src/cddl/lib/libumem/umem.c:197: error: previous definition of
'nofail_cb' was here
/usr/src/cddl/lib/libumem/umem.c:364: error: redefinition of 'nofail_cb'
/usr/src/cddl/lib/libumem/umem.c:197: error: previous definition of
'nofail_cb' was here
/usr/src/cddl/lib/libumem/umem.c:366: error: redefinition of `struct
umem_cache'
/usr/src/cddl/lib/libumem/umem.c:377: error: redefinition of 'umem_alloc'
/usr/src/cddl/lib/libumem/umem.c:210: error: previous definition of
'umem_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:377: error: redefinition of 'umem_alloc'
/usr/src/cddl/lib/libumem/umem.c:210: error: previous definition of
'umem_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:400: error: redefinition of 'umem_zalloc'
/usr/src/cddl/lib/libumem/umem.c:233: error: previous definition of
'umem_zalloc' was here
/usr/src/cddl/lib/libumem/umem.c:400: error: redefinition of 'umem_zalloc'
/usr/src/cddl/lib/libumem/umem.c:233: error: previous definition of
'umem_zalloc' was here
/usr/src/cddl/lib/libumem/umem.c:423: error: redefinition of 'umem_free'
/usr/src/cddl/lib/libumem/umem.c:256: error: previous definition of
'umem_free' was here
/usr/src/cddl/lib/libumem/umem.c:423: error: redefinition of 'umem_free'
/usr/src/cddl/lib/libumem/umem.c:256: error: previous definition of
'umem_free' was here
/usr/src/cddl/lib/libumem/umem.c:431: error: redefinition of
'umem_nofail_callback'
/usr/src/cddl/lib/libumem/umem.c:264: error: previous definition of
'umem_nofail_callback' was here
/usr/src/cddl/lib/libumem/umem.c:431: error: redefinition of
'umem_nofail_callback'
/usr/src/cddl/lib/libumem/umem.c:264: error: previous definition of
'umem_nofail_callback' was here
/usr/src/cddl/lib/libumem/umem.c:439: error: redefinition of
'umem_cache_create'
/usr/src/cddl/lib/libumem/umem.c:272: error: previous definition of
'umem_cache_create' was here
/usr/src/cddl/lib/libumem/umem.c:439: error: redefinition of
'umem_cache_create'
/usr/src/cddl/lib/libumem/umem.c:272: error: previous definition of
'umem_cache_create' was here
/usr/src/cddl/lib/libumem/umem.c:458: error: redefinition of
'umem_cache_alloc'
/usr/src/cddl/lib/libumem/umem.c:291: error: previous definition of
'umem_cache_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:458: error: redefinition of
'umem_cache_alloc'
/usr/src/cddl/lib/libumem/umem.c:291: error: previous definition of
'umem_cache_alloc' was here
/usr/src/cddl/lib/libumem/umem.c:488: error: redefinition of
'umem_cache_free'
/usr/src/cddl/lib/libumem/umem.c:321: error: previous definition of
'umem_cache_free' was here
/usr/src/cddl/lib/libumem/umem.c:488: error: redefinition of
'umem_cache_free'
/usr/src/cddl/lib/libumem/umem.c:321: error: previous definition of
'umem_cache_free' was here
/usr/src/cddl/lib/libumem/umem.c:499: error: redefinition of
'umem_cache_destroy'
/usr/src/cddl/lib/libumem/umem.c:332: error: previous definition of
'umem_cache_destroy' was here
/usr/src/cddl/lib/libumem/umem.c:499: error: redefinition of
'umem_cache_destroy'
/usr/src/cddl/lib/libumem/umem.c:332: error: previous definition of
'umem_cache_destroy' was here
*** Error code 1

Stop in /usr/src/cddl/lib/libumem.
*** Error code 1

Stop in /usr/src/cddl/lib.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.
*** Error code 1

Stop in /usr/src.

Pawel Jakub Dawidek

unread,
Apr 8, 2007, 5:49:31 AM4/8/07
to
On Sun, Apr 08, 2007 at 08:03:11AM +0200, Bruno Damour wrote:
> hello,
>
> After csup, buildworld fails for me in libumem.
> Is this due to zfs import ?
> Or my config ?
>
> Thanks for any clue, i'm dying to try your brand new zfs on amd64 !!
>
> Bruno
>
> FreeBSD vil1.ruomad.net 7.0-CURRENT FreeBSD 7.0-CURRENT #0: Fri Mar 23 07:33:56 CET 2007 ro...@vil1.ruomad.net:/usr/obj/usr/src/sys/VIL1 amd64
>
> make buildworld:
>
> ===> cddl/lib/libumem (all)
> cc -O2 -fno-strict-aliasing -pipe -march=nocona -I/usr/src/cddl/lib/libumem/../../../compat/opensolaris/lib/libumem -D_SOLARIS_C_SOURCE -c /usr/src/cddl/lib/libumem/umem.c
> /usr/src/cddl/lib/libumem/umem.c:197: error: redefinition of 'nofail_cb'
> /usr/src/cddl/lib/libumem/umem.c:30: error: previous definition of 'nofail_cb' was here
> /usr/src/cddl/lib/libumem/umem.c:199: error: redefinition of `struct umem_cache'
> /usr/src/cddl/lib/libumem/umem.c:210: error: redefinition of 'umem_alloc'
> /usr/src/cddl/lib/libumem/umem.c:43: error: previous definition of 'umem_alloc' was here

Did you use my previous patches? There is no cddl/lib/libumem/umem.c is
HEAD, it was it's old location and it was moved to
compat/opensolaris/lib/libumem/. Delete your entire cddl/ directory and
recsup.

Max Laier

unread,
Apr 8, 2007, 1:10:36 PM4/8/07
to
On Saturday 07 April 2007 21:43, Dag-Erling Smørgrav wrote:
> Pawel Jakub Dawidek <p...@FreeBSD.org> writes:
> > Limitations.
> >
> > Currently ZFS is only compiled as kernel module and is only
> > available for i386 architecture. Amd64 should be available very soon,
> > the other archs will come later, as we implement needed atomic
> > operations.
>
> ZFS is now also available on pc98 and amd64.

panic: lock "zfs:&zap->zap_f.zap_num_entries_mtx" 0xffffff006582c260
already initialized

While dump/restoreing /usr to zfs. kgdb trace attached. Let me know if
you need further information.

--
/"\ Best regards, | mla...@freebsd.org
\ / Max Laier | ICQ #67774661
X http://pf4freebsd.love2party.net/ | mlaier@EFnet
/ \ ASCII Ribbon Campaign | Against HTML Mail and News

log.dump_panic

Max Laier

unread,
Apr 8, 2007, 2:13:59 PM4/8/07
to
On Sunday 08 April 2007 19:10, Max Laier wrote:
> On Saturday 07 April 2007 21:43, Dag-Erling Smørgrav wrote:
> > Pawel Jakub Dawidek <p...@FreeBSD.org> writes:
> > > Limitations.
> > >
> > > Currently ZFS is only compiled as kernel module and is only
> > > available for i386 architecture. Amd64 should be available very
> > > soon, the other archs will come later, as we implement needed
> > > atomic operations.
> >
> > ZFS is now also available on pc98 and amd64.
>
> panic: lock "zfs:&zap->zap_f.zap_num_entries_mtx" 0xffffff006582c260
> already initialized
>
> While dump/restoreing /usr to zfs. kgdb trace attached. Let me know
> if you need further information.

The attached diff lets me survive the dump/restore. Not sure if this is
the right fix, but seems like the union messes with mutex initialization.

zfs.dump.diff

Dag-Erling Smørgrav

unread,
Apr 8, 2007, 2:20:44 PM4/8/07
to
Max Laier <m...@love2party.net> writes:
> The attached diff lets me survive the dump/restore. Not sure if
> this is the right fix, but seems like the union messes with mutex
> initialization.

You need to track down where memory for the mutex (or rather zap) was
actually allocated, and stick the memset there. I suspect it
originates on the stack somewhere.

Max Laier

unread,
Apr 8, 2007, 2:43:13 PM4/8/07
to
On Sunday 08 April 2007 20:20, Dag-Erling Smørgrav wrote:
> Max Laier <m...@love2party.net> writes:
> > The attached diff lets me survive the dump/restore. Not sure if
> > this is the right fix, but seems like the union messes with mutex
> > initialization.
>
> You need to track down where memory for the mutex (or rather zap) was
> actually allocated, and stick the memset there. I suspect it
> originates on the stack somewhere.

Well, I assume it is zeroed already, but on the way the other union
members are used which messes up the storage for the mutex. At least
looking at the contents gives me that impression:

> $2 = {zap_objset = 0xffffff0001406410, zap_object = 12660, zap_dbuf =
> 0xffffff005ce892d0, zap_rwlock = {lock_object = { lo_name =
> 0xffffffff8081b416 "zfs:&zap->zap_rwlock", lo_type = 0xffffffff8081b416
> "zfs:&zap->zap_rwlock", lo_flags = 41615360, lo_witness_data = {
> lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, sx_lock =
> 18446742974215086080, sx_recurse = 0}, zap_ismicro = 0, zap_salt =
> 965910969, zap_u = {zap_fat = {zap_phys = 0xffffffff81670000,
> zap_num_entries_mtx = {lock_object = {lo_name = 0x70000 <Address
> 0x70000 out of bounds>, lo_type = 0x0, lo_flags = 2155822976,
> lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}},
> sx_lock = 1, sx_recurse = 0}, zap_block_shift = 0}, zap_micro =
> {zap_phys = 0xffffffff81670000, zap_num_entries = 0, zap_num_chunks =
> 7, zap_alloc_next = 0, zap_avl = { avl_root = 0x0, avl_compar =
> 0xffffffff807f3f80 <mze_compare>, avl_offset = 0, avl_numnodes = 1,
> avl_size = 0}}}}

Matthew Dillon

unread,
Apr 8, 2007, 2:38:14 PM4/8/07
to

:Hi.
:
:I'm happy to inform that the ZFS file system is now part of the FreeBSD

:operating system. ZFS is available in the HEAD branch and will be
:available in FreeBSD 7.0-RELEASE as an experimental feature.

Congratulations on your excellent work, Pawel!

-Matt

Pawel Jakub Dawidek

unread,
Apr 8, 2007, 2:53:12 PM4/8/07
to
On Sun, Apr 08, 2007 at 07:10:36PM +0200, Max Laier wrote:

> On Saturday 07 April 2007 21:43, Dag-Erling Sm?rgrav wrote:
> > Pawel Jakub Dawidek <p...@FreeBSD.org> writes:
> > > Limitations.
> > >
> > > Currently ZFS is only compiled as kernel module and is only
> > > available for i386 architecture. Amd64 should be available very soon,
> > > the other archs will come later, as we implement needed atomic
> > > operations.
> >
> > ZFS is now also available on pc98 and amd64.
>
> panic: lock "zfs:&zap->zap_f.zap_num_entries_mtx" 0xffffff006582c260
> already initialized
>
> While dump/restoreing /usr to zfs. kgdb trace attached. Let me know if
> you need further information.
[...]
> #10 0xffffffff80295755 in panic (fmt=0xffffffff80481bc0 "lock \"%s\" %p already initialized") at /usr/src/sys/kern/kern_shutdown.c:547
> #11 0xffffffff802bd72e in lock_init (lock=0x0, class=0xffffffff80a11000, name=0xa <Address 0xa out of bounds>,
> type=0x1b1196 <Address 0x1b1196 out of bounds>, flags=1048064) at /usr/src/sys/kern/subr_lock.c:201
> #12 0xffffffff807f092a in fzap_upgrade (zap=0xffffff006582c200, tx=0xffffff006591dd00)
> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zap.c:87
> #13 0xffffffff807f42d3 in mzap_upgrade (zap=0xffffff006582c200, tx=0xffffff006591dd00)
> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:361
> #14 0xffffffff807f4cd4 in zap_add (os=0x0, zapobj=18446744071572623360, name=0xffffff00060ebc19 "org.eclipse.jdt_3.2.1.r321_v20060905-R4CM1Znkvre9wC-",
> integer_size=8, num_integers=1, val=0xffffffffaeeb6860, tx=0xffffff006591dd00)
> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:622
> #15 0xffffffff80802d06 in zfs_link_create (dl=0xffffff0065554140, zp=0xffffff005ccfac08, tx=0xffffff006591dd00, flag=1)
> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:564
> #16 0xffffffff8080c01c in zfs_mkdir (ap=0xffffffffaeeb6960) at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1474
> #17 0xffffffff804490f9 in VOP_MKDIR_APV (vop=0x12, a=0xffffffffaeeb6960) at vnode_if.c:1234
> #18 0xffffffff80316195 in kern_mkdir (td=0xffffff000105e000, path=0x5149d1 <Address 0x5149d1 out of bounds>, segflg=15549312, mode=511) at vnode_if.h:653
> #19 0xffffffff8041abd0 in syscall (frame=0xffffffffaeeb6c70) at /usr/src/sys/amd64/amd64/trap.c:825
> #20 0xffffffff8040206b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:272
> #21 0x000000080071969c in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> (kgdb) f 12
> #12 0xffffffff807f092a in fzap_upgrade (zap=0xffffff006582c200, tx=0xffffff006591dd00)
> at /usr/src/sys/modules/zfs/../../contrib/opensolaris/uts/common/fs/zfs/zap.c:87
> 87 mutex_init(&zap->zap_f.zap_num_entries_mtx, NULL, MUTEX_DEFAULT, 0);
> (kgdb) p zap
> $1 = (zap_t *) 0xffffff006582c200
> (kgdb) p *zap

> $2 = {zap_objset = 0xffffff0001406410, zap_object = 12660, zap_dbuf = 0xffffff005ce892d0, zap_rwlock = {lock_object = {
> lo_name = 0xffffffff8081b416 "zfs:&zap->zap_rwlock", lo_type = 0xffffffff8081b416 "zfs:&zap->zap_rwlock", lo_flags = 41615360, lo_witness_data = {
> lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, sx_lock = 18446742974215086080, sx_recurse = 0}, zap_ismicro = 0, zap_salt = 965910969,
> zap_u = {zap_fat = {zap_phys = 0xffffffff81670000, zap_num_entries_mtx = {lock_object = {lo_name = 0x70000 <Address 0x70000 out of bounds>,
> lo_type = 0x0, lo_flags = 2155822976, lo_witness_data = {lod_list = {stqe_next = 0x0}, lod_witness = 0x0}}, sx_lock = 1, sx_recurse = 0},
> zap_block_shift = 0}, zap_micro = {zap_phys = 0xffffffff81670000, zap_num_entries = 0, zap_num_chunks = 7, zap_alloc_next = 0, zap_avl = {
> avl_root = 0x0, avl_compar = 0xffffffff807f3f80 <mze_compare>, avl_offset = 0, avl_numnodes = 1, avl_size = 0}}}}

fzap_upgrade() changes type from 'zap_micro' to 'zap_fat' and union is
used for this (see
sys/contrib/opensolaris/uts/common/fs/zfs/sys/zap_impl.h), that's why we
see this trash:

zap_num_entries_mtx = {lock_object = {lo_name = 0x70000 <Address 0x70000 out of bounds>,
lo_type = 0x0, lo_flags = 2155822976, lo_witness_data = {lod_list = {stqe_next = 0x0},
lod_witness = 0x0}}, sx_lock = 1, sx_recurse = 0},

I already use kmem_zalloc() (note _z_) for zap allocation in
zap_micro.c, so Max is right, that we have to clear this structure here.

I'm quite tired of tracking such problems, because our mechanism for
detecting already initialized locks is too simple (based on one bit), so
I'd prefer to improve it, or just add bzero() to mutex_init().

Pawel Jakub Dawidek

unread,
Apr 8, 2007, 9:07:03 PM4/8/07