ZFS Support?

[Mark Doering]

First off, let me congratulate and thank each and every one of the contributors on this project.  You have built an amazing system.  Your documentation is also top notch.  I have never worked with Fiber Channel prior to last night (with the exception of plugging in the fiber and onlining/initializing/formatting the LUNs presented to my servers from our storage engineers)  Marcin's explanation in the thread here: https://groups.google.com/forum/m/#!topic/esos-users/NDwc_2AlNzs was an eye opener... Through that I was able to configure my HP C7000 chassis brocade 4/24 switch, configure the zoning on the switch, add the hosts in esos, and then follow the wiki for the install and configure and by the end of the night I had a 14.5TB Fibre Channel SAN.... incredible!

On to my question... I was previously using  FreeNAS with a ZFS RAIDZ2 array for my storage that was shared via iSCSI... With my move to Fibre Channel I decided to explore other options which lead me to this fantastic project.  I was wondering if there's any compelling reason you haven't yet included ZFS or if there is similar functionality here that I'm missing... I was able to create a striped lvm volume across my 8 disks, but using mdadm I was unable to use any of the --type raidn flags as I got a warning about something not being included in the kernel and dependencies....

What sort of software based options to I have here to offer some fault tolerance and resiliency to my data array?

[Marcin Czupryniak]

Hi Mark,
good that my explanation helped you to get something that works :) more
will come in the next months
As per your question, right now we don't have ZFS support in ESOS,
because we are focused on SAN environments, and exporting filesystems is
not really our goal; although ZFS can be used to create software RAID
which can act as a container for SCST (file_io mode) I'm personally
against file_io (even if in some situations it can boost performance)
for several reaosns (VFS translation and page caching latencies) which
do not occur with block_io or passtrough SCSI.
I would say that the best option is to get a hardware RAID controller
(with backplane signaling support SGPIO/I2C) to light up the red LEDs in
case of disk failure and battery backup unit for the cache.
If you're building a cheap array you can get used controllers from ebay
starting at 50$, in my experience the adaptec/LSI works best (ex. Dell
PERC H700 512MB [LSI] or Adaptec ASR-52445 28 Ports for around 300$ with
battery and cabling)
I've used both ZFS and Linux RAID (both MD and LVM2) and my experience
with them has been somehow negative (sometimes high CPU usage in I/O
intensive scenarios, higher latency, soft locks for various
miliseconds), but the positive part is that you can avoid expensive RAID
cards for high density low performance cheap arrays.
ESOS should have support for Linux soft RAID (both MD and LVM2) I never
tested the soft RAID for levels different than 1 so will schedule a
couple of tests for the weekend and let you know what's wrong in here.

But in the end I think that ZFS could be a good addition to ESOS as the
Z2 and Z3 has better performance compared to MD RAID6 and there is no
equivalent of Z3 right now (a part from building RAID 60).

- Marc what do you think about having ZFS in ESOS?

= Marcin

> First off, let me congratulate and thank each and every one of the
> contributors on this project. You have built an amazing system. Your
> documentation is also top notch. I have never worked with Fiber
> Channel prior to last night (with the exception of plugging in the
> fiber and onlining/initializing/formatting the LUNs presented to my
> servers from our storage engineers) Marcin's explanation in the
> thread here:
> https://groups.google.com/forum/m/#!topic/esos-users/NDwc_2AlNzs

> <https://groups.google.com/forum/m/#%21topic/esos-users/NDwc_2AlNzs> was

> an eye opener... Through that I was able to configure my HP C7000
> chassis brocade 4/24 switch, configure the zoning on the switch, add
> the hosts in esos, and then follow the wiki for the install and
> configure and by the end of the night I had a 14.5TB Fibre Channel
> SAN.... incredible!
>
>
> On to my question... I was previously using FreeNAS with a ZFS RAIDZ2
> array for my storage that was shared via iSCSI... With my move to
> Fibre Channel I decided to explore other options which lead me to this
> fantastic project. I was wondering if there's any compelling reason
> you haven't yet included ZFS or if there is similar functionality here
> that I'm missing... I was able to create a striped lvm volume across
> my 8 disks, but using mdadm I was unable to use any of the --type
> raidn flags as I got a warning about something not being included in
> the kernel and dependencies....
>
> What sort of software based options to I have here to offer some fault
> tolerance and resiliency to my data array?

> --
> You received this message because you are subscribed to the Google
> Groups "esos-users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to esos-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Marc Smith]

Last I knew the license used for ZFS wasn't compatible with the GPL which is why you only see it in FreeBSD and no main-line Linux distributions, but I'll double-check.

--Marc

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "esos-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+unsubscribe@googlegroups.com.

[Mark Doering]

Marc and Marcin,

  Thank you for your fast replies.  I'm not really too versed in the FOSS licensing schemes out there and their various incompatibilities.  I was able to get the mdadm tool working to create a RAID6 array giving me ~11TB of usable space.

As far as ZFS support.  I know there is a project called ZFSonLinux but again, no idea about the licensing aspect...

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "esos-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.

[Marc Smith]

From the ZFS on Linux FAQ: http://zfsonlinux.org/faq.html#WhatAboutTheLicensingIssue

--snip--

ZFS is licensed under the Common Development and Distribution License (CDDL), and the Linux kernel is licensed under the GNU General Public License Version 2 (GPLv2). While both are free open source licenses they are restrictive licenses. The combination of them causes problems because it prevents using pieces of code exclusively available under one license with pieces of code exclusively available under the other in the same binary. In the case of the kernel, this prevents us from distributing ZFS as part of the kernel binary. However, there is nothing in either license that prevents distributing it in the form of a binary module or in the form of source code.

--snip--

As I had suspected, this is the case: We can't distribute ZFS with ESOS in a binary form. What we could possibly do in the future is provide a separate add-on or module that the ESOS installer scripts prompts users to download (just like the hardware RAID CLI tools).

As Marcin described, I don't think there is a big need for ZFS in the block-level storage arena right now -- its probably more beneficial in NAS type solutions. We can achieve a lot of cool stuff by combing software RAID, LVM, de-duplication file systems, etc.

--Marc

[Mark Doering]

Marc,

  Thank you for at least taking a look.  I'd be interested to hear more about what options are available from a non-hardware RAID standpoint.  Defining storage at such a low level just doesn't lend itself well to the dynamic data needs of most organizations any more as far as large san deployments I don't think...  For base server OS, static content like webpages, etc I think it's great... but I need flexibility in my larger data pool.  While hardware RAID may be faster and possibly more stable (though that's far from a guarantee) it just isn't as smart as something that can interact with the data on a higher level...  I don't think any of the SANs we use in our enterprise are as rigid as hardware RAID on a server... (Presently using Hitachi USP V, AMS 2300, and our newest toy, the HDS VSP...)

[Martino Io]

I think that not having any of the complex software RAID/RAIDlike filesystems (ZFS/GlusterFS/OtherScalableFS) builtin in ESOS doesn't pose a limitation in the flexibility. Using tiered storage architectures you can still have your backend storage on something like ESOS then use whatever filesystem comes best and work on an enclosure like level.

Personally I made a couple of GlusterFS replicated volumes using FC for Block Devices and Infiniband for Gluster replication and it works like a charm, there are advantages of having the Block layer on FC and the highly bandwidth consuming replication offloaded to Infiniband on other hosts, in this way I'm not forced to ditch out my working SAN and for various applications let them both work together (FC SAN for ESX Datastores and Gluster for physical servers).

= Marcin

2013/8/26 Mark Doering <ma...@intervex.net>

[Dan Swartzendruber]

Sorry to necro this - I've been reading back through the list of messages.  Anyway, there are a couple of advantages to exporting zfs-backed storage using esos.

1. ZFS checksums to detect (and repair) damaged data to protect the user.

2. Allow virtually instantaneous snapshots of a SAN volume (for backup purposes.)

 

[Marc Smith]

Those sound like great features for a file system, but for serving up block-level storage, you would then have this situation (starting at the "top"): Your underlying storage device (eg, disk), a file system on top of that (ZFS), then your initiators have the "virtual" block device, and then they run another file system on top of that. The file system on the device, on your initiator side should handle the "safety" aspect. And maybe the ZFS snapshots are better then using LVM snapshots...

The biggest problem with using a file system (ZFS, BTRFS, XFS, whatever) is there is no redundancy. The kernel panics, the system dies, whatever, you're done. Having a block storage outage is not the same as file/network type storage. Especially if you use VMware vSphere... all-paths-down is VERY bad.

Using a hardware solution like the LSI Syncro, or the up-and-coming md-cluster software is where its at... at least for now.

And yes, for some environments like development/testing redundancy probably isn't important and could tolerate a short-lived outage. But for other scenarios, I really think its a requirement.

--Marc

--

You received this message because you are subscribed to the Google Groups "esos-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to esos-users+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dan Swartzendruber]

On 2015-03-30 16:32, 'Marc Smith' via esos-users wrote:
> Those sound like great features for a file system, but for serving up
> block-level storage, you would then have this situation (starting at
> the "top"): Your underlying storage device (eg, disk), a file system
> on top of that (ZFS), then your initiators have the "virtual" block
> device, and then they run another file system on top of that. The file
> system on the device, on your initiator side should handle the
> "safety" aspect. And maybe the ZFS snapshots are better then using LVM
> snapshots...
>
> The biggest problem with using a file system (ZFS, BTRFS, XFS,
> whatever) is there is no redundancy. The kernel panics, the system
> dies, whatever, you're done. Having a block storage outage is not the
> same as file/network type storage. Especially if you use VMware
> vSphere... all-paths-down is VERY bad.
>
> Using a hardware solution like the LSI Syncro, or the up-and-coming
> md-cluster software is where its at... at least for now.
>
> And yes, for some environments like development/testing redundancy
> probably isn't important and could tolerate a short-lived outage. But
> for other scenarios, I really think its a requirement.

I think I was unclear. I wasn't talking about presenting a file on a
filesystem as the target. ZFS allows creating of pseudo-block devices
(like LVM). There are no filesystem semantics involved.

[Dan Swartzendruber]

Note also that zfs checksums are a huge win.  I know of no other filesystem (except btrfs) that detects and fixes bit-rot transparently to the client.

[Marc Smith]

As you said, ZFS uses checksums as its mechanism to detect bit rot. When you use other solutions, they employ other methods for detecting bit rot... hardware RAID controllers have some type of "patrol read" function, Linux md RAID has "scrubbing"... etc. I don't doubt that the mechanism used in ZFS is possibly better, but we've all lived this long without ZFS checksums, I think we'll survive. =)

Anyhow, I'm really not trying to keep ZFS out of ESOS, it just requires work to be done. I may have been more apprehensive some years back about adding ZFS due to the license and then how to distribute binaries, but since that time, we've added other proprietary items to ESOS. We'd just treat ZFS the same: It will be a build-time option. Users can compile their own ESOS, downloading the ZFS package requirements.

It would really be pretty easy to add, and a patch would be much appreciated. If not, I'll have some time in the next week or two and take a look at adding it in.

--Marc

On Mon, Mar 30, 2015 at 4:51 PM, Dan Swartzendruber <dsw...@druber.com> wrote:

Note also that zfs checksums are a huge win.  I know of no other filesystem (except btrfs) that detects and fixes bit-rot transparently to the client.

--

[Jorgen Mourton]

Mark were you able to make any headway with this? I appreciate all your effort in trying to sort this out.

[Marc Smith]

Hi,

Yes, I probably should have updated this thread... ZFS support is in the ESOS master branch now -- as a build option (--enable-zfs). The basics are there and it functions correctly, but the TUI needs a bit more integration to support the provisioning features. So it requires building ESOS from source, but it is in there.

--Marc

[Dan Swartzendruber]

On 2015-07-07 15:19, 'Marc Smith' via esos-users wrote:
> Hi,
>
> Yes, I probably should have updated this thread... ZFS support is in
> the ESOS master branch now -- as a build option (--enable-zfs). The
> basics are there and it functions correctly, but the TUI needs a bit
> more integration to support the provisioning features. So it requires
> building ESOS from source, but it is in there.

It works fine (I am using it in production for awhile now). The only
parts requiring manual action are:

1. create pool
2. create zvol (or dataset for file-based LUNs)
3. create device with something like 'scstadmin -open_dev DEV -handler
vdisk_blockio (or vdisk_fileio) -attributes PATHNAME_OF_LUN_FILE_OR_ZVOL

then, proceed with adding to host group, etc...

[Dan Swartzendruber]

One nuance to keep in mind.  ZFS will not interact with the Linux page cache, so you won't gain anything by using vdisk_fileio for zvols (or file-based LUNs, but I use those for other reasons - zvols can still be a little quirky, yet.)  Another thing: by default, zvols are thick-provisioned (which I like), but file-based LUNs will be thin.  The way around that is to set a reservation on the dataset that the file LUNs live in.

[Brent Bolin]

Dan

Are you using zfs as blockio

How would that be done? Use quota on datasets?

[Dan Swartzendruber]

I create zvols and then create scst device with the CLI then do the rest from the GUI.

>--
>You received this message because you are subscribed to a topic in the Google Groups "esos-users" group.
>To unsubscribe from this topic, visit https://groups.google.com/d/topic/esos-users/1pvFqb8tGdM/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to esos-users+...@googlegroups.com.

[Brent Bolin]

zfs create -V 5gb tank/vol

Cool didn't know you could create specific volumes in pools.  Even has some different attributes then standard datasets (zfs get all)

[Dan Swartzendruber]

On 2015-10-24 15:29, Brent Bolin wrote:
> ZFS CREATE -V 5GB TANK/VOL

and zfs create -s -V 5gb tank/zvol

gives you a thin-provisioned zvol.

[Darren Williams]

I'm a great lover of ZFS but I'm not sure this project is the best place to use it. For both fiber channel and iscsi I would use NAPP-IT anytime on a real ZFS system like Illumos. For me its ESOS for hardware raid, napp-it ZFS and HBA

[Dan Swartzendruber]

Not sure why you think this?  I get a low-overhead iSCSI target exporting storage with all the ZFS benefits.  ???