4x3TB raidz (~9TB usable) read/write performance slower than HFS+

[greeninwi]

Hello,

We recently upgraded and made some changes to one of our Mac Pro systems.  Prior to the upgrade we were running a software RAID0 (OSX controlled) stripe of 4x2TB disks totaling approximately 8TB in usable HFS+ space.  I moved this system to a 4x3TB raidz pool that has approximately 9TB of usable ZFS space.  We are seeing a pretty significant performance hit when reading and writing to these ZFS drives.  We previously would see read and write speeds well into the 400MB/s range, but now are capping out at 200MB/s on a good run.

Additionally, when we were running the HFS+ RAID0 stripe we were only using 32GB of RAM.  Since the ZFS change, we upgraded to 64GB of RAM and are still seeing the performance hit.  This system is being used for DNA sequencing and analysis...so it is fairly specialized.

I am wondering if you can suggest anything for me to do on this system to try and get the speeds back to the 400MB/s read/write range. 

I will be happy to gather any additional information you may need, please ask if you need anything.  I am open to any and all suggestions.

Thanks!

[Fastmail Jason]

Can you post the history from creation to know of the ZFS system.

zpool history

and 

zpool status

and

zfs get all

This will provide as much detail as possible to what has transpired.

--

Jason Belec

Sent from my iPad

--
 
 
 

[greeninwi]

There are two pools, just so you know.

Here is the history:

History for 'Data':

2012-12-10.13:52:22 zpool create Data raidz disk1s2 disk2s2 disk3s2

2012-12-10.14:04:58 zpool import -f 16390525018098050599

2012-12-10.14:09:10 zpool import -f 16390525018098050599

2012-12-11.13:34:40 zpool import -f 16390525018098050599

2012-12-12.07:29:34 zpool import -f 16390525018098050599

2012-12-12.07:35:24 zpool import -f 16390525018098050599

2012-12-12.07:44:26 zpool import -f 16390525018098050599

2012-12-12.07:47:12 zpool import -f 16390525018098050599

2013-01-03.13:09:26 zpool import -f 16390525018098050599

2013-01-07.13:58:52 zpool import -f 16390525018098050599

2013-01-10.13:21:23 zpool import -f 16390525018098050599

2013-01-11.14:26:50 zpool import -f 16390525018098050599

2013-01-11.14:37:06 zpool import -f 16390525018098050599

2013-01-16.12:20:29 zpool import -f 16390525018098050599

2013-01-16.12:57:09 zpool import -f 16390525018098050599

2013-01-16.13:03:00 zpool import -f 16390525018098050599

2013-01-17.14:57:09 zpool import -f 16390525018098050599

History for 'RAIDz':

2012-12-07.12:55:21 zpool create RAIDz raidz disk0s2 disk1s2 disk2s2 disk3s2

2012-12-07.12:56:35 zpool import -f 14082260014033687447

2012-12-07.13:00:06 zpool import -f 14082260014033687447

2012-12-07.13:29:45 zpool import -f 14082260014033687447

2012-12-07.13:35:38 zpool import -f 14082260014033687447

2012-12-07.13:38:17 zpool scrub RAIDz

2012-12-07.13:39:40 zpool online RAIDz disk8s2

2012-12-07.13:51:18 zpool import -f 14082260014033687447

2012-12-07.15:38:13 zpool import -f 14082260014033687447

2012-12-10.12:26:20 zpool import -f 14082260014033687447

2012-12-10.13:40:50 zpool import -f 14082260014033687447

2012-12-10.14:04:56 zpool import -f 14082260014033687447

2012-12-12.07:29:33 zpool import -f 14082260014033687447

2012-12-12.07:35:27 zpool import -f 14082260014033687447

2012-12-12.07:47:08 zpool import -f 14082260014033687447

2013-01-03.13:09:08 zpool import -f 14082260014033687447

2013-01-07.13:58:50 zpool import -f 14082260014033687447

2013-01-10.13:21:22 zpool import -f 14082260014033687447

2013-01-11.14:37:08 zpool import -f 14082260014033687447

2013-01-16.12:20:31 zpool import -f 14082260014033687447

2013-01-16.12:55:21 zpool scrub RAIDz

2013-01-16.12:56:59 zpool import -f 14082260014033687447

2013-01-16.13:03:01 zpool import -f 14082260014033687447

2013-01-17.14:57:10 zpool import -f 14082260014033687447

Status

  pool: Data

 state: ONLINE

 scrub: none requested

config:

NAME         STATE     READ WRITE CKSUM

Data         ONLINE       0     0     0

 raidz1     ONLINE       0     0     0

   disk3s2  ONLINE       0     0     0

   disk2s2  ONLINE       0     0     0

   disk1s2  ONLINE       0     0     0

errors: No known data errors

  pool: RAIDz

 state: ONLINE

 scrub: none requested

config:

NAME         STATE     READ WRITE CKSUM

RAIDz        ONLINE       0     0     0

 raidz1     ONLINE       0     0     0

   disk5s2  ONLINE       0     0     0

   disk4s2  ONLINE       0     0     0

   disk6s2  ONLINE       0     0     0

   disk7s2  ONLINE       0     0     0

errors: No known data errors

Get All

NAME   PROPERTY       VALUE                  SOURCE

Data   type           filesystem             -

Data   creation       Mon Dec 10 13:52 2012  -

Data   used           4.59T                  -

Data   available      796G                   -

Data   referenced     4.59T                  -

Data   compressratio  1.00x                  -

Data   mounted        yes                    -

Data   quota          none                   default

Data   reservation    none                   default

Data   recordsize     128K                   default

Data   mountpoint     /Volumes/Data          default

Data   sharenfs       off                    default

Data   checksum       on                     default

Data   compression    off                    default

Data   atime          on                     default

Data   devices        on                     default

Data   exec           on                     default

Data   setuid         on                     default

Data   readonly       off                    default

Data   zoned          off                    default

Data   snapdir        hidden                 default

Data   aclmode        groupmask              default

Data   aclinherit     secure                 default

Data   canmount       on                     default

Data   shareiscsi     off                    default

Data   xattr          on                     default

Data   copies         1                      default

Data   version        2                      -

RAIDz  type           filesystem             -

RAIDz  creation       Fri Dec  7 12:55 2012  -

RAIDz  used           1.51T                  -

RAIDz  available      6.53T                  -

RAIDz  referenced     1.51T                  -

RAIDz  compressratio  1.00x                  -

RAIDz  mounted        yes                    -

RAIDz  quota          none                   default

RAIDz  reservation    none                   default

RAIDz  recordsize     128K                   default

RAIDz  mountpoint     /Volumes/RAIDz         default

RAIDz  sharenfs       off                    default

RAIDz  checksum       on                     default

RAIDz  compression    off                    default

RAIDz  atime          on                     default

RAIDz  devices        on                     default

RAIDz  exec           on                     default

RAIDz  setuid         on                     default

RAIDz  readonly       off                    default

RAIDz  zoned          off                    default

RAIDz  snapdir        hidden                 default

RAIDz  aclmode        groupmask              default

RAIDz  aclinherit     secure                 default

RAIDz  canmount       on                     default

RAIDz  shareiscsi     off                    default

RAIDz  xattr          on                     default

RAIDz  copies         1                      default

RAIDz  version        2                      -

Any help or guidance you can offer will be greatly appreciated!

Thanks.

[Peter Lai]

I have a couple of questions (I use ZFS but not on mac): does the
current/this version of zfs for mac support ashift?
Also, running a 4-vdev raidz1 is going to be slower than 3,5 or 9
(although that shouldn't explain 2x the loss all by itself).

As for the ram, if there is a way to access the zfs arc stats on mac,
you can look at the size of the cache, its performance, and how it's
being used...

> --
>
>
>

[Daniel Bethe]

> We recently upgraded and made some changes to one of our Mac Pro systems.  Prior to the upgrade we were running a software RAID0 (OSX controlled) stripe of 4x2TB disks totaling approximately 8TB in usable HFS+ space.  I moved this system to a 4x3TB raidz pool that has approximately 9TB of usable ZFS space.

Hi!  So you reconfigured it from pure speed mode, to reliability mode.

>I am wondering if you can suggest anything for me to do on this system to try and get the speeds back to the 400MB/s read/write range. 

Put it back into speed mode.  With three drives, are you wanting maximal speed or maximal reliability?  Either you need to make your drives striped again (zpool create mypool disk1s2 disk2s2 disk3s2), in which case those three drives may be faster than your previous two drives.  Or you need to get another drive and do a 1+0, or get more drives and stripe some raidzs.  There is lots of talk about how to performance optimize your ZFS setup, and they apply because MacZFS is fundamentally just ZFS.  The only performance limit I'm aware of is that MacZFS does not support l2arc.

And make sure you're creating them with 4k alignment, if they're 4k drives.

http://code.google.com/p/maczfs/wiki/FAQ#What_should_I_do_with_4k_(Advanced_Format)_hard_drives?

[Fastmail Jason]

@greeninwi all looks fine, except as Peter has stated you did not build these as recommended for optimum use. I probably wouldn't have called it RaidZ when making such, but... Of course the speed issue is due in part you are no longer raid0, you moved to a structure, ZFS that is more interested in accuracy of data. So your now going to get the speed of a drive or maybe 2 combined and not necessarily of a stripped array.

@Peter we don't have access to that info unless your building ZFS on your system. I have had other members provide me with some interesting options in the past, which I've been playing with and tweaking over the last year with varying results based on RAM and processor.Yes I believe the ashift update is the latest build, should have instructions in the wiki.

--
Jason Belec
Sent from my iPad

> --
>
>
>

[Fastmail Jason]

Taadaaaa! ;)

--
Jason Belec
Sent from my iPad

> --
>
>
>

[Daniel Bethe]

>> As for the ram, if there is a way to access the zfs arc stats on mac,
>> you can look at the size of the cache, its performance, and how it's
>> being used...

> @Peter we don't have access to that info unless your building ZFS on your

Yes, we do.  I'm not sure what building an array has to do with it, Jason, so I don't mean to misunderstand what else you're saying, but we do have that stuff.  Run 'zoink'.  That's an app which was written by Apple, which doesn't exist on any other ZFS platform, which displays your realtime ZFS usage stats.

>> I have a couple of questions (I use ZFS but not on mac): does the
>> current/this version of zfs for mac support ashift?

Yes.

http://code.google.com/p/maczfs/wiki/FAQ#What_should_I_do_with_4k_(Advanced_Format)_hard_drives?

>> Also, running a 4-vdev raidz1 is going to be slower than 3,5 or 9
>> (although that shouldn't explain 2x the loss all by itself).

It wouldn't necessarily be *noticably* slower.  That's just a matter of a minor alignment, and it bears benchmarking against your actual workload if you think you could actually measure it.

[Jason]

Zoink, damn, missed that.

Jason
Sent from my iPhone

> --
>
>
>

[Daniel Bethe]

Just for clarification, I'll note that the performance that you got with a 400MBps raid0 and a 200MBps raidz1 is exactly to be expected from any comparable setup.  If raid5 existed in Apple Software RAID, which it doesn't, you'd have dropped down closer to the speed of one drive as well, because it's calculating parity and stuff.  I have eight 2TB drives on a MacZFS raidz2 and I don't get eight times the speed of one drive.  From everything I've ever read and been told on performance comparisons, you'd have a vaguely similar experience with other raid5 setups.  A speed variation of that size has nothing at all to do with ZFS or with MacZFS.  ZFS's basic data reliability checksumming, and other aspects of software RAID, are barely noticable at all on any remotely modern CPU.  But with ZFS, you then gain some potential performance advantages from ZFS's ARC.  And you gain the forward scalability to add, for example, another three-drive raidz1, which
can automatically stripe and expand into your existing one with a single command.

> --
>

[Daniel Becker]

Not if by "performance" you mean "sequential throughput" (as is the case here) rather than IOPS. Wasn't there another thread about this just recently?

For large (i.e., greater than stripe size) transfers, you should expect to see a drop in write speed because you're effectively writing 33% more data to disk, and depending on how smart of a RAID5 implementation you have, you may see the same decrease in read speed (though this is not strictly necessary). The parity computation for RAID5 is trivial for any reasonably modern CPU.

The other thing to keep in mind with ZFS is that as with any copy-on-write filesystem, what looks like a sequential transfer from the software perspective may not be sequential at all in terms of the corresponding disk accesses, especially with data that is frequently modified (-> fragmentation).

Really, the best way to figure out what's going on is to look at the activity of your raw disks (and CPU) over time while running your benchmarks (e.g. using iostat).

> --
>
>
>

[Daniel Bethe]

> Not if by "performance" you mean "sequential throughput" (as
> is the case here) rather than IOPS. Wasn't there another thread about this
> just recently?

Yeah I pretty much did, speaking in gross generalizations.  ^_^  Which may be more or less grotesque according to a given workload!  YMMV.  Benchmarks away.

[Fastmail Jason]

I just ran a test on a system I'm logged into, 4x1TB raidz and copies are set=2, with a write speed of 170Mb/s and read of 416Mb/s, it's an old system I've had running for a few years with only 6GB RAM and a 2.4 GHz Core 2 Duo processor. Just for reference info as we don't have exactly the same setup. The OS is on an OCZ Vertex 4 256GB SSD.

--
Jason Belec
Sent from my iPad

[greeninwi]

Hi Jason,

Thank you for the information.  I realize that going from a 4 drive stripe to a 4 drive RAID5 (z) requires more processing power and, therefore, is not as fast.  The data we are using needs to be protected though, and running a stripe across so many drives left this machine too vulnerable to drive failures.  If I took the two RAIDz's that I have (one is 4 drive and one is 3 drive) and combined them into one (allowing for one drive failure on a 7 drive RAID) would that get me closer to the speeds I was seeing with the 4 drive HFS+ stripe?

Striping the drives isn't an option anymore.  I am looking to get the best read/write throughput while permitting at least one drive failure.

Any info will be greatly appreciated.

Thanks,
Luke

[Daniel Becker]

On Jan 21, 2013, at 5:56 AM, greeninwi <gree...@gmail.com> wrote:

> Striping the drives isn't an option anymore. I am looking to get the best read/write throughput while permitting at least one drive failure.

I'm guessing there's an unstated capacity constraint here, too; otherwise, you will get the best random I/O performance while satisfying this constraint from a pool of mirrors.

[greeninwi]

So 2-drive stripes in a mirror or vice versa?  The quantity of storage is important since we are sequencing DNA and the data sizes are obscene.  Working within the confines of a Mac Pro makes this a little more difficult as well.  What configuration would you recommend for 7 drives?  I currently have two separate RAIDz configs, but like I said, my throughput is not close to being fast enough the way I currently have it configured.  I need to get in the vicinity of 400MB/s read/write.

Money isn't an obstacle here.

Luke

[Fastmail Jason]

Hi Luke, Daniel's input is valuable, and to expand on that....

I guess what might help you even more is understanding a little more about what your doing or desire to do with one or more pools. The reason being, some ZFS setups are better for speed and others reliability. You can of course combine, however it helps to know what speed you need and for what purpose to best suggest options. 

For example, I run VMs on mirrored pools not raidz as I and others have found this fast and stable. 

Do you need all the data in the same pool, all the time, is that data being accessed more like a database or just files with computations applied and a resulting file stored back? 

As for the specifics you mention as an idea for your pool, you would want a raidz mirrored to another raidz and both would have the same number of drives/sizes. You'll have to test this, and the questions I've posted above will help determine how beneficial this will be. 

If speed is the all consuming goal, then SSD is your only real option. In the last 2 years, all databases I've built we're running on SSD eliminating so many 'tweaks' to enhance performance. The difference is staggering. However we know many are deterred by the 'cost' which is never really put into context of time saved.

Lets see how you answer the above. ;)

--

Jason Belec

Sent from my iPad

--
 
 
 

[Luke Marti]

Hi Jason,

Thanks for your response.  I definitely need to be able to have a drive fail without losing this data, so that is primary...then secondary is speed. 

This is not running VMs, this is all DAS.  It is not being accessed like a DB, it is huge (GBs per) read and write cycle.  One of the pools (the smaller one) is read from, then the larger pool is used as a swap space during the computational stage of the sequencing.  Then all the results are written back to the smaller pool.

We a currently using a 2TB SSD (4 drive stripe) as a "cache" to help speed performance up, but we'd prefer to not use the SSDs as a final solution.  The reason being that we need to recommend these configs to our customers and the capital needed for the SSDs is cumbersome for many.

I realize the time time saved calculation, and internally, we live by that...but when we are doing runs on configs to recommend to customers we like to use all HDDs.

I tried to be relatively thorough in my answers here, but if I am unclear just let me know and I will elaborate more.

Thanks,
Luke

--
 
 
 

--
☮ne l☮ve

[Fastmail Jason]

Ok, I have a far better understanding where you are going with this then. ;)

You should see some feature film data setups, especially for rendering. ;)

OK, so your trying to work a simple, reasonable approach that your clients can mimic. I guess cost does matter (wink, wink).

So the small pool is key, the master. The larger pool is swap and not really necessary to be as safe. Probably a none safe computational pool of stripped drives would be fine for the large pool, however that is your call.

You could try a 2 drive mirror for the small pool, even turn on copies=2 in a second test as it would technically result in 4 copies of everything. Then the speed of the drive and the bus are the determining factors assuming you have good chunk of memory.

Try the following, 

sudo zpool create zpool_name mirror /dev/hd1 /dev/hd2 mirror /dev/hd3 /dev/hd4 mirror /dev/hd5 /dev/hd6

This allows a failure of one drive per mirror, while giving you pretty decent speed while keeping everything safe-issh, safe-er...... Data written will be spread across the pool, and it can of course add more pairs  BUT don't do this after the fact as data from he previous drives won't be redistributed. 

Let me know your results. If faster in anyone of these configurations, we can then look at specific pool functions to turn off for a little more zoom.

--

Jason Belec

Sent from my iPad

--
 
 
 

[Daniel Becker]

A pool consisting of two two-drive mirrors. ZFS does not support nesting VDEVs, so it's not even possible to do it the other way around (in addition, mirrored stripe sets are generally not a good idea as they are less resilient).

In the ideal case (and assuming a beefy enough machine), this will give you 4x the sequential read throughput and 2x the sequential write throughput of a single drive, so with recent HDs you'd probably still be short of you 400MB/s goal for writes.

--
 
 
 

[Jason]

I've got some time this week, I'll run a few configurations and see how close I can get to those desired values without SSDs. 

Jason

Sent from my iPhone

[jasonbelec]

OK, for the way your suggesting you want to do this, I can't see gaining the speed without out some serious tech or tossing the security ZFS provides. I have had success, but not with the structure you wish to utilize. That said I dug up a great article explaining things in detail that might help you choose the best structure.

https://blogs.oracle.com/roch/entry/when_to_and_not_to

[Boyd Waters]

A top-level RAIDz is only one vdev, will have about the same write performance as a single disk.

Read performance can be better than that, but it depends on your data access patterns.

I have 6 disks, and found that my best tradeoff between reliability, performance, and size was to set it up as a concatenated set of 3 pairs.

--
 

---
You received this message because you are subscribed to the Google Groups "zfs-macos" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-macos+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Fastmail Jason]

Personally I also find this the best overall approach for most things. That is what I have set up in the mad science lab, my home and 3 client sites. I don't run VMs on these, use mirrors for that. I have one large 12 disk array I'm playing with currently, testing out various configurations for 512GB SSDs to handle large data, we'll see.... ;)

--

Jason Belec

Sent from my iPad