Setup Advice Needed

[Aaron]

Caution: Newb Alert!

I need to upgrade my large storage needs and think zfs might be better for my needs than a conventional raid setup.

My current setup is 22TB across 8 USB drives.  This sucks, lol.  My immediate needs are for ~20TB connected to a single computer.  I expect this to expand to ~30TB  in the future.  I will be purchasing 2 4-bay JBOD eSATA enclosures and fill them with 3TB hard drives, with the expectation of adding an identical 3rd enclosure later.

How should I plan my filesystem?  None of the content is what I consider irreplaceable.  Ideally I would have 1 redundant drive per enclosure, but if I start with raidz2 I can't upgrade to raidz3 later, correct?  If I start with raidz2 and add a hot spare later, can I survive 3 drive failures then?  I'm guessing not.  In that case I would just stick with raidz2 as opposed to going to raidz3.

How should I arrange pools/volumes etc?  I read that it's better to keep the number of drives in a single pool for performance reasons.  What I want out of all this is 2 volumes - 'Media' and 'Archives' - so that I can direct software to one location for content (and for my own sanity).

The computer for this is a 'hackintosh':

Gigabyte GA-P67A-UD4-B3

Intel i7-2600k

16GB RAM

Thanks for any advice.

[Jason]

Welcome. Yes ZFS will kick the snot out of what you currently have. Assuming you don't intentionally bottleneck anything. ;)

You can get single enclosures that will hold upto 12 drives for a very inexpensive price on Newegg, I just grabbed 2 then added another 12 more drives to each for total of 24 drives with 4 power boxes all inside, allowing me to have a working system and backup/archive system all in one case with redundant power. If anything fails it has backup for drives or power. All connected by esata for traditional systems and for new Thunderbolt systems I can swap the Sonnetech card and box into play. But I like mad science. ;)

If your going to set backup scripts for snapshots, then a RaidZ is great, worst happens you have everything stored elsewhere and a quick restore can occur after repairs. 

You could do 2 RaidZ and mirror them at pool creation or many other combinations. I currently run one at home of 6 drives, 3 mirrors of 2, in RaidZ and having lost 3 drives over a few years, nothing was lost, even when I discovered a broken SATA cable clip. I do also run copies=2 on this pool as it has photos and my wife would never forgive me. All drives have been replaced with larger size resulting in instant pool growth. However many combination options work and I have yet to find a bad setup.

If you build a RaidZ, Raidz2 or Raidz3 you must build with all necessary drives, you cannot add to it after. MacZFS will build with a spare, but doesn't actually recognize it in time of need, you need to manually swap. As long as you scrub regularly you will have plenty of warning and backups just keep the headaches away. BUT I've never lost data on ZFS no matter the problem, it has been a pain sometimes to recover, but it always has been recovered. Backups of snapshots just make it easier. 

You can have a pool per group of devices. You can mount more than one pool, but be careful if your swapping snapshots back and forth or user spaces as you can end up with the wrong one mounted. It gets complicated if your not keeping things simple.

So you seem to be looking for 2 groups of devices, media group with zpool named media and archive group with pool named archive. Whatever devices you put into there and how redundant you want things will determine your overall space. They don't have to be the same, but the archive step should have more and copies set and probably mirroring so you really have an archive.

I have a 'hackintosh' that I experiment with which I setup 18 mths back and it's going strong as any MacMini or iMac so far. ;)

Helpful?

Jason

Sent from my iPad

--
 
 
 

[C. C. Wood]

>Helpful?

Always.

[Aaron]

Helpful?

Extremely.  You made me aware of a couple of misconceptions I had.  So I fired up my FreeBSD virtual machine and added 8 small virtual drives to play with.  I really have two needs, large aggregate storage (Media) and a much smaller set for 'must never lose' data (Archives).  I am going to keep them separate just to make my life simpler.

For the large media storage, I really only want to protect myself against hardware failure.  So perhaps I should forget about raidz# and have a large pool with hot spares.  So 6 x 3TB in a basic pool with 2 x 3TB hot spares would give me 18TB and the ability to survive 2 hard drive failures without data loss.  I could then later add an additional 4 x 3TB drives to this pool to get 10 3TB drives with 2 hot spares for 30TB of storage and can survive 2 drive failures.  Is this correct?

This leaves me with a more general question - raidz1 vs a single hot spare, why would one chose one over the other?

Thanks for the help.

[Aaron]

Thinking about hot spares and reading the FreeBSD man pages, I seem to think I'm way out to lunch. Lol.

[Fastmail Jason]

Ignore hot spares. A scrub run regularly will tell you, that and the pool will run slow going you ample warning to swap in a drive. As long as you backup your snapshots from the beginning you won't have anything to worry about. Then just pull a drive, swap in new one and run the proper command and scrub should show a resilver has commenced. 

--

Jason Belec

Sent from my iPad

--
 
 
 

[Fastmail Jason]

Note: please read up on what a pool can consist of, you can only add drives to mirrors, once a RaidZ123 is built that is it. You grow it by swapping each drive for a larger one. When all are swapped you have a larger pool. ;)

--

Jason Belec

Sent from my iPad

On 2012-12-14, at 3:21 PM, Aaron <aa...@vanhooren.ca> wrote:

--
 
 
 

[Gregg Wonderly]

Hot spares are for raidz configurations.  Mirroring is mirroring.  You can have an extra drive ready to plug in and mirror another vdev onto, if one drive fails.  The issue for macosX, is that there is nothing running in the kernel/background, which detects the disk failure and swaps in your hot spare, in the right way.  In raidZ, you do 'zpool replace old-drive new-drive', in a mirror, you do 'zpool add new-drive good-drive-from-target-vdev'.   

There needs to be some tooling done for MacOS-X to make hot spares work automatically…

Gregg Wonderly

--
 
 
 

[Fastmail Jason]

Yes. I should be so clear someday.... Not going to happen at my advancing years....

--

Jason Belec

Sent from my iPad

--
 
 
 

[Daniel Becker]

Hot spares are assigned to a pool (or multiple pools, actually), not to an individual vdev; it doesn't matter if the pool contains RAIDZ vdevs, mirrors, or a combination thereof. In either case, you can manually replace a failed disk using "zpool replace" ("zpool add" will leave the failed drive in the mirror) instead.

--
 
 
 

[Björn Kahl]

Hi Aaron,

welcome to MacZFS!

Am 14.12.12 19:08, schrieb Jason:

>
> You could do 2 RaidZ and mirror them at pool creation or many other combinations. I currently run one at home of 6 drives, 3 mirrors of 2, in RaidZ and having lost 3 drives over a few years, nothing was lost, even when I discovered a broken SATA cable clip. I do also run copies=2 on this pool as it has photos and my wife would never forgive me. All drives have been replaced with larger size resulting in instant pool growth. However many combination options work and I have yet to find a bad setup.

Just to clarify this:

You can easily combine some mirrored pairs and various raidzX vdevs
into one pool. But you *can* *not* (1) have a mirror made of raidz
devs or vice versa. Such combinations are unsupported.


(1) Ok, technically, the ZIO-Pipeline can serve such and some other
esoteric configurations (and I use it for testing purposes), but there
is *no* *valid* zpool syntax to create such a pool and *absolutely*
*no* support what so ever for such configurations.


Best

Björn

--
| Bjoern Kahl +++ Siegburg +++ Germany |
| "googlelogin@-my-domain-" +++ www.bjoern-kahl.de |
| Languages: German, English, Ancient Latin (a bit :-)) |

[Aaron]

Thanks for all the information!  As mentioned, for some reason I had a real misunderstanding of what a hot spare is.  Things make a lot more sense now and my options are pretty straight forward.

It will take me a little while to get all the new hardware assembled, but I'm sure I'll have more questions once I do.

Thanks again.

[Aaron]

Just following up to say thanks, I have this working on OS X 10.8.2 for one month now with only a few hiccups.

I picked up two SansDigital 5-bay eSATA enclosures on a deal over Christmas and filled them with 10 x 3TB Western Digital Red hard drives.  First issue was realizing that only one of the two eSATA ports on my motherboards supports port multiplier.  So I flashed the RocketRaid cards that came with the enclosures with a JBOD firmware and the supplied kext seems stable so far.  I may upgrade to a nicer eSATA card in the future, but don't consider it a pressing need.

aaron:~ $ zpool status

  pool: Bulldozer

 state: ONLINE

 scrub: none requested

config:

NAME          STATE     READ WRITE CKSUM

Bulldozer     ONLINE       0     0     0

 raidz1      ONLINE       0     0     0

   disk3s2   ONLINE       0     0     0

   disk4s2   ONLINE       0     0     0

   disk5s2   ONLINE       0     0     0

   disk6s2   ONLINE       0     0     0

   disk7s2   ONLINE       0     0     0

   disk8s2   ONLINE       0     0     0

   disk9s2   ONLINE       0     0     0

   disk10s2  ONLINE       0     0     0

   disk11s2  ONLINE       0     0     0

   disk12s2  ONLINE       0     0     0

errors: No known data errors

aaron:~ $ zfs list

NAME        USED  AVAIL  REFER  MOUNTPOINT

Bulldozer  17.4T  6.48T  17.4T  /Volumes/Bulldozer

aaron:~ $ zpool history

History for 'Bulldozer':

2013-01-21.23:27:15 zpool create Bulldozer raidz1 /dev/disk5s2 /dev/disk6s2 /dev/disk7s2 /dev/disk8s2 /dev/disk9s2 /dev/disk10s2 /dev/disk11s2 /dev/disk12s2 /dev/disk13s2 /dev/disk14s2

2013-01-21.23:45:29 zpool import -f 17150110679673639280

2013-02-10.20:34:36 zpool import -f 17150110679673639280

2013-02-10.20:38:31 zpool import -f 17150110679673639280

2013-02-10.20:41:45 zpool import -f 17150110679673639280

Migrating data was no problem once I accepted that Finder wasn't going to get the job done.  After that initial transfer of ~17TB, I have no problems using Finder for day to day stuff.

Thanks for all the help.  After one month it's been great - I don't notice or have to deal with anything, which is the way it should be.  Having said that, I still haven't destroyed the original data, I'll probably have to simulate some disaster situations before I'm comfortable with that.

[Fastmail Jason]

Nice. ;)

--

Jason Belec

Sent from my iPad

--
 
---
You received this message because you are subscribed to the Google Groups "zfs-macos" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zfs-macos+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Gregg Wonderly]

I would go ahead and use 'dd if=/dev/zero of=XXXX' where XXXX is one of your raidz1 disks, to simulate a single disk failure by killing data on the drive.  Then, use scrub on the pool to find the errors, and then repair the errors with a spare disk swap in, or a "zpool replace Bulldozer XXX XXX" to cause a resilver of the drive.  Time how long it takes to do the resilver, and make sure you are content with that time of being exposed for another failure of another disk, which could completely destroy your pool.

With raidz, I'm not really excited about it, without at least raidz2.  Just do some evaluation of recovery modes so that you understand what has to happen, and how you will deal with getting that done.

Gregg

[Jason]

Yes to all that. Another idea is to have another pool you send/receive to to ensure backups exist. I know I do. ;)

Jason

Sent from my iPhone