Slow performance in new USB3 enclosure

[Wonka]

Hi!

Recently transplanted a 9GB raidz1 pool to a new USB3 enclosure (http://www.amazon.com/gp/product/B004WNLPH8/ref=oh_details_o01_s00_i00) on Mac Mini Server 6,2 (2012 model with USB3) and am seeing quite slow performance.

Esp. noticeable with heavy tasks like compressing video or decompressing archives. For example, if compressing a video file existing in the ZFS pool, it takes about 20min. Compressing same file from system's internal SATA drive takes <2m.

To help, here are some details on the pool & the drives it comprises of:

zpool status output:

NAME         STATE     READ WRITE CKSUM

Media        ONLINE       0     0     0

 raidz1     ONLINE       0     0     0

   disk3s2  ONLINE       0     0     0

   disk2s2  ONLINE       0     0     0

   disk4s2  ONLINE       0     0     0

Drives are all matched 3x 3TB HDS723030ALA640.

So questions, yes: 

1. Is this expected performance? 
   
2. If not, can anyone offer tips on isolating the bottleneck? Is it the USB3 enclosure somehow? Need to raise the performance level however i can.
   
3. Which benchmark programs support mac-ZFS implementation? (tried XBench but it stopped on -54 error on the ZFS pool only, internal drives benched just fine).

Thanks for any help on this. In any case, long live Mac-ZFS!! :)

Wonka

[Daniel Becker]

Difficult to say, as "2 minutes" doesn't really mean much by itself. However, a few things to consider:

In terms of raw throughput, 3 SATA-3 links give you almost four times the aggregate throughput as USB 3.0. However, I doubt that that would be a problem with compression (unless you're using a trivial scheme) or encoding (unless you're encoding uncompressed source material), as both would likely be CPU-limited before you hit anywhere near 5 Gbps.

Does your enclosure support UASP (USB-attached SCSI protocol)? Many older and/or cheaper enclosures only support BOT (bulk-only transfer), which means no command queuing and lots of unnecessary round trip times between host controller and device. This will particularly hurt heavy I/O, e.g. if source and destination for your encoding/compression tests are located on the same pool.

In any case, the best way to figure out what's going on is usually looking at "zpool iostat -v 1" and/or "iostat 1" while you're doing your thing.

Wonka

--
 
 
 

[Daniel Bethe]

Hey I guess I'll throw in the standard troubleshooting advice of swapping stuff out.  Move your enclosure to a different system or a different USB controller and try again.  Try another filesystem on the same hardware, to swap out MacZFS.  Boot the same system to another OS.  Ensure that your enclosure is actually negotiating USB 3 and not just 2.  Daniel, fyi, he said it's a new USB 3 enclosure from 2012 so do those phenomena still exist on USB 3, assuming that it is properly using USB 3?

---

From: Daniel Becker <razz...@gmail.com>
To: zfs-...@googlegroups.com
Sent: Thursday, 15 November 2012, 16:25
Subject: Re: [zfs-macos] Slow performance in new USB3 enclosure

--
 
 
 

[Daniel Becker]

On Nov 15, 2012, at 2:36 PM, Daniel Bethe <d...@smuckola.org> wrote:

> Daniel, fyi, he said it's a new USB 3 enclosure from 2012 so do those phenomena still exist on USB 3, assuming that it is properly using USB 3?

My understanding is that UASP is essentially USB 3 only (although a USB 2 version is defined), and that even recent USB 3 controllers and enclosures are not guaranteed to support it (although the controllers used in Macs apparently do). Google should have more info.

[j. matthew]

Thank you all for the feedback; have been in contact with the manufacturer & it took some for me to go through steps to isolate this, and time is not something have had an abundance of, as of late  :(

My understanding is that UASP is essentially USB 3 only (although a USB 2 version is defined), and that even recent USB 3 controllers and enclosures are not guaranteed to support it (although the controllers used in Macs apparently do). Google should have more info.

Correct, manufacturer says that UASP is supported.

Minor update; the pool layout has changed, it is now 5 drives of this type:

Hitachi Deskstar 3.5-Inch 4TB 7200 RPM SATA III 6Gbps Internal Hard Drive Kit (0S03355) 

pool: Media

state: ONLINE

scrub: none requested

config:

NAME         STATE     READ WRITE CKSUM

Media        ONLINE       0     0     0

  raidz1     ONLINE       0     0     0

    disk2s2  ONLINE       0     0     0

    disk3s2  ONLINE       0     0     0

    disk4s2  ONLINE       0     0     0

    disk5s2  ONLINE       0     0     0

    disk6s2  ONLINE       0     0     0

errors: No known data errors

Have attached performance reports of tests done with the manufacturer, as well as the "zpool -iostat -v 1" output from one of the tests; test first writes, then reads.

The manufacturer assured the performance seen is expected; however, am finding real world tests are slower than desired, in direct contrast to the performance specs of the connection method (USB3).

Slow performance is especially noticeable during heavy operations like verifying checksums on files, or compressing & encoding video. Such operations seem to take > 4x as long as on internal drives; understanding yes it won't be 1:1 but would think yeah with USB3 such operations shouldn't be so slow, as Daniel suggests below.

Basically the claim is the performance results reflect 1 drive being accessed, which is expected? Can anyone confirm; is this expected performance from a raidz1 pool of 5 drives with ZFS? 

Was under the impression that with ZFS, data is spread across all drives in the pool, so in my case, performance throughput should be closer to 5 drives, not the performance of 1 drive, as seen.

Am I expecting too much here?

joe

[Björn Kahl]

Am 07.01.13 16:52, schrieb j. matthew:

Unfortunately: Wrong.

ZFS spread data over the drives in stripe configuration.

In raidz or mirror configuration it has to write the same data multiple
times. For a mirror it literately writes the same data twice, for
raidz it has to write the data and the parity.

The exact write pattern depends on a number of factors, including the
incoming write data rate towards the file system. Basically, any
write request of D bytes to a file system made of N disks in raidz
configuration is transformed into N smaller write requests of D/(N-1)
bytes. In case your I/O subsystem can access all N drives
independently and in parallel, then and only then you will see almost
(N-1) times the performance of a single disk (minus some overhead for
data splitting, parity calculation and scheduling of N-1 instead of 1
I/O requests).

In all other cases, for example when your N drives sit in the same
enclosure with one shared link, then you will see the same performance
as with a single disk.

> Am I expecting too much here?

I'd say yes, you expect to much.

To quote https://blogs.oracle.com/roch/entry/when_to_and_not_to:

"Effectively, as a first approximation, an N-disk RAID-Z group will
behave as a single device in terms of delivered random input
IOPS. Thus a 10-disk group of devices each capable of 200-IOPS, will
globally act as a 200-IOPS capable RAID-Z group."


Best

Björn

> On Nov 15,2012, at 16:47, Daniel Becker <razz...@gmail.com> wrote:
>
>>> Daniel, fyi, he said it's a new USB 3 enclosure from 2012 so do those phenomena still exist on USB 3, assuming that it is properly using USB 3?
>>

>> My understanding is that UASP is essentially USB 3 only (although a USB 2 version is defined), and that even recent USB 3 controllers and enclosures are not guaranteed to support it (although the controllers used in Macs apparently do). Google should have more info.
>>

--
| Bjoern Kahl +++ Siegburg +++ Germany |
| "googlelogin@-my-domain-" +++ www.bjoern-kahl.de |
| Languages: German, English, Ancient Latin (a bit :-)) |

[Daniel Becker]

On Jan 7, 2013, at 11:07 AM, Björn Kahl <googl...@bjoern-kahl.de> wrote:

> In all other cases, for example when your N drives sit in the same
> enclosure with one shared link, then you will see the same performance
> as with a single disk.

In terms of throughput (which is what the OP was looking at, I believe), this would only be the case if a single drive could actually saturate the link. For mechanical drives behind USB3, this should not be the case. (Also, if it were, note that write throughput would actually be (N-1)/N of a single disk due to parity.)

> To quote https://blogs.oracle.com/roch/entry/when_to_and_not_to:
>
> "Effectively, as a first approximation, an N-disk RAID-Z group will
> behave as a single device in terms of delivered random input
> IOPS. Thus a 10-disk group of devices each capable of 200-IOPS, will
> globally act as a 200-IOPS capable RAID-Z group."

Note that this is specifically talking about IOPS (i.e., random / small accesses), not sustained throughput, which I believe is what the OP was trying to benchmark.

Assuming sustained throughput for large sequential transfers is indeed what we're interested in, if we assume we're not bottlenecked by the connection, the effective read throughput of a RAID-Z array with N disks is up to N times that of a single drive, while the write throughput is up to (N-1) times that of a single drive (due to the addition of parity). If there is a bottleneck in the connection, the effective read throughput will be limited by the throughput of the connection, while the effective write throughput will be less than (N-1)/N times that.

[j. matthew]

Thanks all for the continued feedback; have to admit am a bit confused by it :)

Bjorn says i shouldn't expect more than what i'm seeing, yet Daniel are you suggesting otherwise?

Basically need to know if this unit is performing up to spec. or if I'd do better to replace it with something else.

Thanks,

j

[Fastmail Jason]

Anyone here can only provide basic assumptions on speed as we would have to have the same setup. That said in many of my mad science setups/tests I have not seen that great a speed boost with USB3 no matter what they claim. Now the easy part, what speed do you want to have? Then it's far easier to provide you a path.


--
Jason Belec
Sent from my iPad

> --
>
>
>

[j. matthew]

Hi Jason,

Thanks for the information. My needs are fairly straight-forward:

- high speed, sufficient that can work directly off the pool (not have to copy files to internal drive to process then copy back to pool, as am doing now).

- compact unit (why i moved from previous self-contained box to mac mini + external enclosure, as mentioned in the OP)

- as much space as possible (4 bay minimum). 

Desire to "future-proof" a bit so i don't have to constantly be thinking about space (for a while anyway) hence the 5 4TB drives, but with this, also need sufficient performance (greater than am seeing).

Hope that helps clear the "path" :)

j

--

[Fastmail Jason]

Well your sort of hurting yourself, large drives are never fast, too much surface to cover on rotating-rust. Speed is from small drives (servers - small surface area) or these days SSDs. The best bet for day to day is a smaller set of rotational drives or SSDs if you could. Then those big drives can be for backups daily from ZFS. 

If that Mac Mini has Thunderbolt you can get some serious speed no matter what, but the connections are still costly.

I would suggest ESATA for everything other than new Thunderbolt tech.

As I said USB3 makes a lot claims and I have tested it with several setups, but ESATA was still better and Thunderbolt has been stunning. For anything really requiring speed I also have found SSDs, even older ones far superior to any rotational-rust drives in similar configuration.

Quality of all parts/components in play also plays a factor along with drivers.

--

Jason Belec

Sent from my iPad

--
 
 
 

[Daniel Becker]

My suggestion would be to figure out what kind of performance you're actually looking for (random vs. sequential) and then methodologically benchmark on step at a time, starting with raw device throughput (dd, iostat, ...) and working your way up. If you can boot into an operating system that has a somewhat more up-to-date ZFS implementation, that might be instructive as well. As it is, you're measuring too many things at once, any number of which may be partially responsible for the results you're seeing.

> --
>
>
>

[Daniel Becker]

On Jan 7, 2013, at 12:40 PM, Fastmail Jason <jason...@belecmartin.com> wrote:

Well your sort of hurting yourself, large drives are never fast, too much surface to cover on rotating-rust.

For a given form factor & RPM, it is actually exactly the opposite, at least as far as sequential throughput is concerned: higher surface density equals higher read/write speeds.

Speed is from small drives (servers - small surface area) or these days SSDs. The best bet for day to day is a smaller set of rotational 

--
 
 
 

[Jason]

No. Sorry. I disagree. But mileage may vary. 

Jason

Sent from my iPhone

[Daniel Becker]

Sequential throughput for a single disk is proportional to the number of R/W heads times the head velocity; to first order, the latter is directly proportional to surface density and RPM. All else being equal, higher capacity drives have either more platters or higher surface density, and therefore yield higher sequential throughput than a lower-capacity drive with the same form factor and RPM. This has nothing to do with mileage.

(IOPS are a different matter, of course, as smaller drives give you more spindles to reach a given total capacity.)

--
 
 
 

[Daniel Becker]

"High speed" is not very meaningful. As was mentioned earlier, it makes a huge different whether you're primarily concerned with large sequential transfers (-> throughput is critical) or small random I/O (-> IOPS are critical). It sounds like you've been benchmarking essentially single-user scenarios that don't involve a lot of seeking so far; if that is indeed your primary use case, throughput is going to be more important than IOPS. I would suggest you run some targeted experiments to see what kind of raw throughput you can get from a single disk in the enclosure, and then repeat the experiment with multiple disks simultaneously.

I.e., do something like this in a Terminal window to test one drive:

sudo -s

dd if=/dev/rdisk2 of=/dev/null bs=1m &

iostat -w 1 disk2

Close terminal window when done to kill background transfer. Then, to test multiple drives concurrently, do this:

sudo -s

for f in 2 3 4 5 6; do ( dd if=/dev/rdisk$f of=/dev/null bs=1m & ); done

iostat -w 1 disk2 disk3 disk4 disk5 disk6

Once again, close terminal window when done to kill background transfers.

Daniel

--
 
 
 

[Gregg Wonderly]

The overall issue is "seek".  If you have 4x4GB disks, or 40x400MB disk, you have different "seek" vs "access" formulas.  I think this is what Jason is  speaking to.  For any particular "data", its location on disk, compared to where the head is, creates a certain latency.  If you just start reading, and read the same "file" contiguously, then a filesystem which "wrote" the data contiguously under the head(s) can provide a reasonable speed up/equalization of transfer rates.

However, for general computer system operation, the more "independent" heads you have, the more "random" I/O speed you can obtain, because the heads can "move" independently, and the OS can schedule "seeks" ahead of the upcoming reads.

Very view people actually use a disk system where "huge" contiguous systems are extremely common.  Yes, the OP was about a "test" which involved large, contiguous I/O performance.

There are additional reliability issues with "data" being so narrow on the service (higher density), that you lose some reliability.  It takes much smaller surface deterioration to create much large data loss.

Gregg Wonderly

[Daniel Becker]

On Jan 7, 2013, at 2:07 PM, Gregg Wonderly <greg...@gmail.com> wrote:

The overall issue is "seek".  If you have 4x4GB disks, or 40x400MB disk, you have different "seek" vs "access" formulas.  I think this is what Jason is  speaking to.  For any particular "data", its location on disk, compared to where the head is, creates a certain latency.  If you just start reading, and read the same "file" contiguously, then a filesystem which "wrote" the data contiguously under the head(s) can provide a reasonable speed up/equalization of transfer rates.

However, for general computer system operation, the more "independent" heads you have, the more "random" I/O speed you can obtain, because the heads can "move" independently, and the OS can schedule "seeks" ahead of the upcoming reads.

I am well aware of that, which is why I explicitly stated that I was talking about sequential throughput in my earlier posts.

Very view people actually use a disk system where "huge" contiguous systems are extremely common.  Yes, the OP was about a "test" which involved large, contiguous I/O performance.

Hence my earlier comments.

There are additional reliability issues with "data" being so narrow on the service (higher density), that you lose some reliability.  It takes much smaller surface deterioration to create much large data loss.

Certainly, but this is an entirely separate aspect and unrelated to performance (at least until the drive is degraded).

--
 
 
 

[j. matthew]

See the original post; ran some specific video compression tests using same file: on ZFS + enclosure conversion yielded a time of about 20m, on internal drive <2min.

If you need more technical tests than that, please bear in mind i am alas but a humble user (with big storage needs); can only speak in the language I know ;)

[Daniel Becker]

Oh, also, did you ever say how much memory your Mac mini has?

<iostat.txt>

<image001.png>

<image002.png>

<image003.png>

On Nov 15,2012, at 16:47, Daniel Becker <razz...@gmail.com> wrote:

Daniel, fyi, he said it's a new USB 3 enclosure from 2012 so do those phenomena still exist on USB 3, assuming that it is properly using USB 3?

My understanding is that UASP is essentially USB 3 only (although a USB 2 version is defined), and that even recent USB 3 controllers and enclosures are not guaranteed to support it (although the controllers used in Macs apparently do). Google should have more info.

On Nov 15, 2012, at 2:36 PM, Daniel Bethe <d...@smuckola.org> wrote:

Hey I guess I'll throw in the standard troubleshooting advice of swapping stuff out.  Move your enclosure to a different system or a different USB controller and try again.  Try another filesystem on the same hardware, to swap out MacZFS.  Boot the same system to another OS.  Ensure that your enclosure is actually negotiating USB 3 and not just 2.  Daniel, fyi, he said it's a new USB 3 enclosure from 2012 so do those phenomena still exist on USB 3, assuming that it is properly using USB 3?

From: Daniel Becker <razz...@gmail.com>
To: zfs-...@googlegroups.com 
Sent: Thursday, 15 November 2012, 16:25
Subject: Re: [zfs-macos] Slow performance in new USB3 enclosure

Difficult to say, as "2 minutes" doesn't really mean much by itself. However, a few things to consider:

In terms of raw throughput, 3 SATA-3 links give you almost four times the aggregate throughput as USB 3.0. However, I doubt that that would be a problem with compression (unless you're using a trivial scheme) or encoding (unless you're encoding uncompressed source material), as both would likely be CPU-limited before you hit anywhere near 5 Gbps.

Does your enclosure support UASP (USB-attached SCSI protocol)? Many older and/or cheaper enclosures only support BOT (bulk-only transfer), which means no command queuing and lots of unnecessary round trip times between host controller and device. This will particularly hurt heavy I/O, e.g. if source and destination for your encoding/compression tests are located on the same pool.

In any case, the best way to figure out what's going on is usually looking at "zpool iostat -v 1" and/or "iostat 1" while you're doing your thing.


On Thu, Nov 15, 2012 at 1:34 PM, Wonka <cas...@gmail.com> wrote:

Hi!

Recently transplanted a 9GB raidz1 pool to a new USB3 enclosure (http://www.amazon.com/gp/product/B004WNLPH8/ref=oh_details_o01_s00_i00) on Mac Mini Server 6,2 (2012 model with USB3) and am seeing quite slow performance.

Esp. noticeable with heavy tasks like compressing video or decompressing archives. For example, if compressing a video file existing in the ZFS pool, it takes about 20min. Compressing same file from system's internal SATA drive takes <2m.

To help, here are some details on the pool & the drives it comprises of:

zpool status output:

NAME         STATE     READ WRITE CKSUM
Media        ONLINE       0     0     0
  raidz1     ONLINE       0     0     0

    disk3s2  ONLINE       0     0     0
    disk2s2  ONLINE       0     0     0
    disk4s2  ONLINE       0     0     0

Drives are all matched 3x 3TB HDS723030ALA640.

So questions, yes: 

• Is this expected performance? 

• If not, can anyone offer tips on isolating the bottleneck? Is it the USB3 enclosure somehow? Need to raise the performance level however i can.

• Which benchmark programs support mac-ZFS implementation? (tried XBench but it stopped on -54 error on the ZFS pool only, internal drives benched just fine).

Thanks for any help on this. In any case, long live Mac-ZFS!! :)

Wonka

--

[j. matthew]

Oh I probably didn't! :$

If it helps, the mac mini has 16GB o' RAM :P

j

[Daniel Becker]

On Jan 7, 2013, at 2:23 PM, "j. matthew" <cas...@gmail.com> wrote:

See the original post; ran some specific video compression tests using same file: on ZFS + enclosure conversion yielded a time of about 20m, on internal drive <2min.

I'm guessing this is not an Apples-to-Apples comparison, though, right? I.e., am I correct in assuming that you're comparing internal drives with HFS+ against the USB enclosure with ZFS here? Do keep in mind that the MacZFS code base hasn't exactly seen a lot of tuning effort, and that it is based on a fairly old version of the ZFS code.

If you need more technical tests than that, please bear in mind i am alas but a humble user (with big storage needs); can only speak in the language I know ;)

Well, if you like, you could just run the commands I listed earlier (they are read-only, so no worries about messing up your data) and post back the results.

On Jan 7,2013, at 15:02, Daniel Becker <razz...@gmail.com> wrote:

I would suggest you run some targeted experiments to see what kind of raw throughput you can get from a single disk in the enclosure, and then repeat the experiment with multiple disks simultaneously.

--