Solaris with ZFS Dedup on HP DL380p G8

[e...@sin-sports.de]

Hi list,

i am trying to install a recent solaris (or one of it's derivates) on a new HP DL380p machine with dual Intel E5-2665 CPUs. Until now without success.

Solaris 11.1 text, booted from cd, panics when starting the installation after selecting the language and keyboard layout.
OpenIndiana 151a does not find the system disks on the P420 Smart Array controller
Solaris 10 1/13 boots and installs fine, but has no zfs dedup, which actually is the main reason to setup this machine.

On Solaris 11 I suspected the driver for the P420 to cause the panic an disabled it in the system BIOS and disabled it in the grub kernel options.

I know this machine has no Oracle certification for any Solaris and HP lists it only as Solaris 10 compatible.

Any hints would be really appreciated!

Kind regards,
Tom

[John D Groenveld]

In article <802817cb-99d0-42c0...@googlegroups.com>,

<e...@sin-sports.de> wrote:
>i am trying to install a recent solaris (or one of it's derivates) on a
>new HP DL380p machine with dual Intel E5-2665 CPUs. Until now without
>success.
>
>Solaris 11.1 text, booted from cd, panics when starting the installation
>after selecting the language and keyboard layout.
>OpenIndiana 151a does not find the system disks on the P420 Smart Array
>controller
>Solaris 10 1/13 boots and installs fine, but has no zfs dedup, which
>actually is the main reason to setup this machine.
>
>On Solaris 11 I suspected the driver for the P420 to cause the panic an
>disabled it in the system BIOS and disabled it in the grub kernel
>options.

Swap the P420 RAID Controller for a H222 HBA?
<URL:http://forums.servethehome.com/raid-controllers-host-bus-adapters/599-lsi-raid-controller-hba-complete-listing-plus-oem-models.html>

John
groe...@acm.org

[e...@sin-sports.de]

I probably would have done so, when the installer of S11.1 had run flawlessly after disabling the P420. But it panics even with disabled controller... I fear, that this won't help.

Tom

[Casper H.S. Dik]

e...@sin-sports.de writes:

>I probably would have done so, when the installer of S11.1 had run flawlessly after disabling the P420. But it panics even with disabled controller... I fear, that this won't help.

Do you have a traceback? (Boot w/ -k so the kernel will stop in the kernel on panic)

You could try disablng VT-x/VT-D (specifically the latter).

Casper

[e...@sin-sports.de]

Hi Casper,

thanks a million :) Disabling VT-D did the trick!

Regards,
Tom

[cindy swearingen]

Be sure you are aware of cautions when using dedup. Make sure your
data is dedupable,
test it first by simulating your workload before enabling it in
production, and carefully
review the memory requirements, here:

http://docs.oracle.com/cd/E26502_01/html/E29007/gazss.html#gjhav

For some workloads and system configurations, compression is an
excellent alternative.

Thanks, Cindy

[e...@sin-sports.de]

Cindy,

I tested this already on a testmachine and in this very particular workload i reached a dedup rate of roughly 1:20! This is why i needed it so badly ;)
Storage is a 5T mirror consisting of 2x8 900G SAS disks on separate LSI controllers in HP D2700 enclosures. 128GB memory should be enough to ensure write performance with dedup for quite some time.

Regards,
Tom

[e...@sin-sports.de]

Hi all,

it is not that i haven't been warned. It is just that i simply do not understand why write performance on the newly created pool ist so horrible...

Hopefully, i'll get some mor advise here. Some basic figures:

root@server12:~# zpool status -D datenhalde
pool: datenhalde
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
datenhalde ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
c11t5000C5005EE0F5D5d0 ONLINE 0 0 0
c12t5000C5005EDBBB95d0 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
c11t5000C5005EE20251d0 ONLINE 0 0 0
c12t5000C5005ED658F1d0 ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
c11t5000C5005ED80439d0 ONLINE 0 0 0
c12t5000C5005EDB23F1d0 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
c11t5000C5005EDA2315d0 ONLINE 0 0 0
c12t5000C5005ED6E049d0 ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
c11t5000C5005EDBB289d0 ONLINE 0 0 0
c12t5000C5005EDB9479d0 ONLINE 0 0 0
mirror-5 ONLINE 0 0 0
c11t5000C5005EDD8385d0 ONLINE 0 0 0
c12t5000C5005ED72855d0 ONLINE 0 0 0
mirror-6 ONLINE 0 0 0
c11t5000C5005ED8759Dd0 ONLINE 0 0 0
c12t5000C5005EE3AB59d0 ONLINE 0 0 0
spares
c11t5000C5005ED6CEADd0 AVAIL
c12t5000C5005EDA2CD5d0 AVAIL

errors: No known data errors

DDT entries 5354008, size 292 on disk, 152 in core

bucket allocated referenced
______ ______________________________ ______________________________
refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE
------ ------ ----- ----- ----- ------ ----- ----- -----
1 3,22M 411G 411G 411G 3,22M 411G 411G 411G
2 1,28M 163G 163G 163G 2,93M 374G 374G 374G
4 440K 54,9G 54,9G 54,9G 2,12M 271G 271G 271G
8 140K 17,5G 17,5G 17,5G 1,39M 177G 177G 177G
16 36,1K 4,50G 4,50G 4,50G 689K 85,9G 85,9G 85,9G
32 6,26K 798M 798M 798M 277K 34,4G 34,4G 34,4G
64 1,92K 244M 244M 244M 136K 16,9G 16,9G 16,9G
128 56 6,52M 6,52M 6,52M 10,5K 1,23G 1,23G 1,23G
256 222 27,5M 27,5M 27,5M 71,0K 8,80G 8,80G 8,80G
512 2 256K 256K 256K 1,38K 177M 177M 177M
1K 4 384K 384K 384K 6,00K 612M 612M 612M
4K 1 512 512 512 4,91K 2,45M 2,45M 2,45M
16K 1 128K 128K 128K 24,9K 3,11G 3,11G 3,11G
512K 1 128K 128K 128K 599K 74,9G 74,9G 74,9G
Total 5,11M 652G 652G 652G 11,4M 1,43T 1,43T 1,43T

root@server12:~# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
datenhalde 5,69T 662G 5,04T 11% 2.22x ONLINE -

root@server12:~# ./arc_summery.pl
System Memory:
Physical RAM: 131021 MB
Free Memory : 18102 MB
LotsFree: 2047 MB

ZFS Tunables (/etc/system):

ARC Size:
Current Size: 101886 MB (arcsize)
Target Size (Adaptive): 103252 MB (c)
Min Size (Hard Limit): 64 MB (zfs_arc_min)
Max Size (Hard Limit): 129997 MB (zfs_arc_max)

ARC Size Breakdown:
Most Recently Used Cache Size: 100% 103252 MB (p)
Most Frequently Used Cache Size: 0% 0 MB (c-p)

ARC Efficency:
Cache Access Total: 124583164
Cache Hit Ratio: 70% 87975485 [Defined State for buffer]
Cache Miss Ratio: 29% 36607679 [Undefined State for Buffer]
REAL Hit Ratio: 103% 128741192 [MRU/MFU Hits Only]

Data Demand Efficiency: 91%
Data Prefetch Efficiency: 29%

CACHE HITS BY CACHE LIST:
Anon: --% Counter Rolled.
Most Recently Used: 74% 65231813 (mru) [ Return Customer ]
Most Frequently Used: 72% 63509379 (mfu) [ Frequent Customer ]
Most Recently Used Ghost: 0% 0 (mru_ghost) [ Return Customer Evicted, Now Back ]
Most Frequently Used Ghost: 0% 0 (mfu_ghost) [ Frequent Customer Evicted, Now Back ]
CACHE HITS BY DATA TYPE:
Demand Data: 15% 13467569
Prefetch Data: 4% 3555720
Demand Metadata: 80% 70648029
Prefetch Metadata: 0% 304167
CACHE MISSES BY DATA TYPE:
Demand Data: 3% 1281154
Prefetch Data: 23% 8429373
Demand Metadata: 73% 26879797
Prefetch Metadata: 0% 17355


root@server12:~# echo "::arc" | mdb -k
hits = 88823429
misses = 37306983
demand_data_hits = 13492752
demand_data_misses = 1281335
demand_metadata_hits = 71470790
demand_metadata_misses = 27578897
prefetch_data_hits = 3555720
prefetch_data_misses = 8429373
prefetch_metadata_hits = 304167
prefetch_metadata_misses = 17378
mru_hits = 66467881
mru_ghost_hits = 0
mfu_hits = 64253247
mfu_ghost_hits = 0
deleted = 41770876
mutex_miss = 172782
hash_elements = 18446744073676992500
hash_elements_max = 18446744073709551615
hash_collisions = 12375174
hash_chains = 18446744073698514699
hash_chain_max = 9
p = 103252 MB
c = 103252 MB
c_min = 64 MB
c_max = 129997 MB
size = 102059 MB
buf_size = 481 MB
data_size = 100652 MB
other_size = 924 MB
l2_hits = 0
l2_misses = 28860232
l2_feeds = 0
l2_rw_clash = 0
l2_read_bytes = 0 MB
l2_write_bytes = 0 MB
l2_writes_sent = 0
l2_writes_done = 0
l2_writes_error = 0
l2_writes_hdr_miss = 0
l2_evict_lock_retry = 0
l2_evict_reading = 0
l2_abort_lowmem = 0
l2_cksum_bad = 0
l2_io_error = 0
l2_hdr_size = 0 MB
memory_throttle_count = 0
meta_used = 1406 MB
meta_max = 1406 MB
meta_limit = 0 MB
arc_no_grow = 1
arc_tempreserve = 0 MB
root@server12:~#

The write-performance is really really slow:

read/write within this pool:
root@server12:/datenhalde/s12test/Bild-DB/Testaktion# /usr/gnu/bin/dd if=Test.tif of=Test2.tif
1885030+1 records in
1885030+1 records out
965135496 bytes (965 MB) copied, 145,923 s, 6,6 MB/s

read from this pool and write to the root-pool:

root@server12:/datenhalde/s12test/Bild-DB/Testaktion# /usr/gnu/bin/dd if=Test.tif of=/tmp/Test2.tif
1885030+1 records in
1885030+1 records out
965135496 bytes (965 MB) copied, 9,51183 s, 101 MB/s
root@server12:/datenhalde/s12test/Bild-DB/Testaktion# /usr/gnu/bin/dd if=FS2013_Fashionation_Beach_06.tif of=FS2013_Test.tif


I just do net get this. Why is it that slow? Am i missing any tunable parameters? From the above figures the ddt should use 5354008*152=776MB in RAM. That should fit easily.

Sorry for the longish post, but i really need some help here, because the real data with much higher dedup ratio is still to be copied to that pool.
Compression is no real alternative, because most of the data will be compressed images and i don't expect to see great compression ratios.

TIA and kind regards,
Tom

[Stefaan A Eeckels]

Hi Tom,

On Tue, 16 Apr 2013 01:31:37 -0700 (PDT)
e...@sin-sports.de wrote:

> Sorry for the longish post, but i really need some help here, because
> the real data with much higher dedup ratio is still to be copied to
> that pool. Compression is no real alternative, because most of the
> data will be compressed images and i don't expect to see great
> compression ratios.

Just a couple of thoughts, no real suggestions unfortunately.

Have you tried to use a different dedup algorithm, e.g. fletcher4 with
verify?

Any idea of how loaded the cpus are during the copy operation? How are
your DL380p systems spec'ed, cpu-wise?

Have you tried to copy from a different device, i.e. from the root pool
to the dedup'ed pool?

Take care,

--
Stefaan Eeckels (Stefaan dot Eeckels at ecc dot lu)