Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

Question about HotPlug and cciss hp storage

326 views
Skip to first unread message

Julien Groselle

unread,
Feb 17, 2012, 5:40:03 AM2/17/12
to
Hi,

After peel the web to find any information about how to add / remove hard drive on Debian.
My Hardware is HP  ProLiant DL380 G7, with HP Smart Array P410i Controller version 3.50.

I had preferred soft RAID solution with mdadm. But the only bad point for me is the Hot Plug.
Could some one help me to add / remove hard drive disk from my server ?

I have tested many solution with "rescan", and i think the solution is not so far...

I'm sure that at least one Debian user use this hardware and want use hot plug.
If not, I'm listening for any information about hot plug on any hardware.

Have all a good day.

JG.

Julien Groselle

unread,
Feb 17, 2012, 6:10:02 AM2/17/12
to
Hello Frank,

Fist of all, thank you for your fast answer.

I use Soft Raid... and not Hard. So the disk array present me all the disks without any RAID or control.
So if i want to remove a HDD, i have to set it faulty and logical remove from mdadm before hard remove HDD physically.

And when i want add the new one... my system doesn't detect it... So i need a little voodoo solution to "force" Debian see the HDD :)

JG.

Le 17 février 2012 11:46, Frank <fr...@dead-link.org> a écrit :
If you use the onboard controller there is no need to do some software voodoo. Just replace a faulty drive by a new one. The rebuild starts automatically.

Your device is /dev/cciss/c0d0

Frank

Frank

unread,
Feb 17, 2012, 6:10:02 AM2/17/12
to
If you use the onboard controller there is no need to do some software voodoo. Just replace a faulty drive by a new one. The rebuild starts automatically.

Your device is /dev/cciss/c0d0

Frank

Am 17.02.2012 um 11:35 schrieb Julien Groselle <julien....@gmail.com>:

--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/B6CB4B59-86E8-4718...@dead-link.org

frank

unread,
Feb 17, 2012, 8:20:02 AM2/17/12
to
On Fri, 2012-02-17 at 12:02 +0100, Julien Groselle wrote:

> I use Soft Raid... and not Hard. So the disk array present me all the
> disks without any RAID or control.
> So if i want to remove a HDD, i have to set it faulty and logical
> remove from mdadm before hard remove HDD physically.

Have you tried to rescan the scsi bus? If so, how have you done it?


Bye
Frank


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1329483607.2...@nero.internal.friendscout24.de

Julien Groselle

unread,
Feb 17, 2012, 9:10:01 AM2/17/12
to
First, we use Soft Raid because we like mdadm so much ! ;)
We have amazing performances and a nice supervision on all the RAID status.

We will keep Software RAID.

The only weak point is that HotPlug.
To Frank, I have tried that :
# echo "rescan" > /proc/driver/cciss/cciss0
And I have a write error : invalid argument

So no, I didn't rescan anything... It's just what I need (a command line to recan all HDD device)

JG.

Christofer C. Bell

unread,
Feb 17, 2012, 11:30:01 AM2/17/12
to
On Fri, Feb 17, 2012 at 8:04 AM, Julien Groselle
<julien....@gmail.com> wrote:
>
> So no, I didn't rescan anything... It's just what I need (a command line to
> recan all HDD device)

Apparently it's as easy as this:

echo "- - -" > /sys/class/scsi_host/host0/scan

Where 0 is the the controller number. This information is taken from here:

http://rackerhacker.com/2009/04/23/re-scan-the-scsi-bus-in-linux-after-hot-swapping-a-drive/

--
Chris


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CAOEVnYvzj4Y0jtJjV+OtTD7K...@mail.gmail.com

frank

unread,
Feb 17, 2012, 11:30:02 AM2/17/12
to
On Fri, 2012-02-17 at 15:04 +0100, Julien Groselle wrote:

> So no, I didn't rescan anything... It's just what I need (a command
> line to recan all HDD device)
>
On Lenny:

apt-get install scsiadd

scsiadd -s

On Squeeze:

apt-get install scsitools

/sbin/rescan-scsi-bus.sh

Cheers
Frank


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1329495006.2...@nero.internal.friendscout24.de

Julien Groselle

unread,
Feb 17, 2012, 11:50:02 AM2/17/12
to
> On Squeeze:
> apt-get install scsitools
> /sbin/rescan-scsi-bus.sh

Humm... Nice ! For now i can't install this tools because of too many unwanted dependencies,
but after installed it on a test server, it seems to give great results.

Thank you for all your answer.
I will make a return about the entire test later.

Maybe you have reduce my work time for this night :P

JG.

Christofer C. Bell

unread,
Feb 17, 2012, 12:00:02 PM2/17/12
to
On Fri, Feb 17, 2012 at 10:34 AM, frank <fr...@dead-link.org> wrote:
> On Fri, 2012-02-17 at 10:26 -0600, Christofer C. Bell wrote:
>
>> echo "- - -" > /sys/class/scsi_host/host0/scan
>
> Have you tried it? Or just copied and pasted?
>
> echo "- - -" > /sys/class/scsi_host/host0/scan
>
> -bash: /sys/class/scsi_host/host0/scan: No such file or directory
> # uname -a
> Linux xxx 2.6.26-2-amd64 #1 SMP Wed Sep 21 03:36:44 UTC 2011 x86_64
> GNU/Linux

I cut and pasted it out of a blog after verifying the command looked
reasonable if it were to be run on my own system. I have no need to
rescan my SCSI bus.

cbell@circe:/sys/class/scsi_host$ ls -l *
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host0 ->
../../devices/pci0000:00/0000:00:1f.1/host0/scsi_host/host0
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host1 ->
../../devices/pci0000:00/0000:00:1f.1/host1/scsi_host/host1
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host2 ->
../../devices/pci0000:00/0000:00:1f.2/host2/scsi_host/host2
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host3 ->
../../devices/pci0000:00/0000:00:1f.2/host3/scsi_host/host3
cbell@circe:/sys/class/scsi_host$ ls -l * */scan
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host0 ->
../../devices/pci0000:00/0000:00:1f.1/host0/scsi_host/host0
--w------- 1 root root 4096 Feb 17 10:25 host0/scan
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host1 ->
../../devices/pci0000:00/0000:00:1f.1/host1/scsi_host/host1
--w------- 1 root root 4096 Feb 17 10:51 host1/scan
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host2 ->
../../devices/pci0000:00/0000:00:1f.2/host2/scsi_host/host2
--w------- 1 root root 4096 Feb 17 10:51 host2/scan
lrwxrwxrwx 1 root root 0 Feb 17 10:24 host3 ->
../../devices/pci0000:00/0000:00:1f.2/host3/scsi_host/host3
--w------- 1 root root 4096 Feb 17 10:51 host3/scan
cbell@circe:/sys/class/scsi_host$ uname -a
Linux circe 2.6.32-5-amd64 #1 SMP Mon Jan 16 16:22:28 UTC 2012 x86_64 GNU/Linux
cbell@circe:/sys/class/scsi_host$

Perhaps your kernel is too old? You realize Lenny is unsupported now, yes?

--
Chris


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CAOEVnYsLxNFJaZVsi4peoCtt...@mail.gmail.com

Julien Groselle

unread,
Feb 17, 2012, 12:00:02 PM2/17/12
to
I already have seen this blog, and the folder /sys/class/scsi_host/ is not populated (on my server, ofc) :
# l /sys/class/scsi_host
total 0

JG

Le 17 février 2012 17:34, frank <fr...@dead-link.org> a écrit :
On Fri, 2012-02-17 at 10:26 -0600, Christofer C. Bell wrote:

> echo "- - -" > /sys/class/scsi_host/host0/scan

Have you tried it? Or just copied and pasted?
echo "- - -" > /sys/class/scsi_host/host0/scan

-bash: /sys/class/scsi_host/host0/scan: No such file or directory
# uname -a
Linux xxx 2.6.26-2-amd64 #1 SMP Wed Sep 21 03:36:44 UTC 2011 x86_64
GNU/Linux

Frank



--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1329496499.2...@nero.internal.friendscout24.de


frank

unread,
Feb 17, 2012, 12:00:02 PM2/17/12
to
On Fri, 2012-02-17 at 10:26 -0600, Christofer C. Bell wrote:

> echo "- - -" > /sys/class/scsi_host/host0/scan

Have you tried it? Or just copied and pasted?

echo "- - -" > /sys/class/scsi_host/host0/scan

-bash: /sys/class/scsi_host/host0/scan: No such file or directory
# uname -a
Linux xxx 2.6.26-2-amd64 #1 SMP Wed Sep 21 03:36:44 UTC 2011 x86_64
GNU/Linux

Frank


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1329496499.2...@nero.internal.friendscout24.de

frank

unread,
Feb 17, 2012, 12:00:02 PM2/17/12
to
On Fri, 2012-02-17 at 10:26 -0600, Christofer C. Bell wrote:
> On Fri, Feb 17, 2012 at 8:04 AM, Julien Groselle
> <julien....@gmail.com> wrote:
> >
> > So no, I didn't rescan anything... It's just what I need (a command line to
> > recan all HDD device)
>
> Apparently it's as easy as this:
>
> echo "- - -" > /sys/class/scsi_host/host0/scan

Sorry, wrong host. It's working too :)

Frank


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1329496922.2...@nero.internal.friendscout24.de

Christofer C. Bell

unread,
Feb 17, 2012, 12:10:01 PM2/17/12
to
On Fri, Feb 17, 2012 at 10:59 AM, Julien Groselle
<julien....@gmail.com> wrote:
> I'm sure my kernel is not so old :)
>
> # uname -r ; cat /etc/debian_version
> 2.6.32-5-amd64
> 6.0.4

Oh, not you, Julien. I was addressing Frank who is running Lenny's
kernel. That said, he seems to have sorted things out on his end, as
well, and it's working for him now. :-) I don't know the state of
support for this feature in Lenny, I only have my own system to look
at which is on Squeeze.

--
Chris


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CAOEVnYuujBZGckOis1w5RK0E=dj2-Fea-TnW1...@mail.gmail.com

Julien Groselle

unread,
Feb 17, 2012, 12:10:02 PM2/17/12
to
I'm sure my kernel is not so old :)

# uname -r ; cat /etc/debian_version
2.6.32-5-amd64
6.0.4

Julien Groselle

unread,
Feb 20, 2012, 5:50:02 AM2/20/12
to
Hi,

I'm trying to understand why this directory is empty : /sys/class/scsi_host/
We have two types of servers, one like that

# uname -r ; cat /etc/debian_version
2.6.32-5-amd64
6.0.4

And other one like that

# uname -r ; cat /etc/debian_version
2.6.39-bpo.2-amd64
6.0.4

On the first one :
# l /sys/class/scsi_host/
total 0

On the second :
# l /sys/class/scsi_host/
total 0
lrwxrwxrwx 1 root root 0 20 févr. 11:06 host0 -> ../../devices/pci0000:00/0000:00:1f.1/host0/scsi_host/host0
lrwxrwxrwx 1 root root 0 20 févr. 11:06 host1 -> ../../devices/pci0000:00/0000:00:1f.1/host1/scsi_host/host1

It drive me crazy !
Could someone explain me this difference ?

Second point, why i have this dependency for the package scsitools ?
# aptitude install scsitools
Les NOUVEAUX paquets suivants vont être installés :
  libdrm-intel1{a} libdrm-radeon1{a} libdrm2{a} libgl1-mesa-dri{a} libgl1-mesa-glx{a} libsgutils2-2{a} libutempter0{a} libxaw7{a} libxmu6{a} libxv1{a} libxxf86dga1{a} libxxf86vm1{a} scsitools sg3-utils{a} tcl8.4{a} tk8.4{a}
  x11-utils{a} xbitmaps{a} xterm{a}

drm... mesa... i don't want this on my server, i just want rescan scsi...
I have installed the package on a test server to read the script /sbin/rescan-scsi-bus, and of course it stop at this line :
for hostdir in /sys/class/scsi_host/host*; do (empty in my case)

This line was my first hope : echo "No SCSI host adapters found in sysfs" ; Oh ! sysfs, nice way to search and it was uninstalled on my production server.
SO i have installed it...
But /sys/class/scsi_host/ is always empty...

Any help ? :)

JG

Stan Hoeppner

unread,
Feb 20, 2012, 9:00:02 AM2/20/12
to
On 2/20/2012 4:47 AM, Julien Groselle wrote:
> Hi,
>
> I'm trying to understand why this directory is empty : /sys/class/scsi_host/
> We have two types of servers, one like that
> # uname -r ; cat /etc/debian_version
> 2.6.32-5-amd64
> 6.0.4
>
> And other one like that
> # uname -r ; cat /etc/debian_version
> 2.6.39-bpo.2-amd64
> 6.0.4
>
> On the first one :
> # l /sys/class/scsi_host/
> total 0
>
> On the second :
> # l /sys/class/scsi_host/
> total 0
> lrwxrwxrwx 1 root root 0 20 févr. 11:06 host0 ->
> ../../devices/pci0000:00/0000:00:1f.1/host0/scsi_host/host0
> lrwxrwxrwx 1 root root 0 20 févr. 11:06 host1 ->
> ../../devices/pci0000:00/0000:00:1f.1/host1/scsi_host/host1
>
> It drive me crazy !
> Could someone explain me this difference ?

The difference is obvious: 2+ years of kernel development between
2.6.32 and 2.6.39--new features added. These are kernel data
structures, not files, after all.

It's interesting that you know where these kernel data structures are in
the filesystem, yet you apparently lack understanding of what they are,
and how they get created in the first place.

> Second point, why i have this dependency for the package scsitools ?
> # aptitude install scsitools
> Les NOUVEAUX paquets suivants vont être installés :
> libdrm-intel1{a} libdrm-radeon1{a} libdrm2{a} libgl1-mesa-dri{a}
> libgl1-mesa-glx{a} libsgutils2-2{a} libutempter0{a} libxaw7{a} libxmu6{a}
> libxv1{a} libxxf86dga1{a} libxxf86vm1{a} scsitools sg3-utils{a} tcl8.4{a}
> tk8.4{a}
> x11-utils{a} xbitmaps{a} xterm{a}

Apparently because this system had at one time a GUI environment, or
part of one, installed. And I would guess based on this that scsitools
has a GUI component available on GUI systems. One of my headless
servers with 2.6.38.6 and Debian 6.0.4 shows only 2 dependencies:

The following NEW packages will be installed:
libsgutils2-2{a} scsitools sg3-utils{a}

> drm... mesa... i don't want this on my server, i just want rescan scsi...
> I have installed the package on a test server to read the script
> /sbin/rescan-scsi-bus, and of course it stop at this line :
> for hostdir in /sys/class/scsi_host/host*; do (empty in my case)
>
> This line was my first hope : echo "No SCSI host adapters found in sysfs" ;
> Oh ! sysfs, nice way to search and it was uninstalled on my production
> server.
> SO i have installed it...
> But /sys/class/scsi_host/ is always empty...
>
> Any help ? :)

cciss is a *block* device driver, not a *SCSI* device driver. Thus
disks attached to a SmartArray controller cannot be directed accessed
via SCSI commands and SCSI tools. Or, at least that's how it used to
be. Apparently this distinction has been blurred between 2.6.32 and
2.6.39. It would seem the 2.6.39 cciss driver allows limited direct
access/manipulation of devices connected to the SmartArray controller
for things such as S.M.A.R.T.

--
Stan



--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F424F8B...@hardwarefreak.com

Julien Groselle

unread,
Feb 20, 2012, 9:50:03 AM2/20/12
to
Thank you for your answer Stan,

Yes, I'm not a specialist about this kernel data structure, but i'm on the way ;)
It's not so easy to find clear information about that.

So for you, if i upgrade my kernel it will be possible to use this script... But i can't reboot production servers, and change kernel version like that.
I prefer ask you again the origin of my need :

I have Soft RAID using mdadm, LVM and Ext4. One of my HDD was in error, so i set it faulty and i remove it form my RAID array.
After i want to replace physically the HDD, so i do it.
But my Debian don't see the new HDD... I don't want to reboot... 
So I started my search to how to implement HotPlug with debian Squeeze and HP servers.

Do you know how I can do this ?

Debian-users told me to rescan scsi, so i have tried to do that. But maybe you have another way to go ?

Thank you in advance.

--
JG.

Stan Hoeppner

unread,
Feb 20, 2012, 10:10:02 AM2/20/12
to
On 2/20/2012 8:43 AM, Julien Groselle wrote:
> Thank you for your answer Stan,
>
> Yes, I'm not a specialist about this kernel data structure, but i'm on the
> way ;)
> It's not so easy to find clear information about that.
>
> So for you, if i upgrade my kernel it will be possible to use this
> script... But i can't reboot production servers, and change kernel version
> like that.
> I prefer ask you again the origin of my need :
>
> I have Soft RAID using mdadm, LVM and Ext4. One of my HDD was in error, so
> i set it faulty and i remove it form my RAID array.
> After i want to replace physically the HDD, so i do it.
> But my Debian don't see the new HDD... I don't want to reboot...
> So I started my search to how to implement HotPlug with debian Squeeze and
> HP servers.
>
> Do you know how I can do this ?
>
> Debian-users told me to rescan scsi, so i have tried to do that. But maybe
> you have another way to go ?

At this point you need to give us all the hardware details of the
machine with the problem. Previously you implied both machines are
HP/Compaq servers both with SmartAray controllers. Now you seem to be
saying the 2.6.32 machine does not have a SmartArray controller.
Knowing exactly how the drives are connected to the system is important
here. You now say you're using mdraid, so this would imply a mobo down
SATA chip or a non-RAID SAS/SATA HBA.

Knowing exactly how the drives are connected (preferably to what chip),
should help us tell you at minimum if hot swapping is even possible with
that hardware. With the SmartArray cards and an HP backplane it
obviously is. With a whitebox server and mobo down SATA ports, hot swap
isn't possible, except with a handful of server boards with real
hardware RAID on the mobo.

--
Stan
--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F426210...@hardwarefreak.com

Julien Groselle

unread,
Feb 20, 2012, 10:30:02 AM2/20/12
to
Like I Said in the first thread :
My Hardware is HP  ProLiant DL380 G7, with HP Smart Array P410i Controller version 3.50.

This is a Smart Array, and it present to the server the disk (RAID0) directly.
So with 8 SAS HDD (8 RAID0) I have made a RAID6. Performances are amazing comparate to hard RAID, but it is not the subject here.

Do you need something else ?

Stan Hoeppner

unread,
Feb 20, 2012, 12:10:02 PM2/20/12
to
On 2/20/2012 9:19 AM, Julien Groselle wrote:
> Like I Said in the first thread :

Few people read every post in every thread. At the point I jumped in to
help, your hardware info was missing. My apologies for not back digging
the thread.

> My Hardware is HP ProLiant DL380 G7, with HP Smart Array P410i Controller
> version 3.50.
>
> This is a Smart Array, and it present to the server the disk (RAID0)
> directly.
> So with 8 SAS HDD (8 RAID0) I have made a RAID6. Performances are amazing
> comparate to hard RAID, but it is not the subject here.
>
> Do you need something else ?

It may not matter at this point, but I'm wondering if you are using
BLK_CPQ_CISS_DA or SCSI_HPSA? Look in your loaded modules list for
something similar to these two kernel drivers. If the former, using
SCSI commands against the 8 block devices as seen by the kernel is not
going to work. If the latter, some of the SCSI tools may work.
Regardless, troubleshooting this path is a dead end. The following is
the path you need to take:

When you --fail[ed] the drive with mdadm, pulled it, and hot plugged a
replacement into the cage, the 410i firmware very likely did not
automatically create a new RAID0 array of this new disk, nor export it
as a usable device to the driver. This is very likely why you are not
finding the new disk anywhere in the sysfs.

So the solution seems rather simple. Run the HP Array Configuration
Utility (ACU). Create a RAID0 array of the new disk and export it, just
as you originally did via the BIOS ACU when you originally configured
the 8 disks. The ACU software is available here:

http://h18000.www1.hp.com/products/servers/proliantstorage/software-management/acumatrix/

Afterwards, run the normal mdadm commands to add the new disk device to
your RAID6 array.

--
Stan
Archive: http://lists.debian.org/4F427C1...@hardwarefreak.com

Stan Hoeppner

unread,
Feb 20, 2012, 2:20:02 PM2/20/12
to
On 2/20/2012 11:00 AM, Stan Hoeppner wrote:

> So the solution seems rather simple. Run the HP Array Configuration
> Utility (ACU). Create a RAID0 array of the new disk and export it, just
> as you originally did via the BIOS ACU when you originally configured
> the 8 disks. The ACU software is available here:
>
> http://h18000.www1.hp.com/products/servers/proliantstorage/software-management/acumatrix/

This will get you there faster.

AMD64:
http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/Debian/pool/non-free/hpacucli_8.70-8.0.2-2_amd64.deb

dpkg -i hpacucli_8.70-8.0.2-2_amd64.deb

i386:
http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/Debian/pool/non-free/hpacucli_8.70-8.0.2-2_i386.deb

dpkg -i hpacucli_8.70-8.0.2-2_i386.deb

--
Stan


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F429AD6...@hardwarefreak.com

Julien Groselle

unread,
Feb 21, 2012, 4:50:02 AM2/21/12
to
Hi Stan,

First, i tell you a huge thanks !
The ACU software is just what i need to better administrate hardware.

Just one problem, the dependecies will "dirty" my server :
# aptitude install lib32gcc1
Les NOUVEAUX paquets suivants vont être installés : 
  lib32gcc1 libc6-i386{a} 

# dpkg -i lib32stdc++6_4.4.5-8_amd64.deb
# dpkg -i hpacucli_8.70-8.0.2-2_amd64.deb

Any solution to avoid install 32bits librairies to my x86_64 server ?
By the way, i have installed all this packqge on a test server, and it seems work fine.
I'm reading documentation about hpacucli to learn how it works. It seems to be clear and simple.

Kind regards.

JG.

Stan Hoeppner

unread,
Feb 21, 2012, 9:10:01 AM2/21/12
to
On 2/21/2012 3:45 AM, Julien Groselle wrote:
> Hi Stan,
>
> First, i tell you a huge thanks !

You're welcome Julien. Sorry I didn't mention it sooner. It took me a
little while to figure out exactly what your problem was, due to
entering the thread so late.

> The ACU software is just what i need to better administrate hardware.

I'm kinda surprised you didn't already know about this tool. It's a
must have. Without it, how can you manage the RAID controller and disks
once the system is in production?

> Just one problem, the dependecies will "dirty" my server :
> # aptitude install lib32gcc1
> Les NOUVEAUX paquets suivants vont être installés :
> lib32gcc1 libc6-i386{a}
>
> # dpkg -i lib32stdc++6_4.4.5-8_amd64.deb
> # dpkg -i hpacucli_8.70-8.0.2-2_amd64.deb
>
> Any solution to avoid install 32bits librairies to my x86_64 server ?

There was a very long discussion about this some time ago. The
consensus seems to be that if you must install a 32bit library, there is
no downside, unless it replaces an existing library that other programs
use, and breaks those programs in the process. I do find it odd that an
amd64 package has a 32bit library dependency. Then again, stuff like
this isn't all that rare when installing commercial software.

> By the way, i have installed all this packqge on a test server, and it
> seems work fine.
> I'm reading documentation about hpacucli to learn how it works. It seems to
> be clear and simple.

Clear and simple is exactly what you want/need in a cli administration
utility.

FYI, the Areca PCIe RAID cards have a 10/100 ethernet port and you
manage them via a built in web interface, out of band. No software to
install, as the kernel driver is in mainline (in recent kernels anyway).
I'm not plugging Areca here, as the cards are apparently lacking
somewhat in the performance department WRT competing cards in the same
price range, and customer support is seriously lacking. Just pointing
out how handy an OOB web management interface can be.

--
Stan


> Le 20 février 2012 20:11, Stan Hoeppner <st...@hardwarefreak.com> a écrit :
>
>> On 2/20/2012 11:00 AM, Stan Hoeppner wrote:
>>
>>> So the solution seems rather simple. Run the HP Array Configuration
>>> Utility (ACU). Create a RAID0 array of the new disk and export it, just
>>> as you originally did via the BIOS ACU when you originally configured
>>> the 8 disks. The ACU software is available here:
>>>
>>>
>> http://h18000.www1.hp.com/products/servers/proliantstorage/software-management/acumatrix/
>>
>> This will get you there faster.
>>
>> AMD64:
>>
>> http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/Debian/pool/non-free/hpacucli_8.70-8.0.2-2_amd64.deb
>>
>> dpkg -i hpacucli_8.70-8.0.2-2_amd64.deb
>>
>> i386:
>>
>> http://downloads.linux.hp.com/SDR/downloads/ProLiantSupportPack/Debian/pool/non-free/hpacucli_8.70-8.0.2-2_i386.deb
>>
>> dpkg -i hpacucli_8.70-8.0.2-2_i386.deb


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F43A4CB...@hardwarefreak.com

Julien Groselle

unread,
Feb 23, 2012, 11:20:02 AM2/23/12
to
Hi everybody,

> You're welcome Julien.  Sorry I didn't mention it sooner.  It took me a
> little while to figure out exactly what your problem was, due to
> entering the thread so late.

No problem, and thank you again.

> I'm kinda surprised you didn't already know about this tool.  It's a
> must have.  Without it, how can you manage the RAID controller and disks
> once the system is in production?

Now i'm sure that is a must have.
Since 4 years to last year, we just had Hardware RAID, so we didn't need to do any actions on HDD...
Now with md RAID we need ! :)

> There was a very long discussion about this some time ago.  The
> consensus seems to be that if you must install a 32bit library, there is
> no downside, unless it replaces an existing library that other programs
> use, and breaks those programs in the process.  I do find it odd that an
> amd64 package has a 32bit library dependency.  Then again, stuff like
> this isn't all that rare when installing commercial software.

Ok, really bad...
But i have another problem, hpacucli don't work with all kernel version !
To avoid details, i show you my results :

2.6.32.5-amd64 : OK
2.6.38-bpo.2-amd64 : NOK
2.6.39_bpo.2-amd64 : NOK
3.2.0.0.bpo.1-amd64 : NOK

With the two kernels 2.6.*bpo* it work well to show config, show everything you want.
To delete, it's Ok too.
But to Create it never give me my shell back ! Better, after the create try, it freeze all disk command like pvdisplay, or mdadm --detail /dev/md0
All this command was just blocked ! I show you what i read in kern.log :

[  960.508752] INFO: task arrayprobe:9011 blocked for more than 120 seconds.
[  960.508811] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  960.508886] arrayprobe      D ffff8801277e17a0     0  9011   9010 0x00000000
[  960.508890]  ffff8801277e17a0 0000000000000086 0000000000000000 ffffffff8160b020
[  960.508895]  0000000000013b40 ffff880129f4dfd8 ffff880129f4dfd8 0000000000013b40
[  960.508899]  ffff8801277e17a0 ffff880129f4c010 ffff880127eccc78 00000001811a3672
[  960.508904] Call Trace:
[  960.508908]  [<ffffffff81332415>] ? __mutex_lock_common+0x127/0x193
[  960.508911]  [<ffffffff8133254d>] ? mutex_lock+0x1a/0x33
[  960.508914]  [<ffffffff811224fc>] ? __blkdev_get+0x7b/0x372
[  960.508918]  [<ffffffff81122aba>] ? blkdev_get+0x2c7/0x2c7
[  960.508921]  [<ffffffff811229b8>] ? blkdev_get+0x1c5/0x2c7
[  960.508924]  [<ffffffff81122aba>] ? blkdev_get+0x2c7/0x2c7
[  960.508927]  [<ffffffff810fa46d>] ? __dentry_open+0x180/0x297
[  960.508930]  [<ffffffff81103537>] ? dget+0x12/0x1e
[  960.508933]  [<ffffffff81105bd1>] ? do_last+0x449/0x543
[  960.508937]  [<ffffffff811071c1>] ? path_openat+0xc6/0x317
[  960.508940]  [<ffffffff811074df>] ? do_filp_open+0x2c/0x75
[  960.508944]  [<ffffffff8110fd94>] ? alloc_fd+0x69/0x10b
[  960.508947]  [<ffffffff810fa1b8>] ? do_sys_open+0x61/0xe8
[  960.508950]  [<ffffffff81339392>] ? system_call_fastpath+0x16/0x1b

With kernel 3.2* it don't find any controlers :
# hpacucli ctrl all show config

Error: No controllers detected.

I'm looking for a solution and i have seen that the version 8.70-8.0 is not the last one.
But i don't find any 64 bit deb file to use... rpm, yes ; i386, yes... 
Of course nothing in your link :(

Still searching...

--
JG

Stan Hoeppner

unread,
Feb 23, 2012, 3:40:02 PM2/23/12
to
On 2/23/2012 10:16 AM, Julien Groselle wrote:

> Now i'm sure that is a must have.
> Since 4 years to last year, we just had Hardware RAID, so we didn't need to
> do any actions on HDD...
> Now with md RAID we need ! :)

RAID 0 arrays are not fault tolerant, so there is nothing the controller
can do when a single drive configured as such fails. RAID 1 mirrors,
however, are fault tolerant.

Thus, the proper way to do what you are attempting to do, with
proprietary RAID cards, is to use hybrid nested hardware/mdraid arrays.
For example, if you want a straight mdraid 10 array but you still want
the RAID card to handle drive fail/swap/rebuild automatically as it did
in the past, you would create multiple RAID 1 mirrors in the controller
and set the rebuild policies as you normally would. Then you create an
mdraid 0 stripe over the virtual drives exported by the controller,
giving you a hybrid soft/hardware RAID 10.

You likely won't see much performance gain with this setup vs. using a
single RAID card with hardware RAID 10. The advantage of this setup
really kicks in when you create the mdraid 0 stripe across many RAID 1
mirrors residing on 2 or more hardware RAID controllers. The 3 main
benefits of this are:

1. Striping can occur across many more spindles than can be achieved
with a single RAID card
2. You keep the hardware write cache benefit
3. Drive failure/replace/rebuild is handled transparently

Obviously it's not feasible to do parity RAID schemes in such a hybrid
setup. If your primary goal of switching to mdraid was to increase the
performance of RAID6, then you simply can't do it with a single RAID
card *and* still have automatic drive failure management. As they say,
there's no such thing as a free lunch.

If RAID6 performance is what you're after, and you want mdraid to be
able to handle the drive failure/replacement automatically without the
HBA getting in the way, then you will need to switch to non-RAID HBAs
that present drives in JBOD/standalone fashion to Linux. LSI makes many
cards suitable for this task. Adaptec has a few as well. They are
relatively inexpensive, $200-300 USD, models with both internal SFF8087
and external SFF8088 ports are available. Give me the specs on your
Proliant, how many drives you're connecting, internal/external, and I'll
shoot you a list of SAS/SATA HBAs that will work the way you want.

> But i have another problem, hpacucli don't work with all kernel version !
> To avoid details, i show you my results :
>
> 2.6.32.5-amd64 : OK
> 2.6.38-bpo.2-amd64 : NOK
> 2.6.39_bpo.2-amd64 : NOK
> 3.2.0.0.bpo.1-amd64 : NOK

This is very common with proprietary vendor software. They have so many
distros to support that they must limit their development and
maintenance efforts to a very limited number of configurations, and
kernel versions. When you look at RHEL kernels for instance, they never
change major numbers during a release lifecycle. So you end up with
things like 2.6.18-274.18.1.el5. This is what is called a "long term
stable kernel". Thus, when a vendor qualifies something like a RAID
card driver or management tools, for RHEL 5, they don't have to worry
about their software breaking as Red Hat updates this kernel over the
life of the release, with things like security patches etc. This is the
main reason why RHEL and SLES are so popular in the enterprise
space--everything 'just works' when vendor BCPs are followed.

To achieve the same level of functionality with Debian, you must stick
with the baseline kernel, 2.6.32-5 and security updates only.

Welcome to the world of "enterprise" hardware.

--
Stan


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F46A3D4...@hardwarefreak.com

Julien Groselle

unread,
Feb 28, 2012, 5:30:03 AM2/28/12
to
Hello everybody,

Frank, we have decided to use soft RAID to optimise our supervision and to optimise our RAID with all the tool that linux provide !
And for me it's better to optimise a RAID5 (RAID6) than use a non optimised RAID10 :
- no more HDD "lost", we have 8 disks, so 4 "real" in RAID10 and 6 in RAID6
- Linux kernel optimisation give us amazing transfert rate, and I/O
- with MD and LVM we can align I/O strictly 

But here is not the discussion.

Return to hpacucli...
I have used this tools to remove 2 HDD "unassigned", every thing was OK until the arrayprobe script give me a warning !
Am I unlucky ??

# arrayprobe 
WARNING Arrayprobe Logical drive 1 on /dev/cciss/c0d0: Logical drive is not configured

# arrayprobe -r 
[...]
Event code 1/0/0 with tag 15
at 2-28-2012 08:26:30
with message: Hot-plug drive removed, Port=1I Box=1 Bay=3 SN=                    

Event code 4/0/0 with tag 16
at 2-28-2012 08:26:30
with message: Physical drive failure, Port=1I Box=1 Bay=3
physical drive 2 has failed with failurecode 20.

Event code 0/0/0
with message: No events to report.

failed to open device /dev/ida/c0d0: No such file or directory
Logical drive 0 on controller /dev/cciss/c0d0 has state 0
Logical drive 1 on controller /dev/cciss/c0d0 has state 2
Logical drive 2 on controller /dev/cciss/c0d0 has state 2
Logical drive 3 on controller /dev/cciss/c0d0 has state 0
WARNING Arrayprobe Logical drive 1 on /dev/cciss/c0d0: Logical drive is not configured

Does my Logical dive 1 mapped on /dev/cciss/c0d0 is not configured... impossible this is my "system RAID"
# hpacucli ctrl slot=0 ld 1 show

Smart Array P410i in Slot 0 (Embedded)

   array A

      Logical Drive: 1
         Size: 136.7 GB
         Fault Tolerance: RAID 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 35132
         Strip Size: 256 KB
         Status: OK
         Array Accelerator: Enabled
         Unique Identifier: 600508B1001C02B880CF0DDEE7FD0FEC
         Disk Name: /dev/cciss/c0d0
         Mount Points: /boot 190 MB
         OS Status: LOCKED
         Logical Drive Label: A00FD4B45001438009E6ADD046E6
         Mirror Group 0:
            physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 146 GB, OK)
         Mirror Group 1:
            physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 146 GB, OK)

So, what's wrong now ?? 
I request your help again on this point, and i hope it will be the last one...

-- 
JG

Stan Hoeppner

unread,
Feb 28, 2012, 12:40:01 PM2/28/12
to
On 2/28/2012 4:20 AM, Julien Groselle wrote:
> Hello everybody,
>
> Frank,

Who is Frank?

> we have decided to use soft RAID to optimise our supervision and to
> optimise our RAID with all the tool that linux provide !
> And for me it's better to optimise a RAID5 (RAID6) than use a non optimised
> RAID10 :
> - no more HDD "lost", we have 8 disks, so 4 "real" in RAID10 and 6 in RAID6
> - Linux kernel optimisation give us amazing transfert rate, and I/O
> - with MD and LVM we can align I/O strictly

So you've made the choice than performance and usable space, above, are
more critical than manageability, below. BTW, if you are using all 8
drives in an mdraid array, what drive(s) are you booting the system
from? The DL380 G7 only has 8 drive slots. Are you booting from the md
array?
Unfortunately this is beyond my experience/expertise at this point.
Unless someone else here can help you, I suggest you contact HP support,
as the problem you're having is with their closed source commercial
SmartArray support tools. You may also want to inquire about this on
one of the mailing lists dealing with storage, such as linux-raid,
linux-scsi, etc, as someone there may have experience with such an HP
hardware setup.

Or you can swap the p410i for an LSI 9211-8i in HBA mode and eliminate
all of your current problems instantly. The only downside is you'll
lose a little write performance due to the lack of write cache.
Everything else is upside. I think I mentioned this option before.
It's ~$250 USD. What's your time worth Julien? How much time have you
spent on this already?

--
Stan


--
To UNSUBSCRIBE, email to debian-us...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/4F4D0F7C...@hardwarefreak.com
0 new messages