After restarting iscsi remote drive are not accesible on IP6 disabled centos 5.4 64bit linux server

[Bangalore]

Hi

i have installed one HP blade server with "Linux version
2.6.18-164.el5 (mock...@builder10.centos.org) (gcc version 4.1.2
20080704 (Red Hat 4.1.2-46)) 0000001 SMP Thu Sep 3 03:28:30 EDT 2009 "
CentOS release 5.4 (Final) .I have disabled the IP6 with these
commands by adding these in /etc/modeprob.conf

alias net-pf-10 off
alias ipv6 off
options ipv6 disable=1

after that i installed iscsi and pointed towards one netapp .then i
created lvm in the system with netapp drive .

when ever i restart a server or restart ISCSCI after that i am not not
able to access the LVM what i created . Please help me

Additional Information dsmg log

EXT3 FS on dm-2, internal journal
EXT3-fs: recovery complete.
EXT3-fs: mounted filesystem with ordered data mode.
cxgb3i: tag itt 0x1fff, 13 bits, age 0xf, 4 bits.
iscsi: registered transport (cxgb3i)
cnic: Unknown symbol __ipv6_addr_type
cnic: Unknown symbol ip6_route_output
iscsi: registered transport (iser)
scsi1 : iSCSI Initiator over TCP/IP
Vendor: NETAPP Model: LUN Rev: 7310
Type: Direct-Access ANSI SCSI revision: 04
SCSI device sdb: 629225472 512-byte hdwr sectors (322163 MB)
sdb: Write Protect is off
sdb: Mode Sense: bd 00 00 08
SCSI device sdb: drive cache: write through
SCSI device sdb: 629225472 512-byte hdwr sectors (322163 MB)
sdb: Write Protect is off
sdb: Mode Sense: bd 00 00 08
SCSI device sdb: drive cache: write through
sdb: sdb1
sd 1:0:0:0: Attached scsi disk sdb
sd 1:0:0:0: Attached scsi generic sg0 type 0



[root@wapdb-node2 aneesh]# service iscsi restart
Logging out of session [sid: 2, target: iqn.1986-03.com.ibm:sn.
135057127, portal: 10.0.104.202,3260]
Logout of [sid: 2, target: iqn.1986-03.com.ibm:sn.135057127, portal:
10.0.104.202,3260]: successful
Stopping iSCSI daemon:
iscsid dead but pid file exists [ OK ]
Starting iSCSI daemon: WARNING: Error inserting libiscsi2 (/lib/
modules/2.6.18-164.el5/kernel/drivers/scsi/libiscsi2.ko): Unknown
symbol in module, or unknown parameter (see dmesg)
FATAL: Error inserting bnx2i (/lib/modules/2.6.18-164.el5/kernel/
drivers/scsi/bnx2i/bnx2i.ko): Unknown symbol in module, or unknown
parameter (see dmesg)
[ OK ]
[ OK ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.
1986-03.com.ibm:sn.135057127, portal: 10.0.104.202,3260]
Login to [iface: default, target: iqn.1986-03.com.ibm:sn.135057127,
portal: 10.0.104.202,3260]: successful
[ OK ]
[root@wapdb-node2 aneesh]#



[root@wapdb-node2 /]# lvdisplay
/dev/MySql/MySql: read failed after 0 of 4096 at 0: Input/output
error
--- Logical volume ---
LV Name /dev/MySql/MySql
VG Name MySql
LV UUID dZ4DCD-qcxM-KnCt-xsrK-KyeM-pndo-0a5T9y
LV Write Access read/write
LV Status available
# open 1
LV Size 300.00 GB
Current LE 76800
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

[Mike Christie]

On 11/19/2010 02:03 AM, Bangalore wrote:
> [root@wapdb-node2 aneesh]# service iscsi restart

Before you run restart, did you unmount filesystems on the lvm device
and did you have lvm release the iscsi/scsi devices? You have to do this
because the logout causes the existing /dev/sdX devices to get deleted.
If you do not have lvm release the scsi devices when you logout then
that lvm/dm device will be referencing a partially deleted device (it is
not totally destroyed because lvm is still referencing it, but the old
device is not going to do IO).

When you login again below, new devices are created. So you have to run
the lvm commands again.

> Logging out of session [sid: 2, target: iqn.1986-03.com.ibm:sn.
> 135057127, portal: 10.0.104.202,3260]
> Logout of [sid: 2, target: iqn.1986-03.com.ibm:sn.135057127, portal:
> 10.0.104.202,3260]: successful
> Stopping iSCSI daemon:
> iscsid dead but pid file exists [ OK ]
> Starting iSCSI daemon: WARNING: Error inserting libiscsi2 (/lib/
> modules/2.6.18-164.el5/kernel/drivers/scsi/libiscsi2.ko): Unknown
> symbol in module, or unknown parameter (see dmesg)

That should not happen. Are you using the iscsi modules that come with
the distro? Send the /var/log/messages when this happens.

[Bangalore]

Dear Mike


These all are LVM package what i have installed on system

[root@wapdb-node1 /]# rpm -qa|grep lvm
lvm2-cluster-2.02.56-7.el5_5.4
system-config-lvm-1.1.5-4.el5
lvm2-2.02.56-8.el5_5.6
[root@wapdb-node1 /]#

i tried by unmounting the device before restarting Iscsi service , But
after that also the result is same as old. After restating of iscsi
system is not able to read the mapped netapp drives . Please see the
entire process which i did


[root@wapdb-node1 ~]# fdisk -l

Disk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes
255 heads, 63 sectors/track, 36468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 36468 292824787+ 8e Linux
LVM

Disk /dev/sda: 322.1 GB, 322163441664 bytes
255 heads, 63 sectors/track, 39167 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 39167 314608896 83 Linux
[root@wapdb-node1 ~]#


[root@wapdb-node1 ~]# fdisk /dev/sda1
Device contains neither a valid DOS partition table, nor Sun, SGI or
OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 39166.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by
w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-39166, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-39166, default 39166):
Using default value 39166

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid
argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

[root@wapdb-node1 ~]# partprobe
[root@wapdb-node1 ~]#



Disk /dev/cciss/c0d0: 299.9 GB, 299966445568 bytes
255 heads, 63 sectors/track, 36468 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/cciss/c0d0p1 * 1 13 104391 83 Linux
/dev/cciss/c0d0p2 14 36468 292824787+ 8e Linux
LVM

Disk /dev/sda: 322.1 GB, 322163441664 bytes
255 heads, 63 sectors/track, 39167 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 1 39167 314608896 83 Linux
[root@wapdb-node1 ~]# pvcreate /dev/sda1
Physical volume "/dev/sda1" successfully created
[root@wapdb-node1 ~]# vgcreate mysql /dev/sda1
Volume group "mysql" successfully created
[root@wapdb-node1 ~]# vgdisplay
--- Volume group ---
VG Name VolGroup01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 279.25 GB
PE Size 32.00 MB
Total PE 8936
Alloc PE / Size 8936 / 279.25 GB
Free PE / Size 0 / 0
VG UUID 9cMTqi-3Brv-0QO2-zCTW-n3MQ-6vhM-pDvCMQ

--- Volume group ---
VG Name mysql
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 1
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 0
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 300.03 GB
PE Size 4.00 MB
Total PE 76808
Alloc PE / Size 0 / 0
Free PE / Size 76808 / 300.03 GB
VG UUID BE4qpy-u1to-yKKj-ECR2-93i0-Hn2r-londS8

[root@wapdb-node1 ~]#



[root@wapdb-node1 ~]# lvcreate -n MySqlDB -L 250GB mysql
Logical volume "MySqlDB" created
[root@wapdb-node1 ~]#

[root@wapdb-node1 ~]# lvdisplay
--- Logical volume ---
LV Name /dev/VolGroup01/LogVol00
VG Name VolGroup01
LV UUID 7s6lRU-5lNO-GwmH-UFc8-5kQk-ZIM3-NCtTa1

LV Write Access read/write
LV Status available
# open 1

LV Size 273.44 GB
Current LE 8750

Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256

Block device 253:0

--- Logical volume ---
LV Name /dev/VolGroup01/LogVol01
VG Name VolGroup01
LV UUID dw9a8f-f65b-4Ga6-mPhn-3tTA-BR8e-H3nEMN

LV Write Access read/write
LV Status available
# open 1

LV Size 5.81 GB
Current LE 186

Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256

Block device 253:1

--- Logical volume ---
LV Name /dev/mysql/MySqlDB
VG Name mysql
LV UUID DwfZUj-3rvQ-66ff-WFvG-XcO0-N34D-uDPQAg

LV Write Access read/write
LV Status available

# open 0
LV Size 250.00 GB
Current LE 64000

Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

[root@wapdb-node1 ~]#
[root@wapdb-node1 ~]# mkfs.ext3 /dev/mysql/MySqlDB
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
32768000 inodes, 65536000 blocks
3276800 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
2000 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208,
4096000, 7962624, 11239424, 20480000, 23887872

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
[root@wapdb-node1 ~]#
/dev/mapper/VolGroup01-LogVol00
265G 3.0G 249G 2% /
/dev/cciss/c0d0p1 99M 13M 82M 14% /boot
tmpfs 2.0G 0 2.0G 0% /dev/shm
[root@wapdb-node1 ~]# mount /dev/mysql/MySqlDB /MySqlDB/
[root@wapdb-node1 ~]# cd /MySqlDB/
[root@wapdb-node1 MySqlDB]# mkdir aneesh
[root@wapdb-node1 MySqlDB]# touch aneesh
[root@wapdb-node1 MySqlDB]# ll
total 20
drwxr-xr-x 2 root root 4096 Nov 29 00:13 aneesh
drwx------ 2 root root 16384 Nov 29 00:11 lost+found
[root@wapdb-node1 MySqlDB]# df -kh
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup01-LogVol00
265G 3.0G 249G 2% /
/dev/cciss/c0d0p1 99M 13M 82M 14% /boot
tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/mapper/mysql-MySqlDB
247G 188M 234G 1% /MySqlDB
[root@wapdb-node1 MySqlDB]#
[root@wapdb-node1 MySqlDB]# cd ..
[root@wapdb-node1 /]# umount /MySqlDB/
[root@wapdb-node1 /]#
[root@wapdb-node1 /]# service iscsi stop
Logging out of session [sid: 1, target: iqn.1986-03.com.ibm:sn.
135057127, portal: 10.0.104.202,3260]
Logout of [sid: 1, target: iqn.1986-03.com.ibm:sn.135057127, portal:
10.0.104.202,3260] successful.
Stopping iSCSI daemon:
[root@wapdb-node1 /]#


[root@wapdb-node1 /]# service iscsi start
iscsid is stopped

Starting iSCSI daemon: WARNING: Error inserting libiscsi2 (/lib/
modules/2.6.18-164.el5/kernel/drivers/scsi/libiscsi2.ko): Unknown
symbol in module, or unknown parameter (see dmesg)

FATAL: Error inserting bnx2i (/lib/modules/2.6.18-164.el5/kernel/
drivers/scsi/bnx2i/bnx2i.ko): Unknown symbol in module, or unknown
parameter (see dmesg)
[ OK ]
[ OK ]
Setting up iSCSI targets: Logging in to [iface: default, target: iqn.
1986-03.com.ibm:sn.135057127, portal: 10.0.104.202,3260]
Login to [iface: default, target: iqn.1986-03.com.ibm:sn.135057127,

portal: 10.0.104.202,3260] successful.
[ OK ]
[root@wapdb-node1 /]#

[root@wapdb-node1 /]# vgdisplay
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 268435390464:
Input/output error
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 268435447808:
Input/output error
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 0: Input/output
error
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 4096: Input/
output error
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 0: Input/output
error
--- Volume group ---
VG Name mysql
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 300.03 GB
PE Size 4.00 MB
Total PE 76808
Alloc PE / Size 64000 / 250.00 GB
Free PE / Size 12808 / 50.03 GB
VG UUID BE4qpy-u1to-yKKj-ECR2-93i0-Hn2r-londS8

--- Volume group ---
VG Name VolGroup01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 3
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 2
Open LV 2
Max PV 0
Cur PV 1
Act PV 1
VG Size 279.25 GB
PE Size 32.00 MB
Total PE 8936
Alloc PE / Size 8936 / 279.25 GB
Free PE / Size 0 / 0
VG UUID 9cMTqi-3Brv-0QO2-zCTW-n3MQ-6vhM-pDvCMQ

[root@wapdb-node1 /]#
[root@wapdb-node1 /]# pvdisplay
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 0: Input/output
error
--- Physical volume ---
PV Name /dev/sdb1
VG Name mysql
PV Size 300.03 GB / not usable 3.25 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 76808
Free PE 12808
Allocated PE 64000
PV UUID KnmFgm-33js-pstw-rfzL-yV0b-EAqt-E4ZBo8

--- Physical volume ---
PV Name /dev/cciss/c0d0p2
VG Name VolGroup01
PV Size 279.26 GB / not usable 9.71 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 8936
Free PE 0
Allocated PE 8936
PV UUID LeVuAv-q7dZ-b64h-lusl-gD5u-bVwl-wTCL19

[root@wapdb-node1 /]#
root@wapdb-node1 /]# lvdisplay
/dev/mysql/MySqlDB: read failed after 0 of 4096 at 0: Input/output

error
--- Logical volume ---

LV Name /dev/mysql/MySqlDB
VG Name mysql
LV UUID DwfZUj-3rvQ-66ff-WFvG-XcO0-N34D-uDPQAg

LV Write Access read/write
LV Status available

# open 0
LV Size 250.00 GB
Current LE 64000

Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2

[root@wapdb-node1 /]# mount /dev/mysql/MySqlDB /MySqlDB/
mount: you must specify the filesystem type
[root@wapdb-node1 /]#



i have seen one error in dsmg

cnic: Unknown symbol __ipv6_addr_type
cnic: Unknown symbol ip6_route_output
iscsi: registered transport (iser)

scsi5 : iSCSI Initiator over TCP/IP

Vendor: NETAPP Model: LUN Rev: 7310
Type: Direct-Access ANSI SCSI revision: 04

SCSI device sda: 629225472 512-byte hdwr sectors (322163 MB)
sda: Write Protect is off
sda: Mode Sense: bd 00 00 08
SCSI device sda: drive cache: write through
SCSI device sda: 629225472 512-byte hdwr sectors (322163 MB)
sda: Write Protect is off
sda: Mode Sense: bd 00 00 08
SCSI device sda: drive cache: write through
sda: sda1
sd 5:0:0:0: Attached scsi disk sda
sd 5:0:0:0: Attached scsi generic sg0 type 0
scsi 4:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
scsi 4:0:0:0: rejecting I/O to dead device
mtrr: type mismatch for e8000000,4000000 old: uncachable new: write-
combining
[root@wapdb-node1 ~]#



i noticed this error dmesg, i think this is becuase of that i have
disabled IP6 by adding
options ipv6 disable=1 in modeprob.conf

cnic: Unknown symbol __ipv6_addr_type
cnic: Unknown symbol ip6_route_output

IP6 disabling is must for this system, How to fix all these issue :(


Regards
Aneesh

> > portal: 10.0.104.202,3260]: successful- Hide quoted text -
>
> - Show quoted text -

[Bangalore]

one more thing

i have tested this in a centos IP6 disabled 5.2 server where am not
getting

cnic: Unknown symbol __ipv6_addr_type
cnic: Unknown symbol ip6_route_output

but am geting "/dev/mysql/mysql: read failed after 0 of 4096 at 0:
Input/output error"





[root@ussd1 /]# pvdisplay
/dev/mysql/mysql: read failed after 0 of 4096 at 0: Input/output
error
--- Physical volume ---
PV Name /dev/sdc1

VG Name mysql
PV Size 300.03 GB / not usable 3.25 MB
Allocatable yes
PE Size (KByte) 4096
Total PE 76808

Free PE 51208
Allocated PE 25600
PV UUID JUPKC0-0p4S-odWF-kF2s-E1Kc-9D5x-KjTWZw

On Nov 19, 1:03 pm, Bangalore <aneeshsu...@gmail.com> wrote:
> Hi
>
>  i have installed one HP blade server with "Linux version

> 2.6.18-164.el5 (mockbu...@builder10.centos.org) (gcc version 4.1.2

[Mike Christie]

On 11/28/2010 01:44 PM, Bangalore wrote:
> i tried by unmounting the device before restarting Iscsi service , But

I said you also need to have lvm release the devices. What you are doing
is unsupported and the results you are getting are expected.

Before running iscsi stop you need to have lvm stop using the iscsi
devices, because when the stop is done it will remove the iscsi devices
and lvm will be referencing bad devices.

After you run iscsi start you need to run something like lvm vgscan so
the lvm devices are rebuilt using the new scsi devices.

I think you want to email the lvm group to see what commands they
reccommend to run.

[Mike Christie]

On 11/29/2010 06:28 PM, Mike Christie wrote:
> On 11/28/2010 01:44 PM, Bangalore wrote:
>> i tried by unmounting the device before restarting Iscsi service , But
>
> I said you also need to have lvm release the devices. What you are doing
> is unsupported and the results you are getting are expected.
>
> Before running iscsi stop you need to have lvm stop using the iscsi
> devices, because when the stop is done it will remove the iscsi devices
> and lvm will be referencing bad devices.
>
> After you run iscsi start you need to run something like lvm vgscan so
> the lvm devices are rebuilt using the new scsi devices.
>
> I think you want to email the lvm group to see what commands they
> reccommend to run.
>

For RHEL 5 based systems here is some more info:
http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Online_Storage_Reconfiguration_Guide/removing_devices.html#clean-device-removal

I do not think you need to do the lvm commands in there, because I think
those are for permanment removal and your situation has the iscsi
devices coming back right away. Maybe you just need to run a vgremove (I
am not sure, so like I said before it is best to ask the lvm people).

Also why are you even running iscsi stop?