iscsitarget && open-iscsi && 'connection timed out' when trying to login
Turbo Fredriksson
have much luck so far.
I can run a discover (and almost anything else):
----- s n i p -----
celia:~# iscsiadm -m discovery -t isns
[...]
192.168.69.8:43070,1 iqn.2011-12.com.bayour:storage.debian.lenny.64
[...]
----- s n i p -----
I have setup the iface (preparing for a multihomed machine):
----- s n i p -----
celia:~# iscsiadm -m iface
default tcp,default,default,unknown
iser iser,default,default,unknown
bnx2i bnx2i,default,default,unknown
iface0
tcp,00:21:97:8c:8f:5f,default,iqn.1993-08.org.debian:01:97d0448a2b3c
celia:~# iscsiadm -m iface -I iface0
iface.iscsi_ifacename = iface0
iface.net_ifacename = default
iface.hwaddress = 00:21:97:8c:8f:5f
iface.transport_name = tcp
iface.initiatorname = iqn.1993-08.org.debian:01:97d0448a2b3c
celia:~# ifconfig eth2
eth2 Link encap:Ethernet HWaddr 00:21:97:8c:8f:5f
inet addr:192.168.69.8 Bcast:192.168.69.255
Mask:255.255.255.0
inet6 addr: fe80::221:97ff:fe8c:8f5f/64 Scope:Link
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:12548813 errors:0 dropped:0 overruns:0 frame:0
TX packets:22819953 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:50000
RX bytes:927858361 (884.8 MiB) TX bytes:32070627200 (29.8
GiB)
Interrupt:44 Base address:0xe000
celia:~#
----- s n i p -----
But when i try to login to the target, it fails:
----- s n i p -----
celia:~# iscsiadm -m node -T
iqn.2011-12.com.bayour:storage.debian.squeeze.64 -l
Logging in to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,3260]
Logging in to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,43070]
Logging in to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,43305]
[hangs there for x (120?) seconds]
iscsiadm: Could not login to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,3260]:
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,43070]:
iscsiadm: initiator reported error (8 - connection timed out)
iscsiadm: Could not login to [iface: iface0, target:
iqn.2011-12.com.bayour:storage.debian.squeeze.64, portal:
192.168.69.8,43305]:
iscsiadm: initiator reported error (8 - connection timed out)
celia:~#
----- s n i p -----
Running iscsid in debug mode, I see some very strange output when
starting up:
----- s n i p -----
celia:/etc/iscsi# ps | egrep 'iet|iscsi|isns'
20218 ? Ssl 0:00 /usr/sbin/isnsd -a 192.168.69.8 -b
192.168.69.255
20238 ? Ss 0:00 /usr/sbin/ietd --address=192.168.69.8
celia:/etc/iscsi# iscsid -f -d8 2>&1 | tee /tmp/x
[...]
iscsid: Cannot resolve host . getaddrinfo error: [Servname not
supported for ai_socktype]
iscsid: cannot resolve host name
iscsid: destroying session
[...]
----- s n i p -----
However, I can clearly see (in /tmp/x) that it - iscsiadm - is actually
connecting, there's
"movement" in iscsid). This keep repeating:
----- s n i p -----
iscsid: thread 0887b9d4 removed from poll_list
iscsid: exec thread 0887b9d4 callback
iscsid: thread 0x887b9d4 schedule: delay 0 state 3
iscsid: thread removed
----- s n i p -----
There is no iptables (everything is in ACCEPT):
----- s n i p -----
celia:~# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
celia:~# iptables -L -n -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
----- s n i p -----
Both the target(s), initiator(s) and iSNSd is running on the same
host and changing from iSNS to sendtarget does not change anything...
Kernel: 3.0.8/x86_64
OS: Debian GNU/Linux Lenny
ZFS: ZoL 0.6.0-rc6/git 20111119
iSCSI: iscsitarget (1.4.20.2-6),
open-iscsi (2.0.870~rc3-0.4.1),
isns (2.1-01+dfsg-3)
--
... but you know as soon as Oracle starts waving its wallet at a
Company it's time to run - fast.
/illumos mailing list
Mike Christie
This is just internal iscsid stuff. It just means the daemon is probing
to see if there have been any events like have we connected to the iscsi
target.
Could you send all of the iscsid log you have?
> iscsid: thread 0887b9d4 removed from poll_list
> iscsid: exec thread 0887b9d4 callback
> iscsid: thread 0x887b9d4 schedule: delay 0 state 3
> iscsid: thread removed
> ----- s n i p -----
>
>
> Both the target(s), initiator(s) and iSNSd is running on the same
> host and changing from iSNS to sendtarget does not change anything...
>
If you do not use ifaces does it work?
Did you change the rp_filter?
Could you also try a newer version of open-iscsi?
Turbo Fredriksson
> I am going to reply on the list, so if we figure it out it is searchable
> on google. Is that ok with you?
Sorry, of course. Used the webclient. It isn't as smart as one could hope :)
>> I'll try. But it's higly probable that this is a 64bit kernel/32bit
>> userland problem...
Is there ANY hope for rescue/success, if this is the case?
--
Ehhhhm - The battle cry of the cronical masturbater.
- Charlie Harper
Mike Christie
> On Tue, 29 Nov 2011 18:42:06 -0600, Mike Christie wrote:
>> Could you send all of the iscsid log you have?
>
> http://bayour.com/misc/iscsid.log.txt
>
>> If you do not use ifaces does it work?
>
Could you send the iscsid log for that?
>
>> Did you change the rp_filter?
>
> Ehhh, no? That's the iptables thing? Then no.
No. Something changed in newer kernels, so you have to set that value
now when using ifaces.
The iface binding feature requires the sysctl setting
net.ipv4.conf.default.rp_filter to be set to 0 or 2. This can be set
in /etc/sysctl.conf by having the line:
net.ipv4.conf.default.rp_filter = N
where N is 0 or 2. Note that when setting this you may have to reboot
the box for the value to take effect.
rp_filter information from Documentation/networking/ip-sysctl.txt:
rp_filter - INTEGER
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path
Each incoming packet is tested against the FIB and if the
interface
is not the best reverse path the packet check will fail.
By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path
Each incoming packet's source address is also tested against
the FIB
and if the source address is not reachable via any interface
the packet check will fail.
However, if this is not even working when not using ifaces that is not
your problem. But you will probably have to set it later when we get the
basics working.
>
>> Could you also try a newer version of open-iscsi?
>
> I'll try. But it's higly probable that this is a 64bit kernel/32bit
> userland
> problem...
>
Normally though we would be failing in some iscsi command to the kernel.
In your logs we are failing before we even hit the 32/64 issue. We are
just not able to do a tcp/ip connection. The iscsi login timer
eventually fires, because the connect() to the target just times out.
For these:
iscsid: Cannot resolve host . getaddrinfo error: [Servname not supported
for ai_socktype]
it looks like you either have partially setup iscsi connection/sessions
(maybe due to iscsiadm failing to clean up properly on a failure) or
maybe you are doing something like iscsi root boot and you are using old
tools without also setting CONFIG_SYSFS_DEPRECATED=y and
CONFIG_SYSFS_DEPRECATED_V2=y in your kernel .config (the new tools
should work instead of building with the depreciated config options).
Turbo Fredriksson
> Could you also try a newer version of open-iscsi?
Got 2.0.872-2 built and installed. Same thing.
http://bayour.com/misc/ietd-3.log.txt
http://bayour.com/misc/iscsid-3.log.txt
--
Geologists recently discovered that "earthquakes" are nothing more than
Bruce Schneier and Chuck Norris communicating via a roundhouse
kick-based cryptosystem.
Turbo Fredriksson
>>> If you do not use ifaces does it work?
>>
>> No.
>
> Could you send the iscsid log for that?
http://bayour.com/misc/ietd-2.log.txt
http://bayour.com/misc/iscsid-2.log.txt
>>> Did you change the rp_filter?
>>
>> Ehhh, no? That's the iptables thing? Then no.
>
> No. Something changed in newer kernels, so you have to set that value
> now when using ifaces.
Ok, it's "always" been set to '1'.
>>> Could you also try a newer version of open-iscsi?
>>
>> I'll try. But it's higly probable that this is a 64bit kernel/32bit
>> userland problem...
>
> Normally though we would be failing in some iscsi command to the
> kernel.
> In your logs we are failing before we even hit the 32/64 issue.
Ok, so that's somewhat good news :). Thanx.
> or maybe you are doing something like iscsi root boot
I am not.
> and you are using old
> tools without also setting CONFIG_SYSFS_DEPRECATED=y and
> CONFIG_SYSFS_DEPRECATED_V2=y in your kernel .config (the new tools
> should work instead of building with the depreciated config options).
----- s n i p -----
celia:~# uname -a
Linux celia 3.0.8+TF.1 #2 SMP Wed Nov 9 18:38:24 CET 2011 x86_64
GNU/Linux
celia:~# grep CONFIG_SYSFS_DEPRECATED /boot/config-`uname -r`
CONFIG_SYSFS_DEPRECATED=y
CONFIG_SYSFS_DEPRECATED_V2=y
----- s n i p -----
Mike Christie
> On Nov 30, 2011, at 1:42 AM, Mike Christie wrote:
>
>> Could you also try a newer version of open-iscsi?
>
>
> Got 2.0.872-2 built and installed. Same thing.
>
> http://bayour.com/misc/ietd-3.log.txt
> http://bayour.com/misc/iscsid-3.log.txt
Can you put this file up again?
Also if you just put the target on a different box does it work?
Turbo Fredriksson
> Run the iet on a separate machine.
Do you mean ietadm or ietd? ietadm works from everywhere. But
ietd MUST run on the physical machine it's currently running
on. That's the one with all the disks, and the whole point of
this exercise is to have the disks 'exported/shared' (?) using
iSCSI from this machine.
Although not absolutly required, it would be ... "very nice"
(to use an understatement :) if I could ALSO run an initiator
on this machine.
> In the logs are you running iet and
> open-iscsi on the same box?
I am.
> You mean if you do
>
> iscsiadm -m discovery -t st -p localhost
> then
> iscsiadm -m node -t yourtarget -p localhost -l
> or
> iscsiadm -m node -t yourtarget -p 127.0.0.1 -l
> then it works?
When I mean localhost, i rather mean the host localhost,
not the address localhost... Hmmm! Not even I understood THAT :)
On the same machine, I use the name of the machine to connect.
As in: The 'iron' (physical host/machine) is called Celia (with
the IP 192.168.69.8). So on Celia I run the target AND the/a
initiator. Celia is running a 64bit kernel, but I haven't upgraded
the userland (not enough time or big enough b**ls :) so that's 32bit.
So on celia:
celia:~# iscsiadm -m discovery -t st -p celia
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.mosix02
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.mosix01
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.freebsd82
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.squeeze.64
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.kfreebsd.64
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.kfreebsd.32
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.live
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.64
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.32.source
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.32
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.xp.pro
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.vista
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.7
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.ubuntu.lucid.launchpad
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp3
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp2
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp1
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.astrix
celia:~# iscsiadm -m node -T iqn.2011-12.com.bayour:storage.debian.lenny.64 -l
Logging in to [iface: default, target: iqn.2011-12.com.bayour:storage.debian.lenny.64, portal: 192.168.69.8,3260]
[this never seem to return - never worked]
However, on lenny64 (which is a VM running in VBox which is
started from a 64bit chroot on Celia - to get VBox working,
this was the only way other than upgrade/reinstall/new machine)
which is 64/64:
lenny64:~# iscsiadm -m discovery -t st -p celia
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp1
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.32.source
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.freebsd82
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.32
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.live
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp2
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.xp.pro
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.vista
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.ubuntu.lucid.launchpad
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.mosix01
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.windows.7
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.kfreebsd.64
192.168.69.8:3260,1 iqn.2011-11.com.bayour:storage.mosix02
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.astrix
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.lenny.64
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.squeeze.64
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.debian.kfreebsd.32
192.168.69.8:3260,1 iqn.2011-12.com.bayour:storage.nexenta.cp3
lenny64:~# iscsiadm -m node -T iqn.2011-12.com.bayour:storage.debian.lenny.64 -l
Logging in to [iface: default, target: iqn.2011-12.com.bayour:storage.debian.lenny.64, portal: 192.168.69.8,3260]
Login to [iface: default, target: iqn.2011-12.com.bayour:storage.debian.lenny.64, portal: 192.168.69.8,3260]: successful
lenny64:~# dmesg | tail | egrep ' sd [0-9].*hardware sector' | tail -n1
[436886.256327] sd 5:0:0:0: [sde] 31457280 512-byte hardware sectors (16106 MB)
lenny64:~# fdisk -l /dev/sde
Disk /dev/sde: 16.1 GB, 16106127360 bytes
64 heads, 32 sectors/track, 15360 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk identifier: 0x2637ddfd
Device Boot Start End Blocks Id System
/dev/sde1 * 1 192 196592 83 Linux
/dev/sde2 193 1147 977920 82 Linux swap / Solaris
/dev/sde3 1148 15360 14554112 83 Linux
lenny64:~#
... which is correct, since I've done it before :). But never on
Celia.
PS. The tings that works on lenny64 ALSO works inside the 64bit
chroot on Celia!
> So for the setup that fails it is 32 bit user/64 bit kernel with both
> the initiator and target running on the same OS instance/box?
Correct!
--
There are no dumb questions,
unless a customer is asking them.
- Unknown