[Iscsitarget-devel] consistent kernel BUG with 1.4.20 branch HEAD, since at least early May 2011
Leho Kraav
I have been running iscsitarget-1.4.20.2 on Gentoo successfully against
OS X and GlobalSAN iSCSI Initiator, their latest version being
4.1.0.279. Stability has been good.
I have 3 volumes, which are a bunch of LVM volumes sitting on a LUKS
encrypted hard drive.
Target iqn.2009-12.com.kraav:mobilitymac.backup
Lun 0 Path=/dev/backup/mobilitymac,Type=fileio
MaxConnections 1 # Number of
connections/session
Target iqn.2009-12.com.kraav:amcbook.backup
Lun 0 Path=/dev/backup/amcbook,Type=fileio
MaxConnections 1 # Number of
connections/session
Target iqn.2009-12.com.kraav:amilo.backup
Lun 0 Path=/dev/backup/amilo,Type=fileio
MaxConnections 1 # Number of
connections/session
Going from 2.6.34 -> 2.6.38 in early May broke iscsitarget and patches
were needed. At that point I gave HEAD first try, compiling SVN
checkout. Immediately I was hit with this kernel BUG when I connect to
target with my OS X initiator.
Aug 19 15:10:06 server kernel: [ 2434.614485] iscsi_trgt: BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.2/work/iscsitarget-1.4.20.2/kernel/iscsi.c:392
assert(req->tio)
Aug 19 15:10:06 server kernel: [ 2434.614494] Pid: 15398, comm: istiod1
Tainted: P 2.6.38-gentoo-r3 #16
Aug 19 15:10:06 server kernel: [ 2434.614497] Call Trace:
Aug 19 15:10:06 server kernel: [ 2434.614507] [<fa559b43>] ?
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614515] [<fa55e015>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614522] [<fa55f4bc>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614529] [<c117c13b>] ?
current_io_context+0x1b/0x30
Aug 19 15:10:06 server kernel: [ 2434.614536] [<fa55b70c>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614543] [<c10284f7>] ?
__wake_up_common+0x47/0x70
Aug 19 15:10:06 server kernel: [ 2434.614549] [<c1032c20>] ?
default_wake_function+0x0/0x10
Aug 19 15:10:06 server kernel: [ 2434.614555] [<fa55b5f0>] ?
worker_thread+0x0/0x2c0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614562] [<c104dd14>] ?
kthread+0x74/0x80
Aug 19 15:10:06 server kernel: [ 2434.614567] [<c104dca0>] ?
kthread+0x0/0x80
Aug 19 15:10:06 server kernel: [ 2434.614572] [<c1003236>] ?
kernel_thread_helper+0x6/0x10
Aug 19 15:10:06 server kernel: [ 2434.614592] ------------[ cut here
]------------
Aug 19 15:10:06 server kernel: [ 2434.614599] kernel BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.2/work/iscsitarget-1.4.20.2/kernel/iscsi.c:392!
Aug 19 15:10:06 server kernel: [ 2434.614600] invalid opcode: 0000 [#1]
PREEMPT SMP
Aug 19 15:10:06 server kernel: [ 2434.614602] last sysfs file:
/sys/devices/platform/it87.656/pwm1
Aug 19 15:10:06 server kernel: [ 2434.614603] Modules linked in:
iscsi_trgt nvidia(P) squashfs vboxnetadp vboxnetflt vboxdrv coretemp
it87 hwmon_vid hwmon nfs nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs
snd_seq snd_seq_device af_packet sco bnep rfcomm l2cap snd_dummy loop
fuse crc32c_intel hid_logitech usbhid btusb bluetooth imon
snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_pcm snd_timer
dvb_usb_dib0700 dib7000p dibx000_common dib0070 rtc_cmos snd dvb_usb
dvb_core rc_core rtc_core rtc_lib processor uhci_hcd skge r8168 sg
i2c_i801 snd_page_alloc button sr_mod firewire_ohci cdrom firewire_core
crc_itu_t pata_jmicron [last unloaded: nvidia]
Aug 19 15:10:06 server kernel: [ 2434.614629]
Aug 19 15:10:06 server kernel: [ 2434.614631] Pid: 15398, comm: istiod1
Tainted: P 2.6.38-gentoo-r3 #16 Gigabyte Technology Co., Ltd.
P55M-UD2/P55M-UD2
Aug 19 15:10:06 server kernel: [ 2434.614634] EIP: 0060:[<fa559b43>]
EFLAGS: 00010282 CPU: 1
Aug 19 15:10:06 server kernel: [ 2434.614636] EIP is at
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614638] EAX: 00000000 EBX:
eab2a3c0 ECX: 00000001 EDX: 00000000
Aug 19 15:10:06 server kernel: [ 2434.614639] ESI: eab2a3dc EDI:
eab2a3c0 EBP: eff69950 ESP: ea5cdf18
Aug 19 15:10:06 server kernel: [ 2434.614640] DS: 007b ES: 007b FS:
00d8 GS: 0000 SS: 0068
Aug 19 15:10:06 server kernel: [ 2434.614642] Process istiod1 (pid:
15398, ti=ea5cc000 task=effb8ec0 task.ti=ea5cc000)
Aug 19 15:10:06 server kernel: [ 2434.614643] Stack:
Aug 19 15:10:06 server kernel: [ 2434.614644] fa563ba0 fa563b44
00000188 fa5630f2 00000000 f286bb00 f4ac5c80 c1513c80
Aug 19 15:10:06 server kernel: [ 2434.614647] effb904c c1513c80
fa55e015 00010000 eab2a3c0 eab2a3dc f0a17000 eff69950
Aug 19 15:10:06 server kernel: [ 2434.614650] fa55f4bc c117c13b
000000d0 00000001 ea833d40 eab2a3c0 eff69940 f0a17000
Aug 19 15:10:06 server kernel: [ 2434.614652] Call Trace:
Aug 19 15:10:06 server kernel: [ 2434.614655] [<fa55e015>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614658] [<fa55f4bc>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614660] [<c117c13b>] ?
current_io_context+0x1b/0x30
Aug 19 15:10:06 server kernel: [ 2434.614662] [<fa55b70c>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614664] [<c10284f7>] ?
__wake_up_common+0x47/0x70
Aug 19 15:10:06 server kernel: [ 2434.614666] [<c1032c20>] ?
default_wake_function+0x0/0x10
Aug 19 15:10:06 server kernel: [ 2434.614669] [<fa55b5f0>] ?
worker_thread+0x0/0x2c0 [iscsi_trgt]
Aug 19 15:10:06 server kernel: [ 2434.614671] [<c104dd14>] ?
kthread+0x74/0x80
Aug 19 15:10:06 server kernel: [ 2434.614673] [<c104dca0>] ?
kthread+0x0/0x80
Aug 19 15:10:06 server kernel: [ 2434.614674] [<c1003236>] ?
kernel_thread_helper+0x6/0x10
Aug 19 15:10:06 server kernel: [ 2434.614675] Code: eb b9 c7 44 24 0c f2
30 56 fa c7 44 24 08 88 01 00 00 c7 44 24 04 44 3b 56 fa c7 04 24 a0 3b
56 fa e8 6a 5d e2 c6 e8 bb 5b e2 c6 <0f> 0b eb fe 89 7c 24 0c c7 44 24
08 fb 00 00 00 c7 44 24 04 e4
Aug 19 15:10:06 server kernel: [ 2434.614692] EIP: [<fa559b43>]
send_data_rsp+0x1d3/0x200 [iscsi_trgt] SS:ESP 0068:ea5cdf18
Aug 19 15:10:06 server kernel: [ 2434.614696] ---[ end trace
91f957db09e02c7f ]---
Just using a few minor kernel-related patches on top of 1.4.20.2 I
managed to get a working setup again.
Today I went to kernel 3.0.2. iscsitarget is apparently broken since
2.6.39 already, but the same patches fix things for 3.0, so that's not
the issue. I did give svn HEAD another try today and was still hit with
this kernel BUG:
Aug 19 14:14:21 server kernel: [ 125.438394] iSCSI Enterprise Target
Software - version 1.4.20.3
Aug 19 14:14:21 server kernel: [ 125.438427] iscsi_trgt: Registered io
type fileio
Aug 19 14:14:21 server kernel: [ 125.438428] iscsi_trgt: Registered io
type blockio
Aug 19 14:14:21 server kernel: [ 125.438429] iscsi_trgt: Registered io
type nullio
Aug 19 14:14:47 server kernel: [ 184.548158] iscsi_trgt: BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.2/work/iscsitarget-1.4.20.2/kernel/iscsi.c:392
assert(req->tio)
Aug 19 14:14:47 server kernel: [ 184.548166] Pid: 4005, comm: istiod1
Tainted: P 3.0.2-pf #1
Aug 19 14:14:47 server kernel: [ 184.548170] Call Trace:
Aug 19 14:14:47 server kernel: [ 184.548180] [<f9514ae3>] ?
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.548189] [<c13910a6>] ?
schedule+0x536/0xbe0
Aug 19 14:14:47 server kernel: [ 184.548196] [<f9518ed5>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.548203] [<f951a34c>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.548210] [<f95166bc>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.548217] [<c1027fd8>] ?
__wake_up_common+0x48/0x70
Aug 19 14:14:47 server kernel: [ 184.548222] [<c102cb10>] ?
try_to_wake_up+0x120/0x120
Aug 19 14:14:47 server kernel: [ 184.548228] [<f95165a0>] ?
wthread_start+0x140/0x140 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.548235] [<c1048834>] ?
kthread+0x74/0x80
Aug 19 14:14:47 server kernel: [ 184.548240] [<c10487c0>] ?
kthread_worker_fn+0x170/0x170
Aug 19 14:14:47 server kernel: [ 184.548246] [<c1394376>] ?
kernel_thread_helper+0x6/0xd
Aug 19 14:14:47 server kernel: [ 184.548263] ------------[ cut here
]------------
Aug 19 14:14:47 server kernel: [ 184.548368] kernel BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.2/work/iscsitarget-1.4.20.2/kernel/iscsi.c:392!
Aug 19 14:14:47 server kernel: [ 184.548460] invalid opcode: 0000 [#1]
PREEMPT SMP
Aug 19 14:14:47 server kernel: [ 184.548551] Modules linked in:
iscsi_trgt fuse af_packet bnep rfcomm nvidia(P) snd_hda_codec_realtek
snd_hda_intel i2c_i801 snd_hda_codec snd_pcm snd_timer snd
snd_page_alloc rtc_cmos skge btusb bluetooth r8169 mii button sg imon
dvb_usb_dib0700 dib7000p dib0070 dvb_usb dvb_core rc_core dibx000_common
processor hid_logitech usbhid sr_mod cdrom firewire_ohci firewire_core
crc_itu_t pata_jmicron uhci_hcd
Aug 19 14:14:47 server kernel: [ 184.549020]
Aug 19 14:14:47 server kernel: [ 184.549084] Pid: 4005, comm: istiod1
Tainted: P 3.0.2-pf #1 Gigabyte Technology Co., Ltd.
P55M-UD2/P55M-UD2
Aug 19 14:14:47 server kernel: [ 184.549186] EIP: 0060:[<f9514ae3>]
EFLAGS: 00010282 CPU: 0
Aug 19 14:14:47 server kernel: [ 184.549263] EIP is at
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.549338] EAX: 00000000 EBX:
f1db80c0 ECX: 00000000 EDX: 00000000
Aug 19 14:14:47 server kernel: [ 184.549418] ESI: f1db80dc EDI:
f1db80c0 EBP: f1c63490 ESP: f0c13f18
Aug 19 14:14:47 server kernel: [ 184.549499] DS: 007b ES: 007b FS:
00d8 GS: 0000 SS: 0068
Aug 19 14:14:47 server kernel: [ 184.549574] Process istiod1 (pid:
4005, ti=f0c12000 task=f0c4f000 task.ti=f0c12000)
Aug 19 14:14:47 server kernel: [ 184.549655] Stack:
Aug 19 14:14:47 server kernel: [ 184.549720] f951e8c0 f951e864
00000188 f951de12 c13910a6 00000001 f0c4de80 00000001
Aug 19 14:14:47 server kernel: [ 184.549835] 00000046 f0c4f000
f9518ed5 00010000 f1db80c0 f1db80dc f2fd6000 f1c63490
Aug 19 14:14:47 server kernel: [ 184.549949] f951a34c f0c4f15c
00000066 00000246 f0c4f000 f1db80c0 f1c63480 f2fd6000
Aug 19 14:14:47 server kernel: [ 184.550065] Call Trace:
Aug 19 14:14:47 server kernel: [ 184.550132] [<c13910a6>] ?
schedule+0x536/0xbe0
Aug 19 14:14:47 server kernel: [ 184.550206] [<f9518ed5>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.550285] [<f951a34c>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.550365] [<f95166bc>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.550443] [<c1027fd8>] ?
__wake_up_common+0x48/0x70
Aug 19 14:14:47 server kernel: [ 184.550517] [<c102cb10>] ?
try_to_wake_up+0x120/0x120
Aug 19 14:14:47 server kernel: [ 184.550592] [<f95165a0>] ?
wthread_start+0x140/0x140 [iscsi_trgt]
Aug 19 14:14:47 server kernel: [ 184.550670] [<c1048834>] ?
kthread+0x74/0x80
Aug 19 14:14:47 server kernel: [ 184.550742] [<c10487c0>] ?
kthread_worker_fn+0x170/0x170
Aug 19 14:14:47 server kernel: [ 184.550818] [<c1394376>] ?
kernel_thread_helper+0x6/0xd
Aug 19 14:14:47 server kernel: [ 184.550892] Code: eb b9 c7 44 24 0c 12
de 51 f9 c7 44 24 08 88 01 00 00 c7 44 24 04 64 e8 51 f9 c7 04 24 c0 e8
51 f9 e8 a4 bc e7 c7 e8 ee ba e7 c7 <0f> 0b eb fe 89 7c 24 0c c7 44 24
08 fb 00 00 00 c7 44 24 04 04
Aug 19 14:14:47 server kernel: [ 184.551404] EIP: [<f9514ae3>]
send_data_rsp+0x1d3/0x200 [iscsi_trgt] SS:ESP 0068:f0c13f18
Aug 19 14:14:47 server kernel: [ 184.551542] ---[ end trace
aa39b6938f772f63 ]---
Aug 19 14:15:33 server sudo: leho : TTY=pts/0 ;
PWD=/secure/home/leho ; USER=root ; COMMAND=/etc/init.d/ietd stop
This BUG hoses the module pretty badly, and clean shutdown will be
impossible. From what I can see, we're crashing in exactly the same
place, so it doesn't seem to be kernel dependent.
After this I went back to building blocks and handpicked only a few
patches for 2.6.39 on top of 1.4.20.2.
* Attached diff: iscsitarget-1.4.20.2-to-gentoopatched-working.diff
This resulted in an apparently working target, except now with this
build my OS X GlobalSAN Initiator totally hoses itself on Disconnecting
from target. As a test I did a 20GB Time Machine backup and that seemed
to have gone over without a hitch. Disconnection has never been an issue
before. So something with the 2.6.39 block-io.c file-io.c patches are
perhaps causing this. Hints for debugging welcome.
I've also attached a diff comparing my currently working build (except
for the disconnect problem) to today's SVN head. Some code in there
messes something up. Hints welcome.
* Attached diff:
iscsitarget-1.4.20.2-gentoopatched-working-to-latest-kbug.diff
--
Leho Kraav, M.Sc.
Arne Redlich
> Hi all
>
>
> I have been running iscsitarget-1.4.20.2 on Gentoo successfully against
> OS X and GlobalSAN iSCSI Initiator, their latest version being
> 4.1.0.279. Stability has been good.
>
> I have 3 volumes, which are a bunch of LVM volumes sitting on a LUKS
> encrypted hard drive.
>
> Target iqn.2009-12.com.kraav:mobilitymac.backup
> Lun 0 Path=/dev/backup/mobilitymac,Type=fileio
> MaxConnections 1 # Number of
> connections/session
>
> Target iqn.2009-12.com.kraav:amcbook.backup
> Lun 0 Path=/dev/backup/amcbook,Type=fileio
> MaxConnections 1 # Number of
> connections/session
>
> Target iqn.2009-12.com.kraav:amilo.backup
> Lun 0 Path=/dev/backup/amilo,Type=fileio
> MaxConnections 1 # Number of
> connections/session
>
>
> Going from 2.6.34 -> 2.6.38 in early May broke iscsitarget and patches
> were needed. At that point I gave HEAD first try, compiling SVN
> checkout. Immediately I was hit with this kernel BUG when I connect to
> target with my OS X initiator.
Leho,
Which branch do you mean with svn HEAD - trunk or 1.4.20?
Thanks,
Arne
------------------------------------------------------------------------------
Get a FREE DOWNLOAD! and learn more about uberSVN rich system,
user administration capabilities and model configuration. Take
the hassle out of deploying and managing Subversion and the
tools developers use with it. http://p.sf.net/sfu/wandisco-d2d-2
_______________________________________________
Iscsitarget-devel mailing list
Iscsitar...@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/iscsitarget-devel
Leho Kraav
Ross S. W. Walker
proper kernel version on Gentoo and the OP pulling from the
wrong HEAD.
If it were really 1.4.20 HEAD the version in the headers would
show 1.4.20.3, if it were from trunk, the headers would show
'trunk'.
Leho,
Can you subversion 1.4.20 HEAD, run a 'make patch' and post the
output so we can see if it's patching the files properly on
Gentoo?
-Ross
______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
Leho Kraav
>
> This looks like a combination of the Makefile not detecting the
> proper kernel version on Gentoo and the OP pulling from the
> wrong HEAD.
>
> If it were really 1.4.20 HEAD the version in the headers would
> show 1.4.20.3, if it were from trunk, the headers would show
> 'trunk'.
>
> Leho,
>
> Can you subversion 1.4.20 HEAD, run a 'make patch' and post the
> output so we can see if it's patching the files properly on
> Gentoo?
I don't think I've made a mistake here. I'm not sure if my attached
diffs came through to the list, though.
iscsitarget-1.4.20.2-gentoopatched-working-to-latest-kbug.diff clearly
shows 1.4.20.3 being patched on. I have all tags, branches and two
different gentoo patchsets copied into git branches, all comparisons
come out right. Kernel BUG was exhibited on the build equivalent to
branches/1.4.20 HEAD.
As for your request, everything seems to build clean in working copy.
"make patch" output was blank. This is what happened:
leho@server 1.4.20 [svn:] $ pwd
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20
$ make distclean
make -C usr clean
make[1]: Entering directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
rm -f *.o ietd ietadm
make[1]: Leaving directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
make -C /lib/modules/3.0.2-pf/build
SUBDIRS=/secure/home/lehois/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel
clean
make[1]: Entering directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
CLEAN
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/.tmp_versions
CLEAN
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/Module.symvers
make[1]: Leaving directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
find . -name \*.orig -exec rm -f \{\} \;
find . -name \*.rej -exec rm -f \{\} \;
find . -name \*~ -exec rm -f \{\} \;
find . -name Module.symvers -exec rm -f \{\} \;
leho@server 1.4.20 [svn:] $ make patch
leho@server 1.4.20 [svn:] $ make -j5
make -C usr
make -C /lib/modules/3.0.2-pf/build
SUBDIRS=/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel
modules
make[1]: Entering directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o ietd.o ietd.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o iscsid.o iscsid.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o conn.o conn.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o session.o session.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o target.o target.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o message.o message.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o ctldev.o ctldev.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o log.o log.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o chap.o chap.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o event.o event.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o param.o param.c
make[1]: Entering directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o plain.o plain.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o isns.o isns.c
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.o
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o md5.o md5.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o sha1.o sha1.c
cc -O2 -fno-inline -Wall -Wstrict-prototypes -I../include -D_GNU_SOURCE
-c -o ietadm.o ietadm.c
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/nthread.o
cc ietd.o iscsid.o conn.o session.o target.o message.o ctldev.o log.o
chap.o event.o param.o plain.o isns.o md5.o sha1.o -o ietd
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/wthread.o
cc ietadm.o param.o -o ietadm
make[1]: Leaving directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/config.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/digest.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/conn.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/session.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/target.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/volume.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iotype.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/file-io.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/null-io.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/target_disk.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/event.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/param.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/block-io.o
CC [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/ua.o
LD [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi_trgt.o
Building modules, stage 2.
MODPOST 1 modules
CC
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi_trgt.mod.o
LD [M]
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi_trgt.ko
make[1]: Leaving directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
--
Leho Kraav, M.Sc.
------------------------------------------------------------------------------
Leho Kraav
>>
>> > Can you subversion 1.4.20 HEAD, run a 'make patch' and post the
>> > output so we can see if it's patching the files properly on
>> > Gentoo?
>>
>> I don't think I've made a mistake here. I'm not sure if my attached
>> diffs came through to the list, though.
>
> to my confusion :) : as you saw yourself, 1.4.20 head compiles cleanly against 3.0.x
> kernels, and the patches you sent look like our *backward
> compatibility* patches. i.e. your patches are the *reverse* (I didn't
> check if they were complete) of the patches we use to make the latest
> IET compile against older kernels. So there must be something wrong
> with the code base you were using.
OK, yes, the attached diffs did include iscsitarget's own patchset
diffs. I was hoping it all wouldnt become too confusing.
>> As for your request, everything seems to build clean in working copy.
>> "make patch" output was blank. This is what happened:
>
> Ok, so far so good. Could you do a make clean, add "-g -O0" to the
> EXTRA_CFLAGS in your working copy's kernel/Makefile and then make and
> make install it?
> (Pls. make sure to remove any artefacts from the gentoo install before
> doing that, i.e. move the ietadm binary, the ietd binary and the
> iscsi_trgt.ko module out of the way)
>
> Once this pristine IET is installed, can you pls. repeat your tests
> and let us know the results? If there are any oopses the
> aforementioned changes to the CFLAGS might help us debugging any
> oopses (if the oops message isn't clear enough already).
I've done two things now:
* 1. Rebuilt iscsitarget as 1.4.20.299 in portage, cutting out *all*
previous worker code patches in ebuild, leaving it fully up to
iscsitarget to take care of compat patching. Was only forced to keep
Makefile augmentations to build against 3.0.2 and to make sure portage
-g -O0 were respected. This should take care of the possible patching
(portage) and then reverse-patching (iscsitarget compat) problem you
brought up, even though I doubt that actually happened - would that
really have gone by error-free? Anyway, this result was FAIL, still same
kernel BUG on connect.
===================================================================
--- usr/Makefile (revision 453)
+++ usr/Makefile (working copy)
@@ -1,4 +1,4 @@
-CFLAGS += -O2 -fno-inline -Wall -Wstrict-prototypes -I../include
+CFLAGS += -fno-inline -Wall -Wstrict-prototypes -I../include
CFLAGS += -D_GNU_SOURCE # required for glibc >= 2.8
PROGRAMS = ietd ietadm
LIBS =
--- a/Makefile
+++ b/Makefile
@@ -40,16 +40,29 @@ KMIN := $(shell echo $(KVER) | \
KREV := $(shell echo $(KVER) | \
sed -e 's/^[0-9][0-9]*\.[0-9][0-9]*\.\([0-9][0-9]*\).*/\1/')
-kver_eq = $(shell [ $(KMAJ)$(KMIN)$(KREV) -eq $(1)$(2)$(3) ] && \
+kver_eq = $(shell [ $(KMAJ) -eq $(1) -a $(KMIN) -eq $(2) -a $(KREV) -eq
$(3) ] && \
echo 1 || echo 0)
-kver_lt = $(shell [ $(KMAJ)$(KMIN)$(KREV) -lt $(1)$(2)$(3) ] && \
+
+kver_lt = $(shell [ $(KMAJ) -lt $(1) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -lt $(2) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -eq $(2) -a $(KREV) -lt $(3) ] && \
echo 1 || echo 0)
-kver_le = $(shell [ $(KMAJ)$(KMIN)$(KREV) -le $(1)$(2)$(3) ] && \
+
+kver_le = $(shell [ $(KMAJ) -lt $(1) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -lt $(2) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -eq $(2) -a $(KREV) -le $(3) ] && \
echo 1 || echo 0)
-kver_gt = $(shell [ $(KMAJ)$(KMIN)$(KREV) -gt $(1)$(2)$(3) ] && \
+
+kver_gt = $(shell [ ( $(KMAJ) -gt $(1) ) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -gt $(2) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -eq $(2) -a $(KREV) -gt $(3) ] && \
echo 1 || echo 0)
-kver_ge = $(shell [ $(KMAJ)$(KMIN)$(KREV) -ge $(1)$(2)$(3) ] && \
+
+kver_ge = $(shell [ ( $(KMAJ) -gt $(1) ) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -gt $(2) -o \
+ $(KMAJ) -eq $(1) -a $(KMIN) -eq $(2) -a $(KREV) -ge $(3) ] && \
echo 1 || echo 0)
+
kver_lk = $(shell [ `echo $(KVER) | egrep $(1)` ] && echo 1 || echo 0)
#
* 2. Attempted to build kernel module with -g -O0. Result is FAIL on
compilation. Which may be a good thing, possible problem source maybe?
$ make
make -C usr
make[1]: Entering directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory
`/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/usr'
make -C /lib/modules/3.0.2-pf/build
SUBDIRS=/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel
modules V=1
make[1]: Entering directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
test -e include/generated/autoconf.h -a -e include/config/auto.conf || (
\
echo; \
echo " ERROR: Kernel configuration is invalid."; \
echo " include/generated/autoconf.h or include/config/auto.conf
are missing.";\
echo " Run 'make oldconfig && make prepare' on kernel src to fix
it."; \
echo; \
/bin/false)
mkdir -p
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/.tmp_versions
; rm -f
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/.tmp_versions/*
make -f scripts/Makefile.build
obj=/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel
gcc
-Wp,-MD,/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/.tio.o.d
-nostdinc -isystem /usr/lib/gcc/i686-pc-linux-gnu/4.4.5/include
-I/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server/arch/x86/include
-Iarch/x86/include/generated -Iinclude -include
include/generated/autoconf.h -D__KERNEL__ -Wall -Wundef
-Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common
-Werror-implicit-function-declaration -Wno-format-security
-fno-delete-null-pointer-checks -O2 -m32 -msoft-float -mregparm=3
-freg-struct-return -mpreferred-stack-boundary=2 -march=i686
-mtune=core2 -Wa,-mtune=generic32 -ffreestanding -DCONFIG_AS_CFI=1
-DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -pipe
-Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx
-mno-sse2 -mno-3dnow -Wframe-larger-than=1024 -fno-stack-protector
-fomit-frame-pointer -Wdeclaration-after-statement -Wno-pointer-sign
-fno-strict-overflow -fconserve-stack
-I/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/../include
-g -O0 -DMODULE -D"KBUILD_STR(s)=#s"
-D"KBUILD_BASENAME=KBUILD_STR(tio)"
-D"KBUILD_MODNAME=KBUILD_STR(iscsi_trgt)" -c -o
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/.tmp_tio.o
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.c
In file included from include/linux/sem.h:81,
from include/linux/sched.h:79,
from include/linux/blkdev.h:6,
from
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi.h:11,
from
/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.c:7:
include/linux/rcupdate.h: In function ‘__kfree_rcu’:
include/linux/rcupdate.h:822: error: size of array ‘type name’ is negative
make[2]: ***
[/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.o]
Error 1
make[1]: ***
[_module_/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel]
Error 2
make[1]: Leaving directory
`/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
make: *** [kernel] Error 2
Arne Redlich
Thanks for posting this - I must have goofed when I tested compilation
against 3.0 kernels. We'll update the code accordingly.
> * 2. Attempted to build kernel module with -g -O0. Result is FAIL on
> compilation. Which may be a good thing, possible problem source maybe?
Hmm, it definitely seems to be a problem with the kernel source code ...
[snip]
> /secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.c
> In file included from include/linux/sem.h:81,
> from include/linux/sched.h:79,
> from include/linux/blkdev.h:6,
> from
> /secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/iscsi.h:11,
> from
> /secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.c:7:
> include/linux/rcupdate.h: In function ‘__kfree_rcu’:
> include/linux/rcupdate.h:822: error: size of array ‘type name’ is negative
> make[2]: ***
... compiling with -O0 against 2.6.39 works, while it's broken against
3.0 )same happens if you take IET out of the equation and try to build
the kernel itself with -O0). The offending code is a static assertion
that is checked at compile time, apparently with a different outcome
for -O0 and -O2.
I'll take a closer look at that.
> [/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel/tio.o]
> Error 1
> make[1]: ***
> [_module_/secure/home/leho/dev/workspace/iscsitarget.svn/branches/1.4.20/kernel]
> Error 2
> make[1]: Leaving directory
> `/mnt/datapool/gentoo/kernel/linux-3.0.2-pf-server'
> make: *** [kernel] Error 2
>
Since you run into the oops with the rebuilt, pristine IET too: can
you post the oops message for that particular version?
Thanks,
Arne
------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
Leho Kraav
>
> Thanks for posting this - I must have goofed when I tested compilation
> against 3.0 kernels. We'll update the code accordingly.
>
>> * 2. Attempted to build kernel module with -g -O0. Result is FAIL on
>> compilation. Which may be a good thing, possible problem source maybe?
>
> Hmm, it definitely seems to be a problem with the kernel source code ...
>
> [snip]
>
> you post the oops message for that particular version?
Aug 20 11:45:02 server kernel: [73137.170905] iscsi_trgt: Registered io
type nullio
Aug 20 11:45:34 server kernel: [73169.500479] iscsi_trgt: BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.299/work/iscsitarget-1.4.20.299/kernel/iscsi.c:392
assert(req->tio)
Aug 20 11:45:34 server kernel: [73169.500488] Pid: 3123, comm: istiod1
Tainted: P 3.0.2-pf #1
Aug 20 11:45:34 server kernel: [73169.500491] Call Trace:
Aug 20 11:45:34 server kernel: [73169.500502] [<f8be4ae3>] ?
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500510] [<c13910a6>] ?
schedule+0x536/0xbe0
Aug 20 11:45:34 server kernel: [73169.500517] [<f8be8ed5>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500524] [<f8bea34c>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500531] [<c117b9cb>] ?
current_io_context+0x1b/0x30
Aug 20 11:45:34 server kernel: [73169.500537] [<f8be66bc>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500544] [<c1027fd8>] ?
__wake_up_common+0x48/0x70
Aug 20 11:45:34 server kernel: [73169.500549] [<c102cb10>] ?
try_to_wake_up+0x120/0x120
Aug 20 11:45:34 server kernel: [73169.500556] [<f8be65a0>] ?
wthread_start+0x140/0x140 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500562] [<c1048834>] ?
kthread+0x74/0x80
Aug 20 11:45:34 server kernel: [73169.500567] [<c10487c0>] ?
kthread_worker_fn+0x170/0x170
Aug 20 11:45:34 server kernel: [73169.500573] [<c1394376>] ?
kernel_thread_helper+0x6/0xd
Aug 20 11:45:34 server kernel: [73169.500594] ------------[ cut here
]------------
Aug 20 11:45:34 server kernel: [73169.500598] kernel BUG at
/var/tmp/portage/sys-block/iscsitarget-1.4.20.299/work/iscsitarget-1.4.20.299/kernel/iscsi.c:392!
Aug 20 11:45:34 server kernel: [73169.500602] invalid opcode: 0000 [#1]
PREEMPT SMP
Aug 20 11:45:34 server kernel: [73169.500607] Modules linked in:
iscsi_trgt coretemp it87 vboxnetadp vboxnetflt vboxdrv dvb_usb_dib0700
dib7000p dib0070 dvb_usb dvb_core dibx000_common imon rc_c
ore hwmon_vid hwmon nfs nfsd lockd nfs_acl auth_rpcgss sunrpc snd_seq
snd_seq_device r8168 squashfs snd_dummy crc32c_intel loop fuse af_packet
bnep rfcomm nvidia(P) btusb bluetooth snd_hda_code
c_realtek i2c_i801 rtc_cmos snd_hda_intel snd_hda_codec snd_pcm
snd_timer snd snd_page_alloc skge button sg processor hid_logitech
usbhid sr_mod cdrom firewire_ohci firewire_core crc_itu_t pata
_jmicron uhci_hcd [last unloaded: i2c_dev]
Aug 20 11:45:34 server kernel: [73169.500666]
Aug 20 11:45:34 server kernel: [73169.500668] Pid: 3123, comm: istiod1
Tainted: P 3.0.2-pf #1 Gigabyte Technology Co., Ltd.
P55M-UD2/P55M-UD2
Aug 20 11:45:34 server kernel: [73169.500671] EIP: 0060:[<f8be4ae3>]
EFLAGS: 00010282 CPU: 1
Aug 20 11:45:34 server kernel: [73169.500673] EIP is at
send_data_rsp+0x1d3/0x200 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500675] EAX: 00000000 EBX:
c5914300 ECX: 00000000 EDX: 00000000
Aug 20 11:45:34 server kernel: [73169.500676] ESI: c591431c EDI:
c5914300 EBP: f04db1d0 ESP: cfefff18
Aug 20 11:45:34 server kernel: [73169.500677] DS: 007b ES: 007b FS:
00d8 GS: 0000 SS: 0068
Aug 20 11:45:34 server kernel: [73169.500679] Process istiod1 (pid:
3123, ti=cfefe000 task=ec046c80 task.ti=cfefe000)
Aug 20 11:45:34 server kernel: [73169.500680] Stack:
Aug 20 11:45:34 server kernel: [73169.500681] f8bee8c8 f8bee868
00000188 f8bede12 c13910a6 00000001 ec007a80 000000d0
Aug 20 11:45:34 server kernel: [73169.500684] f2eb5360 000048ea
f8be8ed5 00010000 c5914300 c591431c c1c0a000 f04db1d0
Aug 20 11:45:34 server kernel: [73169.500687] f8bea34c c117b9cb
000000d0 00000246 cfefbd38 c5914300 f04db1c0 c1c0a000
Aug 20 11:45:34 server kernel: [73169.500690] Call Trace:
Aug 20 11:45:34 server kernel: [73169.500692] [<c13910a6>] ?
schedule+0x536/0xbe0
Aug 20 11:45:34 server kernel: [73169.500694] [<f8be8ed5>] ?
is_volume_reserved+0x45/0x70 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500697] [<f8bea34c>] ?
disk_execute_cmnd+0x23c/0x2b0 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500699] [<c117b9cb>] ?
current_io_context+0x1b/0x30
Aug 20 11:45:34 server kernel: [73169.500702] [<f8be66bc>] ?
worker_thread+0x11c/0x2c0 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500704] [<c1027fd8>] ?
__wake_up_common+0x48/0x70
Aug 20 11:45:34 server kernel: [73169.500706] [<c102cb10>] ?
try_to_wake_up+0x120/0x120
Aug 20 11:45:34 server kernel: [73169.500708] [<f8be65a0>] ?
wthread_start+0x140/0x140 [iscsi_trgt]
Aug 20 11:45:34 server kernel: [73169.500710] [<c1048834>] ?
kthread+0x74/0x80
Aug 20 11:45:34 server kernel: [73169.500712] [<c10487c0>] ?
kthread_worker_fn+0x170/0x170
Aug 20 11:45:34 server kernel: [73169.500714] [<c1394376>] ?
kernel_thread_helper+0x6/0xd
Aug 20 11:45:34 server kernel: [73169.500715] Code: eb b9 c7 44 24 0c 12
de be f8 c7 44 24 08 88 01 00 00 c7 44 24 04 68 e8 be f8 c7 04 24 c8 e8
be f8 e8 a4 bc 7a c8 e8 ee ba 7a c8 <0f> 0b eb fe
89 7c 24 0c c7 44 24 08 fb 00 00 00 c7 44 24 04 04
Aug 20 11:45:34 server kernel: [73169.500732] EIP: [<f8be4ae3>]
send_data_rsp+0x1d3/0x200 [iscsi_trgt] SS:ESP 0068:cfefff18
Aug 20 11:45:34 server kernel: [73169.500735] ---[ end trace
cdc221f287f6d60f ]---
--
Leho Kraav, M.Sc.
------------------------------------------------------------------------------