linux-iscsi and XFS

252 Aufrufe
Direkt zur ersten ungelesenen Nachricht

Harald Kubota

ungelesen,
21.05.2005, 11:56:0921.05.05
an open-...@googlegroups.com, linux-iscsi development team
Hi,

is there something I should know about linux-iscsi (5.0.0.3rcX, v327)
and XFS?
Using ext3 and reiserfs I have no unusual issues (except for bugs which
are quickly
fixed or bugs/incompatibilities on the target side, which are fixed
quickly too).

Using xfs gives me this however:

May 21 23:54:09 burner kernel: Bad page state at free_hot_cold_page (in
process 'swapper', page c1233720)
May 21 23:54:09 burner kernel: flags:0x20000080 mapping:00000000
mapcount:0 count:0
May 21 23:54:09 burner kernel: Backtrace:
May 21 23:54:09 burner kernel: [write_one_page+100/288] bad_page+0x74/0xb0
May 21 23:54:09 burner kernel: [<c013f544>] bad_page+0x74/0xb0
May 21 23:54:09 burner kernel: [__pdflush+277/464]
free_hot_cold_page+0x65/0x130
May 21 23:54:09 burner kernel: [<c013fc95>] free_hot_cold_page+0x65/0x130
May 21 23:54:09 burner kernel: [dev_graft_qdisc+173/208]
skb_release_data+0x9d/0xd0
May 21 23:54:09 burner kernel: [<c03d1add>] skb_release_data+0x9d/0xd0
May 21 23:54:10 burner kernel: [qdisc_graft+157/208] __kfree_skb+0x5d/0x150
May 21 23:54:10 burner kernel: [<c03d1b9d>] __kfree_skb+0x5d/0x150
May 21 23:54:10 burner kernel: [qdisc_graft+32/208] kfree_skbmem+0x10/0x30
May 21 23:54:10 burner kernel: [<c03d1b20>] kfree_skbmem+0x10/0x30
May 21 23:54:10 burner kernel: [pg0+943303240/1067226112]
e1000_clean_tx_irq+0x1f8/0x3a0 [e1000]
May 21 23:54:10 burner kernel: [<f89cfa48>]
e1000_clean_tx_irq+0x1f8/0x3a0 [e1000]
May 21 23:54:10 burner kernel: [sched_setscheduler+220/432]
recalc_task_prio+0x8c/0x160
May 21 23:54:10 burner kernel: [<c0116ebc>] recalc_task_prio+0x8c/0x160
May 21 23:54:10 burner kernel: [pg0+943302495/1067226112]
e1000_clean+0x3f/0x130 [e1000]
May 21 23:54:10 burner kernel: [<f89cf75f>] e1000_clean+0x3f/0x130 [e1000]
May 21 23:54:10 burner kernel: [vc_resize+305/944]
add_interrupt_randomness+0x31/0x40
May 21 23:54:10 burner kernel: [<c02e9c11>]
add_interrupt_randomness+0x31/0x40
May 21 23:54:10 burner kernel: [netlink_broadcast+681/896]
net_rx_action+0x119/0x1a0
May 21 23:54:10 burner kernel: [<c03d8449>] net_rx_action+0x119/0x1a0
May 21 23:54:10 burner kernel: [proc_dostring+227/416]
__do_softirq+0x43/0x90
May 21 23:54:10 burner kernel: [<c0120603>] __do_softirq+0x43/0x90
May 21 23:54:10 burner kernel: [proc_dostring+342/416] do_softirq+0x26/0x30
May 21 23:54:10 burner kernel: [<c0120676>] do_softirq+0x26/0x30
May 21 23:54:10 burner kernel: [proc_doutsstring+117/224]
irq_exit+0x35/0x40
May 21 23:54:10 burner kernel: [<c0120735>] irq_exit+0x35/0x40
May 21 23:54:10 burner kernel: [show_interrupts+494/528] do_IRQ+0x1e/0x30
May 21 23:54:10 burner kernel: [<c010594e>] do_IRQ+0x1e/0x30
May 21 23:54:10 burner kernel: [device_not_available+22/42]
common_interrupt+0x1a/0x20
May 21 23:54:10 burner kernel: [<c0103c72>] common_interrupt+0x1a/0x20
May 21 23:54:10 burner kernel: [default_idle+0/48] default_idle+0x0/0x30
May 21 23:54:10 burner kernel: [<c0101030>] default_idle+0x0/0x30
May 21 23:54:10 burner kernel: [default_idle+35/48] default_idle+0x23/0x30
May 21 23:54:10 burner kernel: [<c0101053>] default_idle+0x23/0x30
May 21 23:54:10 burner kernel: [cpu_idle+66/96] cpu_idle+0x42/0x60
May 21 23:54:10 burner kernel: [<c01010e2>] cpu_idle+0x42/0x60
May 21 23:54:10 burner kernel: [init_centaur+266/480]
start_kernel+0x18a/0x1d0
May 21 23:54:10 burner kernel: [<c05bc83a>] start_kernel+0x18a/0x1d0
May 21 23:54:10 burner kernel: [init_cyrix+32/720]
unknown_bootoption+0x0/0x1e0
May 21 23:54:10 burner kernel: [<c05bc390>] unknown_bootoption+0x0/0x1e0
May 21 23:54:10 burner kernel: Trying to fix it up, but a reboot is needed

And another initiator:

May 22 00:43:27 wbox kernel: Bad page state at free_hot_cold_page (in
process 'swapper', page c1640d20)
May 22 00:43:27 wbox kernel: flags:0x20000084 mapping:00000000
mapcount:0 count:0
May 22 00:43:27 wbox kernel: Backtrace:
May 22 00:43:27 wbox kernel: [<c0140725>] bad_page+0x75/0xb0
May 22 00:43:27 wbox kernel: [<c0140e96>] free_hot_cold_page+0x66/0x120
May 22 00:43:27 wbox kernel: [<c0441ef3>] tcp_in_window+0x303/0x510
May 22 00:43:27 wbox kernel: [<c03e5ec8>] skb_release_data+0x98/0xc0
May 22 00:43:27 wbox kernel: [<c03e5f00>] kfree_skbmem+0x10/0x30
May 22 00:43:27 wbox kernel: [<c0417bf9>] tcp_clean_rtx_queue+0x169/0x450
May 22 00:43:27 wbox kernel: [<c04185e8>] tcp_ack+0xe8/0x650
May 22 00:43:27 wbox kernel: [<c0414c2d>] tcp_event_data_recv+0x2bd/0x300
May 22 00:43:27 wbox kernel: [<c041b1b3>] tcp_rcv_established+0x2d3/0x890
May 22 00:43:27 wbox kernel: [<c0424580>] tcp_v4_do_rcv+0x110/0x130
May 22 00:43:27 wbox kernel: [<c0424dd1>] tcp_v4_rcv+0x831/0x960
May 22 00:43:27 wbox kernel: [<c0407e05>] ip_local_deliver+0xa5/0x2b0
May 22 00:43:27 wbox kernel: [<c0408010>] ip_local_deliver_finish+0x0/0x210
May 22 00:43:27 wbox kernel: [<c040858e>] ip_rcv+0x36e/0x530
May 22 00:43:27 wbox kernel: [<c0408750>] ip_rcv_finish+0x0/0x2e0
May 22 00:43:27 wbox kernel: [<c03ec64d>] netif_receive_skb+0x27d/0x2e0
May 22 00:43:27 wbox kernel: [<c0355aeb>] rtl8139_rx+0x18b/0x370
May 22 00:43:27 wbox kernel: [<c0355f10>] rtl8139_poll+0x60/0x100
May 22 00:43:27 wbox kernel: [<c03ec81a>] net_rx_action+0x6a/0xf0
May 22 00:43:27 wbox kernel: [<c0121641>] __do_softirq+0x41/0xa0
May 22 00:43:27 wbox kernel: [<c01216c6>] do_softirq+0x26/0x30
May 22 00:43:27 wbox kernel: [<c0121785>] irq_exit+0x35/0x40
May 22 00:43:27 wbox kernel: [<c010581e>] do_IRQ+0x1e/0x30
May 22 00:43:27 wbox kernel: [<c0103ba2>] common_interrupt+0x1a/0x20
May 22 00:43:27 wbox kernel: [<c03046a3>] acpi_processor_idle+0x0/0x256
May 22 00:43:27 wbox kernel: [<c03047a2>] acpi_processor_idle+0xff/0x256
May 22 00:43:27 wbox kernel: [<c03046a5>] acpi_processor_idle+0x2/0x256
May 22 00:43:27 wbox kernel: [<c01010e2>] cpu_idle+0x42/0x60
May 22 00:43:27 wbox kernel: [<c05d687d>] start_kernel+0x18d/0x1d0
May 22 00:43:27 wbox kernel: [<c05d63b0>] unknown_bootoption+0x0/0x1f0
May 22 00:43:27 wbox kernel: Trying to fix it up, but a reboot is needed

and then when I did a logout via iscsiadm (machine was wbox):

kernel BUG at mm/slab.c:939!
invalid operand: 0000 [#1]
PREEMPT
Modules linked in: iscsi_tcp scsi_transport_iscsi snd_pcm_oss
snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_via82xx
gameport snd_ac97_codec snd_pcm snd_timer snd_page_alloc snd_mpu401_uart
snd_rawmidi snd_seq_device
CPU: 0
EIP: 0060:[<c014440a>] Tainted: G B VLI
EFLAGS: 00010246 (2.6.12-rc4)
EIP is at kmem_freepages+0x4a/0xb0
eax: 00000000 ebx: 00000008 ecx: 00000006 edx: c1640d20
esi: c17dfc60 edi: f2068000 ebp: f2068000 esp: f7c43ed0
ds: 007b es: 007b ss: 0068
Process events/0 (pid: 3, threadinfo=f7c42000 task=c17f4020)
Stack: c17d7b7c c17d6e10 c17dfc60 c17dfc60 f12d9360 c0144512 c17dfc60
f2068000
c17d6e18 f12d9360 c17dfc60 00000002 c17dfcd0 c0145b85 c17dfc60
f12d9360
00000000 f7c42000 c17dfc7c f7c42000 f7c42000 c17f4144 c063fa40
00000297
Call Trace:
[<c0144512>] slab_destroy+0x52/0xf0
[<c0145b85>] cache_reap+0x125/0x1c0
[<c012cdb0>] worker_thread+0x210/0x2f0
[<c0145a60>] cache_reap+0x0/0x1c0
[<c0118bd0>] default_wake_function+0x0/0x20
[<c0118bd0>] default_wake_function+0x0/0x20
[<c012cba0>] worker_thread+0x0/0x2f0
[<c0131489>] kthread+0xa9/0xf0
[<c01313e0>] kthread+0x0/0xf0
[<c010133d>] kernel_thread_helper+0x5/0x18
Code: 00 40 c1 ea 0c d3 e3 c1 e2 05 8d 4b ff 01 c2 83 f9 ff 74 28 8d b6
00 00 00 00 8d bc 27 00 00 00 00 0f ba 32 07 19 c0 85 c0 75 08 <0f> 0b
ab 03 1c 01 4d c0 83 c2 20 49 83 f9 ff 75 e5 c7 04 24 14

I know XFS is different in many ways from other file systems, so I am
not surprised that if any filesystem shows
unusual behaviour, it's XFS. However this bug is unexpected and pretty
fatal.
It does happen very quick after mounting an XFS filesystem via iSCSI and
doing something. bonnie++ triggered it after some minutes
and once a simple delete of 2 files (one 2GB, one some hundred MB large)
did the job.

Hardware used:
target is my trusty 2.6.11, iet r1137 dual P3-800 with e1000.
initiator "burner" is my usual 2.6.12-rc4, linux-iscsi v327 Athlon 2200+
with e1000.
initiator "wbox" is 2.6.12-rc4, linux-iscsi v327, Celeron 1200, RTL8139
(yeah, 100 Mbit/s)

All have preemptible kernels.

As the target does not seem to play any role here and XFS runs well and
stable on 'normal' disks,
I guess this has something to do with XFS mapping memory for its use
which is not compatible
with linux-iscsi's memory usage. I cannot claim to understand either one
though, so I might be
completely wrong.

Harald

Alex Aizman

ungelesen,
21.05.2005, 12:59:5621.05.05
an open-...@googlegroups.com, linux-iscsi development team, Harald Kubota
Harald Kubota wrote:
>
> Hi,
>
> is there something I should know about linux-iscsi
> (5.0.0.3rcX, v327) and XFS?
> Using ext3 and reiserfs I have no unusual issues (except for
> bugs which are quickly fixed or bugs/incompatibilities on the
> target side, which are fixed quickly too).
>
> Using xfs gives me this however:
>
> May 21 23:54:09 burner kernel: Bad page state at
> free_hot_cold_page (in process 'swapper', page c1233720) May

Both traces result from a combination of OOM (out-of-memory) and XFS. And
possibly e1000 as well, because the latter needs tons of DMA-able memory
(pages). Looks like some core structure, or structures in the kernel get
corrupted, and from then on its all downhill. The right place to discuss
this is LKML: linux-...@vger.kernel.org

Try also this no-brainer, and let know how/if it changes:

echo 65535 > /proc/sys/vm/min_free_kbytes

Dmitry Yusupov

ungelesen,
21.05.2005, 14:13:5621.05.05
an open-...@googlegroups.com, linux-iscsi development team
Just tried to reproduce it on my environment UML/initiator/2.6.13-rc3
vs. UML/target/2.6.13-rc3. Could not reproduce.

-bash-2.05b# /usr/sbin/bonnie++ -s32 -r16 -u root -d .
Using uid:0, gid:0.
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.03 ------Sequential Output------ --Sequential Input- --
Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --
Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %
CP /sec %CP
fedora2.goober. 32M 1276 9 3835 6 1659 2 3440 14 5616 0
927.0 4
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -
Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %
CP /sec %CP
16 245 2 12653 45 293 0 274 3 8441 46
179 0
fedora2.goober.org,32M,1276,9,3835,6,1659,2,3440,14,5616,0,927.0,4,16,245,2,12653,45,293,0,274,3,8441,46,179,0


On Sun, 2005-05-22 at 00:56 +0900, Harald Kubota wrote:
> Hi,
>
> is there something I should know about linux-iscsi (5.0.0.3rcX, v327)
> and XFS?
> Using ext3 and reiserfs I have no unusual issues (except for bugs which
> are quicklyenv

open_iscsi

ungelesen,
21.05.2005, 15:27:3221.05.05
an open-...@googlegroups.com, linux-iscsi development team
Where can I go to get the initiator source?

Eddy

Mike Christie

ungelesen,
21.05.2005, 15:33:3621.05.05
an open-...@googlegroups.com, linux-iscsi development team
Harald Kubota wrote:
> Hi,
>
> is there something I should know about linux-iscsi (5.0.0.3rcX, v327)
> and XFS?
> Using ext3 and reiserfs I have no unusual issues (except for bugs which
> are quickly
> fixed or bugs/incompatibilities on the target side, which are fixed
> quickly too).
>
> Using xfs gives me this however:

I think xfs and iscsi do not play nice
http://www.ussg.iu.edu/hypermail/linux/kernel/0408.3/0061.html

Mike Christie

ungelesen,
21.05.2005, 15:56:5521.05.05
an open-...@googlegroups.com, linux-iscsi development team
open_iscsi wrote:
>
> Where can I go to get the initiator source?
>
> Eddy

On http://www.open-iscsi.org/ you can find the latest development
snapshot http://www.open-iscsi.org/bits/open-iscsi-0.3rc4-325.tar.gz and
links to the subversion source
svn checkout svn://svn.berlios.de/open-iscsi

Harald Kubota

ungelesen,
22.05.2005, 04:04:5822.05.05
an open-...@googlegroups.com, linux-iscsi development team
Alex Aizman wrote:

>Harald Kubota wrote:
>
>
>>Hi,
>>
>>is there something I should know about linux-iscsi
>>(5.0.0.3rcX, v327) and XFS?
>>Using ext3 and reiserfs I have no unusual issues (except for
>>bugs which are quickly fixed or bugs/incompatibilities on the
>>target side, which are fixed quickly too).
>>
>>Using xfs gives me this however:
>>
>>May 21 23:54:09 burner kernel: Bad page state at
>>free_hot_cold_page (in process 'swapper', page c1233720) May
>>
>>
>
>Both traces result from a combination of OOM (out-of-memory) and XFS. And
>possibly e1000 as well, because the latter needs tons of DMA-able memory
>(pages). Looks like some core structure, or structures in the kernel get
>corrupted, and from then on its all downhill. The right place to discuss
>this is LKML: linux-...@vger.kernel.org
>
>Try also this no-brainer, and let know how/if it changes:
>
>echo 65535 > /proc/sys/vm/min_free_kbytes
>
>
>
did not help. Did the min_free_kbytes thing, mounted an iscsi XFS
filesystem, tried to remove some files, got this:

May 22 16:55:50 wbox kernel: XFS mounting filesystem sdd
May 22 16:55:50 wbox kernel: Ending clean XFS mount for filesystem: sdd
May 22 16:56:00 wbox kernel: Bad page state at free_hot_cold_page (in
process 'rm', page c14d9f20)
May 22 16:56:00 wbox kernel: flags:0x20000080 mapping:00000000
mapcount:0 count:0
May 22 16:56:00 wbox kernel: Backtrace:
May 22 16:56:00 wbox kernel: [<c0140725>] bad_page+0x75/0xb0
May 22 16:56:00 wbox kernel: [<c0140e96>] free_hot_cold_page+0x66/0x120
May 22 16:56:00 wbox kernel: [<c0441ef3>] tcp_in_window+0x303/0x510
May 22 16:56:00 wbox kernel: [<c03e5ec8>] skb_release_data+0x98/0xc0
May 22 16:56:00 wbox kernel: [<c03e5f00>] kfree_skbmem+0x10/0x30
May 22 16:56:00 wbox kernel: [<c0417bf9>] tcp_clean_rtx_queue+0x169/0x450
May 22 16:56:00 wbox kernel: [<c04185e8>] tcp_ack+0xe8/0x650
May 22 16:56:00 wbox kernel: [<c041b44f>] tcp_rcv_established+0x56f/0x890

Machine was pretty dead afterwards. I have not tried to ping the machine
unfortunately, so maybe the kernel
was still working. I'll try to ping next time before rebooting.
Before running the rm command, I had 600MB unused memory according to
top. Since the network card is an RTL8139,
filling up 600MB in 10 seconds is difficult, so I don't think it's
out-of-memory related, unless
it's a special kind of memory which is limited. The machine has 1GB of
RAM, so maybe that triggers something.

Harald

Harald Kubota

ungelesen,
22.05.2005, 04:18:4722.05.05
an open-...@googlegroups.com, linux-iscsi development team
After doing the very small bonnie++, I got this again:

May 22 17:09:42 wbox kernel: XFS mounting filesystem sdd
May 22 17:09:42 wbox kernel: Ending clean XFS mount for filesystem: sdd
May 22 17:10:34 wbox kernel: Bad page state at free_hot_cold_page (in
process 'bonnie++', page c1525120)
May 22 17:10:34 wbox kernel: flags:0x20000080 mapping:00000000
mapcount:0 count:0
May 22 17:10:34 wbox kernel: Backtrace:
May 22 17:10:34 wbox kernel: [<c0140725>] bad_page+0x75/0xb0
May 22 17:10:34 wbox kernel: [<c0140e96>] free_hot_cold_page+0x66/0x120
May 22 17:10:34 wbox kernel: [<c0441ef3>] tcp_in_window+0x303/0x510
May 22 17:10:34 wbox kernel: [<c03e5ec8>] skb_release_data+0x98/0xc0
May 22 17:10:34 wbox kernel: [<c03e5f00>] kfree_skbmem+0x10/0x30
May 22 17:10:34 wbox kernel: [<c0417bf9>] tcp_clean_rtx_queue+0x169/0x450
May 22 17:10:34 wbox kernel: [<c04185e8>] tcp_ack+0xe8/0x650
May 22 17:10:34 wbox kernel: [<c041b44f>] tcp_rcv_established+0x56f/0x890
May 22 17:10:34 wbox kernel: [<c0424580>] tcp_v4_do_rcv+0x110/0x130

The same disk via ext3 worked without any issues as expected. Using xfs
broke when bonnie++ tried
to create files.

As Mike stated that xfs has issued with nbd, I assume this is related
and I simply won't use xfs any more when using iSCSI on Linux.

>Machine was pretty dead afterwards. I have not tried to ping the machine
>unfortunately, so maybe the kernel
>was still working.
>
Got an answer to this one: it's dead. Dead like absolutely dead. No ping
possible.

Harald

Alex Aizman

ungelesen,
22.05.2005, 13:44:3022.05.05
an open-...@googlegroups.com, linux-iscsi development team, Harald Kubota
The motivation behind this was not to let XFS to take all the available
memory into page cache, so that it'd create an unhealthy interaction between
kswapd, OOM killer, and the rest.

So, there's something else..

>
> May 22 16:55:50 wbox kernel: XFS mounting filesystem sdd May
> 22 16:55:50 wbox kernel: Ending clean XFS mount for
> filesystem: sdd May 22 16:56:00 wbox kernel: Bad page state
> at free_hot_cold_page (in process 'rm', page c14d9f20) May 22
> 16:56:00 wbox kernel: flags:0x20000080 mapping:00000000
> mapcount:0 count:0 May 22 16:56:00 wbox kernel: Backtrace:
> May 22 16:56:00 wbox kernel: [<c0140725>] bad_page+0x75/0xb0
> May 22 16:56:00 wbox kernel: [<c0140e96>]
> free_hot_cold_page+0x66/0x120 May 22 16:56:00 wbox kernel:
> [<c0441ef3>] tcp_in_window+0x303/0x510 May 22 16:56:00 wbox
> kernel: [<c03e5ec8>] skb_release_data+0x98/0xc0 May 22
> 16:56:00 wbox kernel: [<c03e5f00>] kfree_skbmem+0x10/0x30
> May 22 16:56:00 wbox kernel: [<c0417bf9>]
> tcp_clean_rtx_queue+0x169/0x450 May 22 16:56:00 wbox kernel:
> [<c04185e8>] tcp_ack+0xe8/0x650 May 22 16:56:00 wbox kernel:
> [<c041b44f>] tcp_rcv_established+0x56f/0x890

In the first e-mail you mentioned running preemptible kernel. An obvious
next step - disable preemption at boot time (if possible), or run a kernel
compiled without preemptible patch.

Harald Kubota

ungelesen,
24.05.2005, 10:30:5024.05.05
an open-...@googlegroups.com, linux-iscsi development team
Alex Aizman wrote:

>In the first e-mail you mentioned running preemptible kernel. An obvious
>next step - disable preemption at boot time (if possible), or run a kernel
>compiled without preemptible patch.
>
>
>
I did not think it would make a difference, but I wouldn't be surprised
the first time...
but non-preemptibility makes no difference:

May 24 22:44:49 wbox kernel: Bad page state at free_hot_cold_page (in
process 'X', page c15174e0)
May 24 22:44:49 wbox kernel: flags:0x20000080 mapping:00000000
mapcount:0 count:0
May 24 22:44:49 wbox kernel: Backtrace:
May 24 22:44:49 wbox kernel: [<c0139195>] bad_page+0x75/0xb0
May 24 22:44:49 wbox kernel: [<c01398a6>] free_hot_cold_page+0x66/0xf0
May 24 22:44:49 wbox kernel: [<c0431bb1>] unix_poll+0xb1/0xc0
May 24 22:44:49 wbox kernel: [<c0165024>] poll_freewait+0x44/0x50
May 24 22:44:49 wbox kernel: [<c01654a8>] do_select+0x2a8/0x2c0
May 24 22:44:49 wbox kernel: [<c0165030>] __pollwait+0x0/0xd0
May 24 22:44:49 wbox kernel: [<c01657ed>] sys_select+0x2ed/0x420
May 24 22:44:49 wbox kernel: [<c048f06c>] schedule+0x30c/0x5e0
May 24 22:44:49 wbox kernel: [<c028f362>] copy_to_user+0x42/0x60
May 24 22:44:49 wbox kernel: [<c0103009>] syscall_call+0x7/0xb
May 24 22:44:49 wbox kernel: Trying to fix it up, but a reboot is needed

I tend to think Mike is right. I am just surprised I am the only one
with this problem.

For now I'll drop xfs from the list of useable file systems. ext3 and
reiserfs are ok too.

Harald

Christoph Hellwig

ungelesen,
30.05.2005, 16:54:1930.05.05
an ckno...@science.edu, open-...@googlegroups.com, linux-iscsi development team
On Mon, May 30, 2005 at 03:16:57PM -0500, Carlos Knowlton wrote:
> Meanwhile, is anyone aware of any development being done to fix the
> XFS-iSCSI issue?

First thing we need is an agreement whether the iscsi code is right
assuming that it never gets passed kmalloced pages or whether is XFS
is right in passing them. It's difficult because handling them makes
life much easier for XFS and not handling them makes life easier for
iscsi. Note that we might see more usage of s/g buffers of kmalloc'ed
pages once we get rid of the non-s/g I/O path in the scsi layer, so
personally I'd suggest iscsi should handle it (and yes, I'm biassed
;-)). To make that handling easier bios will probably need a flag
to tell that we're using kmalloc'ed memory.

Dmitry Yusupov

ungelesen,
31.05.2005, 11:31:3931.05.05
an open-...@googlegroups.com, ckno...@science.edu, linux-iscsi development team
On Mon, 2005-05-30 at 22:54 +0200, Christoph Hellwig wrote:
> On Mon, May 30, 2005 at 03:16:57PM -0500, Carlos Knowlton wrote:
> > Meanwhile, is anyone aware of any development being done to fix the
> > XFS-iSCSI issue?
>
> First thing we need is an agreement whether the iscsi code is right
> assuming that it never gets passed kmalloced pages or whether is XFS
> is right in passing them.

iscsi_tcp is just a pass-through layer between scsi-ml and tcp/ip stack.
we get pages as struct scatterlist * and than passing them down
one-by-one via tcp_sendpage() api. That means we never touch or modify
struct page* on our own. So, to make XFS work we need either modify XFS
code or TCP/IP code.

> It's difficult because handling them makes
> life much easier for XFS and not handling them makes life easier for
> iscsi. Note that we might see more usage of s/g buffers of kmalloc'ed
> pages once we get rid of the non-s/g I/O path in the scsi layer, so
> personally I'd suggest iscsi should handle it (and yes, I'm biassed
> ;-)). To make that handling easier bios will probably need a flag
> to tell that we're using kmalloc'ed memory.

ok.

another limitation currently is that we can not handle more than one
page on non-s/g I/O path. It works OK in most cases. Since we are moving
to eliminate all non-s/g I/O pathes, I wonder if could just leave this
code as is?

Christoph Hellwig

ungelesen,
31.05.2005, 12:21:4031.05.05
an open-...@googlegroups.com, ckno...@science.edu, linux-iscsi development team
On Tue, May 31, 2005 at 08:31:39AM -0700, Dmitry Yusupov wrote:
> iscsi_tcp is just a pass-through layer between scsi-ml and tcp/ip stack.
> we get pages as struct scatterlist * and than passing them down
> one-by-one via tcp_sendpage() api. That means we never touch or modify
> struct page* on our own. So, to make XFS work we need either modify XFS
> code or TCP/IP code.

Or do more than just a little pass-through. If networking people don't
want to handle non-refcounted pages in tcp_sendpage you need to switch
to tcp_sendmsg for those.

> ok.
>
> another limitation currently is that we can not handle more than one
> page on non-s/g I/O path.

That needs fixing as such requests can happen using ioctls easily.

Dmitry Yusupov

ungelesen,
31.05.2005, 22:03:4431.05.05
an open-...@googlegroups.com, ckno...@science.edu, linux-iscsi development team
fixed now. r363

Dmitry Yusupov

ungelesen,
02.06.2005, 16:01:5302.06.05
an open-...@googlegroups.com, ckno...@science.edu, Harald Kubota, Christoph Hellwig, linux-iscsi development team
On Tue, 2005-05-31 at 18:21 +0200, Christoph Hellwig wrote:
> On Tue, May 31, 2005 at 08:31:39AM -0700, Dmitry Yusupov wrote:
> > iscsi_tcp is just a pass-through layer between scsi-ml and tcp/ip stack.
> > we get pages as struct scatterlist * and than passing them down
> > one-by-one via tcp_sendpage() api. That means we never touch or modify
> > struct page* on our own. So, to make XFS work we need either modify XFS
> > code or TCP/IP code.
>
> Or do more than just a little pass-through. If networking people don't
> want to handle non-refcounted pages in tcp_sendpage you need to switch
> to tcp_sendmsg for those.

ok. this is fixed now. r364.

basically, we check for page_count() before decide what send method to
use. in our tests it passes XFS + bonnie++ combination which was failing
before as observed by Harald Kubota. Thanks! And give it a try!

Harald Kubota

ungelesen,
05.06.2005, 07:04:4305.06.05
an open-...@googlegroups.com, linux-iscsi development team
Dmitry Yusupov wrote:

>On Tue, 2005-05-31 at 18:21 +0200, Christoph Hellwig wrote:
>
>
>>On Tue, May 31, 2005 at 08:31:39AM -0700, Dmitry Yusupov wrote:
>>
>>
>>>iscsi_tcp is just a pass-through layer between scsi-ml and tcp/ip stack.
>>>we get pages as struct scatterlist * and than passing them down
>>>one-by-one via tcp_sendpage() api. That means we never touch or modify
>>>struct page* on our own. So, to make XFS work we need either modify XFS
>>>code or TCP/IP code.
>>>
>>>
>>Or do more than just a little pass-through. If networking people don't
>>want to handle non-refcounted pages in tcp_sendpage you need to switch
>>to tcp_sendmsg for those.
>>
>>
>
>ok. this is fixed now. r364.
>
>basically, we check for page_count() before decide what send method to
>use. in our tests it passes XFS + bonnie++ combination which was failing
>before as observed by Harald Kubota. Thanks! And give it a try!
>
>
Getting closer, but not there yet.

I used r366 on 2.6.12-rc5 (target is iet 0.4.9 on 2.6.11). No longer bad
crashed, but it's not working 100%.
On my 100mbit/s host I get kernel panics caused by the RTL8139 driver.
The machine dies at this point completely.
The output on the console suggests this is a network driver problem.

On my other machine with e1000 card (so far solid and fast) a bonnie++
run yields:

burner ~ # cat /root/xfs.out
burner iscsi # bonnie++ -s 2000 -u nobody -d t
Using uid:65534, gid:65534.
Writing a byte at a time...done
Writing intelligently...
done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...I/O error in filesystem ("sde") meta-data
dev sde block 0xc042c0 ("xlog_iodone") error 5 buf count 27648
Filesystem "sde": Log I/O Error Detected. Shutting down filesystem: sde
Please umount the filesystem, and rectify the problem(s)
I/O error in filesystem ("sde") meta-data dev sde block 0xc042f6
("xlog_iodone") error 5 buf count 32768
I/O error in filesystem ("sde") meta-data dev sde block 0xc04336
("xlog_iodone") error 5 buf count 32768
I/O error in filesystem ("sde") meta-data dev sde block 0xc04376
("xlog_iodone") error 5 buf count 10240
Can't delete file U0009515
Cleaning up test directory after error.
Bonnie: drastic I/O error (rmdir): Input/output error

I attached the syslog output as it's too long and I am not sure what
might be helpful or not.
Here's a short excerpt:

Jun 5 19:44:48 burner kernel: iscsi1: detected conn error (1011)
Jun 5 19:44:48 burner iscsid: connected local port 42059 to
192.168.11.60:3260
Jun 5 19:44:48 burner iscsid: deleting a scheduled/waiting thread!
Jun 5 19:44:48 burner iscsid: bound iSCSI connection (handle
0x0xf4be7a00) to session (handle 0x0xf702ee04)
Jun 5 19:44:48 burner iscsid: sending login PDU with current stage 0,
next stage 1, transit 0x80, isid 0x00023d000000
Jun 5 19:44:48 burner iscsid: >
InitiatorName=iqn.2004-12.org.athome.burner:01.c633334712
Jun 5 19:44:48 burner iscsid: > InitiatorAlias=burner
Jun 5 19:44:48 burner iscsid: >
TargetName=iqn.2004-12.org.athome.s800:s800.test
Jun 5 19:44:48 burner iscsid: > SessionType=Normal
Jun 5 19:44:48 burner iscsid: > AuthMethod=CHAP,None
Jun 5 19:44:48 burner iscsid: send PDU began for hdr 48 bytes and data
172 bytes
Jun 5 19:44:48 burner iscsid: wrote 48 bytes of PDU header
Jun 5 19:44:48 burner iscsid: wrote 172 bytes of PDU data
Jun 5 19:44:48 burner iscsid: send PDU finished for conn (handle
0xf4be7a00)
Jun 5 19:44:48 burner iscsid: detected poll event 1
Jun 5 19:44:48 burner iscsid: message real length is 72 bytes,
recv_handle 0x806fa10
Jun 5 19:44:48 burner iscsid: detected iSCSI connection (handle
0xf4be7a00) error (1011) state (2)
Jun 5 19:44:48 burner iscsid: deleting a scheduled/waiting thread!
Jun 5 19:44:48 burner iscsid: re-opening session 1 (reopen_cnt 1)
Jun 5 19:44:48 burner iscsid: connection 0x0xf4be7a00 is stopped for
recovery
Jun 5 19:44:48 burner iscsid: disconnecting conn 0xb7d654b4, fd 7
Jun 5 19:44:48 burner iscsid: flushing per-connection events
Jun 5 19:44:48 burner iscsid: set TCP recv window size to 524288,
actually got 262142
Jun 5 19:44:48 burner iscsid: set TCP send window size to 524288,
actually got 262142
Jun 5 19:44:48 burner iscsid: connecting to 192.168.11.60:3260 ip_length 4

Any ideas wha causes this? I can make a ethereal dump if that helps.

Harald


syslog.xfs.bz2

Harald Kubota

ungelesen,
05.06.2005, 09:43:3205.06.05
an open-...@googlegroups.com, linux-iscsi development team
Harald Kubota wrote:

>Getting closer, but not there yet.
>
>
>
Before someone suggest to test this particular hardware/software
combination using non-XFS:
ext2, ext3 and reiserfs worked well. And jfs too. I think it's the first
time I ever used this :-)
After testing all, I tried xfs again, only to get the same messages.

Harald

Dmitry Yusupov

ungelesen,
05.06.2005, 11:23:5305.06.05
an open-...@googlegroups.com, linux-iscsi development team
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Thanks for the test!!!

I think now it points to interoperability problem. (I assume target is
not died and when it happen, network pings) Which is not necessary XFS
related. XFS just triggered it. I tried to reproduce it on my side, but
no luck. To debug this one, we definitely need ethereal dump on
Initiator side.
Allen antworten
Dem Autor antworten
Weiterleiten
0 neue Nachrichten