6.0BETA3 panic in ip_output (vlan/RIP related?)
Gavin Atkinson
Hi,
I've just managed to panic an amd64 machine running 6.0BETA3.
wiggum# ifconfig vlan76 destroy
wiggum# Aug 31 12:02:48 wiggum routed[244]: IP_DROP_MEMBERSHIP ALLHOSTS: Can't assign requested address
wiggum#
wiggum# ifconfig vlan76 create
wiggum# ifconfig vlan76 vlan 76 vlandev bge0
wiggum# ifconfig vlan76 inet x.y.76.59 netmask 255.255.254.0
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x8:0xffffffff80429420
stack pointer = 0x10:0xffffffffb260b600
frame pointer = 0x10:0xffffffffb260b710
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 244 (routed)
[thread pid 244 tid 100077 ]
Stopped at strlen: cmpb $0,0(%rdi)
db> tr
Tracing pid 244 tid 100077 td 0xffffff0078c74980
strlen() at strlen
vsnprintf() at vsnprintf+0x2e
panic() at panic+0x14b
_mtx_lock_flags() at _mtx_lock_flags+0xd6
ip_output() at ip_output+0x692
rip_output() at rip_output+0x161
rip_send() at rip_send+0x65
sosend() at sosend+0x654
kern_sendit() at kern_sendit+0x104
sendit() at sendit+0x66
sendto() at sendto+0x54
syscall() at syscall+0x4b2
Xfast_syscall() at Xfast_syscall+0xa8
--- syscall (133, FreeBSD ELF64, sendto), rip = 0x800799dfc, rsp =
0x7fffffffeb28, rbp = 0x413112 ---
db>
(kgdb) where
#23 0xffffffff803d3d5e in vsnprintf (str=0x0, size=0, format=0x0, ap=0x0) at /usr/src/sys/kern/subr_prf.c:408
#24 0xffffffff803b3efb in panic (fmt=0xffffffff80615639 "%s @ %s:%d") at /usr/src/sys/kern/kern_shutdown.c:520
#25 0xffffffff803ab6e6 in _mtx_lock_flags (m=0xffffff00622dec78, opts=0, file=0xffffffff80628080 "/usr/src/sys/netinet/ip_output.c",
line=296) at /usr/src/sys/kern/kern_mutex.c:268
#26 0xffffffff80464a52 in ip_output (m=0xffffff005e402300, opt=0xffffff0042432000, ro=0xffffffffb260b8d0, flags=32, imo=0xffffff007b8aa500,
inp=0xffffff0061bf2000) at /usr/src/sys/netinet/ip_output.c:296
#27 0xffffffff80465791 in rip_output (m=0xffffff005e402300, so=0x0, dst=64) at /usr/src/sys/netinet/raw_ip.c:320
#28 0xffffffff80466535 in rip_send (so=0xffffff0061ccf000, flags=0, m=0xffffff005e402300, nam=0xffffff007b5b90f0, control=0x0, td=0x0)
at /usr/src/sys/netinet/raw_ip.c:785
#29 0xffffffff803f95c4 in sosend (so=0xffffff0061ccf000, addr=0xffffff007b5b90f0, uio=0xffffffffb260ba80, top=0xffffff005e402300,
control=0x0, flags=0, td=0xffffff0078c74980) at /usr/src/sys/kern/uipc_socket.c:829
#30 0xffffffff80400534 in kern_sendit (td=0xffffff0078c74980, s=5, mp=0xffffffffb260bb50, flags=0, control=0x0, segflg=8)
at /usr/src/sys/kern/uipc_syscalls.c:772
#31 0xffffffff804016f6 in sendit (td=0xffffff0078c74980, s=5, mp=0xffffffffb260bb50, flags=0) at /usr/src/sys/kern/uipc_syscalls.c:712
#32 0xffffffff80401ab4 in sendto (td=0x0, uap=0x0) at /usr/src/sys/kern/uipc_syscalls.c:830
#33 0xffffffff80570042 in syscall (frame=
{tf_rdi = 5, tf_rsi = 140737488350080, tf_rdx = 8, tf_rcx = 0, tf_r8 = 140737488350016, tf_r9 = 16, tf_rax = 133, tf_rbx = 5367808, tf)
at /usr/src/sys/amd64/amd64/trap.c:796
#34 0xffffffff8055d468 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:272
(kgdb) f 25
#25 0xffffffff803ab6e6 in _mtx_lock_flags (m=0xffffff00622dec78, opts=0, file=0xffffffff80628080 "/usr/src/sys/netinet/ip_output.c",
line=296) at /usr/src/sys/kern/kern_mutex.c:268
268 KASSERT(m->mtx_object.lo_class == &lock_class_mtx_sleep,
(kgdb) l
263 void
264 _mtx_lock_flags(struct mtx *m, int opts, const char *file, int line)
265 {
266
267 MPASS(curthread != NULL);
268 KASSERT(m->mtx_object.lo_class == &lock_class_mtx_sleep,
269 ("mtx_lock() of spin mutex %s @ %s:%d", m->mtx_object.lo_name,
270 file, line));
271 WITNESS_CHECKORDER(&m->mtx_object, opts | LOP_NEWORDER | LOP_EXCLUSIVE,
272 file, line);
(kgdb) up
#26 0xffffffff80464a52 in ip_output (m=0xffffff005e402300, opt=0xffffff0042432000, ro=0xffffffffb260b8d0, flags=32, imo=0xffffff007b8aa500,
inp=0xffffff0061bf2000) at /usr/src/sys/netinet/ip_output.c:296
296 IN_LOOKUP_MULTI(ip->ip_dst, ifp, inm);
(kgdb) l
291 if (ia != NULL)
292 ip->ip_src = IA_SIN(ia)->sin_addr;
293 }
294
295 IN_MULTI_LOCK();
296 IN_LOOKUP_MULTI(ip->ip_dst, ifp, inm);
297 if (inm != NULL &&
298 (imo == NULL || imo->imo_multicast_loop)) {
299 IN_MULTI_UNLOCK();
300 /*
I've got the core file if anyone wants any more info.
Gavin
Yar Tikhiy
>
> I've just managed to panic an amd64 machine running 6.0BETA3.
>
> wiggum# ifconfig vlan76 destroy
> wiggum# Aug 31 12:02:48 wiggum routed[244]: IP_DROP_MEMBERSHIP ALLHOSTS: Can't assign requested address
> wiggum#
> wiggum# ifconfig vlan76 create
> wiggum# ifconfig vlan76 vlan 76 vlandev bge0
> wiggum# ifconfig vlan76 inet x.y.76.59 netmask 255.255.254.0
>
>
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 0; apic id = 00
> instruction pointer = 0x8:0xffffffff80429420
> stack pointer = 0x10:0xffffffffb260b600
> frame pointer = 0x10:0xffffffffb260b710
> code segment = base 0x0, limit 0xfffff, type 0x1b
> = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags = interrupt enabled, resume, IOPL = 0
> current process = 244 (routed)
Thanks for reporting this! The problem seems known and has
to do with a deficiency in our multicast code WRT interface
removal/re-insertion. It is on the to-do list of our networking
gurus and hopefully will be dealt with RSN, after IP multicast
code locking and cleanup are complete.
--
Yar
Yar Tikhiy
>
> Thanks for reporting this! The problem seems known and has
> to do with a deficiency in our multicast code WRT interface
> removal/re-insertion. It is on the to-do list of our networking
> gurus and hopefully will be dealt with RSN, after IP multicast
> code locking and cleanup are complete.
BTW, there is a number of PR's related to this issue.
Here are their numbers for reference:
kern/77665
kern/78227
kern/82882
--
Yar
Gavin Atkinson
It's good to hear that the problem is understood, however it seems that
this panic is trivial to recreate for anyone running routed, and
therefore 6.0-RELEASE may well be unusable for me. I've just got this
second panic from the same machine which looks like it may also be
related to the multicast code.
wiggum# reboot
Sep 1 18:05:05
wiggum reboot: r
Faooted by root
Sep 1 18:05:05 wiggum syslogd: exiting on signal 15
tal trap 9: general protection fault while in kernel mode
cpuid = 1; apic id = 01
instruction pointer = 0x8:0xffffffff80429420
stack pointer = 0x10:0xffffffffb49c4590
frame pointer = 0x10:0xffffffffb49c46a0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 244 (routed)
[thread pid 244 tid 100101 ]
Stopped at strlen: cmpb $0,0(%rdi)
db> tr
Tracing pid 244 tid 100101 td 0xffffff0061d1b980
strlen() at strlen
vsnprintf() at vsnprintf+0x2e
panic() at panic+0x14b
_mtx_lock_flags() at _mtx_lock_flags+0xd6
if_delmulti() at if_delmulti+0x3f
in_delmulti() at in_delmulti+0x4b
ip_freemoptions() at ip_freemoptions+0x30
in_pcbdetach() at in_pcbdetach+0x182
udp_detach() at udp_detach+0x51
soclose() at soclose+0x1a0
soo_close() at soo_close+0x3f
fdrop_locked() at fdrop_locked+0xa1
closef() at closef+0x31
fdfree() at fdfree+0x48a
exit1() at exit1+0x302
sys_exit() at sys_exit+0xe
syscall() at syscall+0x4b2
Xfast_syscall() at Xfast_syscall+0xa8
Given I've accidentally been able to panic this machine twice in two
days, is there any chance a fix or at least a band-aid will be in place
before the release?
Thanks,
Gavin
Yar Tikhiy
It's a shame, but I can't tell for sure. The problem in the IP
multicast code is really easy to hit, but it will be rather hard
to fix it without major changes to the code. The workaround is to
avoid destroying interfaces with multicast groups still joined on
them. Both panics you saw seemed to result from the code trying
to use memory structures of a now-deceased interface.
--
Yar
Robert Watson
>> Thanks for reporting this! The problem seems known and has to do with
>> a deficiency in our multicast code WRT interface removal/re-insertion.
>> It is on the to-do list of our networking gurus and hopefully will be
>> dealt with RSN, after IP multicast code locking and cleanup are
>> complete.
>
> It's good to hear that the problem is understood, however it seems that
> this panic is trivial to recreate for anyone running routed, and
> therefore 6.0-RELEASE may well be unusable for me. I've just got this
> second panic from the same machine which looks like it may also be
> related to the multicast code.
I believe I've chatted with Gleb about this some, but want to confirm that
I understand the problem here: this occurs when an interface is removed
while IP multicast membership is still present for multicast groups on the
interface. When the multicast socket is closed, then the kernel panics
because it has a now invalid cached pointer to the interface structure
(now freed), which cases an assertion failure because the mutex code
detects that it is operating on an invalid mutex.
So it sounds like we need to figure out how the multicast code should
behave on interface removal -- I wonder what other operating systems do
here? Do they simply invalidate current membership related with the
interface, or do they leave the multicast sockets in a state such that if
the interface comes back, the memberships are re-bound?
Robert N M Watson
Gavin Atkinson
> On Thu, 1 Sep 2005, Gavin Atkinson wrote:
>>> Thanks for reporting this! The problem seems known and has to do with a
>>> deficiency in our multicast code WRT interface removal/re-insertion. It is
>>> on the to-do list of our networking gurus and hopefully will be dealt with
>>> RSN, after IP multicast code locking and cleanup are complete.
>>
>> It's good to hear that the problem is understood, however it seems that
>> this panic is trivial to recreate for anyone running routed, and therefore
>> 6.0-RELEASE may well be unusable for me. I've just got this second panic
>> from the same machine which looks like it may also be related to the
>> multicast code.
>
> I believe I've chatted with Gleb about this some, but want to confirm that I
> understand the problem here: this occurs when an interface is removed while
> IP multicast membership is still present for multicast groups on the
> interface. When the multicast socket is closed, then the kernel panics
> because it has a now invalid cached pointer to the interface structure (now
> freed), which cases an assertion failure because the mutex code detects that
> it is operating on an invalid mutex.
I don't explicitely use multicast anywhere in my setup. I run routed and
get my default gateway via RIP, which seems to involve the multicast code.
I've followed the code through and as far as I can tell, your belief is
correct (at least for the second panic). The first panic occured when
creating an interface, but I suspect that was more because routed noticed
that interfaces had changed and attempted to send packets out, and fell
over when it stumbled over the invalid mutex in the destroyed vlan.
I have coredumps from both the panics if they will help at all, although I
suspect it'll be just as easy for you to recreate it at your end - run
routed without arguments and do something with interfaces - delete and
recreate vlan interfaces seem the easiest way, although I guess inserting
or removing a cardbus card would also be bad.
> So it sounds like we need to figure out how the multicast code should behave
> on interface removal -- I wonder what other operating systems do here? Do
> they simply invalidate current membership related with the interface, or do
> they leave the multicast sockets in a state such that if the interface comes
> back, the memberships are re-bound?
I can't help here, I'm afraid.
Gavin
Robert Watson
On Sat, 3 Sep 2005, Robert Watson wrote:
> I believe I've chatted with Gleb about this some, but want to confirm
> that I understand the problem here: this occurs when an interface is
> removed while IP multicast membership is still present for multicast
> groups on the interface. When the multicast socket is closed, then the
> kernel panics because it has a now invalid cached pointer to the
> interface structure (now freed), which cases an assertion failure
> because the mutex code detects that it is operating on an invalid mutex.
>
> So it sounds like we need to figure out how the multicast code should
> behave on interface removal -- I wonder what other operating systems do
> here? Do they simply invalidate current membership related with the
> interface, or do they leave the multicast sockets in a state such that
> if the interface comes back, the memberships are re-bound?
I've now committed a regression test for this bug:
src/tools/regression/netinet/msocket_ifnet_remove
Which basically simulates the removal of an interface while in use for
multicast, resulting in a similar panic to the one of the ones you've
reported. An if_disc discard interface is used. It tests both raw and
UDP socket variants, and should panic 6.x and 7.x boxes; it may panic 4.x
and 5.x, but may just corrupt kernel memory silently.
I believe the solution for now is that on ifnet tear-down, we will need to
walk the various pcb lists and trim references to the multicast address.
I chatted a little with Bill Fenner today about what the application
semantics should be, and likely we need to substantially change the way
IPv4 and IPv6 multicast handle group membership for sockets in order to
get the "right" behavior, so a panic work-around for 6.0 is the right
thing to do, even though it won't be the final answer.
I should have an opportunity to look into a possible solution for this in
the next few days.
Robert N M Watson
Yar Tikhiy
>
> I believe I've chatted with Gleb about this some, but want to confirm that
> I understand the problem here: this occurs when an interface is removed
> while IP multicast membership is still present for multicast groups on the
> interface. When the multicast socket is closed, then the kernel panics
> because it has a now invalid cached pointer to the interface structure
> (now freed), which cases an assertion failure because the mutex code
> detects that it is operating on an invalid mutex.
I have exactly the same notion of the issue.
> So it sounds like we need to figure out how the multicast code should
> behave on interface removal -- I wonder what other operating systems do
> here? Do they simply invalidate current membership related with the
> interface, or do they leave the multicast sockets in a state such that if
> the interface comes back, the memberships are re-bound?
The idea of keeping such "orphaned" multicast sockets around seems
to come from the way we deal with ordinary, unicast, sockets, whose
local address has been deleted: such sockets just wait for the
address to be re-assigned to some interface in the system (a different
one, perhaps.)
However, multicast group membership is bound to a specific interface
by design. Therefore I fail to see how we can revive it on a
different, newly created, interface even if it may have the same
name and type as the old one used to have. E.g., a dial-up user
disconnects, ppp0 is destroyed; is ppp0 to appear when another user
connects the same as the old instance of ppp0? And here is the
opposite case: you replace a PCMCIA Ethernet card in your notebook
with another one of a different hardware type; how can the system
tell that the old and new interfaces are virtually the same? I'm
afraid that the system should lose its multicast group membership
on a destroyed interface because there will be no such interface
in the system again.
In the case of routing daemons, which are among the major consumers
of IP multicast, such a daemon will notice this or some other
interface re-appearing and join groups it needs on it again. Other
multicast software can just fail upon its interface departure, or
wait until an interface with an IP address known to the software
re-appears. I'm afraid we cannot hide this in the kernel because,
in general, an IP address alone is insufficient to determine an
interface to join a multicast group on in the presence of "unnumbered"
interfaces.
--
Yar
Robert Watson
On Tue, 6 Sep 2005, Mark Tinguely wrote:
>> So it sounds like we need to figure out how the multicast code should
>> behave on interface removal -- I wonder what other operating systems do
>> here? Do they simply invalidate current membership related with the
>> interface, or do they leave the multicast sockets in a state such that
>> if the interface comes back, the memberships are re-bound?
>
> In the case of a non-local multicast sessions, the saved multicast
> socket state would need to keep a timestamp of the last a multicast
> router IGMP session probe to detect the possibility of session pruning.
I was assuming that, at the very least, it would be necessary to issue a
new IGMP join when binding the socket to a new interface...
Robert N M Watson
Mark Tinguely
> So it sounds like we need to figure out how the multicast code should
> behave on interface removal -- I wonder what other operating systems do
> here? Do they simply invalidate current membership related with the
> interface, or do they leave the multicast sockets in a state such that if
> the interface comes back, the memberships are re-bound?
In the case of a non-local multicast sessions, the saved multicast socket
state would need to keep a timestamp of the last a multicast router IGMP
session probe to detect the possibility of session pruning.
--Mark Tinguely.