XEN3_DOM0 5.99.55 on grinds to a halt shortly after boot (repeatedly)
Johan Ihrén
This is 5.99.55 from about July 20th/21st.
Let me start with the observation that this is hardware related, as the same exact 5.99.55 disk works like a charm in an older Core 2 Quad machine. But when I boot from this disk in a machine with a brand new Core i7 + H67 chipset it works "less well".
If I boot the standard XEN3_DOM0 kernel I typically get to the login prompt and usually manage to login and give one or two commands before the machine basically stops. When I break into DDB it looks like this (typed by hand, no serial console):
Stopped in pid 0.2 (system)
breakpoint()
wskbd_translate{}
wskbd_input()
pckbd_input()
pckbcintr()
evtchn_do_event()
call_evtchn_do_event()
hypervisor_callback()
idle_loop()
The interesting thing is that this doesn't look bad to me. This seems to be mostly identical to what I'd see if I break into DDB on a machine that's working just fine. So what is it doing? I don't know.
Some additional information:
1. If I boot GENERIC it doesn't hang, only XEN3_DOM0 hangs. Furthermore, when GENERIC is idling, no services, nothing it is still seeing ~10% interrupts on CPU0. That seems to be an awful lot...
2. XEN3_DOM0 grinds to a halt also in single user, although it may take slightly longer to get there.
3. "Grinds to a halt" is not the same as "hangs". There is something going on, it is just that it has to be measured in geological time units.
* Once it happened during fsck and I decided to leave it to it. fsck of a 30GB filesystem took several hours (but did complete).
* I typed "root<RETURN>" at the login prompt when the machine had gone catatonic and nothing happened. An hour later there was still no change. But the next morning there was the expected "Password:" prompt on the console ;-)
Suggestions for what to try next would be much appreciated.
Regards,
Johan Ihrén
--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-...@muc.de
Jean-Yves Migeon
> This is 5.99.55 from about July 20th/21st.
>
> Let me start with the observation that this is hardware related, as
> the same exact 5.99.55 disk works like a charm in an older Core 2 Quad
> machine. But when I boot from this disk in a machine with a brand new
> Core i7 + H67 chipset it works "less well".
>
> If I boot the standard XEN3_DOM0 kernel I typically get to the login
> prompt and usually manage to login and give one or two commands before
> the machine basically stops. When I break into DDB it looks like this
> (typed by hand, no serial console):
>
> Stopped in pid 0.2 (system)
> breakpoint()
> wskbd_translate{}
> wskbd_input()
> pckbd_input()
> pckbcintr()
> evtchn_do_event()
> call_evtchn_do_event()
> hypervisor_callback()
> idle_loop()
>
> The interesting thing is that this doesn't look bad to me. This seems
> to be mostly identical to what I'd see if I break into DDB on a
> machine that's working just fine. So what is it doing? I don't know.
You are correct.
> 1. If I boot GENERIC it doesn't hang, only XEN3_DOM0 hangs.
> Furthermore, when GENERIC is idling, no services, nothing it is still
> seeing ~10% interrupts on CPU0. That seems to be an awful lot...
>
> 2. XEN3_DOM0 grinds to a halt also in single user, although it may
> take slightly longer to get there.
>
> 3. "Grinds to a halt" is not the same as "hangs". There is something
> going on, it is just that it has to be measured in geological time
> units.
It's very likely to be interrupt related, yes.
> * Once it happened during fsck and I decided to leave it to it. fsck
> of a 30GB filesystem took several hours (but did complete).
>
> * I typed "root<RETURN>" at the login prompt when the machine had
> gone catatonic and nothing happened. An hour later there was still no
> change. But the next morning there was the expected "Password:"
> prompt on the console ;-)
>
> Suggestions for what to try next would be much appreciated.
At ddb(4) prompt, try "show event". You may also try "show event" first,
then "continue", then break again in ddb and show event again.
If there's an event counter ridiculously high (rate or total count), try
looking to which interrupt line it correspond to via "dmesg" (also
through ddb prompt).
--
Jean-Yves Migeon
jeanyve...@free.fr
Johan Ihrén
On Jul 30, 2011, at 00:07 , Jean-Yves Migeon wrote:
> On 29.07.2011 23:28, Johan Ihrén wrote:
>> This is 5.99.55 from about July 20th/21st.
>>
>> 1. If I boot GENERIC it doesn't hang, only XEN3_DOM0 hangs.
>> Furthermore, when GENERIC is idling, no services, nothing it is still
>> seeing ~10% interrupts on CPU0. That seems to be an awful lot...
>>
>> 2. XEN3_DOM0 grinds to a halt also in single user, although it may
>> take slightly longer to get there.
>>
>> 3. "Grinds to a halt" is not the same as "hangs". There is something
>> going on, it is just that it has to be measured in geological time
>> units.
>
> It's very likely to be interrupt related, yes.
>
>> * Once it happened during fsck and I decided to leave it to it. fsck
>> of a 30GB filesystem took several hours (but did complete).
>>
>> * I typed "root<RETURN>" at the login prompt when the machine had
>> gone catatonic and nothing happened. An hour later there was still no
>> change. But the next morning there was the expected "Password:"
>> prompt on the console ;-)
>>
>> Suggestions for what to try next would be much appreciated.
>
> At ddb(4) prompt, try "show event". You may also try "show event" first,
> then "continue", then break again in ddb and show event again.
All counters are in the tens or hundreds except for two:
event type 1: vcpu0 ioapic0 pin 20 = 708325251
event type 1: vcpu0 clock = 814442
I suspect that at least the first one qualifies as "ridiculously high" ;-)
> If there's an event counter ridiculously high (rate or total count), try
> looking to which interrupt line it correspond to via "dmesg" (also
> through ddb prompt).
I'm not entirely sure what to look for here, but looking for ioapic "pins" I notice that a few pins (16-19) attach to various PCI devices. "Pin 20" is not one of them.
However, pin 20 is mentioned later on:
pciide0: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
...
pciide1: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
Regards,
Johan
PS. When booting GENERIC to compare then I find this in dmesg:
pciide0: using ioapic0 pin 20 for native-PCI interrupt
...
pciide1: using ioapic0 pin 20 for native-PCI interrupt
I.e. the "event channel 7" part is unique for the XEN3_DOM0 kernel.
Jean-Yves Migeon
>> At ddb(4) prompt, try "show event". You may also try "show event" first,
>> then "continue", then break again in ddb and show event again.
>
> All counters are in the tens or hundreds except for two:
>
> event type 1: vcpu0 ioapic0 pin 20 = 708325251
> event type 1: vcpu0 clock = 814442
>
> I suspect that at least the first one qualifies as "ridiculously high" ;-)
Yep, the pin 20 is way too high.
> I'm not entirely sure what to look for here, but looking for ioapic
> "pins" I notice that a few pins (16-19) attach to various PCI devices.
> "Pin 20" is not one of them.
>
> However, pin 20 is mentioned later on:
> pciide0: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
> ...
> pciide1: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
I suppose that you have a harddrive connected to pciide0 (somewhere in
dmesg you probably have an "atabus0 at pciide0").
H67 is likely to have an AHCI mode -- could you try that one out,
please? It's generally configured in BIOS.
> PS. When booting GENERIC to compare then I find this in dmesg:
>
> pciide0: using ioapic0 pin 20 for native-PCI interrupt
> ...
> pciide1: using ioapic0 pin 20 for native-PCI interrupt
>
> I.e. the "event channel 7" part is unique for the XEN3_DOM0 kernel.
Yes; Xen virtualizes interrupts, and event channel are their
counterpart. Channels are not specific to virtual devices, but also real
ones.
--
Jean-Yves Migeon
jeanyve...@free.fr
Johan Ihrén
On Jul 30, 2011, at 12:55 , Jean-Yves Migeon wrote:
> On 30.07.2011 01:42, Johan Ihrén wrote:
>>> At ddb(4) prompt, try "show event". You may also try "show event" first,
>>> then "continue", then break again in ddb and show event again.
>>
>> All counters are in the tens or hundreds except for two:
>>
>> event type 1: vcpu0 ioapic0 pin 20 = 708325251
>> event type 1: vcpu0 clock = 814442
>>
>> I suspect that at least the first one qualifies as "ridiculously high" ;-)
>
> Yep, the pin 20 is way too high.
>
>> I'm not entirely sure what to look for here, but looking for ioapic
>> "pins" I notice that a few pins (16-19) attach to various PCI devices.
>> "Pin 20" is not one of them.
>>
>> However, pin 20 is mentioned later on:
>> pciide0: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
>> ...
>> pciide1: using ioapic0 pin 20, event channel 7 for native-PCI interrupt
>
> I suppose that you have a harddrive connected to pciide0 (somewhere in
> dmesg you probably have an "atabus0 at pciide0").
>
> H67 is likely to have an AHCI mode -- could you try that one out,
> please? It's generally configured in BIOS.
AHCI seems to work like a charm (only nuisance was that in AHCI-mode the disks were numbered differently). Subjective speed up about a gazillion times and now this box performs I expected it to (right now quite happily running 28 DOMUs striped across 3 SSDs...).
But pciide should of course also work so I guess there's still a bug in there somewhere? I'd be more than happy to help with further testing if anyone wants to look at that (except next week when I'm offline for vacation). This box is intended for testing.
Jean-Yves: Many, many thanks for your quick help.
Regards,
Johan
Jean-Yves Migeon
> AHCI seems to work like a charm (only nuisance was that in AHCI-mode
> the disks were numbered differently). Subjective speed up about a
> gazillion times and now this box performs I expected it to (right now
> quite happily running 28 DOMUs striped across 3 SSDs...).
>
> But pciide should of course also work so I guess there's still a bug
> in there somewhere?
Yes, there is one, but it's difficult to say where it lies. Interrupt
storms causes range from bad configuration and driver bugs to pure
hardware errors, and it's difficult to say where to look: at least three
parties are involved here, from chipset to hypervisor and dom0 kernel.
All interrupts are typically ACK by hypervisor, which schedules it for
later handling by dom0 on its event channel. I don't know why it keeps
spamming/signaling the dom0 though; it could also depend on a Xen
revision that includes a quirk fix for the chipset, or that pciide(4)
improperly configures the hardware (or need a fix specific to the H67).
> I'd be more than happy to help with further
> testing if anyone wants to look at that (except next week when I'm
> offline for vacation). This box is intended for testing.
Well, I am about to be offline for vacation for two weeks too, so good
timing :)
> Jean-Yves: Many, many thanks for your quick help.
You're welcome.
--
Jean-Yves Migeon
jeanyve...@free.fr