Why Linux kernel forced to enter X2APIC mode( just because of booting cpu has supported x2apic) without depending on BIOS' setting in MSR->x2apic enablement bit ?

[Zhang, Lin-Bao (Linux Kernel R&D)]

Hi Suresh and other guys ,

In 3.4.4/3.6.6 ,I found a x2apic issue . if I am wrong , sorry first , and welcome your correction . thanks for your forwarding other maintainers.
I am testing a server , its BIOS is like this:
a) If BIOS think the system is of x2apic , it will set x2apic enablement bit in MSR and create x2apic ACPI tables and pass control to OS with x2apic mode
b) If BIOS feel the system doesn't meet x2apic conditions , it will not set x2apic enablement bit in MSR ,and pass control to OS with xapic mode.

It seems that MSR should be interface to OS to use. OS should leverage this, but it seems not.
In linux kernel source code ,
We see:
void check_x2apic(void)
{
if (x2apic_enabled()) { // this depends on 2 conditions : cpu has supported x2apic or not, bios setting MSR bit or not .
pr_info("x2apic enabled by BIOS, switching to x2apic ops\n");
x2apic_preenabled = x2apic_mode = 1;
}
It seems that Linux kernel will follow BIOS' result in early booting.

But I found that in other source : enable_IR_x2apic() , linux kernel obviously discard BIOS' result like this:

#define x2apic_supported() (cpu_has_x2apic) // just depends on if booting CPU has supported x2apic
Enable_IR_x2apic {
........
x2apic_enabled = 1;
if (x2apic_supported() && !x2apic_mode) { //x2apic_mode depends on BIOS's MSR.
x2apic_mode = 1;
enable_x2apic(); // OS will write MSR->x2apic enablement bit and print
pr_info("Enabled x2apic\n");
}
......
}

void enable_x2apic(void)
{
u64 msr;

rdmsrl(MSR_IA32_APICBASE, msr);
if (x2apic_disabled) {
__disable_x2apic(msr);
return;
}

if (!x2apic_mode)
return;

if (!(msr & X2APIC_ENABLE)) {
printk_once(KERN_INFO "Enabling x2apic\n");
// linux kernel will write MSR->x2apic_enable ,work around BIOS , this is reasonable ?
wrmsrl(MSR_IA32_APICBASE, msr | X2APIC_ENABLE);
}
}
I am very surprised for this ,why BIOS has claimed that BIOS didn't support x2apic , but Linux kernel will force enabling x2apic
(although booting CPU has supported x2apic) ?
I have one machine , in booting log ,
There is no " x2apic enabled by BIOS, switching to x2apic ops "
But later , linux kernel will print
" [ 0.464649] parse_iosapics: ecap f0207e
[ 0.469728] IOAPIC id 8 under DRHD base 0xa8000000 IU 0
[ 0.576731] IOAPIC id 0 under DRHD base 0xa8000000 IOMMU 0
[ 0.584065] parse_io_apic: io_apic 2 nr 2
[ 0.589590] Enabled IRQ remapping in x2apic mode
[ 0.595663] Enabling x2apic
[ 0.599337] Enabled x2apic
[ 0.603007] Switched APIC routing to cluster x2apic." ( I can add x2apic_phys to force using physical mode).


1, I quite worry about this , try to think , BIOS didn't enable x2apic , it just did enable xAPIC mode ,
it didn't create ACPI tables of x2apic. In this case, how Linux kernel can run normally in x2apic mode ?
in fact, in this machine , Linux kernel didn't run into some issues ,but I still think this is a potential defect.
2, why Linux kernel first depends on MSR and later will discard MSR ? is there any history ?
Linux kernel wants to completely discard MSR checking in future ?
3, anybody test machines which support x2apic ? do you have similar problems ?
in these machines , will you check MSR bit ?


-- Bob(LinBao Zhang)
HP linux kernel enginner


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[Yinghai Lu]

On Tue, Dec 18, 2012 at 1:50 AM, Zhang, Lin-Bao (Linux Kernel R&D)
<linbao...@hp.com> wrote:
> Hi Suresh and other guys ,
>
> In 3.4.4/3.6.6 ,I found a x2apic issue . if I am wrong , sorry first , and welcome your correction . thanks for your forwarding other maintainers.
> I am testing a server , its BIOS is like this:
> a) If BIOS think the system is of x2apic , it will set x2apic enablement bit in MSR and create x2apic ACPI tables and pass control to OS with x2apic mode

kernel will check if DMAR table is right or not, if intr-remapping
could be enabled, kernel will stay with x2apic.
if intr-remapping can not be enabled (bad dmar table), kernel will
switch back to xapic mode, but if BSP's X2APIC is
bigger than 255, then you will get kernel panic.

> b) If BIOS feel the system doesn't meet x2apic conditions , it will not set x2apic enablement bit in MSR ,and pass control to OS with xapic mode.

kernel will check if cpuid support x2apic, if it supports x2apic, it
will check if DMAR/intr-remapping could be enabled, if so
kernel will switch to x2apic. otherwise it will stay with xapic.

So you need to make cpuid show does not support x2apic --- check with
intel they have way to do that.
or you can pass "nox2apic" in boot command line.

Yinghai

[Zhang, Lin-Bao (Linux Kernel R&D)]

Hi Yinghai ,
Thanks very much for your reply. I also checked some other previous emails about x2apic patches.

> From: yhlu....@gmail.com [mailto:yhlu....@gmail.com] On Behalf Of
> Yinghai Lu

I just worry about this case,

> > b) If BIOS feel the system doesn't meet x2apic conditions , it will not set
> x2apic enablement bit in MSR ,and pass control to OS with xapic mode.
>
> kernel will check if cpuid support x2apic, if it supports x2apic, it will check if
> DMAR/intr-remapping could be enabled, if so kernel will switch to x2apic.
> otherwise it will stay with xapic.
>
> So you need to make cpuid show does not support x2apic --- check with intel
> they have way to do that.

My concerns is BIOS creates different ACPI tables in xAPIC mode and X2APIC mode.
BIOS will auto detect if the system meets x2APIC condition , if no , it will not create right ACPI tables for this mode and don't set MSR bit.
For this case (BIOS didn't set MSR , but BSP supports x2apic ) , I want to know
1, For Linux kernel, which ACPI tables will be used for x2apic and XAPIC ?
2, why Linux kernel will set x2apic enablement bit in CPU msr ? generally speaking , BIOS set it and OS get it.
If OS get it , who will read it ? BIOS ? I consult BIOS designer , he said , their BIOS calling service(just like UEFI run time service)
will also judge MSR again , so I don't know why Linux kernel will set MSR bit , unless OS has special target .

> or you can pass "nox2apic" in boot command line.

Yeah, I think this would be fine if we really want to use xAPIC mode instead of x2apic mode.
But I am puzzled why Linux kernel will force to enter x2apic just depends on BSP has supported x2apic ?
"BSP supports x2apic" condition is enough for judging if OS can enable x2apic ?


Thanks very much !

[Zhang, Lin-Bao (Linux Kernel R&D)]

Hi Yinghai ,

Actually ,my question like this:
OS can really enable x2apic correctly without BIOS' help in this case :
BIOS has claimed definitely it doesn't enable x2apic ,but OS feel it can enable x2apic(based on BSP supports x2apic) ).

If Linux don't need BIOS' help , how did linux kernel reach this action?
I am reading this related code, I just see Linux kernel judge cpu_has_x2apic() ,if yes, it will set MSR, certainly , IR works right.
We can think BIOS' supporting is not necessary ?

Thanks very much .

-- Bob(LinBao Zhang)
HP linux kernel enginner

[Yinghai Lu]

On Wed, Dec 19, 2012 at 2:13 AM, Zhang, Lin-Bao (Linux Kernel R&D)
<linbao...@hp.com> wrote:
>> > b) If BIOS feel the system doesn't meet x2apic conditions , it will not set
>> x2apic enablement bit in MSR ,and pass control to OS with xapic mode.
>>
>> kernel will check if cpuid support x2apic, if it supports x2apic, it will check if
>> DMAR/intr-remapping could be enabled, if so kernel will switch to x2apic.
>> otherwise it will stay with xapic.
>>
>> So you need to make cpuid show does not support x2apic --- check with intel
>> they have way to do that.
> My concerns is BIOS creates different ACPI tables in xAPIC mode and X2APIC mode.
> BIOS will auto detect if the system meets x2APIC condition , if no , it will not create right ACPI tables for this mode and don't set MSR bit.
> For this case (BIOS didn't set MSR , but BSP supports x2apic ) , I want to know
> 1, For Linux kernel, which ACPI tables will be used for x2apic and XAPIC ?

two MADT?

why two?

according to Intel, if your apic id < 255, even cpu is in x2apic state
already, you only have xapic entry in MADT.

> 2, why Linux kernel will set x2apic enablement bit in CPU msr ? generally speaking , BIOS set it and OS get it.
> If OS get it , who will read it ? BIOS ? I consult BIOS designer , he said , their BIOS calling service(just like UEFI run time service)
> will also judge MSR again , so I don't know why Linux kernel will set MSR bit , unless OS has special target .

because os does not trust BIOS.

BIOS ONLY SHOULD only do thing it should do:
1. initialize memory.
2. assign some resource to pci devices bar
3. provide memory mem-map to bootloader and os.

If it don't want OS to use x2apic, disable that in CPUID.

>
>> or you can pass "nox2apic" in boot command line.
> Yeah, I think this would be fine if we really want to use xAPIC mode instead of x2apic mode.
> But I am puzzled why Linux kernel will force to enter x2apic just depends on BSP has supported x2apic ?
> "BSP supports x2apic" condition is enough for judging if OS can enable x2apic ?

if system support x2apic, os will try use x2apic at first, unless BIOS
mess it up.

[Yinghai Lu]

On Wed, Dec 19, 2012 at 2:36 AM, Zhang, Lin-Bao (Linux Kernel R&D)
<linbao...@hp.com> wrote:
> Hi Yinghai ,
>
> Actually ,my question like this:
> OS can really enable x2apic correctly without BIOS' help in this case :
> BIOS has claimed definitely it doesn't enable x2apic ,but OS feel it can enable x2apic(based on BSP supports x2apic) ).

how does BIOS cleaim?

disable that in CPUID?

>
> If Linux don't need BIOS' help , how did linux kernel reach this action?

need DMAR table to enable intr remapping.
--- ioapic only have 8 bits for dst.

[H. Peter Anvin]

On 12/18/2012 09:49 AM, Yinghai Lu wrote:
>
> So you need to make cpuid show does not support x2apic --- check with
> intel they have way to do that.
>

No, but there is a flag in ACPI that "this BIOS is broken with x2APIC".
Setting that flag as opposed to fixing the bugs, though is very much not
recommended, however; the lack of interrupt routing means there are ways
for a rogue device to compromise the security provided by VT-d, see:

http://www.invisiblethingslab.com/resources/2011/Software%20Attacks%20on%20Intel%20VT-d.pdf

-hpa

[Zhang, Lin-Bao (Linux Kernel R&D)]

Hi Yinghai,

Maybe two days ago, mis-understand the usage of MSR.
Today, I carefully researched Intel? 64 Architecture x2APIC Specification which is open for all people.

I think current core argument is our BIOS did following judgement ,
but Linux kernel did not(Linux seems just judge BSP supports x2apic mode or not by CPUID method)
++++++++++++
2.9 SYSTEM TRANSITIONS
This section describes implications for the x2APIC across system state transitions -
specifically initialization and booting.
The default will be for the BIOS to pass the control to the OS with the local x2APICs
in xAPIC mode if all x2APIC IDs reported by CPUID.0BH:EDX are less than 255, and
in x2APIC mode if there are any logical processor reporting its x2APIC ID at 255 or
greater.
++++++++++++
Our BSP really supports x2apic mode, however,as this spec, our BIOS think some of configuration of server
could not meet x2apic conditions ,So didn't place local xAPICs into x2apic mode.
But linux kernel indeed enable x2apic mode(just believe BSP, didn't do above judgement) . so this is the conflict.

I carefully read Linux kernel source code, I didn't find related code.
BIOS and OS both can configure and enable local xAPICs into x2APIC mode ,but they should follow
the same spec . from the spec , we should consider all cores or logical CPUs .
if I lost some source code , sorry first , could you point me these code ? thanks very much!

Following are some clarifications for old questions. Thanks very much for Yinghai and others' help.
From spec:
++++++++++++++++
2.2 DETECTING AND ENABLING X 2APIC
A processor’s support to operate its local APIC in the x2APIC mode can be detected
by querying the extended feature flag information reported by CPUID. When CPUID
is executed with EAX = 1, the returned value in ECX[Bit 21] indicates processor’s
support for the x2APIC mode. If CPUID.(EAX=01H):ECX[Bit 21] is set, then the local
APIC in the processor supports the x2APIC capability and can be placed into the
x2APIC mode. This bit is set only when the x2APIC hardware is present.
? System software can place the local APIC in the x2APIC mode by setting the
x2APIC mode enable bit (bit 10) in the IA32_APIC_BASE MSR at MSR address
01BH. The layout for the IA32_APIC_BASE MSR is shown in Figure 2-1.
++++++++++++++++
So ,as above words:
1, CPUID is just executed to query if CPU has supported x2apic mode.
It is read only. It is decided by CPU vendor.
In kernel source , cpu_has_x2apic() should leverage CPUID interface.

2, bit 11 and 10 of IA32_APIC_BASE MSR are used to switch to x2apic (EN=1, EXTD=1).
So I begin to understand that , OS certainly can control MSR to make CPU enter x2apic mode.
BIOS also can , OS can also read it and write it as we need.

> -----Original Message-----
> From: yhlu....@gmail.com [mailto:yhlu....@gmail.com] On Behalf Of
> Yinghai Lu

> Sent: 2012年12月20日 0:06
>
> how does BIOS claim?
>
Sorry for words , BIOS didn't claim , it just didn't set MSR(en=1 , EXTD=1). OS will read it first.
This just can indicate that before BIOS passing control to OS, local x2APICs running in xAPIC mode.


> disable that in CPUID?
>
No , CPUID is read only , "When CPUID is executed with EAX = 1, the returned value in ECX[Bit 21] indicates processor’s
support for the x2APIC mode."
BIOS just will query ,can set. OS also did like this.

[H. Peter Anvin]

On 12/20/2012 08:46 AM, Yinghai Lu wrote:

> On Thu, Dec 20, 2012 at 2:38 AM, Zhang, Lin-Bao (Linux Kernel R&D)
> <linbao...@hp.com> wrote:
>> I think current core argument is our BIOS did following judgement ,
>> but Linux kernel did not(Linux seems just judge BSP supports x2apic mode or not by CPUID method)
>

> kernel checks that, if the BSP in x2apic mode, kernel will check if
> intr-remapping can be enabled, otherwise
> it will switch back to xapic mode.

>
>> Our BSP really supports x2apic mode, however,as this spec, our BIOS think some of configuration of server
>> could not meet x2apic conditions ,So didn't place local xAPICs into x2apic mode.
>> But linux kernel indeed enable x2apic mode(just believe BSP, didn't do above judgement) . so this is the conflict.
>

> so your BIOS is wrong, it does not enable intr-remapping.
>

FWIW, I bought a *server* today and it didn't have interrupt remapping
and x2APIC, I would consider it defective and return it.

-hpa

[Yinghai Lu]

On Thu, Dec 20, 2012 at 2:38 AM, Zhang, Lin-Bao (Linux Kernel R&D)
<linbao...@hp.com> wrote:
> I think current core argument is our BIOS did following judgement ,
> but Linux kernel did not(Linux seems just judge BSP supports x2apic mode or not by CPUID method)

kernel checks that, if the BSP in x2apic mode, kernel will check if
intr-remapping can be enabled, otherwise

it will switch back to xapic mode.

> Our BSP really supports x2apic mode, however,as this spec, our BIOS think some of configuration of server
> could not meet x2apic conditions ,So didn't place local xAPICs into x2apic mode.
> But linux kernel indeed enable x2apic mode(just believe BSP, didn't do above judgement) . so this is the conflict.

so your BIOS is wrong, it does not enable intr-remapping.

Yinghai