X2APIC support

[Slawa Olhovchenkov]

Does STABLE support X2APIC?
I see X2APIC related commits in CURRENT, what is status for STABLE?
I am try to enable X2APIC support on X10DRi and see kernel trap on
boot.

_______________________________________________
freebsd...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"

[Konstantin Belousov]

On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> Does STABLE support X2APIC?
> I see X2APIC related commits in CURRENT, what is status for STABLE?
> I am try to enable X2APIC support on X10DRi and see kernel trap on
> boot.

x2APIC is only supported in HEAD. The code to parse processor x2APIC
structures in MADT was committed, but AFAIK never tested on real hardware.

There are no plans to merge this to stable/10.

[Slawa Olhovchenkov]

On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:

> On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > Does STABLE support X2APIC?
> > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > boot.
>
> x2APIC is only supported in HEAD. The code to parse processor x2APIC
> structures in MADT was committed, but AFAIK never tested on real hardware.
>
> There are no plans to merge this to stable/10.

I am try now on real hardware and got kernel fault at boot when set in
BIOS X2APIC_OPT_OUT Flag to [Enable].
https://s9.postimg.org/6mt8bu21b/fault2.jpg

0xffffffff80537e74 is /usr/src/sys/kern/subr_smp.c:959

Boot success if X2APIC_OPT_OUT Flag set to [Disable]
Supermicro X10DRi, dual E5-2650v4

PS: addr2line don't support debug symbol location at /usr/lib/debug

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 02:27:24PM +0300, Slawa Olhovchenkov wrote:
> On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:
>
> > On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > > Does STABLE support X2APIC?
> > > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > > boot.
> >
> > x2APIC is only supported in HEAD. The code to parse processor x2APIC
> > structures in MADT was committed, but AFAIK never tested on real hardware.
> >
> > There are no plans to merge this to stable/10.
>
> I am try now on real hardware and got kernel fault at boot when set in
> BIOS X2APIC_OPT_OUT Flag to [Enable].
> https://s9.postimg.org/6mt8bu21b/fault2.jpg

How do you know that the issue is due to that flag ? I doubt that BIOS
would provide direct knob to manage that flag, since requesting that
out-out only makes sense if BIOS is unable to handle x2APIC mode, e.g.
in SMI handlers. How precisely the BIOS setting called which you frobbed ?

Show complete verbose dmesg from the first line to backtrace.

>
> 0xffffffff80537e74 is /usr/src/sys/kern/subr_smp.c:959
And this is a topology probe code, which should not be affected by the
x2APIC mode at all, except possibly due to longer format of the APIC IDs.

I added Andrey to Cc:, he might give some more guidance.

>
> Boot success if X2APIC_OPT_OUT Flag set to [Disable]
> Supermicro X10DRi, dual E5-2650v4

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 02:27:24PM +0300, Slawa Olhovchenkov wrote:
> > On Sat, Dec 12, 2015 at 03:35:13PM +0200, Konstantin Belousov wrote:
> >
> > > On Sat, Dec 12, 2015 at 04:06:15PM +0300, Slawa Olhovchenkov wrote:
> > > > Does STABLE support X2APIC?
> > > > I see X2APIC related commits in CURRENT, what is status for STABLE?
> > > > I am try to enable X2APIC support on X10DRi and see kernel trap on
> > > > boot.
> > >
> > > x2APIC is only supported in HEAD. The code to parse processor x2APIC
> > > structures in MADT was committed, but AFAIK never tested on real hardware.
> > >
> > > There are no plans to merge this to stable/10.
> >
> > I am try now on real hardware and got kernel fault at boot when set in
> > BIOS X2APIC_OPT_OUT Flag to [Enable].
> > https://s9.postimg.org/6mt8bu21b/fault2.jpg
> How do you know that the issue is due to that flag ?

Just by turn on and off this flag and try boot at both cases.

> I doubt that BIOS would provide direct knob to manage that flag, since requesting that
> out-out only makes sense if BIOS is unable to handle x2APIC mode, e.g.
> in SMI handlers. How precisely the BIOS setting called which you frobbed ?

As I point before: 'X2APIC_OPT_OUT Flag'.
Hidden before 'X2APIC' set to '[Enable]'

https://s15.postimg.org/3k5l7z397/bios_x2.jpg

> Show complete verbose dmesg from the first line to backtrace.

Booting...
\01;00Table 'FACP' at 0x79b189a8
Table 'APIC' at 0x79b18ab8
APIC: Found table at 0x79b18ab8
APIC: Using the MADT enumerator.
MADT: Found CPU APIC ID 0 ACPI ID 0: enabled
SMP: Added CPU 0 (AP)
MADT: Found CPU APIC ID 2 ACPI ID 2: enabled
SMP: Added CPU 2 (AP)
MADT: Found CPU APIC ID 4 ACPI ID 4: enabled
SMP: Added CPU 4 (AP)
MADT: Found CPU APIC ID 6 ACPI ID 6: enabled
SMP: Added CPU 6 (AP)
MADT: Found CPU APIC ID 8 ACPI ID 8: enabled
SMP: Added CPU 8 (AP)
MADT: Found CPU APIC ID 10 ACPI ID 10: enabled
SMP: Added CPU 10 (AP)
MADT: Found CPU APIC ID 16 ACPI ID 16: enabled
SMP: Added CPU 16 (AP)
MADT: Found CPU APIC ID 18 ACPI ID 18: enabled
SMP: Added CPU 18 (AP)
MADT: Found CPU APIC ID 20 ACPI ID 20: enabled
SMP: Added CPU 20 (AP)
MADT: Found CPU APIC ID 22 ACPI ID 22: enabled
SMP: Added CPU 22 (AP)
MADT: Found CPU APIC ID 24 ACPI ID 24: enabled
SMP: Added CPU 24 (AP)
MADT: Found CPU APIC ID 26 ACPI ID 26: enabled
SMP: Added CPU 26 (AP)
MADT: Found CPU APIC ID 32 ACPI ID 32: enabled
SMP: Added CPU 32 (AP)
MADT: Found CPU APIC ID 34 ACPI ID 34: enabled
SMP: Added CPU 34 (AP)
MADT: Found CPU APIC ID 36 ACPI ID 36: enabled
SMP: Added CPU 36 (AP)
MADT: Found CPU APIC ID 38 ACPI ID 38: enabled
SMP: Added CPU 38 (AP)
MADT: Found CPU APIC ID 40 ACPI ID 40: enabled
SMP: Added CPU 40 (AP)
MADT: Found CPU APIC ID 42 ACPI ID 42: enabled
SMP: Added CPU 42 (AP)
MADT: Found CPU APIC ID 48 ACPI ID 48: enabled
SMP: Added CPU 48 (AP)
MADT: Found CPU APIC ID 50 ACPI ID 50: enabled
SMP: Added CPU 50 (AP)
MADT: Found CPU APIC ID 52 ACPI ID 52: enabled
SMP: Added CPU 52 (AP)
MADT: Found CPU APIC ID 54 ACPI ID 54: enabled
SMP: Added CPU 54 (AP)
MADT: Found CPU APIC ID 56 ACPI ID 56: enabled
SMP: Added CPU 56 (AP)
MADT: Found CPU APIC ID 58 ACPI ID 58: enabled
SMP: Added CPU 58 (AP)
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p305117 #0: Wed Aug 31 17:58:35 UTC 2016
s...@storage01.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
Table 'FACP' at 0x79b189a8
Table 'APIC' at 0x79b18ab8
Table 'FPDT' at 0x79b18c70
Table 'FIDT' at 0x79b18cb8
Table 'SPMI' at 0x79b18d58
Table 'MCFG' at 0x79b18da0
Table 'UEFI' at 0x79b18de0
Table 'MCEJ' at 0x79b18e28
Table 'HPET' at 0x79b18f58
Table 'WDDT' at 0x79b18f90
Table 'SSDT' at 0x79b18fd0
Table 'SSDT' at 0x79b30020
Table 'SSDT' at 0x79b32670
Table 'PRAD' at 0x79b326d8
Table 'DMAR' at 0x79b32798
Table 'HEST' at 0x79b328e8
Table 'BERT' at 0x79b32990
Table 'ERST' at 0x79b329c0
Table 'EINJ' at 0x79b32bf0
ACPI: No SRAT table found
PPIM 0: PA=0xb8000, VA=0xffffffff81610000, size=0x8000, mode=0
VT(vga): text 80x25
Preloaded elf kernel "/boot/kernel.VSTREAM/kernel" at 0xffffffff814a8000.
Preloaded /boot/entropy "/boot/entropy" at 0xffffffff814a8ac0.
Preloaded elf obj module "/boot/kernel.VSTREAM/zfs.ko" at 0xffffffff814a8b10.
Preloaded elf obj module "/boot/kernel.VSTREAM/krpc.ko" at 0xffffffff814a9300.
Preloaded elf obj module "/boot/kernel.VSTREAM/opensolaris.ko" at 0xffffffff814a9970.
Preloaded elf obj module "/boot/kernel.VSTREAM/if_igb.ko" at 0xffffffff814a9fa8.
Preloaded elf obj module "/boot/kernel.VSTREAM/if_lagg.ko" at 0xffffffff814aa618.
Preloaded elf obj module "/boot/kernel.VSTREAM/ukbd.ko" at 0xffffffff814aacc8.
Preloaded elf obj module "/boot/kernel.VSTREAM/usb.ko" at 0xffffffff814ab378.
Preloaded elf obj module "/boot/kernel.VSTREAM/umass.ko" at 0xffffffff814abaa8.
Preloaded elf obj module "/boot/kernel.VSTREAM/accf_http.ko" at 0xffffffff814ac098.
Preloaded elf obj module "/boot/kernel.VSTREAM/cc_htcp.ko" at 0xffffffff814ac610.
Preloaded elf obj module "/boot/kernel.VSTREAM/uhci.ko" at 0xffffffff814acc40.
Preloaded elf obj module "/boot/kernel.VSTREAM/ohci.ko" at 0xffffffff814ad1f0.
Preloaded elf obj module "/boot/kernel.VSTREAM/ehci.ko" at 0xffffffff814ad7a0.
Preloaded elf obj module "/boot/kernel.VSTREAM/xhci.ko" at 0xffffffff814add50.
Calibrating TSC clock ... TSC clock: 2200043851 Hz
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
XSAVE Features=0x1<XSAVEOPT>
VT-x: Basic Features=0xda0400<SMM,INS/OUTS,TRUE>
Pin-Based Controls=0xff<ExtINT,NMI,VNMI,PreTmr,PostIntr>
Primary Processor Controls=0xfff9fffe<INTWIN,TSCOff,HLT,INVLPG,MWAIT,RDPMC,RDTSC,CR3-LD,CR3-ST,CR8-LD,CR8-ST,TPR,NMIWIN,MOV-DR,IO,IOmap,MTF,MSRmap,MONITOR,PAUSE>
Secondary Processor Controls=0x77fff<APIC,EPT,DT,RDTSCP,x2APIC,VPID,WBINVD,UG,APIC-reg,VID,PAUSE-loop,RDRAND,INVPCID,VMFUNC,VMCS,XSAVES>
Exit Controls=0xda0400<PAT-LD,EFER-SV,PTMR-SV>
Entry Controls=0xda0400
EPT Features=0x6334141<XO,PW4,UC,WB,2M,1G,INVEPT,AD,single,all>
VPID Features=0xf01<INVVPID,individual,single,all,single-globals>
TSC: P-state invariant, performance statistics
Data TLB: 2 MByte or 4 MByte pages, 4-way set associative, 32 entries and a separate array with 1 GByte pages, 4-way set associative, 4 entries
Data TLB: 4 KB pages, 4-way set associative, 64 entries
Instruction TLB: 2M/4M pages, fully associative, 8 entries
Instruction TLB: 4KByte pages, 8-way set associative, 128 entries
64-Byte prefetching
Shared 2nd-Level TLB: 4 KByte /2 MByte pages, 6-way associative, 1536 entries. Also 1GBbyte pages, 4-way, 16 entries
L2 cache: 256 kbytes, 8-way associative, 64 bytes/line
real memory = 137438953472 (131072 MB)
Physical memory chunk(s):
0x0000000000010000 - 0x0000000000095fff, 548864 bytes (134 pages)
0x0000000000100000 - 0x00000000001fffff, 1048576 bytes (256 pages)
0x00000000014e9000 - 0x000000007918bfff, 2009739264 bytes (490659 pages)
0x0000000100000000 - 0x0000001fabf78fff, 131734147072 bytes (32161657 pages)
avail memory = 133408907264 (127228 MB)
Table 'FACP' at 0x79b189a8
Table 'APIC' at 0x79b18ab8
Table 'FPDT' at 0x79b18c70
Table 'FIDT' at 0x79b18cb8
Table 'SPMI' at 0x79b18d58
Table 'MCFG' at 0x79b18da0
Table 'UEFI' at 0x79b18de0
Table 'MCEJ' at 0x79b18e28
Table 'HPET' at 0x79b18f58
Table 'WDDT' at 0x79b18f90
Table 'SSDT' at 0x79b18fd0
Table 'SSDT' at 0x79b30020
Table 'SSDT' at 0x79b32670
Table 'PRAD' at 0x79b326d8
Table 'DMAR' at 0x79b32798
DMAR: Found table at 0x79b32798
x2APIC available but disabled by DMAR table
Event timer "LAPIC" quality 600
LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
ACPI APIC Table: <ALASKA A M I >
Package ID shift: 5
L3 cache ID shift: 5
L2 cache ID shift: 1
L1 cache ID shift: 1
Core ID shift: 1
kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80537e74
stack pointer = 0x28:0xffffffff814b4a60
frame pointer = 0x28:0xffffffff814b4a70
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff805272e7 at kdb_backtrace+0x67
#1 0xffffffff804dd662 at vpanic+0x182
#2 0xffffffff804dd4d3 at panic+0x43
#3 0xffffffff807a3791 at trap_fatal+0x351
#4 0xffffffff807a3983 at trap_pfault+0x1e3
#5 0xffffffff807a2f0c at trap+0x26c
#6 0xffffffff80787ca1 at calltrap+0x8
#7 0xffffffff8083b52a at topo_probe+0x61a
#8 0xffffffff8078fe81 at cpu_mp_start+0x1b1
#9 0xffffffff805382ca at mp_start+0x3a
#10 0xffffffff80465cd8 at mi_startup+0x118
#11 0xffffffff8028dfac at btext+0x2c
Uptime: 1s

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> As I point before: 'X2APIC_OPT_OUT Flag'.
> Hidden before 'X2APIC' set to '[Enable]'
>
> https://s15.postimg.org/3k5l7z397/bios_x2.jpg
>
> > Show complete verbose dmesg from the first line to backtrace.

And post the verbose dmesg for successfull boot as well, please.

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > As I point before: 'X2APIC_OPT_OUT Flag'.
> > Hidden before 'X2APIC' set to '[Enable]'
> >
> > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> >
> > > Show complete verbose dmesg from the first line to backtrace.
>
> And post the verbose dmesg for successfull boot as well, please.

http://m.uploadedit.com/ba3s/147275109329.txt

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 08:31:49PM +0300, Slawa Olhovchenkov wrote:
> On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:
>
> > On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > > As I point before: 'X2APIC_OPT_OUT Flag'.
> > > Hidden before 'X2APIC' set to '[Enable]'
> > >
> > > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> > >
> > > > Show complete verbose dmesg from the first line to backtrace.
> >
> > And post the verbose dmesg for successfull boot as well, please.
>
> http://m.uploadedit.com/ba3s/147275109329.txt

From pure curiousity, if you enable hyperthreading, does system survives
frobbing of X2APIC_OPT_OUT ?

Also, please show the output of acpidump -t.

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 08:50:49PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 08:31:49PM +0300, Slawa Olhovchenkov wrote:
> > On Thu, Sep 01, 2016 at 08:26:32PM +0300, Konstantin Belousov wrote:
> >
> > > On Thu, Sep 01, 2016 at 03:13:00PM +0300, Slawa Olhovchenkov wrote:
> > > > On Thu, Sep 01, 2016 at 02:45:00PM +0300, Konstantin Belousov wrote:
> > > > As I point before: 'X2APIC_OPT_OUT Flag'.
> > > > Hidden before 'X2APIC' set to '[Enable]'
> > > >
> > > > https://s15.postimg.org/3k5l7z397/bios_x2.jpg
> > > >
> > > > > Show complete verbose dmesg from the first line to backtrace.
> > >
> > > And post the verbose dmesg for successfull boot as well, please.
> >
> > http://m.uploadedit.com/ba3s/147275109329.txt
>
> >From pure curiousity, if you enable hyperthreading, does system survives
> frobbing of X2APIC_OPT_OUT ?

Sorry, don't cleanly understund, what combination of BIOS setting I am need to probe?
And what I am need to check?

> Also, please show the output of acpidump -t.

/*
RSD PTR: OEM=ALASKA, ACPI_Rev=2.0x (2)
XSDT=0x0000000079ae50a0, length=36, cksum=197
*/
/*
XSDT: Length=188, Revision=1, Checksum=37,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
Entries={ 0x0000000079b189a8, 0x0000000079b18ab8, 0x0000000079b18c70, 0x0000000079b18cb8, 0x0000000079b18d58, 0x0000000079b18da0, 0x0000000079b18de0, 0x0000000079b18e28, 0x0000000079b18f58, 0x0000000079b18f90, 0x0000000079b18fd0, 0x0000000079b30020, 0x0000000079b32670, 0x0000000079b326d8,
0x0000000079b32798, 0x0000000079b328e8, 0x0000000079b32990, 0x0000000079b329c0, 0x0000000079b32bf0 }
*/
/*
FACP: Length=268, Revision=5, Checksum=66,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
FACS=0x79fa8f80, DSDT=0x79ae51f0
INT_MODEL=APIC
Preferred_PM_Profile=Enterprise Server (4)
SCI_INT=9
SMI_CMD=0xb2, ACPI_ENABLE=0xa0, ACPI_DISABLE=0xa1, S4BIOS_REQ=0x0
PSTATE_CNT=0x0
PM1a_EVT_BLK=0x400-0x403
PM1a_CNT_BLK=0x404-0x405
PM2_CNT_BLK=0x450-0x450
PM_TMR_BLK=0x408-0x40b
GPE0_BLK=0x420-0x42f
P_LVL2_LAT=101 us, P_LVL3_LAT=1001 us
FLUSH_SIZE=1024, FLUSH_STRIDE=16
DUTY_OFFSET=1, DUTY_WIDTH=3
DAY_ALRM=13, MON_ALRM=0, CENTURY=50
IAPC_BOOT_ARCH={LEGACY_DEVICES,NO_ASPM}
Flags={WBINVD,C1_SUPPORTED,SLEEP_BUTTON,S4_RTC_WAKE,RESET_REGISTER,PLATFORM_CLOCK}
RESET_REG=0xcf9:0[8] (IO), RESET_VALUE=0x6
*/
/*
FACS: Length=64, HwSig=0x0ee12c64, Firm_Wake_Vec=0x00000000
Global_Lock=
Flags=
Version=2
*/
/*
DSDT: Length=210868, Revision=2, Checksum=0,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=INTL, Creator Revision=0x20091013
*/
/*
APIC: Length=436, Revision=3, Checksum=176,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
Local APIC ADDR=0xfee00000
Flags={PC-AT}

Type=Local APIC
ACPI CPU=0
Flags={ENABLED}
APIC ID=0

Type=Local APIC
ACPI CPU=2
Flags={ENABLED}
APIC ID=2

Type=Local APIC
ACPI CPU=4
Flags={ENABLED}
APIC ID=4

Type=Local APIC
ACPI CPU=6
Flags={ENABLED}
APIC ID=6

Type=Local APIC
ACPI CPU=8
Flags={ENABLED}
APIC ID=8

Type=Local APIC
ACPI CPU=10
Flags={ENABLED}
APIC ID=10

Type=Local APIC
ACPI CPU=16
Flags={ENABLED}
APIC ID=16

Type=Local APIC
ACPI CPU=18
Flags={ENABLED}
APIC ID=18

Type=Local APIC
ACPI CPU=20
Flags={ENABLED}
APIC ID=20

Type=Local APIC
ACPI CPU=22
Flags={ENABLED}
APIC ID=22

Type=Local APIC
ACPI CPU=24
Flags={ENABLED}
APIC ID=24

Type=Local APIC
ACPI CPU=26
Flags={ENABLED}
APIC ID=26

Type=Local APIC
ACPI CPU=32
Flags={ENABLED}
APIC ID=32

Type=Local APIC
ACPI CPU=34
Flags={ENABLED}
APIC ID=34

Type=Local APIC
ACPI CPU=36
Flags={ENABLED}
APIC ID=36

Type=Local APIC
ACPI CPU=38
Flags={ENABLED}
APIC ID=38

Type=Local APIC
ACPI CPU=40
Flags={ENABLED}
APIC ID=40

Type=Local APIC
ACPI CPU=42
Flags={ENABLED}
APIC ID=42

Type=Local APIC
ACPI CPU=48
Flags={ENABLED}
APIC ID=48

Type=Local APIC
ACPI CPU=50
Flags={ENABLED}
APIC ID=50

Type=Local APIC
ACPI CPU=52
Flags={ENABLED}
APIC ID=52

Type=Local APIC
ACPI CPU=54
Flags={ENABLED}
APIC ID=54

Type=Local APIC
ACPI CPU=56
Flags={ENABLED}
APIC ID=56

Type=Local APIC
ACPI CPU=58
Flags={ENABLED}
APIC ID=58

Type=Local APIC NMI
ACPI CPU=0
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=2
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=4
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=6
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=8
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=10
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=16
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=18
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=20
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=22
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=24
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=26
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=32
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=34
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=36
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=38
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=40
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=42
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=48
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=50
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=52
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=54
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=56
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=Local APIC NMI
ACPI CPU=58
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}

Type=IO APIC
APIC ID=1
INT BASE=0
ADDR=0x00000000fec00000

Type=IO APIC
APIC ID=2
INT BASE=24
ADDR=0x00000000fec01000

Type=IO APIC
APIC ID=3
INT BASE=48
ADDR=0x00000000fec40000

Type=INT Override
BUS=0
IRQ=0
INTR=2
Flags={Polarity=conforming, Trigger=conforming}

Type=INT Override
BUS=0
IRQ=9
INTR=9
Flags={Polarity=active-hi, Trigger=level}
*/
/*
FPDT: Length=68, Revision=1, Checksum=156,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
*/
/*
FIDT: Length=156, Revision=1, Checksum=90,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=AMI, Creator Revision=0x10013
*/
/*
SPMI: Length=65, Revision=5, Checksum=228,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x0,
Creator ID=AMI., Creator Revision=0x0
*/
/*
MCFG: Length=60, Revision=1, Checksum=97,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=MSFT, Creator Revision=0x97

Base Address=0x0000000080000000
Segment Group=0x0000
Start Bus=0
End Bus=255
*/
/*
UEFI: Length=66, Revision=1, Checksum=247,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1072009,
Creator ID=, Creator Revision=0x0
*/
/*
MCEJ: Length=304, Revision=1, Checksum=118,
OEMID=INTEL, OEM Table ID=, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x100000d
*/
/*
HPET: Length=56, Revision=1, Checksum=77,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x20091013
HPET Number=0
ADDR=0x00000000fed00000:0[64] (Memory) HW Rev=0x1
Comparators=7
Counter Size=1
Legacy IRQ routing capable={TRUE}
PCI Vendor ID=0x8086
Minimal Tick=14318
Flags=0x00
*/
/*
WDDT: Length=64, Revision=1, Checksum=133,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x0,
Creator ID=INTL, Creator Revision=0x20091013
*/
/*
SSDT: Length=94285, Revision=2, Checksum=215,
OEMID=ALASKA, OEM Table ID=PmMgt, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x20120913
*/
/*
SSDT: Length=9802, Revision=2, Checksum=113,
OEMID=ALASKA, OEM Table ID=SpsNm, OEM Revision=0x2,
Creator ID=INTL, Creator Revision=0x20120913
*/
/*
SSDT: Length=100, Revision=2, Checksum=247,
OEMID=ALASKA, OEM Table ID=SpsNvs, OEM Revision=0x2,
Creator ID=INTL, Creator Revision=0x20120913
*/
/*
PRAD: Length=191, Revision=2, Checksum=8,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x2,
Creator ID=INTL, Creator Revision=0x20120913
*/
/*
DMAR: Length=336, Revision=1, Checksum=39,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x20091013
Host Address Width=46
Flags={INTR_REMAP}

Type=DRHD
Length=104
Flags=
Segment=0
Address=0x00000000fbffc000
Device Scope:
Type=IOAPIC
Length=8
EnumerationId=3
StartBusNumber=128
Path={5:4}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:0}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:1}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:2}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:3}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:4}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:5}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:6}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=128
Path={4:7}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=128
Path={1:0}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=128
Path={3:0}

Type=DRHD
Length=40
Flags={INCLUDE_ALL}
Segment=0
Address=0x00000000c7ffc000
Device Scope:
Type=IOAPIC
Length=8
EnumerationId=1
StartBusNumber=240
Path={31:7}

Type=IOAPIC
Length=8
EnumerationId=2
StartBusNumber=0
Path={5:4}

Type=HPET
Length=8
EnumerationId=0
StartBusNumber=240
Path={15:0}

Type=RMRR
Length=48
Segment=0
BaseAddress=0x000000007bc3c000
LimitAddress=0x000000007bc4bfff
Device Scope:
Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=0
Path={20:0}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=0
Path={26:0}

Type=PCI Endpoint Device
Length=8
EnumerationId=0
StartBusNumber=0
Path={29:0}

Type=ATSR
Length=56
Flags=
Segment=0
Device Scope:
Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=0
Path={1:0}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=0
Path={2:0}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=0
Path={3:0}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=0
Path={3:2}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=128
Path={1:0}

Type=PCI Sub-Hierarchy
Length=8
EnumerationId=0
StartBusNumber=128
Path={3:0}

Type=RHSA
Length=20
BaseAddress=0x00000000c7ffc000
ProximityDomain=0x00000000

Type=RHSA
Length=20
BaseAddress=0x00000000fbffc000
ProximityDomain=0x00000001
*/
/*
HEST: Length=168, Revision=1, Checksum=26,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x1
*/
/*
BERT: Length=48, Revision=1, Checksum=141,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x1
*/
/*
ERST: Length=560, Revision=1, Checksum=33,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x1
*/
/*
EINJ: Length=336, Revision=1, Checksum=62,
OEMID=ALASKA, OEM Table ID=A M I, OEM Revision=0x1,
Creator ID=INTL, Creator Revision=0x1
*/

[Konstantin Belousov]

On Thu, Sep 01, 2016 at 09:00:14PM +0300, Slawa Olhovchenkov wrote:
> Sorry, don't cleanly understund, what combination of BIOS setting I am need to probe?
> And what I am need to check?

Set 'Hyper-Threading' to Enabled.
Set 'X2APIC_OPT_OUT' to Enabled.
Try to boot.

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 09:19:15PM +0300, Konstantin Belousov wrote:

> On Thu, Sep 01, 2016 at 09:00:14PM +0300, Slawa Olhovchenkov wrote:
> > Sorry, don't cleanly understund, what combination of BIOS setting I am need to probe?
> > And what I am need to check?
>
> Set 'Hyper-Threading' to Enabled.
> Set 'X2APIC_OPT_OUT' to Enabled.
> Try to boot.

Crashed at same point.

[Slawa Olhovchenkov]

On Thu, Sep 01, 2016 at 09:37:29PM +0200, sth...@nethelp.no wrote:

> > > > Sorry, don't cleanly understund, what combination of BIOS setting I am need to probe?
> > > > And what I am need to check?
> > >
> > > Set 'Hyper-Threading' to Enabled.
> > > Set 'X2APIC_OPT_OUT' to Enabled.
> > > Try to boot.
> >
> > Crashed at same point.
>

> A comment about X2APIC, on a different type of system: I had to disable
> X2APIC on HP ProLiant DL360 Gen9 servers in order to avoid 10.3-STABLE
> crashing during boot.
>
> Steinar Haug, Nethelp consulting, sth...@nethelp.no

In this test I am use 11.0

[sth...@nethelp.no]

> > > Sorry, don't cleanly understund, what combination of BIOS setting I am need to probe?
> > > And what I am need to check?
> >
> > Set 'Hyper-Threading' to Enabled.
> > Set 'X2APIC_OPT_OUT' to Enabled.
> > Try to boot.
>
> Crashed at same point.

A comment about X2APIC, on a different type of system: I had to disable
X2APIC on HP ProLiant DL360 Gen9 servers in order to avoid 10.3-STABLE
crashing during boot.

Steinar Haug, Nethelp consulting, sth...@nethelp.no

[Andriy Gapon]

Interesting. Could you please do 'list *topo_probe+0x61a' in kgdb, so that I
can see what code is being executed when the trap happens? Also, disassembly of
the function could be useful as well.

Wait...
Kostik, I see one strange thing which is common to both successful and
unsuccessful configurations. All "SMP: Added CPU..." lines have "AP" in them.
It seems like the platform does not tell explicitly tell which CPU is the BSP,
see cpu_add() function. This can break quite a few assumption. And I am not
even sure how the successful scenario works.
Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
set based on the current CPU's Local APIC ID. I suspect then that this
information is incorrect in the failing case.

Slawa,
my guess can be checked by adding a printf to cpu_mp_start() right after
boot_cpu_id assignment.

> #8 0xffffffff8078fe81 at cpu_mp_start+0x1b1
> #9 0xffffffff805382ca at mp_start+0x3a
> #10 0xffffffff80465cd8 at mi_startup+0x118
> #11 0xffffffff8028dfac at btext+0x2c
> Uptime: 1s

--
Andriy Gapon

[Slawa Olhovchenkov]

(kgdb) list *topo_probe+0x61a
0xffffffff8083b52a is in topo_probe (/usr/src/sys/x86/x86/mp_x86.c:540).
535 topo_layers[layer].subtype);
536 }
537 }
538
539 parent = &topo_root;
540 for (layer = 0; layer < nlayers; ++layer) {
541 node_id = boot_cpu_id >> topo_layers[layer].id_shift;
542 node = topo_find_node_by_hwid(parent, node_id,
543 topo_layers[layer].type,
544 topo_layers[layer].subtype);
Current language: auto; currently minimal

> can see what code is being executed when the trap happens? Also, disassembly of
> the function could be useful as well.

(kgdb) x/40i *topo_probe+0x600
0xffffffff8083b510 <topo_probe+1536>: and $0xf8,%al
0xffffffff8083b512 <topo_probe+1538>: movslq -0x4(%r12),%rcx
0xffffffff8083b517 <topo_probe+1543>: mov %rbx,%rdi
0xffffffff8083b51a <topo_probe+1546>: callq 0xffffffff80537e30 <topo_find_node_by_hwid>
0xffffffff8083b51f <topo_probe+1551>: mov %rax,%rbx
0xffffffff8083b522 <topo_probe+1554>: mov %rbx,%rdi
0xffffffff8083b525 <topo_probe+1557>: callq 0xffffffff80537e70 <topo_promote_child>
0xffffffff8083b52a <topo_probe+1562>: add $0xc,%r12
0xffffffff8083b52e <topo_probe+1566>: dec %r14d
0xffffffff8083b531 <topo_probe+1569>: jne 0xffffffff8083b500 <topo_probe+1520>
0xffffffff8083b533 <topo_probe+1571>: movb $0x1,0xffffffff80dfa664
0xffffffff8083b53b <topo_probe+1579>: add $0x68,%rsp
0xffffffff8083b53f <topo_probe+1583>: pop %rbx
0xffffffff8083b540 <topo_probe+1584>: pop %r12
0xffffffff8083b542 <topo_probe+1586>: pop %r13
0xffffffff8083b544 <topo_probe+1588>: pop %r14
0xffffffff8083b546 <topo_probe+1590>: pop %r15
0xffffffff8083b548 <topo_probe+1592>: pop %rbp
0xffffffff8083b549 <topo_probe+1593>: retq
0xffffffff8083b54a <topo_probe+1594>: nopw 0x0(%rax,%rax,1)

> Wait...
> Kostik, I see one strange thing which is common to both successful and
> unsuccessful configurations. All "SMP: Added CPU..." lines have "AP" in them.

for #1..#23
no line 'SMP: AP CPU #0 Launched!'

> It seems like the platform does not tell explicitly tell which CPU is the BSP,
> see cpu_add() function. This can break quite a few assumption. And I am not
> even sure how the successful scenario works.

# mptable

===============================================================================

MPTable

-------------------------------------------------------------------------------

MP Floating Pointer Structure:

location: BIOS
physical address: 0x000fd050
signature: '_MP_'
length: 16 bytes
version: 1.4
checksum: 0x27
mode: Virtual Wire

-------------------------------------------------------------------------------

MP Config Table Header:

physical address: 0x000fcaa0
signature: 'PCMP'
base table length: 1228
version: 1.4
checksum: 0x95
OEM ID: 'A M I'
Product ID: 'ALASKA'
OEM table pointer: 0x00000000
OEM table size: 0
entry count: 112
local APIC address: 0xfee00000
extended table length: 220
extended table checksum: 72

-------------------------------------------------------------------------------

MP Config Base Table Entries:

--
Processors: APIC ID Version State Family Model Step Flags
0 0x15 BSP, usable 6 15 1 0xbfebfbff
2 0x15 AP, usable 6 15 1 0xbfebfbff
4 0x15 AP, usable 6 15 1 0xbfebfbff
6 0x15 AP, usable 6 15 1 0xbfebfbff
8 0x15 AP, usable 6 15 1 0xbfebfbff
10 0x15 AP, usable 6 15 1 0xbfebfbff
16 0x15 AP, usable 6 15 1 0xbfebfbff
18 0x15 AP, usable 6 15 1 0xbfebfbff
20 0x15 AP, usable 6 15 1 0xbfebfbff
22 0x15 AP, usable 6 15 1 0xbfebfbff
24 0x15 AP, usable 6 15 1 0xbfebfbff
26 0x15 AP, usable 6 15 1 0xbfebfbff
32 0x15 AP, usable 6 15 1 0xbfebfbff
34 0x15 AP, usable 6 15 1 0xbfebfbff
36 0x15 AP, usable 6 15 1 0xbfebfbff
38 0x15 AP, usable 6 15 1 0xbfebfbff
40 0x15 AP, usable 6 15 1 0xbfebfbff
42 0x15 AP, usable 6 15 1 0xbfebfbff
48 0x15 AP, usable 6 15 1 0xbfebfbff
50 0x15 AP, usable 6 15 1 0xbfebfbff
52 0x15 AP, usable 6 15 1 0xbfebfbff
54 0x15 AP, usable 6 15 1 0xbfebfbff
56 0x15 AP, usable 6 15 1 0xbfebfbff
58 0x15 AP, usable 6 15 1 0xbfebfbff

> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
> set based on the current CPU's Local APIC ID. I suspect then that this
> information is incorrect in the failing case.
>
> Slawa,
> my guess can be checked by adding a printf to cpu_mp_start() right after
> boot_cpu_id assignment.

System now in early production and I can't be reboot often.

[Konstantin Belousov]

On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:

Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
are initialized. BIOS might reprogram LAPIC Ids, so reading from
CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
it might be in the x2APIC state, since 8 bits are not enough for 32bit
x2APIC Id.

[Andriy Gapon]

On 04/09/2016 18:14, Konstantin Belousov wrote:
> On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
>> Kostik, I see one strange thing which is common to both successful and
>> unsuccessful configurations. All "SMP: Added CPU..." lines have "AP" in them.
>> It seems like the platform does not tell explicitly tell which CPU is the BSP,
>> see cpu_add() function. This can break quite a few assumption. And I am not
>> even sure how the successful scenario works.
>> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
>> set based on the current CPU's Local APIC ID. I suspect then that this
>> information is incorrect in the failing case.
>>
> Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
> are initialized. BIOS might reprogram LAPIC Ids, so reading from
> CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
> it might be in the x2APIC state, since 8 bits are not enough for 32bit
> x2APIC Id.

Hmm, I am not sure how what you are saying here is relevant to the problem.
I believe that cpu_mp_start() is executed (on the BSP) after the BSP's LAPIC is
initialized. So, the code should just work.

My theory was that the BSP's LAPIC ID was incorrectly programmed by BIOS (firmware).

[Konstantin Belousov]

On Sun, Sep 04, 2016 at 06:49:43PM +0300, Andriy Gapon wrote:
> On 04/09/2016 18:14, Konstantin Belousov wrote:
> > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> >> Kostik, I see one strange thing which is common to both successful and
> >> unsuccessful configurations. All "SMP: Added CPU..." lines have "AP" in them.
> >> It seems like the platform does not tell explicitly tell which CPU is the BSP,
> >> see cpu_add() function. This can break quite a few assumption. And I am not
> >> even sure how the successful scenario works.
> >> Ah... I see that there is a backup code in cpu_mp_start() where boot_cpu_id is
> >> set based on the current CPU's Local APIC ID. I suspect then that this
> >> information is incorrect in the failing case.
> >>
> > Well, there is no easy way to read the LAPIC Id of BSP before LAPICs
> > are initialized. BIOS might reprogram LAPIC Ids, so reading from
> > CPUID[1].EBX[31:24] might return incorrect data. Even more incorrect
> > it might be in the x2APIC state, since 8 bits are not enough for 32bit
> > x2APIC Id.
>
> Hmm, I am not sure how what you are saying here is relevant to the problem.
> I believe that cpu_mp_start() is executed (on the BSP) after the BSP's LAPIC is
> initialized. So, the code should just work.

The order is madt_probe()->madt_probe_cpus()->madt_setup_local().

madt_probe() and madt_probe_cpus() are called from apic_init9) at
SI_SUB_TUNABLEs, and madt_probe_cpus()->madt_add_cpu()->lapic_create()->
cpu_add() is how the SMP: ... lines are printed.

The madt_setup_local() code is called from apic_setup_local() at SI_SUB_CPU,
this is where APIC window is mapped or x2APIC mode is enabled by the call
to native_lapic_init().

You cannot get BSP LAPIC Id earlier than native_lapic_init() was executed.

mp_start()->cpu_mp_start() is called right after the madt_setup_local().

>
> My theory was that the BSP's LAPIC ID was incorrectly programmed by BIOS (firmware).

This is possible, of course. But it would not affect "SMP: Added CPU ..."
lines.

[Andriy Gapon]

On 04/09/2016 19:29, Konstantin Belousov wrote:
> This is possible, of course. But it would not affect "SMP: Added CPU ..."
> lines.

Well, looking at the code it seems that only if mptable is used, then those
lines are expected to correctly identify a BSP. With MADT there is no
information to identify the BSP and that is supposed to happen in cpu_mp_start().


static void
madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
{
struct lapic_info *la;

/*
* The MADT does not include a BSP flag, so we have to let the
* MP code figure out which CPU is the BSP on its own.
*/
..

In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
problematic configuration.

--
Andriy Gapon

[Slawa Olhovchenkov]

On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:

> On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > DMAR: Found table at 0x79b32798
> > x2APIC available but disabled by DMAR table
>
> > Event timer "LAPIC" quality 600
> > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > ACPI APIC Table: <ALASKA A M I >

Is this significant? On carsh case present 'x2APIC available but
disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
116268019 tsc 2200043851)'

On successful boot both messages absent:

Table 'DMAR' at 0x79b32798

DMAR: Found table at 0x79b32798

Event timer "LAPIC" quality 600

ACPI APIC Table: <ALASKA A M I >

[Konstantin Belousov]

On Tue, Sep 06, 2016 at 04:13:05PM +0300, Slawa Olhovchenkov wrote:
> On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
>
> > On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > > DMAR: Found table at 0x79b32798
> > > x2APIC available but disabled by DMAR table
> >
> > > Event timer "LAPIC" quality 600
> > > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > > ACPI APIC Table: <ALASKA A M I >
>
> Is this significant? On carsh case present 'x2APIC available but
> disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
> 116268019 tsc 2200043851)'

Significant for what ? First message mostly repeat the setting in the
BIOS which you frobbed. The second message indicates that xAPIC
timings for ICR reads were calibrated, i.e. x2APIC mode was indeed
disabled.

[Slawa Olhovchenkov]

On Tue, Sep 06, 2016 at 05:08:28PM +0300, Konstantin Belousov wrote:

> On Tue, Sep 06, 2016 at 04:13:05PM +0300, Slawa Olhovchenkov wrote:
> > On Sun, Sep 04, 2016 at 11:19:16AM +0300, Andriy Gapon wrote:
> >
> > > On 01/09/2016 15:13, Slawa Olhovchenkov wrote:
> > > > DMAR: Found table at 0x79b32798
> > > > x2APIC available but disabled by DMAR table
> > >
> > > > Event timer "LAPIC" quality 600
> > > > LAPIC: ipi_wait() us multiplier 1 (r 116268019 tsc 2200043851)
> > > > ACPI APIC Table: <ALASKA A M I >
> >
> > Is this significant? On carsh case present 'x2APIC available but
> > disabled by DMAR table' and 'LAPIC: ipi_wait() us multiplier 1 (r
> > 116268019 tsc 2200043851)'
> Significant for what ? First message mostly repeat the setting in the

For cause of crash.
May be not all of code correct understund this combinations of setting?

> BIOS which you frobbed. The second message indicates that xAPIC
> timings for ICR reads were calibrated, i.e. x2APIC mode was indeed
> disabled.

OK

[Slawa Olhovchenkov]

On Sun, Sep 04, 2016 at 08:14:07PM +0300, Andriy Gapon wrote:

> On 04/09/2016 19:29, Konstantin Belousov wrote:
> > This is possible, of course. But it would not affect "SMP: Added CPU ..."
> > lines.
>
> Well, looking at the code it seems that only if mptable is used, then those
> lines are expected to correctly identify a BSP. With MADT there is no
> information to identify the BSP and that is supposed to happen in cpu_mp_start().
>
>
> static void
> madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
> {
> struct lapic_info *la;
>
> /*
> * The MADT does not include a BSP flag, so we have to let the
> * MP code figure out which CPU is the BSP on its own.
> */

> ...

>
> In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
> And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
> problematic configuration.

For next day or two I am have new server with same hardware before put
in prodution.
Can I do some test for you?

[Andriy Gapon]

On 12/09/2016 12:39, Slawa Olhovchenkov wrote:
> On Sun, Sep 04, 2016 at 08:14:07PM +0300, Andriy Gapon wrote:
>
>> On 04/09/2016 19:29, Konstantin Belousov wrote:
>>> This is possible, of course. But it would not affect "SMP: Added CPU ..."
>>> lines.
>>
>> Well, looking at the code it seems that only if mptable is used, then those
>> lines are expected to correctly identify a BSP. With MADT there is no
>> information to identify the BSP and that is supposed to happen in cpu_mp_start().
>>
>>
>> static void
>> madt_add_cpu(u_int acpi_id, u_int apic_id, u_int flags)
>> {
>> struct lapic_info *la;
>>
>> /*
>> * The MADT does not include a BSP flag, so we have to let the
>> * MP code figure out which CPU is the BSP on its own.
>> */
>> ...
>>
>> In other words, those "SMP: Added CPU ..." are truly a cosmetic issue.
>> And it's my guess (just a guess) that BSP LAPIC ID is incorrect in the
>> problematic configuration.
>
> For next day or two I am have new server with same hardware before put
> in prodution.
> Can I do some test for you?
>

From my earlier email:

"my guess can be checked by adding a printf to cpu_mp_start() right after
boot_cpu_id assignment".

--
Andriy Gapon

[Slawa Olhovchenkov]

I am not kernel developer: please point what I am need insert and file
for edit.

[Andriy Gapon]

On 12/09/2016 19:44, Slawa Olhovchenkov wrote:
> I am not kernel developer: please point what I am need insert and file
> for edit.

In sys/amd64/amd64/mp_machdep.c (assuming you use amd64), in function
cpu_mp_start, in this place
/* Set boot_cpu_id if needed. */
if (boot_cpu_id == -1) {
boot_cpu_id = PCPU_GET(apic_id);
cpu_info[boot_cpu_id].cpu_bsp = 1;
} else

right after boot_cpu_id = PCPU_GET(apic_id) line please insert
printf("boot_cpu_id = %d\n", boot_cpu_id);

--
Andriy Gapon

[Slawa Olhovchenkov]

On Mon, Sep 12, 2016 at 08:05:33PM +0300, Andriy Gapon wrote:

> On 12/09/2016 19:44, Slawa Olhovchenkov wrote:
> > I am not kernel developer: please point what I am need insert and file
> > for edit.
>
> In sys/amd64/amd64/mp_machdep.c (assuming you use amd64), in function
> cpu_mp_start, in this place
> /* Set boot_cpu_id if needed. */
> if (boot_cpu_id == -1) {
> boot_cpu_id = PCPU_GET(apic_id);
> cpu_info[boot_cpu_id].cpu_bsp = 1;
> } else
>
> right after boot_cpu_id = PCPU_GET(apic_id) line please insert
> printf("boot_cpu_id = %d\n", boot_cpu_id);

VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)

Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
XSAVE Features=0x1<XSAVEOPT>

VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

TSC: P-state invariant, performance statistics

real memory = 137438953472 (131072 MB)

avail memory = 133407973376 (127227 MB)

Event timer "LAPIC" quality 600

ACPI APIC Table: <ALASKA A M I >

boot_cpu_id = 255

kernel trap 12 with interrupts disabled


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = ff
fault virtual address = 0x0
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80537e74

stack pointer = 0x28:0xffffffff814b3a60
frame pointer = 0x28:0xffffffff814b3a70

code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 0 ()
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff805272e7 at kdb_backtrace+0x67
#1 0xffffffff804dd662 at vpanic+0x182
#2 0xffffffff804dd4d3 at panic+0x43

#3 0xffffffff807a37a1 at trap_fatal+0x351
#4 0xffffffff807a3993 at trap_pfault+0x1e3
#5 0xffffffff807a2f1c at trap+0x26c
#6 0xffffffff80787ca1 at calltrap+0x8
#7 0xffffffff8083b53a at topo_probe+0x61a
#8 0xffffffff8078fe93 at cpu_mp_start+0x1c3

#9 0xffffffff805382ca at mp_start+0x3a
#10 0xffffffff80465cd8 at mi_startup+0x118
#11 0xffffffff8028dfac at btext+0x2c
Uptime: 1s

[Andriy Gapon]

On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> boot_cpu_id = 255

I think that this points towards the BIOS not configuring the BSP LAPIC
correctly when you select that combination of BIOS options.
That's weird, but I fail to find any other explanation.

If wonder what would happen if you disable X2APIC_OPT_OUT and set
hw.x2apic_enable=0 in loader.conf or at the loader prompt.

--
Andriy Gapon

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:

> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> > boot_cpu_id = 255
>
> I think that this points towards the BIOS not configuring the BSP LAPIC
> correctly when you select that combination of BIOS options.
> That's weird, but I fail to find any other explanation.
>
> If wonder what would happen if you disable X2APIC_OPT_OUT and set
> hw.x2apic_enable=0 in loader.conf or at the loader prompt.

Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.

[Andriy Gapon]

On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
>
>> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
>>> boot_cpu_id = 255
>>
>> I think that this points towards the BIOS not configuring the BSP LAPIC
>> correctly when you select that combination of BIOS options.
>> That's weird, but I fail to find any other explanation.
>>
>> If wonder what would happen if you disable X2APIC_OPT_OUT and set
>> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
>
> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
>

This doesn't answer my question and doesn't add any new information, correct?

--
Andriy Gapon

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:

> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> > On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
> >
> >> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> >>> boot_cpu_id = 255
> >>
> >> I think that this points towards the BIOS not configuring the BSP LAPIC
> >> correctly when you select that combination of BIOS options.
> >> That's weird, but I fail to find any other explanation.
> >>
> >> If wonder what would happen if you disable X2APIC_OPT_OUT and set
> >> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
> >
> > Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
> >
>
> This doesn't answer my question and doesn't add any new information, correct?

I am don't see any question. I am see only suggestion of workaround.
Sorry if missunderstund.
You need some additional debug print?

[Andriy Gapon]

On 13/09/2016 15:42, Slawa Olhovchenkov wrote:
> On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:
>
>> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
>>> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
>>>
>>>> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
>>>>> boot_cpu_id = 255
>>>>
>>>> I think that this points towards the BIOS not configuring the BSP LAPIC
>>>> correctly when you select that combination of BIOS options.
>>>> That's weird, but I fail to find any other explanation.
>>>>
>>>> If wonder what would happen if you disable X2APIC_OPT_OUT and set
>>>> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
>>>
>>> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
>>>
>>
>> This doesn't answer my question and doesn't add any new information, correct?
>
> I am don't see any question. I am see only suggestion of workaround.
> Sorry if missunderstund.
> You need some additional debug print?
>

I already knew from you that not enabling X2APIC_OPT_OUT fixed the problem.

"If wonder what would happen if you disable X2APIC_OPT_OUT and set
hw.x2apic_enable=0 in loader.conf or at the loader prompt."

This was a question even though there is no question mark.


--
Andriy Gapon

[Slawa Olhovchenkov]

On Tue, Sep 13, 2016 at 03:57:39PM +0300, Andriy Gapon wrote:

> On 13/09/2016 15:42, Slawa Olhovchenkov wrote:
> > On Tue, Sep 13, 2016 at 03:38:17PM +0300, Andriy Gapon wrote:
> >
> >> On 13/09/2016 15:11, Slawa Olhovchenkov wrote:
> >>> On Tue, Sep 13, 2016 at 03:04:13PM +0300, Andriy Gapon wrote:
> >>>
> >>>> On 12/09/2016 20:53, Slawa Olhovchenkov wrote:
> >>>>> boot_cpu_id = 255
> >>>>
> >>>> I think that this points towards the BIOS not configuring the BSP LAPIC
> >>>> correctly when you select that combination of BIOS options.
> >>>> That's weird, but I fail to find any other explanation.
> >>>>
> >>>> If wonder what would happen if you disable X2APIC_OPT_OUT and set
> >>>> hw.x2apic_enable=0 in loader.conf or at the loader prompt.
> >>>
> >>> Boot OK w/o hw.x2apic_enable=0, only with disable X2APIC_OPT_OUT.
> >>>
> >>
> >> This doesn't answer my question and doesn't add any new information, correct?
> >
> > I am don't see any question. I am see only suggestion of workaround.
> > Sorry if missunderstund.
> > You need some additional debug print?
> >
>
> I already knew from you that not enabling X2APIC_OPT_OUT fixed the problem.
> "If wonder what would happen if you disable X2APIC_OPT_OUT and set
> hw.x2apic_enable=0 in loader.conf or at the loader prompt."
> This was a question even though there is no question mark.

boot failed:

set hw.x2apic_enable=0
loading required module 'krpc'
/boot/kernel.VSTREAM/krpc.ko size 0x2a210 at 0x134e000
loading required module 'opensolaris'
^@/boot/kernel.VSTREAM/opensolaris.ko size 0xadb8 at 0x1379000
/boot/kernel.VSTREAM/if_igb.ko size 0x69f10 at 0x1384000
can't find 'if_ixgbe'
/boot/kernel.VSTREAM/if_lagg.ko size 0x150c0 at 0x13ee000^M ^@
+/boot/kernel.VSTREAM/ukbd.ko size 0xe128 at 0x1404000
loading required module 'usb'
/boot/kernel.VSTREAM/usb.ko size 0x458b0 at 0x1413000^M|
/boot/kernel.VSTREAM/umass.ko size 0xaa10 at 0x1459000
/boot/kernel.VSTREAM/accf_http.ko size 0x2710 at 0x1464000
/boot/kernel.VSTREAM/uhci.ko size 0xd508 at 0x1467000
/boot/kernel.VSTREAM/ohci.ko size 0xc9d0 at 0x1475000^M
/boot/kernel.VSTREAM/ehci.ko size 0xfc40 at 0x1482000
/boot/kernel.VSTREAM/xhci.ko size 0x11068 at 0x1492000
/boot/kernel.VSTREAM/cc_htcp.ko size 0x3a70 at 0x14a4000
Booting...
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p305117 #0: Mon Sep 12 20:38:53 MSK 2016
s...@edge21.int.integros.com:/usr/obj/usr/src/sys/VSTREAM amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU)

[Andriy Gapon]

Thank you!
It seems like exactly the same behavior that happens when you toggle that BIOS
option.

My theory is that in both cases, hw.x2apic_enable=0 and X2APIC_OPT_OUT is on,
the BIOS turns on x2APIC mode and transitions to OS in that mode.
In the case when X2APIC_OPT_OUT is on it's clearly a BIOS bug.
But maybe we could do a little bit better in both cases. At the very least we
could detect the situation and panic with a helpful message (e.g. "x2APIC mode
is disabled but turn on by BIOS"). Perhaps we could even try to downgrade to
xAPIC mode.

--
Andriy Gapon

[Slawa Olhovchenkov]

Goggling on X2APIC_OPT_OUT take same result: other OS in this case
downgrade to xAPIC mode.

[Konstantin Belousov]

On Tue, Sep 13, 2016 at 05:54:26PM +0300, Andriy Gapon wrote:

If hw.x2apic_enable=0 and machine booted to the stage of topo probe,
BIOS definitely did not made hand-off with x2APIC enabled. Any access
to the LAPIC registers page in x2APIC mode faults.

And X2APIC_OPT_OUT also does not result in x2APIC mode hand-off, for the
same reason. System would panic much earlier, while dmesg indicates that
LAPIC ICR calibration was succesful.

It is invalid LAPIC Id or a bug in topo code or combination of issues.

[Andriy Gapon]

On 13/09/2016 18:22, Konstantin Belousov wrote:
> Any access
> to the LAPIC registers page in x2APIC mode faults.

Is this a fact?
I read the following in the specification:

In x2APIC mode, the memory mapped interface is not available and any
access to the MMIO interface will behave similar to that of a legacy xAPIC
in globally disabled state.

But I couldn't find what actually happens for the legacy xAPIC in globally
disabled state. For AMD processors it is documented that if xAPIC is disabled
then accessing the APIC memory range works the same as accessing the regular
memory. That can be different for Intel processors, of course.

--
Andriy Gapon

[Konstantin Belousov]

On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
> On 13/09/2016 18:22, Konstantin Belousov wrote:
> > Any access
> > to the LAPIC registers page in x2APIC mode faults.
>
> Is this a fact?
> I read the following in the specification:
>
> In x2APIC mode, the memory mapped interface is not available and any
> access to the MMIO interface will behave similar to that of a legacy xAPIC
> in globally disabled state.
>
> But I couldn't find what actually happens for the legacy xAPIC in globally
> disabled state. For AMD processors it is documented that if xAPIC is disabled
> then accessing the APIC memory range works the same as accessing the regular
> memory. That can be different for Intel processors, of course.

Look at the native_lapic_init(), very beginning of the function. If
x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
attributes are changed.

Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
because no LAPIC accesses are done until native_lapic_enable_x2apic() is
called. This just happens, I thought about more organized receipt of the
current LAPIC mode. Issue is that decision for LAPIC mode is performed
by madt enumerator which pref. should not read LAPIC config (too early).
And it is not clear at all, what to do if there is reason to use xAPIC,
as checked by madt_setup_local(), but we are in x2APIC mode already.

Anyway, returning to the original issue, I do not believe that the
hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
is specified. Kernel would fault in that case.

[Slawa Olhovchenkov]

As I assume, other OS don't fault in this case.

[Andriy Gapon]

On 14/09/2016 14:36, Konstantin Belousov wrote:
> On Tue, Sep 13, 2016 at 06:52:19PM +0300, Andriy Gapon wrote:
>> On 13/09/2016 18:22, Konstantin Belousov wrote:
>>> Any access
>>> to the LAPIC registers page in x2APIC mode faults.
>>
>> Is this a fact?
>> I read the following in the specification:
>>
>> In x2APIC mode, the memory mapped interface is not available and any
>> access to the MMIO interface will behave similar to that of a legacy xAPIC
>> in globally disabled state.
>>
>> But I couldn't find what actually happens for the legacy xAPIC in globally
>> disabled state. For AMD processors it is documented that if xAPIC is disabled
>> then accessing the APIC memory range works the same as accessing the regular
>> memory. That can be different for Intel processors, of course.
>
> Look at the native_lapic_init(), very beginning of the function. If
> x2apic_mode is non-zero, lapic_map is cleared after DMAP page cache
> attributes are changed.

Right, but we are talking about the case where x2apic_mode *is zero* while the
hardware in x2APIC mode.

> Right now, if BIOS pass control to OS in x2APIC mode, thinks just work
> because no LAPIC accesses are done until native_lapic_enable_x2apic() is
> called. This just happens, I thought about more organized receipt of the
> current LAPIC mode. Issue is that decision for LAPIC mode is performed
> by madt enumerator which pref. should not read LAPIC config (too early).
> And it is not clear at all, what to do if there is reason to use xAPIC,
> as checked by madt_setup_local(), but we are in x2APIC mode already.

Yes. As I mentioned earlier we should at least panic with an informative
message. Maybe we could add some code to so something smarter later.

> Anyway, returning to the original issue, I do not believe that the
> hand-off on that machine occured in the x2APIC mode when X2APIC_OPT_OUT
> is specified. Kernel would fault in that case.

I think that it should be easy to check that if Slawa is still willing to try
another diagnostic patch.

diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
index 203e9d00e8acc..37ac03fb9d811 100644
--- a/sys/x86/x86/local_apic.c
+++ b/sys/x86/x86/local_apic.c
@@ -429,6 +429,12 @@ native_lapic_init(vm_paddr_t addr)
if (x2apic_mode) {
native_lapic_enable_x2apic();
lapic_map = NULL;
+ } else if ((cpu_feature2 & CPUID2_X2APIC) != 0) {
+ r = rdmsr(MSR_APICBASE);
+ printf("MSR_APICBASE = 0x%16jx\n", (uintmax_t)r);
+ if ((r & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
+ (APICBASE_X2APIC | APICBASE_ENABLED))
+ printf("x2APIC is prohibited but turned on by BIOS\n");
}

/* Setup the spurious interrupt handler. */



--
Andriy Gapon

[Andriy Gapon]

On 13/09/2016 17:59, Slawa Olhovchenkov wrote:
> Goggling on X2APIC_OPT_OUT take same result: other OS in this case
> downgrade to xAPIC mode.

It still does not make any sense for BIOS to turn on x2APIC and then instruct
the OS that x2APIC must not be used.

--
Andriy Gapon

[Slawa Olhovchenkov]

What combination of BIOS setting need?

[Andriy Gapon]

The one that causes the crash.

[Slawa Olhovchenkov]

VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)

Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
XSAVE Features=0x1<XSAVEOPT>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)

MSR_APICBASE = 0x fee00d00

x2APIC is prohibited but turned on by BIOS

#7 0xffffffff8083b57a at topo_probe+0x61a

#8 0xffffffff8078fe93 at cpu_mp_start+0x1c3
#9 0xffffffff805382ca at mp_start+0x3a
#10 0xffffffff80465cd8 at mi_startup+0x118
#11 0xffffffff8028dfac at btext+0x2c
Uptime: 1s

[Andriy Gapon]

On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> MSR_APICBASE = 0x fee00d00
> x2APIC is prohibited but turned on by BIOS

Kostik, ^^^^^

P.S. the format string for the value should have been 0x%016jx.

[Konstantin Belousov]

On Wed, Sep 14, 2016 at 05:02:21PM +0300, Andriy Gapon wrote:
> On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> > MSR_APICBASE = 0x fee00d00
> > x2APIC is prohibited but turned on by BIOS
>
> Kostik, ^^^^^

Well, the following might work, but I have no good idea what to do
when BIOS does handoff with x2APIC enabled and directs us to not
enable it. Switching to xAPIC mode is not an option, I suspect.

diff --git a/sys/x86/acpica/madt.c b/sys/x86/acpica/madt.c
index 241a769..3a93fd6 100644
--- a/sys/x86/acpica/madt.c
+++ b/sys/x86/acpica/madt.c
@@ -138,7 +138,6 @@ madt_setup_local(void)

madt = pmap_mapbios(madt_physaddr, madt_length);

if ((cpu_feature2 & CPUID2_X2APIC) != 0) {

- x2apic_mode = 1;
reason = NULL;

/*
@@ -150,21 +149,17 @@ madt_setup_local(void)
if (dmartbl_physaddr != 0) {
dmartbl = acpi_map_table(dmartbl_physaddr,
ACPI_SIG_DMAR);
- if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0) {
- x2apic_mode = 0;
+ if ((dmartbl->Flags & ACPI_DMAR_X2APIC_OPT_OUT) != 0)
reason = "by DMAR table";
- }
acpi_unmap_table(dmartbl);
}
if (vm_guest == VM_GUEST_VMWARE) {
vmware_hvcall(VMW_HVCMD_GETVCPU_INFO, p);
if ((p[0] & VMW_VCPUINFO_VCPU_RESERVED) != 0 ||
- (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0) {
- x2apic_mode = 0;
- reason = "inside VMWare without intr redirection";
- }
+ (p[0] & VMW_VCPUINFO_LEGACY_X2APIC) == 0)
+ reason =
+ "inside VMWare without intr redirection";
} else if (vm_guest == VM_GUEST_XEN) {
- x2apic_mode = 0;
reason = "due to running under XEN";
} else if (vm_guest == VM_GUEST_NO &&
CPUID_TO_FAMILY(cpu_id) == 0x6 &&
@@ -184,13 +179,21 @@ madt_setup_local(void)
if (!strcmp(hw_vendor, "LENOVO") ||
!strcmp(hw_vendor,
"ASUSTeK Computer Inc.")) {
- x2apic_mode = 0;
reason =
"for a suspected SandyBridge BIOS bug";
}
freeenv(hw_vendor);
}
}
+ if (reason != NULL && lapic_is_x2apic()) {
+ if (bootverbose)
+ printf("x2APIC should be disabled %s but "
+ "already enabled by BIOS; enabling.\n",
+ reason);
+ reason = NULL;
+ }
+ if (reason == NULL)
+ x2apic_mode = 1;
TUNABLE_INT_FETCH("hw.x2apic_enable", &x2apic_mode);
if (!x2apic_mode && reason != NULL && bootverbose)
printf("x2APIC available but disabled %s\n", reason);
diff --git a/sys/x86/include/apicvar.h b/sys/x86/include/apicvar.h
index 1ddb69e..09c3a63 100644
--- a/sys/x86/include/apicvar.h
+++ b/sys/x86/include/apicvar.h
@@ -206,6 +206,7 @@ struct apic_ops {
void (*create)(u_int, int);
void (*init)(vm_paddr_t);
void (*xapic_mode)(void);
+ bool (*is_x2apic)(void);
void (*setup)(int);
void (*dump)(const char *);
void (*disable)(void);
@@ -268,6 +269,13 @@ lapic_xapic_mode(void)
apic_ops.xapic_mode();
}

+static inline bool
+lapic_is_x2apic(void)
+{
+
+ return (apic_ops.is_x2apic());
+}
+
static inline void
lapic_setup(int boot)
{
diff --git a/sys/x86/x86/local_apic.c b/sys/x86/x86/local_apic.c
index cd774df..d9a3453 100644
--- a/sys/x86/x86/local_apic.c
+++ b/sys/x86/x86/local_apic.c
@@ -269,6 +269,16 @@ native_lapic_enable_x2apic(void)
wrmsr(MSR_APICBASE, apic_base);
}

+static bool
+native_lapic_is_x2apic(void)
+{
+ uint64_t apic_base;
+
+ apic_base = rdmsr(MSR_APICBASE);
+ return ((apic_base & (APICBASE_X2APIC | APICBASE_ENABLED)) ==
+ (APICBASE_X2APIC | APICBASE_ENABLED));
+}
+
static void lapic_enable(void);
static void lapic_resume(struct pic *pic, bool suspend_cancelled);
static void lapic_timer_oneshot(struct lapic *);
@@ -329,6 +339,7 @@ struct apic_ops apic_ops = {
.create = native_lapic_create,
.init = native_lapic_init,
.xapic_mode = native_lapic_xapic_mode,
+ .is_x2apic = native_lapic_is_x2apic,
.setup = native_lapic_setup,
.dump = native_lapic_dump,
.disable = native_lapic_disable,
diff --git a/sys/x86/xen/xen_apic.c b/sys/x86/xen/xen_apic.c
index 4d7a39b..45c3c18 100644
--- a/sys/x86/xen/xen_apic.c
+++ b/sys/x86/xen/xen_apic.c
@@ -139,6 +139,13 @@ xen_pv_lapic_disable(void)

}

+static bool
+xen_pv_lapic_is_x2apic(void)
+{
+
+ return (false);
+}
+
static void
xen_pv_lapic_eoi(void)
{
@@ -351,6 +358,7 @@ struct apic_ops xen_apic_ops = {
.create = xen_pv_lapic_create,
.init = xen_pv_lapic_init,
.xapic_mode = xen_pv_lapic_disable,
+ .is_x2apic = xen_pv_lapic_is_x2apic,
.setup = xen_pv_lapic_setup,
.dump = xen_pv_lapic_dump,
.disable = xen_pv_lapic_disable,

[Andriy Gapon]

On 14/09/2016 19:08, Konstantin Belousov wrote:
> Well, the following might work, but I have no good idea what to do
> when BIOS does handoff with x2APIC enabled and directs us to not
> enable it. Switching to xAPIC mode is not an option, I suspect.

The specification describes a way to transition from x2APIC to xAPIC mode, but
it's not trivial, because it requires checking all CPU IDs and disabling CPUs
with too high ones.

Personally, I am not comfortable with your patch. First, it seems that the
patch does not cover hw.x2apic_enable=0. And I don't like that we leave x2APIC
mode enabled even when we otherwise would not enable it just because BIOS.

All in all, if we can't go x2APIC -> xAPIC, then I would prefer a meaningful
panic over a "compromise".

Just my opinion. The topic probably warrants a wider discussion and a proper
review.

Slawa,
thank you for your persistence and testing.

--
Andriy Gapon

[Slawa Olhovchenkov]

On Wed, Sep 14, 2016 at 07:08:02PM +0300, Konstantin Belousov wrote:

> On Wed, Sep 14, 2016 at 05:02:21PM +0300, Andriy Gapon wrote:
> > On 14/09/2016 15:49, Slawa Olhovchenkov wrote:
> > > MSR_APICBASE = 0x fee00d00
> > > x2APIC is prohibited but turned on by BIOS
> >
> > Kostik, ^^^^^
>
> Well, the following might work, but I have no good idea what to do
> when BIOS does handoff with x2APIC enabled and directs us to not
> enable it. Switching to xAPIC mode is not an option, I suspect.

CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x21cbfbb<FSGSBASE,TSCADJ,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,RDSEED,ADX,SMAP,PROCTRACE>
XSAVE Features=0x1<XSAVEOPT>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 137438953472 (131072 MB)
avail memory = 133407973376 (127227 MB)

Event timer "LAPIC" quality 600
ACPI APIC Table: <ALASKA A M I >

boot_cpu_id = 0
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 12 core(s)
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
random: entropy device external interface
module_register_init: MOD_LOAD (vesa, 0xffffffff807bf2f0, 0) error 19
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
netmap: loaded module
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <ALASKA A M I > on motherboard
acpi0: Power Button (fixed)
[...]

boot OK, w/o any APIC-related messages