3.4-rc smpboot: do_boot_cpu failed(-1) to wakeup CPU#1

[shrybman]

All 3.4-rc kernels I have tried displayed this issue.


This report looks similar https://lkml.org/lkml/2015/9/26/165.
I also have a Core 2 cpu on a Asus P5B (not deluxe) board.



..
 ACPI: Core revision 20150818
 ACPI: 1 ACPI AML tables successfully acquired and loaded
 Mount-cache hash table entries: 8192 (order: 4, 65536 bytes)
 Mountpoint-cache hash table entries: 8192 (order: 4, 65536 bytes)
 CPU: Physical Processor ID: 0
 CPU: Processor Core ID: 0
 mce: CPU supports 6 MCE banks
 CPU0: Thermal monitoring enabled (TM2)
 process: using mwait in idle threads
 Last level iTLB entries: 4KB 128, 2MB 4, 4MB 4
 Last level dTLB entries: 4KB 256, 2MB 0, 4MB 32, 1GB 0
 Freeing SMP alternatives memory: 20K (ffffffff819b5000 - ffffffff819ba000)
 ftrace: allocating 18583 entries in 73 pages
 ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
 smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
 Performance Events: PEBS fmt0-, 4-deep LBR, Core2 events, Intel PMU driver.
 perf_event_intel: PEBS disabled due to CPU errata
 ... version:                2
 ... bit width:              40
 ... generic registers:      2
 ... value mask:             000000ffffffffff
 ... max period:             000000007fffffff
 ... fixed-purpose events:   3
 ... event mask:             0000000700000003
 x86: Booting SMP configuration:
 .... node  #0, CPUs:      #1
 smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
 x86: Booted up 1 node, 1 CPUs
 smpboot: Total of 1 processors activated (4272.42 BogoMIPS)
 devtmpfs: initialized


I took a guess and got lucky.


Reverting "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted" resolved the issue.


Regards,


Shane

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

[shrybman]

> All 3.4-rc kernels I have tried displayed this issue.

Stupid typo, that should be kernel v4.3-rc.

[Donald Parsons]

On Wed, 2015-10-14 at 12:20 -0400, shrybman wrote:
> All 3.4-rc kernels I have tried displayed this issue.

> This report looks similar https://lkml.org/lkml/2015/9/26/165.
> I also have a Core 2 cpu on a Asus P5B (not deluxe) board.

> ...

As a followup, I updated compiler to gcc-5.2 (no help or change).
I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:
"cpu_init_udelay=10000"
which did not help getting missing CPU back online.

So all linux-4.3.0-rc[15] lose the second CPU core.

Don

[shrybman]

.snip..

> >  smpboot: Total of 1 processors activated (4272.42 BogoMIPS)
> >  devtmpfs: initialized
> > I took a guess and got lucky.
> > Reverting "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted" resolved the issue.
>
>
> As a followup, I updated compiler to gcc-5.2 (no help or change).
> I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:
>        "cpu_init_udelay=10000"
> which did not help getting missing CPU back online.

Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert
and atomic init_deasserted"  patch?

[Brown, Len]

Donald, Shane,
Thanks for reporting this.

> > I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:

> >        "cpu_init_udelay=10000"

> > which did not help getting missing CPU back online.

right, if the issue is caused by the patch below,
that cmdline will not help.

> Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert

> and atomic init_deasserted"  patch?

Yes, please let me know if reverting that patch helps you too.

You have similar hardware:

Shane:

smpboot: CPU0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (fam: 06, model: 0f, stepping: 06)

Donald:

CPU : Intel Core 2 CPU 6600 @ 2.4GHz

I think I can get ahold of a core2 6xxx box tomorrow.
Please send me your .config directly, and I'll see if I can reproduce the issue.

thanks,
-Len

N‹§²æìr¸›yúèšØb²X¬¶Ç§vØ^–)Þº{.nÇ+‰·¥Š{±‘êçzX§¶ ›¡Ü¨}©ž²Æ zÚ&j:+v‰¨¾ «‘êçzZ+€Ê+zf£¢·hšˆ§~†­†Ûiÿûàz¹ ®w¥¢¸?™¨è­Ú&¢)ߢ f”ù^jÇ«y§m…á@A«a¶Ú ÿ0¶ìh® å’i

[Donald Parsons]

On Thu, 2015-10-15 at 02:09 +0000, Brown, Len wrote:
> Donald, Shane,
> Thanks for reporting this.
>
> > > I also tried suggested /vmlinuz-4.3.0-rc3 parameter in grub:

> > > "cpu_init_udelay=10000"
> > > which did not help getting missing CPU back online.
>
> right, if the issue is caused by the patch below,
> that cmdline will not help.
>
> > Did you try reverting the "x86/smpboot: Remove APIC.wait_for_init_deassert

> > and atomic init_deasserted" patch?

>
> Yes, please let me know if reverting that patch helps you too.

How? Please send a patch or git cmd(s). I have the
git/stable/linux-stable.git on my PC. Thanks.

> You have similar hardware:
>
> Shane:
>
> smpboot: CPU0: Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
>
> Donald:
>
> CPU : Intel Core 2 CPU 6600 @ 2.4GHz

Mine is also same (fam: 06, model: 0f, stepping: 06).

> I think I can get ahold of a core2 6xxx box tomorrow.
> Please send me your .config directly, and I'll see if I can reproduce the issue.

Will do.

Don

[Brown, Len]

> > > Did you try reverting the "x86/smpboot: Remove
> APIC.wait_for_init_deassert
> > > and atomic init_deasserted" patch?
> >
> > Yes, please let me know if reverting that patch helps you too.
>
> How? Please send a patch or git cmd(s). I have the
> git/stable/linux-stable.git on my PC. Thanks.

git log calls it this:

commit 656bba306827a44ed73b3f93f75bb3147de17fae
Author: Len Brown <len....@intel.com>
Date: Sun Aug 16 11:45:48 2015 -0400

x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted

So you want to simply do this:

$ git revert 656bba306827a44ed73b3f93f75bb3147de17fae

build and test.

cheers,
-Len

[shrybman]

> Donald, Shane,
> Thanks for reporting this.

Sure! Thanks for looking into it.

> You have similar hardware:
>
> Shane:
>
> smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06, model: 0f, stepping: 06)
>
> Donald:
>
> CPU : Intel Core 2 CPU 6600 @ 2.4GHz
>
> I think I can get ahold of a core2 6xxx box tomorrow.

Maybe the mobo/bios? We both have the same Asus P5B as well. Donald said he has the deluxe version while I have the plain version.

[Donald Parsons]

On Thu, 2015-10-15 at 02:53 +0000, Brown, Len wrote:
> > > > Did you try reverting the "x86/smpboot: Remove
> > APIC.wait_for_init_deassert
> > > > and atomic init_deasserted" patch?
> > >
> > > Yes, please let me know if reverting that patch helps you too.
> >
> > How? Please send a patch or git cmd(s). I have the
> > git/stable/linux-stable.git on my PC. Thanks.
>
> git log calls it this:
>
> commit 656bba306827a44ed73b3f93f75bb3147de17fae
> Author: Len Brown <len....@intel.com>
> Date: Sun Aug 16 11:45:48 2015 -0400
>
> x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted
>
> So you want to simply do this:
>
> $ git revert 656bba306827a44ed73b3f93f75bb3147de17fae
>
> build and test.

I did the revert in linux-stable (last tag being v4.3-rc4) gave a revert
description so it would be applied.

built and tested. Result: did not help, still missing the second core.

I am sure the revert happened but I am uncertain as to how to build in a
git tree. I think I did it correctly
cp .config into linux-stable tree
edit Makefile to add -rev to: EXTRAVERSION = -rc4-rev
make oldconfig
make -j3 >> ../mak/mak_430-rc4-rev.log 2>&1 &

The before/after kernels are:
3965280 Oct 4 15:20 vmlinuz-4.3.0-rc4
3965408 Oct 15 13:45 vmlinuz-4.3.0-rc4-rev
The reverted vmlinuz is 128 bytes bigger; .config's are identical. I
see the revert adds back in some code so sizes seem appropriate.

Don

[shrybman]

> > build and test.
>
> I did the revert in linux-stable (last tag being v4.3-rc4) gave a revert
> description so it would be applied.
>
> built and tested.  Result:  did not help, still missing the second core.

Hmm. I have booted this kernel with the reverted patch a couple of times
now and it seems good. I will rebuild without the revert and test that? ...

Ok, tested without the revert and still missing the second core. So, the
revert seems to reproducibly fix the problem here.    I dunno.


Shane

[Brown, Len]

> > You have similar hardware:
> >
> > Shane:
> >
> > smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam: 06,
> model: 0f, stepping: 06)
> >
> > Donald:
> >
> > CPU : Intel Core 2 CPU 6600 @ 2.4GHz
> >
> > I think I can get ahold of a core2 6xxx box tomorrow.

Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz

is working for me with latest upstream.
(It is on an Intel D975 XBX motherboard)

please send me the .config you are using
that fails and I'll try that.

thanks,
-Len

N����ìr�›y���Øb²X��ǧvØ^–)޺{.nÇ+���Š{���çzX�¶ ��ܨ}���ƠzÚ&j:+v��¾ ���çzZ+�Ê+zf��·h��§~���Ûi��àz¹ ®w��¸?����Ú&¢)ߢ f�ù^jǫy§m�á@A«a�Ú ÿ0�ìh® �’i

[Brown, Len]

> > > You have similar hardware:
> > >
> > > Shane:
> > >

> > > smpboot: CPU0: Intel(R) Core(TM)2 CPU          6400  @ 2.13GHz (fam:

> 06,
> > model: 0f, stepping: 06)
> > >
> > > Donald:
> > >
> > > CPU : Intel Core 2 CPU 6600 @ 2.4GHz
> > >
> > > I think I can get ahold of a core2 6xxx box tomorrow.
>

> Intel(R) Core(TM)2 CPU E6800 @ 2.93GHz
>
> is working for me with latest upstream.
> (It is on an Intel D975 XBX motherboard)

Good news - I reproduced the failure on a similar box, an Intel D975xbx2:

[ 0.000000] Linux version 4.3.0-rc5+ (lenb@z87) (gcc version 4.9.2 20150212 (Red Hat 4.9.2-6) (GCC) ) #375 SMP Thu Oct 15 18:17:04 EDT 2015
..
[ 0.084000] smpboot: CPU0: Intel(R) Core(TM)2 Quad CPU @ 2.66GHz (family: 0x6, model: 0xf, stepping: 0x7)
[ 0.084000] Performance Events: PEBS fmt0-, 4-deep LBR, Core2 events, Intel PMU driver.
[ 0.084000] perf_event_intel: PEBS disabled due to CPU errata
[ 0.084000] ... version: 2
[ 0.084000] ... bit width: 40
[ 0.084000] ... generic registers: 2
[ 0.084000] ... value mask: 000000ffffffffff
[ 0.084000] ... max period: 000000007fffffff
[ 0.084000] ... fixed-purpose events: 3
[ 0.084000] ... event mask: 0000000700000003
[ 0.084000] x86: Booting SMP configuration:
[ 0.084000] .... node #0, CPUs: #1
[ 10.080003] smpboot: do_boot_cpu failed(-1) to wakeup CPU#1
[ 10.080175] NMI watchdog: enabled on all CPUs, permanently consumes one hw-PMU counter.
[ 10.080334] #2 #3
[ 10.084017] x86: Booted up 1 node, 3 CPUs
[ 10.084120] smpboot: Total of 3 processors activated (16001.58 BogoMIPS)

¢éì¹» ®&Þ~º&¶ ¬–+-±éݶ ¥Šw®žË›±Êâmébžìdz¹Þ–)í…æèw* jg¬±¨ ¶‰šŽŠÝ¢j/ êäz¹Þ–Šà2ŠÞ™¨è­Ú&¢)ß¡«a¶Ú þø ®G« éh® æj:+v‰¨Šwè†Ù¥>Wš±êÞiÛaxP jØm¶ŸÿÃ-» +ƒùdš_

[Brown, Len]

> I did the revert in linux-stable (last tag being v4.3-rc4) gave a revert
> description so it would be applied.
>
> built and tested. Result: did not help, still missing the second core.

Same result here.

upstream failed to bring up CPU #1 on 5/5 boots

Revert "x86/smpboot: Remove APIC.wait_for_init_deassert and atomic init_deasserted"

This reverts commit 656bba306827a44ed73b3f93f75bb3147de17fae.

Still fails the same way.

Adding "cpu_init_udelay=10000"

does not help.

commence bisect...

cheers,
-Len

[Brown, Len]

try booting upstream with "cpu_init_udelay=1".
If it works, then it actually implicates this commit:

a9bcaa02a5104ace6a9d9e4a9cd9192a9e7744d6
("x86/smpboot: Remove SIPI delays from cpu_up()")

Unfortunately the commit message for that on is erroneous --
"cpu_init_udelay=10000" is actually a NO-OP,
because that matches the compiled-in default.
Indeed, any non-zero value bug 10000 should work.

thanks,
-Len

｢鴈ｹｻ ｮ&ﾞ~ｺ&ｶ ｬ�-ｱ鰡ｶ ･学ｮ寨岾ﾊ穃饕樌dzｹﾞ�奛跖w* jgｬｱｨ ｶ凹至ﾝ｢j/�艘ｹﾞ槙�巌勣隴ﾚ&｢)ﾟ｡ｫaｶﾚ �ｮGｫ晞hｮ 詼:+v鴎学閹ﾙ･>W坡�iﾛaxP jﾘmｶ�ﾃ-ｻ +�d喟

[shrybman]

> try booting upstream with "cpu_init_udelay=1".
> If it works, then it actually implicates this commit:
>
> a9bcaa02a5104ace6a9d9e4a9cd9192a9e7744d6
> ("x86/smpboot: Remove SIPI delays from cpu_up()")
>
> Unfortunately the commit message for that on is erroneous --
> "cpu_init_udelay=10000" is actually a NO-OP,
> because that matches the compiled-in default.
> Indeed, any non-zero value bug 10000 should work.

That does seem to work. Heh, I tried that with 10000 last week.


Thanks,


Shane