System clock runs too fast after 2.6.27 -> 2.6.28.1 upgrade
Miklos Vajna
We just recently upgraded from Linux 2.6.27 to 2.6.28.1 and we noticed
that on some i686 machines the system clocks runs way faster than
necessary, they count about 70 minutes in an hour.
The strange fact is that using the same config on some other i686
machines the problem is not reproducible.
We had no messages on boot using the quiet option with 2.6.27, now there
is a
IO APIC resources could be not be allocated.
with 2.6.28. I'm not sure if this is related or not.
I'm attaching the dmesg of my laptop (where the bug is reproducible),
the config and the lspci output.
Before I started to write this mail I tried the followings:
- Booting with 'noapic notsc clocksource=pit'.
- Booting with 'noapic notsc clocksource=jiffies'.
- Booting with 'nohz=off'.
Is this a known problem?
Let me know if I missed any required information I should provide.
Thanks!
john stultz
> We just recently upgraded from Linux 2.6.27 to 2.6.28.1 and we noticed
> that on some i686 machines the system clocks runs way faster than
> necessary, they count about 70 minutes in an hour.
>
> The strange fact is that using the same config on some other i686
> machines the problem is not reproducible.
>
> We had no messages on boot using the quiet option with 2.6.27, now there
> is a
>
> IO APIC resources could be not be allocated.
>
> with 2.6.28. I'm not sure if this is related or not.
>
> I'm attaching the dmesg of my laptop (where the bug is reproducible),
> the config and the lspci output.
Could you send the output from:
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
For both kernels on the affected hardware ?
Also could you attach the dmesg for 2.6.27 on the same hardware?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majo...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Miklos Vajna
> Could you send the output from:
> cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> cat /sys/devices/system/clocksource/clocksource0/available_clocksource
>
> For both kernels on the affected hardware ?
Output when no related kernel parameters are used:
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc pit jiffies
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
acpi_pm
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
acpi_pm jiffies tsc
So it seems the problem is that acpi_pm is no longer available.
> Also could you attach the dmesg for 2.6.27 on the same hardware?
dmesg-2.6.27 and config-2.6.27 attached.
john stultz
> On Wed, Jan 21, 2009 at 12:00:57PM -0800, john stultz <john...@us.ibm.com> wrote:
> > Could you send the output from:
> > cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> > cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> >
> > For both kernels on the affected hardware ?
>
> Output when no related kernel parameters are used:
>
> 2.6.28.1:
>
> $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> tsc
> $ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> tsc pit jiffies
>
> 2.6.27.10:
>
> $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
> acpi_pm
> $ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
> acpi_pm jiffies tsc
>
> So it seems the problem is that acpi_pm is no longer available.
Yep. Its there, but I see in the 2.6.28 dmesg:
PM-Timer running at invalid rate: 87% of normal - aborting.
Which means its being disqualified.
Looking at the dmesg diff I see:
-TSC: PIT calibration confirmed by PMTIMER.
-TSC: using PMTIMER calibration value
-Detected 1499.820 MHz processor.
+TSC: PIT calibration deviates from PMTIMER: 1309343 1499790.
+TSC: Using PIT calibration value
+Detected 1309.343 MHz processor.
So it seems the PIT is having some sort of issue, which causes the TSC
to be miscalibrated (apparently causing the fast timekeeping in 2.6.28
you're seeing) and in addition it also is disqualifying the acpi_pm
clocksource.
Alas, the circular "what do we trust" issue bites us again.
The TSC calibrate code was reworked recently, but that should have been
in 2.6.27 (if I'm reading the git commit dates properly), so right off I
don't know what to blame.
Ingo/Thomas: Any ideas here?
Miklos: Can you try bisecting this down some?
thanks
-john
Ingo Molnar
* Miklos Vajna <vmi...@frugalware.org> wrote:
> On Wed, Jan 21, 2009 at 01:34:06PM -0800, john stultz <john...@us.ibm.com> wrote:
> > Miklos: Can you try bisecting this down some?
>
> I did it two times when I finally realised that it depends on the
> config. So currently I have two config files, with "-good" the clock is
> fine, with "-bad" it's too fast.
>
> Of course the diff contains unrelated changes as well, I'm trying to
> find which one is the problematic one.
these look suspect in the good->bad diff:
# CONFIG_X86_VOYAGER is not set
# CONFIG_X86_GENERICARCH is not set
# CONFIG_X86_VSMP is not set
-# CONFIG_X86_VISWS is not set
-# CONFIG_X86_RDC321X is not set
+CONFIG_X86_VISWS=y
+CONFIG_X86_RDC321X=y
CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
-# CONFIG_PARAVIRT_GUEST is not set
+CONFIG_PARAVIRT_GUEST=y
+# CONFIG_VMI is not set
+CONFIG_KVM_CLOCK=y
+CONFIG_KVM_GUEST=y
+CONFIG_LGUEST_GUEST=y
+CONFIG_PARAVIRT=y
+CONFIG_PARAVIRT_CLOCK=y
+# CONFIG_PARAVIRT_DEBUG is not set
CONFIG_MEMTEST=y
Ingo
Miklos Vajna
> Miklos: Can you try bisecting this down some?
I did it two times when I finally realised that it depends on the
john stultz
> On Wed, Jan 21, 2009 at 01:34:06PM -0800, john stultz <john...@us.ibm.com> wrote:
> > Miklos: Can you try bisecting this down some?
>
> I did it two times when I finally realised that it depends on the
> config. So currently I have two config files, with "-good" the clock is
> fine, with "-bad" it's too fast.
Could you send the same info I requested before (available_clocksources,
current_clocksource, dmesg) for both good and bad configs? Also make
sure they're on the same kernel version (as the configs are from
slightly different kernel versions).
> Of course the diff contains unrelated changes as well, I'm trying to
> find which one is the problematic one.
Bisecting the config difference is probably the most reliable way, but
you might try some shots in the dark first:
CONFIG_X86_VISWS: If I recall, this is support for SGI workstations that
had different apics then most systems, and might be related.
CONFIG_PARAVIRT_GUEST: I really haven't been paying much attention to
the paravirt bits, but this has some clock related subconfig options
that might be affecting it.
Miklos Vajna
> > I did it two times when I finally realised that it depends on the
> > config. So currently I have two config files, with "-good" the clock is
> > fine, with "-bad" it's too fast.
>
> Could you send the same info I requested before (available_clocksources,
> current_clocksource, dmesg) for both good and bad configs? Also make
> sure they're on the same kernel version (as the configs are from
> slightly different kernel versions).
Good:
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
acpi_pm jiffies tsc
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
acpi_pm
Bad:
$ cat /sys/devices/system/clocksource/clocksource0/available_clocksource
tsc pit jiffies
$ cat /sys/devices/system/clocksource/clocksource0/current_clocksource
tsc
> CONFIG_X86_VISWS: If I recall, this is support for SGI workstations that
> had different apics then most systems, and might be related.
>
> CONFIG_PARAVIRT_GUEST: I really haven't been paying much attention to
> the paravirt bits, but this has some clock related subconfig options
> that might be affecting it.
I just did a new build with CONFIG_X86_VISWS and CONFIG_X86_RDC321X
disabled, but I kept CONFIG_PARAVIRT_GUEST enabled and that works fine
as well. So CONFIG_PARAVIRT_GUEST seems to be unrelated.
(I built these on a faster machine, that's why the gcc version differs,
but that does not matter, I hope.)
Thank you both for your help! :)
Miklos Vajna
> these look suspect in the good->bad diff:
>
> # CONFIG_X86_VOYAGER is not set
> # CONFIG_X86_GENERICARCH is not set
> # CONFIG_X86_VSMP is not set
> -# CONFIG_X86_VISWS is not set
> -# CONFIG_X86_RDC321X is not set
> +CONFIG_X86_VISWS=y
> +CONFIG_X86_RDC321X=y
> CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y
> -# CONFIG_PARAVIRT_GUEST is not set
> +CONFIG_PARAVIRT_GUEST=y
> +# CONFIG_VMI is not set
> +CONFIG_KVM_CLOCK=y
> +CONFIG_KVM_GUEST=y
> +CONFIG_LGUEST_GUEST=y
> +CONFIG_PARAVIRT=y
> +CONFIG_PARAVIRT_CLOCK=y
> +# CONFIG_PARAVIRT_DEBUG is not set
> CONFIG_MEMTEST=y
Hmm it seems my reply to this mail was lost, so:
First I build with CONFIG_X86_VISWS, CONFIG_X86_RDC321X and
CONFIG_PARAVIRT_GUEST disabled, then the clock became normal again.
Then later I tried just disabling CONFIG_X86_VISWS and
CONFIG_X86_RDC321X, and that was enough as well.
Thank you!
john stultz
> On Thu, Jan 22, 2009 at 12:03:14PM -0800, john stultz
> > had different apics then most systems, and might be related.
> >
> disabled, but I kept CONFIG_PARAVIRT_GUEST enabled and that works fine
> as well. So CONFIG_PARAVIRT_GUEST seems to be unrelated.
Ah. CONFIG_X86_RDC321X is indeed the issue.
>From arch/x86/include/asm/timex.h:
#ifdef CONFIG_X86_ELAN
# define PIT_TICK_RATE 1189200 /* AMD Elan has different frequency! */
#elif defined(CONFIG_X86_RDC321X)
# define PIT_TICK_RATE 1041667 /* Underlying HZ for R8610 */
#else
# define PIT_TICK_RATE 1193182 /* Underlying HZ */
#endif
This gives the kernel an incorrect slower frequency for the PIT hardware
on your system and is why the TSC calibration is off and why the ACPI PM
gets thrown out.
I'd think this would ideally be more of a subarch in the config
selection like the AMD Elan is, so folks know it really is not just a
extra-support feature and in fact limits what hardware you can use the
resulting kernel on.
Florian, any thoughts on this? Would a patch like the following
(untested) be ok by you?
thanks
-john
Signed-off-by: John Stultz <john...@us.ibm.com>
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 862adb9..6d8ac94 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -300,6 +300,16 @@ config X86_ELAN
If unsure, choose "PC-compatible" instead.
+config X86_RDC321X
+ bool "RDC R-321x SoC"
+ depends on X86_32
+ select M486
+ select X86_REBOOTFIXUPS
+ help
+ This option is needed for RDC R-321x system-on-chip, also known
+ as R-8610-(G).
+ If you don't have one of these chips, you should say N here.
+
config X86_VOYAGER
bool "Voyager (NCR)"
depends on X86_32 && (SMP || BROKEN) && !PCI
@@ -380,16 +390,6 @@ config X86_VISWS
A kernel compiled for the Visual Workstation will run on general
PCs as well. See <file:Documentation/sgi-visws.txt> for details.
-config X86_RDC321X
- bool "RDC R-321x SoC"
- depends on X86_32
- select M486
- select X86_REBOOTFIXUPS
- help
- This option is needed for RDC R-321x system-on-chip, also known
- as R-8610-(G).
- If you don't have one of these chips, you should say N here.
-
config SCHED_OMIT_FRAME_POINTER
def_bool y
prompt "Single-depth WCHAN output"
Ingo Molnar
* john stultz <john...@us.ibm.com> wrote:
> On Fri, 2009-01-23 at 00:57 +0100, Miklos Vajna wrote:
> > On Thu, Jan 22, 2009 at 12:03:14PM -0800, john stultz
> > > CONFIG_X86_VISWS: If I recall, this is support for SGI workstations that
> > > had different apics then most systems, and might be related.
> > >
> > I just did a new build with CONFIG_X86_VISWS and CONFIG_X86_RDC321X
> > disabled, but I kept CONFIG_PARAVIRT_GUEST enabled and that works fine
> > as well. So CONFIG_PARAVIRT_GUEST seems to be unrelated.
>
> Ah. CONFIG_X86_RDC321X is indeed the issue.
>
> >From arch/x86/include/asm/timex.h:
>
> #ifdef CONFIG_X86_ELAN
> # define PIT_TICK_RATE 1189200 /* AMD Elan has different frequency! */
> #elif defined(CONFIG_X86_RDC321X)
> # define PIT_TICK_RATE 1041667 /* Underlying HZ for R8610 */
> #else
> # define PIT_TICK_RATE 1193182 /* Underlying HZ */
> #endif
>
> This gives the kernel an incorrect slower frequency for the PIT hardware
> on your system and is why the TSC calibration is off and why the ACPI PM
> gets thrown out.
>
> I'd think this would ideally be more of a subarch in the config
> selection like the AMD Elan is, so folks know it really is not just a
> extra-support feature and in fact limits what hardware you can use the
> resulting kernel on.
a better solution is that if RDC is set [it is unset by default] we should
turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If a real
RDC or ELAN system is running, that variable is changed during early
bootup, in the init functions for those platforms.
Ingo
john stultz
> * john stultz <john...@us.ibm.com> wrote:
>> I'd think this would ideally be more of a subarch in the config
>> selection like the AMD Elan is, so folks know it really is not just a
>> extra-support feature and in fact limits what hardware you can use the
>> resulting kernel on.
>
> a better solution is that if RDC is set [it is unset by default] we should
> turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If a real
> RDC or ELAN system is running, that variable is changed during early
> bootup, in the init functions for those platforms.
Took a quick look at this, and I don't see any detection code for
either RDC. The X86_ELAN config also has no conditionals, but maybe
the elan cpufreq code can help here.
Florian: Is there any way we can check very early (ie: in setup_arch)
for an RDC box?
Robert Schwebel, Sven Geggus: Either of you still maintaining the ELAN
code? Will the following from elanfreq_cpu_init catch all ELAN chips?
if ((c->x86_vendor != X86_VENDOR_AMD) ||
(c->x86 != 4) || (c->x86_model != 10))
thanks
-john
Ingo Molnar
* john stultz <john...@us.ibm.com> wrote:
> On Fri, Jan 23, 2009 at 3:04 AM, Ingo Molnar <mi...@elte.hu> wrote:
> > * john stultz <john...@us.ibm.com> wrote:
> >> I'd think this would ideally be more of a subarch in the config
> >> selection like the AMD Elan is, so folks know it really is not just a
> >> extra-support feature and in fact limits what hardware you can use the
> >> resulting kernel on.
> >
> > a better solution is that if RDC is set [it is unset by default] we should
> > turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If a real
> > RDC or ELAN system is running, that variable is changed during early
> > bootup, in the init functions for those platforms.
>
> Took a quick look at this, and I don't see any detection code for either
> RDC. The X86_ELAN config also has no conditionals, but maybe the elan
> cpufreq code can help here.
i'd suggest to just remove the #ifdef for now (it's causing problems) and
add the variable - we can add the quirks for those two platforms later on,
separately from your patch(es).
Ingo
Florian Fainelli
> On Fri, Jan 23, 2009 at 3:04 AM, Ingo Molnar <mi...@elte.hu> wrote:
> > * john stultz <john...@us.ibm.com> wrote:
> >> I'd think this would ideally be more of a subarch in the config
> >> selection like the AMD Elan is, so folks know it really is not just a
> >> extra-support feature and in fact limits what hardware you can use the
> >> resulting kernel on.
> >
> > a better solution is that if RDC is set [it is unset by default] we
> > should turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If a
> > real RDC or ELAN system is running, that variable is changed during early
> > bootup, in the init functions for those platforms.
>
> Took a quick look at this, and I don't see any detection code for
> either RDC. The X86_ELAN config also has no conditionals, but maybe
> the elan cpufreq code can help here.
>
> Florian: Is there any way we can check very early (ie: in setup_arch)
> for an RDC box?
Sorry for answering so late.
We cannot rely on the cpuid since there is not one, nor the bios, since most
(if not all) boards use RedBoot as a bootloader. If we have PCI initiliazed,
we could check the presence of a R6040 Ethernet MAC.
--
Cordialement, Florian Fainelli
------------------------------
Florian Fainelli
> * john stultz <john...@us.ibm.com> wrote:
> > On Fri, Jan 23, 2009 at 3:04 AM, Ingo Molnar <mi...@elte.hu> wrote:
> > > * john stultz <john...@us.ibm.com> wrote:
> > >> I'd think this would ideally be more of a subarch in the config
> > >> selection like the AMD Elan is, so folks know it really is not just a
> > >> extra-support feature and in fact limits what hardware you can use the
> > >> resulting kernel on.
> > >
> > > a better solution is that if RDC is set [it is unset by default] we
> > > should turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If
> > > a real RDC or ELAN system is running, that variable is changed during
> > > early bootup, in the init functions for those platforms.
> >
> > Took a quick look at this, and I don't see any detection code for either
> > RDC. The X86_ELAN config also has no conditionals, but maybe the elan
> > cpufreq code can help here.
>
> i'd suggest to just remove the #ifdef for now (it's causing problems) and
> add the variable - we can add the quirks for those two platforms later on,
> separately from your patch(es).
Fine with me. Ingo, is there any infrastructure yet to get an early quirk set
the PIT tick rate accordingly ?
Robert Schwebel
> On Fri, Jan 23, 2009 at 3:04 AM, Ingo Molnar <mi...@elte.hu> wrote:
> > * john stultz <john...@us.ibm.com> wrote:
> >> I'd think this would ideally be more of a subarch in the config
> >> selection like the AMD Elan is, so folks know it really is not just a
> >> extra-support feature and in fact limits what hardware you can use the
> >> resulting kernel on.
> >
> > a better solution is that if RDC is set [it is unset by default] we should
> > turn PIT_TICK_RATE into a variable, defaulting it to 1193182. If a real
> > RDC or ELAN system is running, that variable is changed during early
> > bootup, in the init functions for those platforms.
>
> Took a quick look at this, and I don't see any detection code for
> either RDC. The X86_ELAN config also has no conditionals, but maybe
> the elan cpufreq code can help here.
>
> Florian: Is there any way we can check very early (ie: in setup_arch)
> for an RDC box?
>
> Robert Schwebel, Sven Geggus: Either of you still maintaining the ELAN
> code? Will the following from elanfreq_cpu_init catch all ELAN chips?
>
> if ((c->x86_vendor != X86_VENDOR_AMD) ||
> (c->x86 != 4) || (c->x86_model != 10))
Nope, I don't have systems available any more with the ELAN.
rsc
--
Pengutronix e.K. | |
Industrial Linux Solutions | http://www.pengutronix.de/ |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0 |
Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-5555 |
Ingo Molnar
Not yet but it should be easy - do you have the time to implement it?
Ingo Molnar
* Florian Fainelli <florian....@telecomint.eu> wrote:
> [...]
yep, an early PCI quirk for that would suffice.
Ingo
Florian Fainelli
> * Florian Fainelli <florian....@telecomint.eu> wrote:
> > Le Saturday 24 January 2009 10:14:04 Ingo Molnar, vous avez écrit :
> > > * john stultz <john...@us.ibm.com> wrote:
> > > > On Fri, Jan 23, 2009 at 3:04 AM, Ingo Molnar <mi...@elte.hu> wrote:
> > > > > * john stultz <john...@us.ibm.com> wrote:
> > > > >> I'd think this would ideally be more of a subarch in the config
> > > > >> selection like the AMD Elan is, so folks know it really is not
> > > > >> just a extra-support feature and in fact limits what hardware you
> > > > >> can use the resulting kernel on.
> > > > >
> > > > > a better solution is that if RDC is set [it is unset by default] we
> > > > > should turn PIT_TICK_RATE into a variable, defaulting it to
> > > > > 1193182. If a real RDC or ELAN system is running, that variable is
> > > > > changed during early bootup, in the init functions for those
> > > > > platforms.
> > > >
> > > > Took a quick look at this, and I don't see any detection code for
> > > > either RDC. The X86_ELAN config also has no conditionals, but maybe
> > > > the elan cpufreq code can help here.
> > >
> > > i'd suggest to just remove the #ifdef for now (it's causing problems)
> > > and add the variable - we can add the quirks for those two platforms
> > > later on, separately from your patch(es).
> >
> > Fine with me. Ingo, is there any infrastructure yet to get an early
> > quirk set the PIT tick rate accordingly ?
>
> Not yet but it should be easy - do you have the time to implement it?
Not until a week, sorry.
Thanks
--
Cordialement, Florian Fainelli
------------------------------
Ingo Molnar
ok - i've done the interim patch below. Please send a quirk patch for
RDC's non-standard PIT frequency.
Ingo
------------------->
From e1b4d1143651fb3838be1117785b6e0386fa151f Mon Sep 17 00:00:00 2001
From: Ingo Molnar <mi...@elte.hu>
Date: Sun, 25 Jan 2009 16:57:00 +0100
Subject: [PATCH] x86: use standard PIT frequency
the RDC and ELAN platforms use slighly different PIT clocks, resulting in
a timex.h hack that changes PIT_TICK_RATE during build time. But if a
tester enables any of these platform support .config options, the PIT
will be miscalibrated on standard PC platforms.
So use one frequency - in a subsequent patch we'll add a quirk to allow
x86 platforms to define different PIT frequencies.
Signed-off-by: Ingo Molnar <mi...@elte.hu>
---
arch/x86/include/asm/timex.h | 13 ++++---------
1 files changed, 4 insertions(+), 9 deletions(-)
diff --git a/arch/x86/include/asm/timex.h b/arch/x86/include/asm/timex.h
index 1287dc1..b5c9d45 100644
--- a/arch/x86/include/asm/timex.h
+++ b/arch/x86/include/asm/timex.h
@@ -1,18 +1,13 @@
-/* x86 architecture timex specifications */
#ifndef _ASM_X86_TIMEX_H
#define _ASM_X86_TIMEX_H
#include <asm/processor.h>
#include <asm/tsc.h>
-#ifdef CONFIG_X86_ELAN
-# define PIT_TICK_RATE 1189200 /* AMD Elan has different frequency! */
-#elif defined(CONFIG_X86_RDC321X)
-# define PIT_TICK_RATE 1041667 /* Underlying HZ for R8610 */
-#else
-# define PIT_TICK_RATE 1193182 /* Underlying HZ */
-#endif
-#define CLOCK_TICK_RATE PIT_TICK_RATE
+/* The PIT ticks at this frequency (in HZ): */
+#define PIT_TICK_RATE 1193182
+
+#define CLOCK_TICK_RATE PIT_TICK_RATE
#define ARCH_HAS_READ_CURRENT_TIMER