Clock not moving in virtual machine
Rob Farmer
I have a VPS from rootbsd.net which is running current, though I don't
update it very often. I just built and installed a new world and
kernel and now the clock will not move from the time the system was
booted, ie:
# date
Thu Jul 15 16:15:58 PDT 2010
<wait a minute>
# date
Thu Jul 15 16:15:58 PDT 2010
I have an old kernel from May 27 which doesn't have this problem. I
noticed some clock related stuff changing in current in the last
couple of weeks and suspect that their VM setup doesn't play well with
these changes (their site says they use Xen, but several boot messages
refer to QEMU). Officially, I think they only support running 8.0 so I
thought I would ask here if anyone has any ideas before putting in a
support request.
Here's a diff of the dmesgs - I can post full copies if needed but
didn't want to start with a ridicously long message:
--- dmesg.old 2010-07-15 17:45:20.000000000 -0700
+++ dmesg.new 2010-07-15 17:46:45.000000000 -0700
@@ -2,10 +2,9 @@
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
-FreeBSD 9.0-CURRENT #0: Thu May 27 01:26:08 PDT 2010
+FreeBSD 9.0-CURRENT #0: Thu Jul 15 16:13:28 PDT 2010
rfa...@peridot.predatorlabs.net:/usr/obj/usr/src/sys/PERIDOT i386
-Timecounter "i8254" frequency 1193182 Hz quality 0
-CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2261.02-MHz
686-class CPU)
+CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2261.59-MHz
686-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 6 Model = 1a Stepping = 5
Features=0x1781fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x80182201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,<b31>>
@@ -13,9 +12,10 @@
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 268435456 (256 MB)
-avail memory = 252473344 (240 MB)
-ACPI Error: A valid RSDP was not found (20100428/tbxfroot-309)
+avail memory = 252444672 (240 MB)
+ACPI Error: A valid RSDP was not found (20100702/tbxfroot-309)
MPTable: <_HVMCPU_ XEN >
+Event timer "LAPIC" frequency 0 Hz quality 500
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 0 package(s) x 16 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
@@ -26,7 +26,7 @@
ioapic0: Assuming intbase of 0
ioapic0 <Version 1.1> irqs 0-47 on motherboard
kbd1 at kbdmux0
-ACPI Error: A valid RSDP was not found (20100428/tbxfroot-309)
+ACPI Error: A valid RSDP was not found (20100702/tbxfroot-309)
ACPI: Table initialisation failed: AE_NOT_FOUND
ACPI: Try disabling either ACPI or apic support.
pcib0: <Host to PCI bridge> pcibus 0 on motherboard
@@ -81,7 +81,10 @@
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
-atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
+atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
+atrtc0: [FILTER]
+Event timer "RTC" frequency 32768 Hz quality 0
+Starting kernel event timers: LAPIC @ 200Hz, RTC @ 128Hz
Timecounters tick every 5.000 msec
ad0: 10240MB <QEMU HARDDISK 0.10.2> at ata0-master WDMA2
SMP: AP CPU #1 Launched!
Thanks,
--
Rob Farmer
_______________________________________________
freebsd...@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-curre...@freebsd.org"
Alexander Motin
> I have a VPS from rootbsd.net which is running current, though I don't
> update it very often. I just built and installed a new world and
> kernel and now the clock will not move from the time the system was
> booted, ie:
> # date
> Thu Jul 15 16:15:58 PDT 2010
> <wait a minute>
> # date
> Thu Jul 15 16:15:58 PDT 2010
>
> I have an old kernel from May 27 which doesn't have this problem. I
> noticed some clock related stuff changing in current in the last
> couple of weeks and suspect that their VM setup doesn't play well with
> these changes (their site says they use Xen, but several boot messages
> refer to QEMU). Officially, I think they only support running 8.0 so I
> thought I would ask here if anyone has any ideas before putting in a
> support request.
>
> Here's a diff of the dmesgs - I can post full copies if needed but
> didn't want to start with a ridicously long message:
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> FreeBSD/SMP: 0 package(s) x 16 core(s) x 2 SMT threads
> cpu0 (BSP): APIC ID: 0
Probably not related, but funny. :) So you have two CPUs?
> @@ -81,7 +81,10 @@
> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
> ppc0: [ITHREAD]
> ppbus0: <Parallel port bus> on ppc0
> -atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
> +atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
> +atrtc0: [FILTER]
> +Event timer "RTC" frequency 32768 Hz quality 0
> +Starting kernel event timers: LAPIC @ 200Hz, RTC @ 128Hz
> Timecounters tick every 5.000 msec
Everything seems reasonable there. Try to collect more information:
sysctl kern.timecounter
sysctl kern.eventtimer
vmstat -ia
systat -vm 1 (presence and frequencies of interrupts)
It could be a bug in emulation of some timers or bug in respective timer
driver, which was not triggered before last changes. You may try switch
to different timecounter by setting kern.timecounter.hardware, or
different eventtimers by setting kern.eventtimer.timer1 and
kern.eventtimer.timer2 sysctls.
--
Alexander Motin
Rob Farmer
> Rob Farmer wrote:
>> I have a VPS from rootbsd.net which is running current, though I don't
>> update it very often. I just built and installed a new world and
>> kernel and now the clock will not move from the time the system was
>> booted, ie:
>> # date
>> Thu Jul 15 16:15:58 PDT 2010
>> <wait a minute>
>> # date
>> Thu Jul 15 16:15:58 PDT 2010
>>
>> I have an old kernel from May 27 which doesn't have this problem. I
>> noticed some clock related stuff changing in current in the last
>> couple of weeks and suspect that their VM setup doesn't play well with
>> these changes (their site says they use Xen, but several boot messages
>> refer to QEMU). Officially, I think they only support running 8.0 so I
>> thought I would ask here if anyone has any ideas before putting in a
>> support request.
>>
>> Here's a diff of the dmesgs - I can post full copies if needed but
>> didn't want to start with a ridicously long message:
>
>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
>> FreeBSD/SMP: 0 package(s) x 16 core(s) x 2 SMT threads
>> cpu0 (BSP): APIC ID: 0
>
> Probably not related, but funny. :) So you have two CPUs?
Yes, there's 2 CPUs. It also gives the non uniform processors warning,
but it has been like that forever.
>
>> @@ -81,7 +81,10 @@
>> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
>> ppc0: [ITHREAD]
>> ppbus0: <Parallel port bus> on ppc0
>> -atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
>> +atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
>> +atrtc0: [FILTER]
>> +Event timer "RTC" frequency 32768 Hz quality 0
>> +Starting kernel event timers: LAPIC @ 200Hz, RTC @ 128Hz
>> Timecounters tick every 5.000 msec
>
> Everything seems reasonable there. Try to collect more information:
> sysctl kern.timecounter
> sysctl kern.eventtimer
> vmstat -ia
> systat -vm 1 (presence and frequencies of interrupts)
>
> It could be a bug in emulation of some timers or bug in respective timer
> driver, which was not triggered before last changes. You may try switch
> to different timecounter by setting kern.timecounter.hardware, or
> different eventtimers by setting kern.eventtimer.timer1 and
> kern.eventtimer.timer2 sysctls.
>
> --
> Alexander Motin
>
This is all on the new (not-working) kernel in single user mode:
# sysctl kern.timecounter
kern.timecounter.tick: 1
kern.timecounter.choice: TSC(-100) dummy(-1000000)
kern.timecounter.hardware: dummy
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.TSC.counter: 205772785
kern.timecounter.tc.TSC.frequency: 2261052646
kern.timecounter.tc.TSC.quality: -100
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 1
kern.timecounter.tc.TSC.counter changes everytime (205772785,
3200717147, 1205899870, ...) but I can't see any pattern.
# sysctl kern.eventtimer
kern.eventtimer.choice: LAPIC(500) RTC(0)
kern.eventtimer.et.LAPIC.flags: 15
kern.eventtimer.et.LAPIC.frequency: 50001404
kern.eventtimer.et.LAPIC.quality: 500
kern.eventtimer.et.RTC.flags: 1
kern.eventtimer.et.RTC.frequency: 32768
kern.eventtimer.et.RTC.quality: 0
kern.eventtimer.timer2: RTC
kern.eventtimer.timer1: LAPIC
kern.eventtimer.singlemul: 4
# vmstat -ia
interrupt total rate
??? 0 0
irq1: atkbd0 339 339
stray irq1 0 0
irq0: 0 0
stray irq0 0 0
irq3: 0 0
stray irq3 0 0
irq4: uart0 0 0
stray irq4 0 0
irq5: re0 0 0
stray irq5 0 0
irq6: 0 0
stray irq6 0 0
irq7: ppc0 0 0
stray irq7 0 0
irq8: atrtc0 24463 24463
stray irq8 0 0
irq9: 0 0
stray irq9 0 0
irq10: re1 0 0
stray irq10 0 0
irq11: 0 0
stray irq11 0 0
irq12: psm0 0 0
stray irq12 0 0
irq13: 0 0
stray irq13 0 0
irq14: ata0 224 224
stray irq14 0 0
irq15: ata1 0 0
stray irq15 0 0
irq16: 0 0
stray irq16 0 0
irq17: 0 0
stray irq17 0 0
irq18: 0 0
stray irq18 0 0
irq19: 0 0
stray irq19 0 0
irq20: 0 0
stray irq20 0 0
irq21: 0 0
stray irq21 0 0
irq22: 0 0
stray irq22 0 0
irq23: 0 0
stray irq23 0 0
irq24: 0 0
stray irq24 0 0
irq25: 0 0
stray irq25 0 0
irq26: 0 0
stray irq26 0 0
irq27: 0 0
stray irq27 0 0
irq28: 0 0
stray irq28 0 0
irq29: 0 0
stray irq29 0 0
irq30: 0 0
stray irq30 0 0
irq31: 0 0
stray irq31 0 0
irq32: 0 0
stray irq32 0 0
irq33: 0 0
stray irq33 0 0
irq34: 0 0
stray irq34 0 0
irq35: 0 0
stray irq35 0 0
irq36: 0 0
stray irq36 0 0
irq37: 0 0
stray irq37 0 0
irq38: 0 0
stray irq38 0 0
irq39: 0 0
stray irq39 0 0
irq40: 0 0
stray irq40 0 0
irq41: 0 0
stray irq41 0 0
irq42: 0 0
stray irq42 0 0
irq43: 0 0
stray irq43 0 0
irq44: 0 0
stray irq44 0 0
irq45: 0 0
stray irq45 0 0
irq46: 0 0
stray irq46 0 0
irq47: 0 0
stray irq47 0 0
cpu0:timer 38223 38223
cpu1:timer 38220 38220
Total 101469 101469
systat -ia cycles between:
526 total
atkbd0 1
128 atrtc0 8
ata0 irq14
199 cpu0:timer
199 cpu1:timer
530 total
atkbd0 1
128 atrtc0 8
ata0 irq14
201 cpu0:timer
201 cpu1:timer
If I set kern.eventtimer.timer1="RTC", dmesg says:
Starting kernel event timers: RTC @ 200Hz, LAPIC @ 128Hz
Event timer "LAPIC" is dead.
Starting kernel event timers: RTC @ 800Hz, NONE @ 0Hz
The clock still doesn't move though.
On the old kernel I had i8254 in the dmesg and that appears to be what was used:
kern.timecounter.tick: 1
kern.timecounter.choice: TSC(-100) i8254(0) dummy(-1000000)
kern.timecounter.hardware: i8254
kern.timecounter.stepwarnings: 0
kern.timecounter.tc.i8254.mask: 65535
kern.timecounter.tc.i8254.counter: 51889
kern.timecounter.tc.i8254.frequency: 1193182
kern.timecounter.tc.i8254.quality: 0
kern.timecounter.tc.TSC.mask: 4294967295
kern.timecounter.tc.TSC.counter: 266155055
kern.timecounter.tc.TSC.frequency: 2261010847
kern.timecounter.tc.TSC.quality: -100
kern.timecounter.smp_tsc: 0
kern.timecounter.invariant_tsc: 1
Also, in the old kernel, there was nothing on irq8 in vmstat:
irq8: 0 0
stray irq8 0 0
At the loader prompt I tried entering set
kern.timecounter.hardware="i8254" and set
kern.timecounter.hardware="TSC" and they appear set correctly if I
type show, but when I boot, the sysctl is back to dummy.
I have zero knowledge of this kind of thing, but I looks to me like I
need to be using i8254 and the new kernel is not detecting this
device. Is there a hint or something I can set to try and pick up this
device again?
Thanks,
--
Rob Farmer
Alexander Motin
>>> @@ -81,7 +81,10 @@
>>> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
>>> ppc0: [ITHREAD]
>>> ppbus0: <Parallel port bus> on ppc0
>>> -atrtc0: <AT Real Time Clock> at port 0x70 irq 8 on isa0
>>> +atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
>>> +atrtc0: [FILTER]
>>> +Event timer "RTC" frequency 32768 Hz quality 0
>>> +Starting kernel event timers: LAPIC @ 200Hz, RTC @ 128Hz
>>> Timecounters tick every 5.000 msec
>> Everything seems reasonable there. Try to collect more information:
>> sysctl kern.timecounter
>> sysctl kern.eventtimer
>> vmstat -ia
>> systat -vm 1 (presence and frequencies of interrupts)
>>
>> It could be a bug in emulation of some timers or bug in respective timer
>> driver, which was not triggered before last changes. You may try switch
>> to different timecounter by setting kern.timecounter.hardware, or
>> different eventtimers by setting kern.eventtimer.timer1 and
>> kern.eventtimer.timer2 sysctls.
>
>
> # sysctl kern.timecounter
> kern.timecounter.tick: 1
> kern.timecounter.choice: TSC(-100) dummy(-1000000)
> kern.timecounter.hardware: dummy
> kern.timecounter.stepwarnings: 0
> kern.timecounter.tc.TSC.mask: 4294967295
> kern.timecounter.tc.TSC.counter: 205772785
> kern.timecounter.tc.TSC.frequency: 2261052646
> kern.timecounter.tc.TSC.quality: -100
> kern.timecounter.smp_tsc: 0
> kern.timecounter.invariant_tsc: 1
>
> kern.timecounter.tc.TSC.counter changes everytime (205772785,
> 3200717147, 1205899870, ...) but I can't see any pattern.
It is probably hard to see pattern due to to very high clock frequency.
But TSC timecounter is unreliable even on real SMP systems. What it
counts on virtual SMP - even bigger question. As system seems never uses
timecounters with negative quality - you've left with
kern.timecounter.hardware=dummy - that's why time is not going. As last
resort you may try to set sysctl kern.timecounter.hardware=TSC in run time.
Previously you were using i8254 timecounter, but now you lost it as your
virtual machine PnP BIOS doesn't announce it. QEMU does the same, but
instead it gives HPET which your emulator seems also doesn't. To force
i8254 presence add to the /boot/device.hints lines:
hint.attimer.0.at="isa"
hint.attimer.0.port="0x40"
hint.attimer.0.irq="0"
--
Alexander Motin
Alexander Motin
> On Fri, Jul 16, 2010 at 11:47:23PM +0300, Alexander Motin wrote:
>> It is probably hard to see pattern due to to very high clock frequency.
>> But TSC timecounter is unreliable even on real SMP systems. What it
>> counts on virtual SMP - even bigger question. As system seems never uses
>> timecounters with negative quality - you've left with
>> kern.timecounter.hardware=dummy - that's why time is not going. As last
>> resort you may try to set sysctl kern.timecounter.hardware=TSC in run time.
>
> as the timecounter in /etc/sysctl.conf - I've since found it should be
> possible to also set kern.timecounter.smp_tsc=1 in /boot/loader.conf to let
> the TSC be chosen. The system's now been running for a day and I've not had
> any warnings about the clock going backward, and since the time has
> remained correct I guess Xen synchronises with the host?
I have no idea about TSC in XEN, but QEMU just passes TSC from the
physical CPU. So if host' TSCs are not synchronized - value may jump
accidentally when QEMU process migrates between CPUs.
> Should I still switch back to using the i8254?
I would say it depends. i8254 frequency is always known, while TSC
depends on calibration. Calibration on virtual machine I think much
less precise then on physical. Same time, if i8254 also used as event
timer, timestamp calculation algorithm is not very trivial there and I
am not sure it is absolutely reliable. So I would probably used i8254 as
time counter and LAPIC+RTC as event timers.
Rob Farmer
> On Fri, Jul 16, 2010 at 11:47:23PM +0300, Alexander Motin wrote:
>>
>> It is probably hard to see pattern due to to very high clock frequency.
>> But TSC timecounter is unreliable even on real SMP systems. What it
>> counts on virtual SMP - even bigger question. As system seems never uses
>> timecounters with negative quality - you've left with
>> kern.timecounter.hardware=dummy - that's why time is not going. As last
>> resort you may try to set sysctl kern.timecounter.hardware=TSC in run time.
>
> I came across the same problem on rootbsd a few days ago, and set the TSC
> as the timecounter in /etc/sysctl.conf - I've since found it should be
> possible to also set kern.timecounter.smp_tsc=1 in /boot/loader.conf to let
> the TSC be chosen. The system's now been running for a day and I've not had
> any warnings about the clock going backward, and since the time has
> switch back to using the i8254?
>
> Bruce Cran
>
Setting kern.timecounter.smp_tsc=1 works for me. I put it in
/boot/loader.conf so it would automatically work for single user mode
too. I don't think the host automatically synchronizes the clock -
their website recommends running ntp and I saw the clock drift a fair
amount before I started doing that.
Thanks for the tip.
--
Rob Farmer
Bruce Cran
>
> It is probably hard to see pattern due to to very high clock frequency.
> But TSC timecounter is unreliable even on real SMP systems. What it
> counts on virtual SMP - even bigger question. As system seems never uses
> timecounters with negative quality - you've left with
> kern.timecounter.hardware=dummy - that's why time is not going. As last
> resort you may try to set sysctl kern.timecounter.hardware=TSC in run time.
I came across the same problem on rootbsd a few days ago, and set the TSC
as the timecounter in /etc/sysctl.conf - I've since found it should be
possible to also set kern.timecounter.smp_tsc=1 in /boot/loader.conf to let
the TSC be chosen. The system's now been running for a day and I've not had
any warnings about the clock going backward, and since the time has
remained correct I guess Xen synchronises with the host? Should I still
switch back to using the i8254?
--
Bruce Cran