System clock ticking too fast on BeagleBones

[Džiugas Baltrūnas]

Hello,

We are using tens of BeagleBones running the same 3.2.30-psp23 kernel and all powered using a 5V/2A power adaptor. On almost one third of nodes system clock is ticking too fast, approximatelly 4 times faster (i.e. Fri Nov 30 16:15:19 UTC 2012 which is correct time versus Mon Dec  3 17:02:37 UTC 2012). We are also running ntpd on all nodes, but larger time difference causes it to panic and exit (-g flag helps, but only for a very short time).

After some investigation I've noticed that some of the nodes boot with 32 KHz clock and some with 24MHz clock, but there is no rule that a particular clock is causing a time skew:

[node1]

$ grep -i clock /var/log/dmesg

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] omap_dm_timer_switch_src: Switching to HW default clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low power state

[    0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz

[    0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms

[    0.173176] Switching to clocksource gp timer

[    2.273396] omap_rtc omap_rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

[node2]

$ grep -i clock /var/log/dmesg

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz

[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms

[    0.174926] Switching to clocksource gp timer

[    2.253112] omap_rtc omap_rtc: setting system clock to 2012-10-25 19:56:51 UTC (1351195011)

I tried to compare /sys/kernel/debug/clock/summary files among nodes (after having mounted debugfs as per http://processors.wiki.ti.com/index.php/AM335x_PSP_User's_Guide#Clock_Management_details) and I noticed that timer1_fck has different parent clock:

dziugas@evalg-data1:/tmp/clock$ diff -u1 node1.summary node2.summary 

--- node1.summary 2012-11-09 10:40:01.660509296 +0100

+++ node2.summary 2012-11-09 10:42:02.932847298 +0100

@@ -73,3 +73,3 @@

 timer2_fck                     sys_clkin_ck                   24000000   1

-timer1_fck                     sys_clkin_ck                   24000000   1

+timer1_fck                     clk_32768_ck                   32768      1

 timer0_fck                     clk_rc32k_ck                   32000      0

@@ -170,3 +170,3 @@

 tclkin_ck                      none                           12000000   0

-sys_clkin_ck                   virt_24m_ck                    24000000   8

+sys_clkin_ck                   virt_24m_ck                    24000000   7

 virt_26m_ck                    none                           26000000   0

@@ -177,2 +177,2 @@

 clk_32khz_ck                   clkdiv32k_ick                  32768      0

-clk_32768_ck                   none                           32768      1

+clk_32768_ck                   none                           32768      2

So I suspect that for some reason parent clock is changed after the boot, but system clock is still calculating jiffies according to the frequency that was available during the boot and it causes a time drift.

Does anybody else have the same problem and could assist finding a solution for it? Recent clock changes by TI mainly deal with suspend feature (http://e2e.ti.com/support/dsp/sitara_arm174_microprocessors/f/791/p/180128/654964.aspx) which we do not use, so there should be something else causing this clock problem. Our nodes are placed in different physical locations, so unreliable power source could hardly be the cause. For now we plan to run ntpdate every minute instead of ntpd, but this is a workaround rather than solution.

Thanks,

Džiugas

[Juanjo]

I've not noted clock skews on mines but I have seen that sometimes the clock source is 32Khz XTAL and other the 24Mhz XTAL.

We run Debian wheezy witch calls ntpdate when Internet connection is up.

root@bb:/home/juanjo# date

Sat Dec  1 11:31:32 CLST 2012

root@bb:/home/juanjo# hwclock 

Sat 01 Dec 2012 11:31:36 AM CLST  -0.910919 seconds

root@bb:/home/juanjo# w

 11:31:48 up 3 days, 19:50,  1 user,  load average: 0.01, 0.02, 0.05

USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT

juanjo   pts/0    10.10.100.20     11:28    0.00s  0.90s  0.73s sshd: juanjo [priv]

Kernel logs:

[    0.000000] IRQ: Found an INTC at 0xfa200000 (revision 5.0) with 128 interrupts

[    0.000000] Total of 128 interrupts on 1 active controller

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz

[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms

One detail, in my Kernel I have RTC as a module waiting to implement my own. So some things RTC related are not called at the same time and I boot with fixrtc:

[    0.000000] Kernel command line: console=ttyO0,115200n8 fixrtc root=/dev/mmcblk0p2 ro rootfstype=ext4 rootwait ip=none

As I have the RTC as a module it does get load after rootfs is mounted:

[    1.822662] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)

[    1.830566] Waiting for root device /dev/mmcblk0p2...

[    1.844207] mmc0: host does not support reading read-only switch. assuming write-enable.

[    1.854644] mmc0: new high speed SD card at address b368

[    1.860931] mmcblk0: mmc0:b368 AF UD 1.86 GiB 

[    1.867523]  mmcblk0: p1 p2

[    1.955352] EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null)

[    1.963989] VFS: Mounted root (ext4 filesystem) readonly on device 179:2.

[    1.981475] devtmpfs: mounted

[    1.985015] Freeing init memory: 240K

[    3.236602] udevd[147]: starting version 175

[    4.549011] omap_rtc omap_rtc: rtc core: registered omap_rtc as rtc0

[Vaibhav Hiremath]

On 11/30/2012 10:03 PM, Džiugas Baltrūnas wrote:
> Hello,
>
> We are using tens of BeagleBones running the same 3.2.30-psp23 kernel
> and all powered using a 5V/2A power adaptor. On almost one third of
> nodes system clock is ticking too fast, approximatelly 4 times faster
> (i.e. Fri Nov 30 16:15:19 UTC 2012 which is correct time versus Mon Dec
> 3 17:02:37 UTC 2012). We are also running ntpd on all nodes, but larger
> time difference causes it to panic and exit (-g flag helps, but only for
> a very short time).
>
> After some investigation I've noticed that some of the nodes boot with
> 32 KHz clock and some with 24MHz clock, but there is no rule that a
> particular clock is causing a time skew:
>

There is specific reason behind configuring them.

Timer 1 is used for system timers (clocksource) and we need it tobe
ticking across suspend/resume cycle, so we need persistent clock.
In case of AM335x, RTC 32k clock is the only clock available which is
persistent.

> --- node1.summary2012-11-09 10:40:01.660509296 +0100
> +++ node2.summary2012-11-09 10:42:02.932847298 +0100

You can try to change the timer1 parent and see if it fixes this issue.
To change the parent of the timer by,

Modify "arch/arm/mach-omap2/timer.c" with,

diff --git a/arch/arm/mach-omap2/timer.c b/arch/arm/mach-omap2/timer.c
index 1af3871..91abfd3 100644
--- a/arch/arm/mach-omap2/timer.c
+++ b/arch/arm/mach-omap2/timer.c
@@ -420,7 +420,7 @@ OMAP_SYS_TIMER(3)
OMAP_SYS_TIMER_INIT(3_secure, OMAP3_SECURE_TIMER, OMAP3_CLKEV_SOURCE,
2, OMAP3_MPU_SOURCE)
OMAP_SYS_TIMER(3_secure)
-OMAP_SYS_TIMER_INIT(3_am33xx, 2, OMAP4_MPU_SOURCE, 1, AM33XX_RTC32K_SOURCE)
+OMAP_SYS_TIMER_INIT(3_am33xx, 2, OMAP4_MPU_SOURCE, 1, OMAP4_MPU_SOURCE)
OMAP_SYS_TIMER(3_am33xx)
#endif

Thanks,
Vaibhav

[Juanjo]

So basically getting a 24Mhz clock is a bug ?

[    0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz

[    0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms

[    0.173176] Switching to clocksource gp timer

Because I also noted some of my Beagles take GPTIMER1 as 24Mhz and others as 32Khz

[Hiremath, Vaibhav]

On Wed, Dec 05, 2012 at 05:31:57, Juanjo wrote:
> So basically getting a 24Mhz clock is a bug ?
>

No its not a bug.

If you have 32Khz crystal connected to RTC module, then timer will be
sourced from RTC-32Khz clock. And we have fall back mechanism for this, if
timer fails to get 32khz clock then it falls back to 24Mhz crystal clock.

> [ 0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz
> [ 0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps
> every 178956ms
> [ 0.173176] Switching to clocksource gp timer
>
>
> Because I also noted some of my Beagles take GPTIMER1 as 24Mhz and
> others as 32Khz
>

Most likely, either you don’t have 32k crystal OR you don’t have below patch
in u-boot
http://arago-project.org/git/projects/?p=u-boot-am33x.git;a=commit;h=5fdfe4d225485ac6a63b874143f1a784d5394f55


Thanks,
Vaibhav

> <http://processors.wiki.ti.com/index.php/AM335x_PSP_User's_Guide#Clock_M
> anagement_details> )

> <http://e2e.ti.com/support/dsp/sitara_arm174_microprocessors/f/791/p/180
> 128/654964.aspx> )

[Džiugas Baltrūnas]

Hello,

Thanks for the feedback. Indeed, none of the 3.2.x kernels from Robert Nelson's tree (https://github.com/RobertCNelson/linux-dev/tree/am33x-v3.2/patches) do have the patch you've pointed (it does not even contain arch/arm/cpu/armv7 folder).

However, I'm also wondering if it is really the case that some cape-less BeagleBones of the same revision (we are using mostly Rev. 6) are equipped with 32 Khz crystal while others not?

I've just installed v3.2_AM335xPSP_04.06.00.09-rc2 and I can reproduce the following (feeding power from a power adaptor, no mini USB cable connected):

1. If the board is booted when power plug was previously unplugged, clock is always 24 MHz:

# grep -i clock /var/log/dmesg

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] omap_dm_timer_switch_src: Switching to HW default clocksource(sys_clkin_ck) for timer1, this may impact timekeeping in low power state

[    0.000000] OMAP clocksource: GPTIMER1 at 24000000 Hz

[    0.000000] sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 178956ms

[    0.139246] PTP clock support registered

[    0.141670] Switching to clocksource gp timer

[    1.543804] clock: disabling unused clocks to save power

[    1.564633] omap_rtc am33xx-rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

# grep timer1 /sys/kernel/debug/clock/summary 

timer1_ick                     l4_wkup_gclk                   100000000  1

timer1_fck                     sys_clkin_ck                   24000000   1

2. If the reboot is triggered by pressing the reset button or issuing a 'reboot' command, clock is always 32 KHz:

# grep -i clock /var/log/dmesg

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz

[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms

[    0.141387] PTP clock support registered

[    0.143798] Switching to clocksource gp timer

[    1.530700] clock: disabling unused clocks to save power

[    1.551361] omap_rtc am33xx-rtc: setting system clock to 2012-12-10 16:33:40 UTC (1355157220)

# grep timer1 /sys/kernel/debug/clock/summary 

timer1_ick                     l4_wkup_gclk                   100000000  1

timer1_fck                     clk_32768_ck                   32768      1

And this makes to believe that the selection of the clocksource is somehow stateful, even though board is powered from the same source all the time. And since our nodes are not likely to be hit by any suspension mechanism, I still don't get what makes a clocksource to change while the board is up and running.

Thanks,

Džiugas

[Robert Nelson]

On Mon, Dec 10, 2012 at 11:25 AM, Džiugas Baltrūnas <dzi...@simula.no> wrote:
> Hello,
>
> Thanks for the feedback. Indeed, none of the 3.2.x kernels from Robert
> Nelson's tree
> (https://github.com/RobertCNelson/linux-dev/tree/am33x-v3.2/patches) do have
> the patch you've pointed (it does not even contain arch/arm/cpu/armv7
> folder).

Well, Vaibhav's patch does not touch the kernel tree.. It's for the
bootloader (u-boot), it also happens to be included in the u-boot
"v2013.01-rc1" release..

So... If (and only if) you are running my demo images, look at:

/boot/uboot/tools

is there an "update.sh" file?
if so run "sudo ./update.sh" then..

sudo ./bootloader_update.sh --use-beta-bootloader

Regards,

--
Robert Nelson
http://www.rcn-ee.com/

[Hiremath, Vaibhav]

This is interesting data pointers,

I do not have BeagleBone in front of me right now, let me try this out
tomorrow once I get into office, will share my observation.

Thanks,
Vaibhav

[Džiugas Baltrūnas]

Sorry, I haven't noticed the patch was for u-boot and thanks for pointing this out.

I've retested the initial clock selection using u-boot from arago-project (so with the mentioned patch in place) and I can confirm that the clock source is always 32 MHz no matter how the board was booted:

# grep -i clock /var/log/dmesg

[    0.000000] OMAP clockevent source: GPTIMER2 at 24000000 Hz

[    0.000000] OMAP clocksource: GPTIMER1 at 32768 Hz

[    0.000000] sched_clock: 32 bits at 32kHz, resolution 30517ns, wraps every 131071999ms

[    0.141113] PTP clock support registered

[    0.143066] Switching to clocksource gp timer

[    1.395019] clock: disabling unused clocks to save power

[    1.414581] omap_rtc am33xx-rtc: setting system clock to 2000-01-01 00:00:00 UTC (946684800)

So I'm wondering to know whether the consistence of the initial clocksource will solve the original problem of the system clock start ticking to fast, i.e. what are the conditions for system clock not to change while the system is running?

Džiugas

--
For more options, visit http://beagleboard.org/discuss

[Atul Sengar]

The best solution to this problem is (if locally connected)
Install Local NTP server and put "service ntp restart" in an infinite loop with sleep time 30 seconds approx. by writing a code in "/etc/init.d/rc.local" file. Reboot system and time will be synchronized with the server computer.