Supermicro SYS-1029UX-LL1-S16 system clock running too fast

[Himanshu Sharma]

Hi guys

We recenty got new Supermicros SYS-1029UX-LL1-S16 20 core servers for testing. We are running it with RedHat 7.4. In our testing we have observed that the system clock on these servers is running way too fast.  

Evidence of observation

1. Chrony

We sync our servers using chronyd and the frequency for this one is something we havent seen anywhere. I am not sure that clock can be slewed to accommodate this much frequency offset. (SYS-1029UX-LL1-S16 is gaining 26 millisecond every second).

> chronyc tracking 

Reference ID    : xxxxxxxx

Stratum         : 4

Ref time (UTC)  : Fri May 11 08:48:20 2018

System time     : 0.000001912 seconds fast of NTP time

Last offset     : +0.000001952 seconds

RMS offset      : 0.000001457 seconds

Frequency       : 26814.393 ppm fast

Residual freq   : +0.134 ppm

Skew            : 0.124 ppm

Root delay      : 0.194362879 seconds

Root dispersion : 0.000912781 seconds

Update interval : 4.0 seconds

Leap status     : Normal

It is taking forever to adjust clock rate to accomodate this much frequency difference.

2. PTPD

We also tried running ptpd to sync time with max_offset_ppm = 1000 to adjust for maximum possible frequency difference, but this was also futile. 

> /var/run/ptpd2.event.log

2018-05-11 14:21:21.249612 ptpd2[113396].enp94s0 (info)      (slv) TimingService.PTP0: acquired clock control

2018-05-11 14:21:49.976752 ptpd2[113396].enp94s0 (critical)  (slv) Offset above 1 second (1.001336627 s). Clock will step.

2018-05-11 14:21:48.975450 ptpd2[113396].enp94s0 (warning)   (slv) Stepped the system clock to: 05/11/18 14:21:48.975425504

2018-05-11 14:21:49.169540 ptpd2[113396].enp94s0 (notice)    (lstn_reset) Now in state: PTP_LISTENING

2018-05-11 14:21:51.022561 ptpd2[113396].enp94s0 (info)      (lstn_reset) UTC offset is now 37

2018-05-11 14:21:51.022639 ptpd2[113396].enp94s0 (info)      (lstn_reset) New best master selected: 88f031fffec32ec1(10.30.128.1)/0

2018-05-11 14:21:51.022788 ptpd2[113396].enp94s0 (notice)    (slv) Now in state: PTP_SLAVE, Best master: 88f031fffec32ec1(10.30.128.1)/0 (IPv4:172.27.210.123)

2018-05-11 14:21:51.024175 ptpd2[113396].enp94s0 (notice)    (slv) Received first Sync from Master

2018-05-11 14:21:52.057232 ptpd2[113396].enp94s0 (notice)    (slv) Received first Delay Response from Master

2018-05-11 14:21:52.057246 ptpd2[113396].enp94s0 (notice)    (slv) Received new Delay Request interval 1 from Master (was: 0)

2018-05-11 14:21:59.257989 ptpd2[113396].enp94s0 (notice)    (slv) Servo: Going to slew the clock with the maximum frequency adjustment

2018-05-11 14:22:28.029476 ptpd2[113396].enp94s0 (critical)  (slv) Offset above 1 second (1.011173248 s). Clock will step.

2018-05-11 14:22:27.018337 ptpd2[113396].enp94s0 (warning)   (slv) Stepped the system clock to: 05/11/18 14:22:27.18314450

2018-05-11 14:22:27.196373 ptpd2[113396].enp94s0 (notice)    (lstn_reset) Now in state: PTP_LISTENING

2018-05-11 14:22:29.056742 ptpd2[113396].enp94s0 (info)      (lstn_reset) UTC offset is now 37

2018-05-11 14:22:29.056797 ptpd2[113396].enp94s0 (info)      (lstn_reset) New best master selected: 88f031fffec32ec1(10.30.128.1)/0

2018-05-11 14:22:29.056843 ptpd2[113396].enp94s0 (notice)    (slv) Now in state: PTP_SLAVE, Best master: 88f031fffec32ec1(10.30.128.1)/0 (IPv4:172.27.210.123)

2018-05-11 14:22:29.063858 ptpd2[113396].enp94s0 (notice)    (slv) Received first Sync from Master

2018-05-11 14:22:30.066198 ptpd2[113396].enp94s0 (notice)    (slv) Received first Delay Response from Master

2018-05-11 14:22:30.066215 ptpd2[113396].enp94s0 (notice)    (slv) Received new Delay Request interval 1 from Master (was: 0)

Approximately every 40 seconds the clock has to be adjusted back 1 second to accomodate for the time gained which cannot be offset by the frequency adjustment. 

There is clearly nothing we can do using only frequency adjustment to accommodate a clock frequency this high. We also dont want the clock to go back in time because it can affect some of the mathematical calculations based on time difference which are written (not by me) on the premise that time moves only forward (which indeed it should :)).

Would really appreciate any solutions to rectify this issue. 

[Wojciech Kudla]

This is interesting. 

I know this is not what you need but for the sake of experiment, what happens if you switch the clock source from tsc to hpet? What are all the clock sources available on that system? 

Can you please post the relevant entries from dmesg? 

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Himanshu Sharma]

Hi Wojciech

cat /sys/devices/system/clocksource/clocksource0/available_clocksource

tsc acpi_pm 

Switching clocksource to acpi_pm resolved the clock frequency issue

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

acpi_pm

chronyc tracking 

Reference ID    : AC1D0218 (bkc2)

Stratum         : 4

Ref time (UTC)  : Fri May 11 11:23:28 2018

System time     : 0.000000000 seconds fast of NTP time

Last offset     : +0.000005277 seconds

RMS offset      : 0.000004877 seconds

Frequency       : 36.614 ppm fast

Residual freq   : +0.048 ppm

Skew            : 3.755 ppm

Root delay      : 0.193692103 seconds

Root dispersion : 0.015338245 seconds

Update interval : 2.0 seconds

Leap status     : Normal

However it increased the cost of clock_gettime(CLOCK_REALTIME, ..); function call to 1.5 micros which used to be just 20 nanos with tsc.  clock_gettime(CLOCK_REALTIME, ..) is a very critical function and it needs to be most optimised. So 1.5 micros cant work.

On some further investigation with my team we observed that the TSC frequency on server is higher than promised by the CPU provider.

vendor_id    : GenuineIntel

cpu family    : 6

model        : 85

model name    : Intel(R) Xeon(R) Gold 6144 CPU @ 3.50GHz

We calculated the observed TSC ticks by averaging the tsc ticks (rdtsc) over a period of 10 seconds. (see code attached)

TSC ticks per sec = 3596432020 (which is exactly the drift I reported in my starting post) .

26814.393 ppm ~ 1/40

(1-3500/3596) ~ 1/40

This is an overclocked server and this overshooting of TSC ticks is making the system clock gain time I guess .... 

[Silviu F]

Hello,

Do you have a solution? We have exactly the same server and the same problem with Linux 3.10.0-693.el7.x86_64.

[Wojciech Kudla]

I started seeing a strange clock issue, also on a supermicro rig, so will post here as this might be somewhat related. 

Some time after the boot (may range from seconds to hours) the clocksource watchdog reports tsc instability and switches to acpi_pm which practically ruins performance for the case I'm dealing with (mostly due to clock_gettime() calls not going via vDSO). 

[46370.620454] clocksource: timekeeping watchdog on CPU11: Marking clocksource 'tsc' as unstable because the skew is too large:
[46370.620464] clocksource: 'acpi_pm' wd_now: ba2933 wd_last: 2e55b4 mask: ffffff
[46370.620466] clocksource: 'tsc' cs_now: 6556533cf505b cs_last: 65563c5994731 mask: ffffffffffffffff
[46370.620473] tsc: Marking TSC unstable due to clocksource watchdog
[46370.620485] TSC found unstable after boot, most likely due to broken BIOS. Use 'tsc=unstable'.
[46370.620489] sched_clock: Marking unstable (46371036033892, -415557641)<-(46374206972364, -3586490522)
[46370.620716] clocksource: Override clocksource tsc is unstable and not HRT compatible - cannot switch while in HRT/NOHZ mode
[46370.620754] clocksource: Switched to clocksource acpi_pm

I tired to force tsc with clocksource=tsc on the boot line but that got ignored. 

The kernel is 4.15.17 with custom config for dynamic ticks and rcu offloading. 

I spoke to a vendor about this and they confirmed it's most likely a bios issue they have already a patch for. 

I'd contact your HW supplier to see if this is related. 

Hope this helps. 

--

[Silviu F]

We bought 2 same servers form supermicro, 1 month apart, the first doesn't have this issue, only difference it has an older Bios :

        Vendor: American Megatrends Inc.

        Version: 2.0b

        Release Date: 03/14/2018

The problematic server has:

          Vendor: American Megatrends Inc.

        Version: 2.0c

        Release Date: 04/12/2018

What BIOS do you have?

Silviu

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Silviu F]

Also, i don't see the tsc instability you are mentioning. 

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsubscribe...@googlegroups.com.

[Wojciech Kudla]

You are probably on a CPU with invariant TSC so experimenting with locking C-states and P-state drivers won't give you the answers you are looking for.

Also, modern linux kernels will perform TSC calibration with PIT (or other reliable source of ticks) upon boot, so overclocking should not be an issue here.

Can you post the result of: dmesg | egrep -i "tsc|clock"

 

2018-06-20 18:01 GMT+01:00 Silviu F <sfod...@gmail.com>:

This is the full info of the one that works:

BIOS Information

        Vendor: American Megatrends Inc.

        Version: 2.0b

        Release Date: 03/14/2018

        Address: 0xF0000

        Runtime Size: 64 kB

        ROM Size: 16384 kB

        Characteristics:

                PCI is supported

                BIOS is upgradeable

                BIOS shadowing is allowed

                Boot from CD is supported

                Selectable boot is supported

                BIOS ROM is socketed

                EDD is supported

                5.25"/1.2 MB floppy services are supported (int 13h)

                3.5"/720 kB floppy services are supported (int 13h)

                3.5"/2.88 MB floppy services are supported (int 13h)

                Print screen service is supported (int 5h)

                Serial services are supported (int 14h)

                Printer services are supported (int 17h)

                ACPI is supported

                USB legacy is supported

                BIOS boot specification is supported

                Targeted content distribution is supported

                UEFI is supported

        BIOS Revision: 5.12

DOESN'T WORK:

BIOS Information

        Vendor: American Megatrends Inc.

        Version: 2.0c

        Release Date: 04/12/2018

        Address: 0xF0000

        Runtime Size: 64 kB

        ROM Size: 16384 kB

        Characteristics:

                PCI is supported

                BIOS is upgradeable

                BIOS shadowing is allowed

                Boot from CD is supported

                Selectable boot is supported

                BIOS ROM is socketed

                EDD is supported

                5.25"/1.2 MB floppy services are supported (int 13h)

                3.5"/720 kB floppy services are supported (int 13h)

                3.5"/2.88 MB floppy services are supported (int 13h)

                Print screen service is supported (int 5h)

                Serial services are supported (int 14h)

                Printer services are supported (int 17h)

                ACPI is supported

                USB legacy is supported

                BIOS boot specification is supported

                Targeted content distribution is supported

                UEFI is supported

        BIOS Revision: 5.12

On Wed, Jun 20, 2018 at 11:57 AM, Wojciech Kudla <wojciec...@gmail.com> wrote:

I'm on:

Handle 0x0000, DMI type 0, 24 bytes

BIOS Information

        Vendor: American Megatrends Inc.

        Version: 3.0a

        Release Date: 03/21/2018

        Address: 0xF0000

        Runtime Size: 64 kB

        ROM Size: 8192 kB

        Characteristics:

                PCI is supported

                BIOS is upgradeable

                BIOS shadowing is allowed

                Boot from CD is supported

                Selectable boot is supported

                BIOS ROM is socketed

                EDD is supported

                5.25"/1.2 MB floppy services are supported (int 13h)

                3.5"/720 kB floppy services are supported (int 13h)

                3.5"/2.88 MB floppy services are supported (int 13h)

                Print screen service is supported (int 5h)

                8042 keyboard services are supported (int 9h)

                Serial services are supported (int 14h)

                Printer services are supported (int 17h)

                ACPI is supported

                USB legacy is supported

                BIOS boot specification is supported

                Targeted content distribution is supported

                UEFI is supported

        BIOS Revision: 5.11

[Wojciech Kudla]

Just a thought... You mentioned you're running 3.10.x (which I'm guessing is RHEL 7.x). Can you try a more recent kernel?

[Silviu F]

[    0.000000] hpet clockevent registered

[    0.000000] tsc: Detected 3500.000 MHz processor

[    0.121369] TSC deadline timer enabled

[    0.800876] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.807968] acpi PNP0A08:01: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.813341] acpi PNP0A08:02: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.818394] acpi PNP0A08:03: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.822363] acpi PNP0A08:06: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.823399] acpi PNP0A08:07: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.826451] acpi PNP0A08:08: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.828079] acpi PNP0A08:09: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI]

[    0.836608] Switched to clocksource hpet

[    2.226606] Switched to clocksource tsc

[    2.234657] rtc_cmos 00:00: setting system clock to 2018-06-19 17:10:06 UTC (1529428206)

[    2.430456] PTP clock support registered

[Himanshu Sharma]

I have followed up with supermicro guys for this as well but they have suggested this is a bug with latest redhat kernels. Redhat 7.2 or less will run fine as per their suggestion. Haven't tested it yet though

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/oG9vLZVYjVA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mechanical-symp...@googlegroups.com.

[Wojciech Kudla]

Got similar reply from a vendor. 

If downgrading is not an option for you then adding tsc=reliable to kernel boot line seems to address the problem. It's more of a nasty work around than anything else, but does the trick if you're fine with blacklisting tsc from clock watchdog. 

[Silviu F]

I am running CentOS Linux release 7.4.1708 (Core) on 2 servers from SuperMicro one works fine, the other doesn't. Reverted bios to version 2.0b, no fix. Will probably get a return. 

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/oG9vLZVYjVA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Silviu F]

Updated the to CentOS 7.5 same problem. At this point i think somehow is hardware related

[Wojciech Kudla]

Have you tried Himanshu's suggestion and downgraded to rhel 7.2 or earlier? 

Or is it simply not an option for you? 

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/oG9vLZVYjVA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mechanical-symp...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

[Silviu F]

Not an option

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "mechanical-sympathy" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mechanical-sympathy/oG9vLZVYjVA/unsubscribe.

To unsubscribe from this group and all its topics, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-sympathy+unsub...@googlegroups.com.

[Eddie]

Running on Debian 10, kernel 5.0.21-1, I've tried two adding to /proc/cmdline both

a) tsc=reliable 

b) tsc=unstable

not using one of these options gives a huge time drift to the system.

using A worked, but TSC is not as precise as HPET.

using B is best, because time does not drift and clocksource is now HPET.

Mobo is X11SCM-F, BIOS:

        Vendor: American Megatrends Inc.
        Version: 1.0b
        Release Date: 05/17/2019

        Address: 0xF0000
        Runtime Size: 64

kB
        ROM Size: 32 MB
        Characteristics:

                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is support

ed

        BIOS Revision: 5.13

[Wojciech Kudla]

There's a good reason why tsc is the default clock source in Linux.

It's more precise and cheaper to probe. 

Given that most modern platforms offer nonstop_tsc, I'd give it another shot and run with:

hpet=disable clocksource=tsc

In your kernel stanza. 

Adding tsc=reliable disables clock sync hence the time drift you're observing. 

--
You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/6b5e68ce-ea5f-4dad-aace-d696c05e4316%40googlegroups.com.

[Eddie]

having nothing in the kernel cmdline is when I was getting drift.
"tsc=reliable" worked (no drift), but tsc is a much less preferable clock source in modern platforms. see https://superuser.com/a/393978/68374

per debian 10 advice, as seen in dmesg, I am sticking with tsc=unstable

[Wojciech Kudla]

I have a few issues with the post you're referring to. It seems to completely ignore the existence of constant_tsc and invariant_tsc. The problems it talks about do not exist on modern day platforms; it's just an outdated post. 

http://btorpey.github.io/blog/2014/02/18/clock-sources-in-linux/

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/reference_guide/chap-timestamping#sect-Hardware_clocks

https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/tsc-resynchronization

--

You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mechanical-symp...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/mechanical-sympathy/711aaed9-032b-4618-90f2-d8e79d700148%40googlegroups.com.