Regarding error:"New cluster time, XXX, is too far from this node's wall clock t

[Khush Boo]

Hi Folks,

I have my mongo db instance running as a single node cluster configuration.

If i change a system date to lets say 2/3 months ahead,

and then set it back to current date.

my db stops responding and

and shell  shows error 

"New cluster time, 169603669, is too far from this node's wall clock time, 154950669".

Is there is any way to recover from this error ?

how do i get back to working state.

Thanks in advance,

Khush

[Khush Boo]

Specifically getting,

com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting to connect. Client view of cluster state is {type=UNKNOWN, servers=[{address=localhost:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoCommandException: Command failed with error 205: 'New cluster time, 1627201609, is too far from this node's wall clock time, 1564043336.' on server localhost:27017. The full response is { "operationTime" : { "$timestamp" : { "t" : 1627201609, "i" : 1 } }, "ok" : 0.0, "errmsg" : "New cluster time, 1627201609, is too far from this node's wall clock time, 1564043336.", "code" : 205, "codeName" : "ClusterTimeFailsRateLimiter", "$clusterTime" : { "clusterTime" : { "$timestamp" : { "t" : 1627201609, "i" : 1 } }, "signature" : { "hash" : { "$binary" : "AAAAAAAAAAAAAAAAAAAAAAAAAAA=", "$type" : "00" }, "keyId" : { "$numberLong" : "0" } } } }}}]

[Kevin Adistambha]

Hi,

If i change a system date to lets say 2/3 months ahead,>and then set it back to current date.

From the error message:

New cluster time, 1627201609, is too far from this node's wall clock time, 1564043336.

it’s not set to 2/3 months ahead. Rather, it’s set to 2 years ahead:

> new Date(1564043336 * 1000)
ISODate("2019-07-25T08:28:56Z")

> new Date(1627201609 * 1000)
ISODate("2021-07-25T08:26:49Z")

To determine the approximate order of operation in a distributed system, MongoDB uses a Lamport clock, with a defined maximum tolerance of forward/backward movement of 1 year (as per MongoDB 4.0.11). Lamport clocks have a property of always-increasing timestamp. Realistically, any part of your cluster should not have a time difference of more than 1 year. If you do, then this error will appear.

If you’re faced with this situation, you have two possibilities:

1. Set your client’s time to match the cluster’s time. This may or may not be a realistic solution, since all your operations will be logged as happening way forward in the future.
2. Dump & restore the whole database, and fix the server’s time to the correct time.

I’m guessing that you’re testing failure modes, but since most Linux distros have ntp or similar service by default, your clock should not differ by more than 1 year.

Best regards,
Kevin

​