Time Syncronization

[Evan Ruff]

Hey guys,

I'm curious to hear about any strategies people have used to employ time synchronization among the AppEngine instances. I'm working on a project that is looking to have a pretty important time component and the current strategy I'm looking at is something long the lines of:

1. Record System.currentTimeMils()

2. URLFetch to an NTP server somewhere

3. Note return time

4. Correct NTP time using the start time to get request time.

Is this how other people are handling it? Is there another way that might be simpler? I'm a little hesitant to run an URLFetch for each request, obviously.

Thanks!

E

[Barry Hunter]

If accuracy is that important to you, using URLFetch would not be accurate enough. 

If you use URLFetch, and the request takes say 400ms, you've no idea where the actual timestamp belongs in that window. In theory its the middle but because you don't know the relative delays of the send/receive phases, its not guaranteed. 

In theory, you could use the experimental Sockets API, to actually use the NTP protocol itself, actually get a more accurate value. 

But it would take time (many seconds) for the time to actually sync. So you would need to say run a backend/module, that maintains time, locked via NTP to a server somewhere. Which your front end apps queries via an API. In theory a API connection to the back end is less susceptible to network delays than to a external NTP server. And it might be possible maintain the current skew, in a global veriable that persists between requests, so every request doesnt 'have' to make the RPC. 

.... but appengine already undertakes to maintain server time via NTP anyway, so it wont really give you any benefit. (so this is all bit of a thought experiment!) 

This might also be of interest:

https://www.google.com/search?q=%22Google+Production+time%22

E

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Richard]

My experience is that GAE backends are horrible when it comes to the 'NTP correct' time.  I have seen servers as much as 20 minutes out.  For the last 6 months or so, they have been much better, so maybe Google got its act together.

Since I run a MMO synchronized client (android) server (GAE) game, I need everyone to be correct to within 1-2 seconds.  The following code works pretty well:

    url = "http://www.worldtimeserver.com"
    format = '%a, %d %b %Y %H:%M:%S %Z'
    correction = 0
    try:
        orig_timestamp = time.mktime(time.gmtime(time.time()))
        result = urllib2.urlopen(url)
        date_string = result.headers['Date']
        recv_timestamp = time.mktime(time.gmtime(time.time()))
        dest_timestamp = time.mktime(time.strptime(date_string, format))

        correction = int((recv_timestamp - dest_timestamp)/2%180.)

Reason for the %180 in there is the games are on a 3 minute boundary.

I rarely see more than 1-3 seconds of 'correction' needing to be applied nowadays.

[Chad Vincent]

My current project isn't very time-sensitive, but my previous one we saw servers up to 45 minutes slow.  Made it hard to make fetches from Amazon S3 buckets with a 30-minute signed token when it was already expired by 15 minutes...

[alex]

Evan, if the question here is whether currentTimeMils() returns correct time, it's all managed for you, obviously including correct time. This is not an EC2 instance.

Not that it probably matters here but Google introduced a concept called "true time", which IMO even more accurate than NTP in some sense. I don't know if servers that run appspot apps use it at some level, but I know for sure Google team use it in Spanner.

Anyway, why would you want to adjust current system time, do you have evidence that it's not accurate enough?

[Evan Ruff]

Hey guys,

Thanks a lot for the interesting ideas about handling this issue. So for in my application, I need to be very sensitive (mils) about the arrival time of requests to resolve conflicts in data assignment.

Basically, I'm worried about two requests coming in around the same time, hitting two different instances, then having to resolve which request was "first".

I've thought about creating a memcache-based semaphore based around the actual data ID that will be assigned, but I thought I'd see if there was a slicker way that anyone had used.

Thanks,

E

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/PPbVax3k1s0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.

[Rafael]

Another idea to speed up synchronization on a high concurrency transactional level, would be to create a single java backend instance that takes in tasks from the front ends and process it in a singular queue. That way you know the time is just from one server and you also know its from a single threaded queue.

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.

[Richard]

One other item.

I have had DeadlineExceeded Errors when calling time.time() in Python.

Everything is managed for you.  That means ANYTHING that relies on the 'machine' can fail randomly (and does!!!!)

[Barry Hunter]

Basically, I'm worried about two requests coming in around the same time, hitting two different instances, then having to resolve which request was "first".

You could use cloud-sql :) 

Or even

http://stackoverflow.com/questions/3985812/how-to-implement-autoincrement-on-google-appengine

[alex]

> Everything is managed for you. That means ANYTHING that relies on the
> 'machine' can fail randomly (and does!!!!)

this implication is totally wrong.

[Barry Hunter]

Everything is managed for you.  That means ANYTHING that relies on the 'machine' can fail randomly (and does!!!!)

http://www.zdnet.com/google-at-scale-everything-breaks-3040093061/

 

[Evan Ruff]

Hey Guys,

Thanks so much for the ideas and links. I've been doing some more thinking around this, and was wondering if fashioning a single-threaded pull queue would serve my purposes?

Basically, when a request comes in, push it to the pull queue, do the linear assignment of the value through a pull, then do the additional processing/conflict management in a threaded task?

Could adding the tasks to a pull queue give me order of arrival?

Thanks,

E