Any guarantees on time sync among servers?

[jbdhl]

I know that the app engine servers are synchronized with NTP so that
their clocks should be almost perfectly synchronized. Should. But can
I rely on that? Are there any mechanisms that decouples a node if its
clock gets e.g. more than N seconds out of sync the rest (or some
master node)? If so, what is the number N?

In my app I need to receive objects that has changed since last call
to the server, and normally I would have used some sequence/serial
datatype for that. But such a datatype isn't effective in a
distributed environment so I'll try to "hack" the same functionality
using a Date instead, and then just "handle it" if the client receives
a few already received objects when polling the server. This may
happen if the server clocks aren't perfectly in sync.

[Ikai L (Google)]

The autogenerated ID is not guaranteed to be sequential, but it is guaranteed to be unique. It seems fairly unlikely that the times would be out of sync, but you can use Memcache's INCR on values. This is atomic and returns the next higher integer value.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
Ikai Lan
Developer Programs Engineer, Google App Engine
http://googleappengine.blogspot.com | http://twitter.com/app_engine

[jbdhl]

Thanks. But using memcache would give me a somewhat "fragile" counter
as it's memory-only. However, I stumbled upon Jeff Scudder's
implementation of a sharded counter, which is exactly what I'm looking
for. But... Shouldn't the code that 1) fetches a shard, and 2)
increases it, be encapsulated in a transaction in order to prevent
other users to make an update between 1) and 2)? Is his implementation
really "thread safe"?

The increment-code can be seen in
demos/shardedcounter/src/com/google/appengine/demos/shardedcounter/
v3/ShardedCounter.java
in the App Engine SDK.

[Ikai L (Google)]

It can be, but there's a fundamental tradeoff to doing so: you'd be transacting on every single count increment. It's just a matter of how much volatility you can accept:

1. Most volatile: Only use Memcache and INCR

2. Still volatile but generally okay: Use Memcache with INCR and periodic flushing

3. Better with a minor race condition, mitigated by number of shards: sharded counter

4. Best but expensive: transactional updates on each sharded counter

Something like pageview counts, for instance, may fall under #2, since you just need a number that is good enough - as this number grows, you have a higher margin of inaccuracy that you'd accept. For a counter that isn't frequently incremented, you'd have to consider whether sharding the counter is worthwhile.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[jbdhl]

Nice points. I actually think approach 2) will suffice. Thanks alot!

[kazunori_279]

FYI, here's my another proposal as an alternative to the sharded
counter: LogCounter.

http://groups.google.com/group/google-appengine/browse_thread/thread/da02ebd3001223f7/622bc30435097715?show_docid=622bc30435097715

The basic idea is to use MemcacheService#increment to get unique and
sequential integer values, while writing logs for them to obtain a
durability. While it should be as durable as the option 4, while it
should be more scalable and faster because the log writing can be in
parallel and there's no point of race condition.

Thanks,

Kaz

[Ikai L (Google)]

That link goes to a groups post.

I wouldn't use logging for this. Right now we are saving logs for what looks like 90 days, but this may change such that logging is only until you run out of space. Also - you can't programmatically filter on values in logs at the application level easily. 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[kazunori_279]

Hi Ikai,

Oops, I pasted a wrong link. Here's my code:

http://code.google.com/p/sth-samples/source/browse/trunk/sth-samples/src/jp/co/sth/samples/service/CounterService.java

By "Log" I do not mean the log messages, but actual entities which
works as journals of the counter updates (or whatever you write on
memcache). Instead of updating existing entities (which can be a
bottleneck), it would be scalable to add new entities to gain the
durability.

Thanks,

Kaz

On Apr 1, 2:24 am, "Ikai L (Google)" <ika...@google.com> wrote:
> That link goes to a groups post.
>
> I wouldn't use logging for this. Right now we are saving logs for what looks
> like 90 days, but this may change such that logging is only until you run
> out of space. Also - you can't programmatically filter on values in logs at
> the application level easily.
>
>
>
> On Tue, Mar 30, 2010 at 8:12 PM, kazunori_279 <kazunori...@gmail.com> wrote:
> > FYI, here's my another proposal as an alternative to the sharded
> > counter: LogCounter.
>

> >http://groups.google.com/group/google-appengine/browse_thread/thread/...

>
> > The basic idea is to use MemcacheService#increment to get unique and
> > sequential integer values, while writing logs for them to obtain a
> > durability. While it should be as durable as the option 4, while it
> > should be more scalable and faster because the log writing can be in
> > parallel and there's no point of race condition.
>
> > Thanks,
>
> > Kaz
>
> > On Mar 27, 11:10 pm, jbdhl <jbirksd...@gmail.com> wrote:
> > > Nice points. I actually think approach 2) will suffice. Thanks alot!
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsu...@googlegroups.com>