Is App Engine suitable for write intensive applications?

[Andrei-Ştefăniţă Cocorean]

Hi,

I want to develop a web service which will be used by a few thousand clients to save new pieces of data at a rate of aprox. 1 post/minute. I've been looking at what App Engine has to offer but I'm not sure if it's the right platform for this kind of system. At a glance it seems the datastore is optimized for efficient reads, but not so much for write intensive applications.

I'd appreciate any help on deciding whether it's worth to try and build this service on App Engine. Also, if you implemented something similar I'd be interested in hearing what problems you encountered along the way.

Regards,
Andrei

[Ikai Lan (Google)]

Yes, 1 post a minute is fine. Even 1 write a second is fine.

Where you will run into issues is when your application does something like ~hundreds of writes a second. There are very few systems that will do this well, and you will have to start looking into sharding your writes - but for now this is a premature optimization.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/bW3pKqHKcSEJ.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Robert Kluin]

One of the things that amazes me so much about App Engine is that it can handle hundreds, or thousands, of writes per second. I've seen it do this numerous times.  It takes some knowledge to sustain those rates, but it is very possible.



Robert

--
------
Robert Kluin
Ezox Systems, LLC

[Andrei-Ştefăniţă Cocorean]

I think my initial message was a bit unclear. I meant that each client
sends on average 1 post/minute and the system has to support several
thousand clients. Hundreds of writes per second may not be that far
away in the future.

Do you know some best practices in order to sustain the rates you
mentioned? I thought about performing writes from task queues to try
to reduce datastore contention by keeping a constant rate of
operations (regardless of how many clients post to the service at the
same time).

Does the number of entities in a given group have any impact on the
performance of subsequent writes (transactional or not) to that group?


Regards,
Andrei

[Jeff Schnitzer]

There are really only two hard issues that will limit your write rate,
assuming adequate budget:

1) Limit of one write per second per entity group.

This one is usually really easy to work around; just break your app up
into many entity groups. Most apps tend to naturally break down by
user or by business or some other categorization that has a low
per-unit write rate. As long as you are cognizant of query
eventuality and XG transaction limits, you can scale up to any write
rate. If each user is a separate entity group, it's like each user is
running on it's own little database.

The problems tend to appear when you need accurate runtime counts
across changing data. At this point look into sharded counters. You
shouldn't need to do this often.

2) Throughput limit on tablet splits for increasing index values

You won't hit this until many hundreds of writes per second. The
problem is when you have an index on a more-or-less monotonically
increasing field like say a timestamp. When the index is updated, the
writes will always be to the end of the table... and you'll get a "hot
tablet" that will split (causing a delay), then another "hot tablet"
since you're always writing to the end. The HRD helps in that it
gives you a multiple of the total write rate, but you still get a
limit.

Ikai wrote about this (and drew some awesome cartoons) here:

http://ikaisays.com/2011/01/25/app-engine-datastore-tip-monotonically-increasing-values-are-bad/

If you design with these two issues in mind, you shouldn't have any
problem doing thousands of writes per second... or whatever you can
afford.

Jeff

2012/1/29 Andrei-Ştefăniţă Cocorean <andrei....@gmail.com>:

[Robert Kluin]

Hi Andrei,
I just wanted to comment on one little thing in addition to all the
stuff Jeff said. I'd not suggest using the task-queue to do all
writes by default, but it is a good idea. I would suggest that you
try the write, then if you hit an exception (over quota, timeout,
deadline, internal error, etc...) defer the write to a task. I've had
extremely good luck with this approach on very high sustained write
rate apps in the past. If you're wanting to return to the client as
fast as possible, set a low deadline for the db put in the user
request. That way if you hit a little latency flare-up, you have more
control of the user experience.

Robert

2012/1/29 Andrei-Ştefăniţă Cocorean <andrei....@gmail.com>:

[Andrei-Ştefăniţă Cocorean]

Thank you for the feedback. It helped me better understand what the
platform offers and how to take advantage of its capabilities.

Regards,
Andrei

On Jan 30, 7:40 am, Robert Kluin <robert.kl...@gmail.com> wrote:
> Hi Andrei,
>   I just wanted to comment on one little thing in addition to all the
> stuff Jeff said.  I'd not suggest using the task-queue to do all
> writes by default, but it is a good idea.  I would suggest that you
> try the write, then if you hit an exception (over quota, timeout,
> deadline, internal error, etc...) defer the write to a task.  I've had
> extremely good luck with this approach on very high sustained write
> rate apps in the past.  If you're wanting to return to the client as
> fast as possible, set a low deadline for the db put in the user
> request.  That way if you hit a little latency flare-up, you have more
> control of the user experience.
>
> Robert
>

> 2012/1/29 Andrei-ªtefãniþã Cocorean <andrei.cocor...@gmail.com>:

> >> > On Thu, Jan 26, 2012 at 1:57 PM, Andrei-ªtefãniþã Cocorean <

[Richard Watson]

Jeff wrote:

> 2) Throughput limit on tablet splits for increasing index values 

How do you avoid timestamps if you would otherwise use them, for example needing to query by the time something occurred? I know Itai put a couple tips into his post, just asking how others have solved it.

Richard

[Jeff Schnitzer]

Wild guess: There are very few people who hit this write rate limit
so nobody really thinks about it. Even Ikai's advice is to ignore it
until it becomes a problem, because for 99.9% of apps it never will
be.

Jeff

[Robert Kluin]

If you do hit it, it can get quite complicated. It is also a royal
PITA to figure out that is what's happening. The basic idea will be
to somehow "shard" the index, so that writes aren't all going to the
"end" of the index. How you do that can be quite app dependent.

Robert

[WGuerlich]

I know, I'm going to hit the write limit with a timestamp I need to update on every write and which needs to be indexed.

As an alternative to sharding: What do you think about adding time jitter to the timestamp, that is, changing time randomly by a couple seconds? In my application the timestamp being off by a couple senconds wouldn't pose a problem.

Now what I need to know is: How many index entries can I expect to go into one tablet? This is needed to estimate the amount of jitter necessary to avoid hitting the same tablet on every write.

Any insights on this?

Wolfram

[Richard Watson]

Thanks Robert, I suspected you'd come across it.  I was re-reading Itai's post and just thinking about the shard suggestion.

(Jeff, definitely not going to pre-optimize, but I reckon one benefit of working with GAE is that we get to learn about designing for apps at scale. So...thought experiment.)

It seems obvious that fewer shards allow you to query across the full answer set with the least amount of queries. E.g. if you'll often query across users for a single multi-user customer, best would be to have a prefix that is shared for that customer, rather than one user. That way you don't have to sew the result sets together. (Does using a namespace handle this automatically or does it only affect keys? Where are the 'built-in' shard-points? Accounts must be sharded, otherwise the tablet covering the "current" timestamp would explode.)

One approach might be to change the data format (or store the value in an unindexed field) to just dump the data with minimal contention, and then later process it via a task to index it correctly. You get safer insert and can fix indexing when you can manage the process more carefully. Using this method you don't have to do anything different until you near huge write counts, and then adjust your app slightly to delay indexing.

Richard

[Jeff Schnitzer]

On Wed, Feb 1, 2012 at 2:48 AM, Richard Watson <richard...@gmail.com> wrote:
>
> It seems obvious that fewer shards allow you to query across the full answer
> set with the least amount of queries. E.g. if you'll often query across
> users for a single multi-user customer, best would be to have a prefix that
> is shared for that customer, rather than one user. That way you don't have
> to sew the result sets together. (Does using a namespace handle this
> automatically or does it only affect keys? Where are the 'built-in'
> shard-points? Accounts must be sharded, otherwise the tablet covering the
> "current" timestamp would explode.)

The "sharding" in GAE-land works a little differently from the way you think.

There's the notion of an Entity Group, which is probably closest to a
traditional data federation, but with a twist: you typically create
zillions of tiny entity groups, say, one for each customer. The
sharding is quite transparent; you only notice it when you write to
the same EG too fast or you try to run transactions across EGs.

The kind of sharding you would have to do to escape the hot tablet
problem is sharding the values of a particular field. The index
tablets span all EGs. So you might create 4 "versions" of the login
timestamp (say, prefixed with a different letter) and then issue four
queries when you want to query for the last 100 people that logged in.
In this case, you just pick a random prefix every time you write the
field... there's no need to make it stable.

Jeff

[Richard Watson]

I haven't mixed it up with entity groups, I refer only to sharding in the context of an index. Apologies if my use of the terminology confused the issue in any way.

My point was that the entries in the index are surely already sharded by some kind of prefix. There's no way all apps that save current millis in a field are being indexed on the same tablet. They must have some type of separation, which could maybe be achieved by prefix.  For example, the index entry could be something like [AppId][NamespaceId][EntityName][FieldName][123456789], which would remove contention between apps.  If [Namespace] is in there, then any apps that use namespaces already have a built-in natural shard point for index entries.

And I think the point about adding the indexed field value later is still valid.  Save the value in an unindexed field (e.g. unindexedTimestamp), and then update the indexed field (indexedTimestamp) via a task later, controlling the frequency of saves so it reduced impact on the index tablets.  That way you don't mess about with prefixes and adding result sets together.

Richard

[1] Does an index entry look like this:

[AppId][NamespaceId][

[Robert Kluin]

Hey Richard,
I believe indexes are written similar to what you've stated there.
You can use namespaces to mitigate this problem. The other way to
"shard" would be to prefix the value with some known value -- like a
letter. That way you spread the load across multiple index shards.
Both of these result in the same basic pain point though, to do a
"global" query you need to run a query within each of the shards then
join the results.

In the past I've worked around this problem in several different
ways. The best is to see if there is a natural way to "shard" the
index such that you won't need to do queries across the shards. If
you can do that, you're done. It also depends on what the problem
value is used for. In some cases it may not need to be so precise, so
you can spread the load around a bit by randomizing the value within
some acceptable range of error. I've also used the task-queue to
control the rate at which the problem entities get written. This can
help if you've got very bursty write rates.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/3Ltu7gGC8EsJ.

[Robert Kluin]

So I'd say don't worry about it unless you actually hit this problem.
If you do know you'll hit it, see if you have a way to "shard" the
timestamp, by account, user, or region, etc..., to relieve some of the
pressure. If you must have a global timestamp, I'd say keep it as
simple as possible, until you hit the issue. At that point you can
figure out a fix.

When I have timestamps on high write-rate entities that are
non-critical, for example "expiration" times that are used only for
cleanup, I'll sometimes add a random jitter of several hours to spread
the writes out a bit.  I'd be surprised if changing it by a few
seconds helped much -- but it could. Keep in mind, there will already
be some degree of randomness since the instance clocks have some
slight variation. If you're hitting this issue, I'd give it a shot
though. If it works it could at least buy you some time to get a
better fix.

I don't think there is a fixed number of rows per shard. I think it
is split up by data size, and I don't think the exact number is
publicly documented. Maybe you can roughly figure it out via
experimentation.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/r0SVTq6i4iEJ.

[Ikai Lan (Google)]

Thanks for the answers, Robert.

Shard size isn't determined by amount of data, but by access patterns. An example of an anti-pattern that will cause a shard size imbalance would be an entity write every time a user takes an action - but you never do anything with this data. Since the data just kind of accumulates, the shard never splits (unless it hits some hardware bound, which I've never really seen happen yet with GAE data).

As a final note, it takes a LOT of writes before this sort of thing happens, and I sometimes regret writing that blog post because anytime you write a blog post about scalability patterns, it invites people to prematurely implement them (Brett Slatkin's video generated an endless number of questions from people doing sub 1 QPS). We've done launches on the YouTube/Google homepage (http://blog.golang.org/2011/12/from-zero-to-go-launching-on-google.html) that haven't required us to make these changes because they did fine under load testing. I'd invest more energy in figuring out the right way to load test, then trying to figure out the bottlenecks when you hit limits with real data.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

[Robert Kluin]

Yeah Ikai is completely correct. I should have noted more clearly
that this is not something I even waste time worrying about until I
think I'm actually hitting it, which is not often. In the few cases
where I do think I've bumped into it, it is a writing thousands of
entities per second type of thing -- which is not very common.

It is interesting that sharding is determined by access patterns. Is
that something you can elaborate on at all? ;)

Robert

[Ikai Lan (Google)]

Robert, I'll see what I can do. No promises on an ETA. It isn't in one of the white papers?

http://labs.google.com/papers/bigtable.html

Oh what the heck ... the link is broken. Let me see what's up.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

[Ikai Lan (Google)]

Okay, looks like these whitepapers are at research.google.com now:

http://research.google.com/archive/bigtable.html

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

[Ikai Lan (Google)]

"A Bigtable cluster stores a number of tables. Each ta-

ble consists of a set of tablets, and each tablet contains

all data associated with a row range. Initially, each table

consists of just one tablet. As a table grows, it is auto-

matically split into multiple tablets, each approximately

100-200 MB in size by default."

Also: 100-200 MB is the default configuration. It might not be what we have configured. This is *really really* detailed, though.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com

[Robert Kluin]

Hey Ikai,
That's what I was actually remembering, that a tablet would be
around 100 or 200mb. I couldn't remember where I'd read that though
-- thanks for the link.

Robert

[WGuerlich]

Thanks, Ikai and Robert for shedding some light on this. I agree with you about premature optimization.

But now I know how to avoid these hotspots in the first place and since there is a natural opportunity  for sharding in my application you've probably saved me a lot of headaches further down the road. Since I'm planning to port an existing app I know for sure that thousand of writes per seconds are a requirement.

With tablets beeing as big as 200MB there's really no point in spreading timestamps over a few seconds as I suggested. A couple of hours would probably do.

Wolfram

[Robert Kluin]

Cool. Good luck. Let us know once you get the app up and going. I
always like to see high-usage apps.

Robert

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/3FJONwpMYXcJ.

[Gopal Patel]

if data is independent of each other , yes you can easily reach that speed of writing.

and unless it is ok to lost a one or two data point ( in case of memcache eviction ) i recommend directly writing to datastore.

On Wed, Oct 24, 2012 at 5:46 AM, Bradley Ford <wom...@gmail.com> wrote:

Hi All,

I hope I can get some advice on a similar topic to that posted above.

I need to record time series data at high-frequency and timestamps are very important.

During a peak period for each data point it might be recording at 1 second update rates per data point. But during the off-peak periods I expect 10-30 second update rates are to be expected.

My thinking is that each data-point could be its own tablet; this will result in between 100,000 to 1,000,000 tablets.

The other point too note is that it is ok if I have a delay before data is wrote to the database; so I could dump to the database after I have collected every 10 or 20 data points. (Possibly using Memcache)

From my reading generally and on this thread it should be possible if I can split each data point onto its own tablet; I am still unsure how I can make the cost manageable  In particular if I can use Memcache as a method to reduce the number of writes.

Best Regards

Brad

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/36DvbBBlMbAJ.