How to speed up the task queue execution?

[dflorey]

I'm using task queue to execute a lot of transactions on different
entity groups in parallel.
But even if I specify 50/s execution rate I end up with only ~120
executed tasks per minute.
I don't get any "too much contention..." exceptions so I have no clue
where the bottleneck is.
Any ideas how to speed things up?

[Tristan]

Are you actually queuing 50 tasks per second? If you only queue 120
tasks per minute, you'll only execute 120 per minute.

[dflorey]

:-)
Yes, I can see like 1000 waiting tasks in the task queue. They are
scheduled within few seconds but need ~10 minutes to get executed even
though each one is just doing more or less nothing - just changing a
few entities in an entity group.

[dflorey]

And: It is running fast like hell in dev environment. Just a few
seconds to run the full bunch of tasks.

[Tristan Slominski]

:)
What do the logs say? I think tasks are like any other http request, are you starting new processes a bunch of time? That's 7-8 seconds each time. That may be the cause.

On Jun 22, 2010 4:53 PM, "dflorey" <daniel...@gmail.com> wrote:

And: It is running fast like hell in dev environment. Just a few
seconds to run the full bunch of tasks.

On Jun 22, 11:51 pm, dflorey <daniel.flo...@gmail.com> wrote:
> :-)

> Yes, I can see like 1000 wait...

[Martin Webb]

I want to find out what typical costs are being paid for running apps on app engine say per 10,000 for 100,000 users. I read a post today that gave an example - and it was an example where it concluded that 10,000 users would cost $24000 a month - which is crazy - i cant imagine one can make $2 per user from advertising.
Can anyone shed any light on what the typical costs might be. Obviously it depends on the app/storage - but i'm talking about typical examples. lets say to run an app like twitter per 10,000 users.
Any help much appreciated.

 

Regards

 

 

Martin Webb

[Jaroslav Záruba]

I'm curious too. Also I'd love to read the post you mentioned. Pretty please!

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Ikai L (Google)]

It's highly unlikely 10,000 users would cost that much money. There are extremely large customers paying a fraction of that.

Here's a recent blog post from the developer of the Humble Indie Bundle:

http://googleappengine.blogspot.com/2010/06/how-app-engine-served-humble-indie.html

1 million uniques for $71 USD. Granted, the usage pattern isn't the same as Twitter, where each user may send out 100 tweets a day (who does this? I am a pretty heavy tweeter and I don't come close to this), but even that traffic profile shouldn't come anywhere near $24k.

--
Ikai Lan 
Developer Programs Engineer, Google App Engine

Blog: http://googleappengine.blogspot.com 

Twitter: http://twitter.com/app_engine

Reddit: http://www.reddit.com/r/appengine

[Tristan]

Yeah... looking at it again I can't even tell you where I got the
numbers, it looks like I just made them up...

Here's the thread with the corrected numbers (why did this thread get
hijacked?)

http://groups.google.com/group/google-appengine/browse_thread/thread/82ac1d0574a43d3c/593b3dbace36e16a#593b3dbace36e16a

On Jun 22, 7:35 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> It's highly unlikely 10,000 users would cost that much money. There are
> extremely large customers paying a fraction of that.
>
> Here's a recent blog post from the developer of the Humble Indie Bundle:
>

> http://googleappengine.blogspot.com/2010/06/how-app-engine-served-hum...

>
> 1 million uniques for $71 USD. Granted, the usage pattern isn't the same as
> Twitter, where each user may send out 100 tweets a day (who does this? I am
> a pretty heavy tweeter and I don't come close to this), but even that
> traffic profile shouldn't come anywhere near $24k.
>
> On Tue, Jun 22, 2010 at 4:47 PM, Jaroslav Záruba

> <jaroslav.zar...@gmail.com>wrote:

>
>
>
>
>
> > I'm curious too. Also I'd love to read the post you mentioned. Pretty
> > please!
>

> > On Wed, Jun 23, 2010 at 1:44 AM, Martin Webb <spydre...@yahoo.co.uk>wrote:
>
> >> I want to find out what typical costs are being paid for running apps on
> >> app engine say per 10,000 for 100,000 users. I read a post today that gave
> >> an example - and it was an example where it concluded that 10,000 users
> >> would cost $24000 a month - which is crazy - i cant imagine one can make $2
> >> per user from advertising.
> >> Can anyone shed any light on what the typical costs might be. Obviously it
> >> depends on the app/storage - but i'm talking about typical examples. lets
> >> say to run an app like twitter per 10,000 users.
> >> Any help much appreciated.
>
> >> Regards
>

> >> *Martin Webb*

>
> >>  --
> >> You received this message because you are subscribed to the Google Groups
> >> "Google App Engine" group.
> >> To post to this group, send email to google-a...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[Ikai L (Google)]

The broken number in that thread is 0.01 CPU hours per user per day. You do realize that is 36 seconds of computation time per User, right? If we were to be aggressive and assume 3.6 seconds instead, it brings those off the cuff costs of 100,000 users (who are obviously very heavily engaged, as they are returning every day and consuming 3.6s of CPU time) to $300. Not bad. But the real numbers are probably lower.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Tristan]

I disagree with your assessment that 0.01 CPU is unrealistic. Here's
an example.

User sends a request for a page.. cold start, need new instance ~
10,000 ms = 10s.
User stares at it for 2 seconds, clicks an interaction... we're
unlucky, new instance ~10,000 ms = 10s.
User executes a task that delegates something to the queue. Oh no,
start of a task, new instance ~10,000 ms = 10s.

And that's where 30s of computation went.

In low usage apps this happens all the time.

I'm gonna write out the math again since I don't trust my numbers:

60 seconds * 60 minutes * 24 hours = 86,400.

*If* (this is not a valid assumption, but worst case scenario) 10,000
users were evenly distributed over the 24 hours, there will be 8
seconds between each user visit, enough time for the above 30s usage
scenario to take place repetitively.

Even making more realistic assumptions, I would argue that there is
still a quite likely chance that each user could use 36s, or 0.01 CPU
hours in a twitter / facebook / user intensive application.

Funny, but it could be the case that there is a threshold of user
numbers that once reached drastically reduces your costs due to
instances being alive the whole time.

On Jun 22, 7:52 pm, "Ikai L (Google)" <ika...@google.com> wrote:
> The broken number in that thread is 0.01 CPU hours per user per day. You do
> realize that is 36 seconds of computation time per User, right? If we were
> to be aggressive and assume 3.6 seconds instead, it brings those off the
> cuff costs of 100,000 users (who are obviously very heavily engaged, as they
> are returning every day and consuming 3.6s of CPU time) to $300. Not bad.
> But the real numbers are probably lower.
>

> On Tue, Jun 22, 2010 at 5:37 PM, Tristan <tristan.slomin...@gmail.com>wrote:
>
>
>
>
>
> > Yeah... looking at it again I can't even tell you where I got the
> > numbers, it looks like I just made them up...
>
> > Here's the thread with the corrected numbers (why did this thread get
> > hijacked?)
>

> >http://groups.google.com/group/google-appengine/browse_thread/thread/...

> > > >> google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

> > e...@googlegroups.com>
> > > >> .
> > > >> For more options, visit this group at
> > > >>http://groups.google.com/group/google-appengine?hl=en.
>
> > > >  --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "Google App Engine" group.
> > > > To post to this group, send email to google-a...@googlegroups.com
> > .
> > > > To unsubscribe from this group, send email to

> > > > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com><google-appengine%2Bunsubscrib

[Martin Webb]

ha ha ha !
i think every app engine developer must have choked on their late night coffee, looked at the the half eaten muffin beside their keyboard and wondered if they will ever be able to afford to eat again, if their app attracts a minimal 10,000 users at 24 thousand USD a month.
Hey its bought a smile to my face - the figs that are now being talked about seem far more practical - ill stop crunching my cpu cycles and thrashing my code to speed things up - and continue to drink and eat this nights quota of coffee and muffins in peace.
Thx to all her commented.

 

Regards

 

 

Martin Webb

 

The information contained in this email is confidential and may contain proprietary information. It is meant solely for the intended recipient. Access to this email by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted in reliance on this, is prohibited and may be unlawful. No liability or responsibility is accepted if information or data is, for whatever reason corrupted or does not reach its intended recipient. No warranty is given that this email is free of viruses. The views expressed in this email are, unless otherwise stated, those of the author

 

 

---

From: Tristan <tristan....@gmail.com>
To: Google App Engine <google-a...@googlegroups.com>
Sent: Wed, 23 June, 2010 1:37:28
Subject: [google-appengine] Re: typical monthly costs for running an app

> >> google-appengine+unsub...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/google-appengine?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengine+unsub...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

> > .
> > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> Blog:http://googleappengine.blogspot.com
> Twitter:http://twitter.com/app_engine
> Reddit:http://www.reddit.com/r/appengine

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.

To unsubscribe from this group, send email to google-appengine+unsub...@googlegroups.com.

[Tim Hoffman]

Hi

Yes you could see all the restarts but only if you are getting only a
few users and visiting infrequently.

The minute you start getting busy you are very unlikely to be getting
all the restarts

T

[Tristan Slominski]

What's the criteria for instance recycle? Isn't it 1 request per second?

> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[djidjadji]

Do they address different entity groups or just a select few?

You can only do sustained about 1 transaction per second per entity
group (a post by Nick Johnson).
Maybe you get by doing 2 per second resulting in 120 updates finishing
in 1 minutes.

Have a look at the talk of Bret Slatkin of I/O 2010, the fan-in
solution might be something to use.

2010/6/23 Tristan Slominski <tristan....@gmail.com>:

[Geoffrey Spear]

On Jun 22, 9:49 pm, Tristan <tristan.slomin...@gmail.com> wrote:
> User sends a request for a page.. cold start, need new instance  ~
> 10,000 ms = 10s.
> User stares at it for 2 seconds, clicks an interaction... we're
> unlucky, new instance ~10,000 ms = 10s.
> User executes a task that delegates something to the queue. Oh no,
> start of a task, new instance ~10,000 ms = 10s.

If your instances are taking 10000ms of CPU to start up, you're doing
something horribly wrong. Period.

[Jaroslav Záruba]

That's pretty brave statement given how much the start times vary. Sometimes my app needs 3s, sometimes it takes 9s.

 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.

To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Ross M Karchner]

I would love to know how much traffic (or better, how many "topics"
and subscriptions) PubSubHubBub serves, and how much it costs (even if
nobody really pays for it).

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.

> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Harshal Patil]

Good question Ross. :) 

+1

[dflorey]

Back to my initial question... ;-)
@djidjadji: Thanks for your post! Do you know more about this "1
second limitiaon"?
I'm working only on separate entity groups (but they are of the same
kind) only.
I've been in that talk at the Google IO but I'm still confused why I'm
not able to execute more than ~2 tasks per second.
Is there some global transaction/s limit per app?
Or is the problem just that my app has low traffic for a long time and
then I'm starting ~1500 concurrent tasks?
But doing this for some time, I do not get any warnings about new
allocated nodes, but it is still not getting faster.

On Jun 23, 5:45 pm, Harshal <p.hars...@gmail.com> wrote:
> Good question Ross. :)
>
> +1
>

> On Wed, Jun 23, 2010 at 9:11 PM, Ross M Karchner <rosskarch...@gmail.com>wrote:
>
>
>
> > I would love to know how much traffic (or better, how many "topics"
> > and subscriptions) PubSubHubBub serves, and how much it costs (even if
> > nobody really pays for it).
>

> > On Wed, Jun 23, 2010 at 11:09 AM, Geoffrey Spear <geoffsp...@gmail.com>

> > wrote:
>
> > > On Jun 22, 9:49 pm, Tristan <tristan.slomin...@gmail.com> wrote:
> > >> User sends a request for a page.. cold start, need new instance  ~
> > >> 10,000 ms = 10s.
> > >> User stares at it for 2 seconds, clicks an interaction... we're
> > >> unlucky, new instance ~10,000 ms = 10s.
> > >> User executes a task that delegates something to the queue. Oh no,
> > >> start of a task, new instance ~10,000 ms = 10s.
>
> > > If your instances are taking 10000ms of CPU to start up, you're doing
> > > something horribly wrong.  Period.
>
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > > To post to this group, send email to google-a...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

> > > For more options, visit this group at
> >http://groups.google.com/group/google-appengine?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Google App Engine" group.
> > To post to this group, send email to google-a...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>
> > .

[djidjadji]

It's not a 1 second limitation. But if you hammer with concurrent
transactions on a single entity group you get a lot of "collisions"
and retries and that slows the whole process down. A rule of thumb is
don't try to update a single entity group more then once a second for
a long period of time. It's not about the kind but the individual
instances of a kind. If you have 50 request/s and they all want to
update 50 different objects with transactions it should work. It might
slow down a bit if they all hit the same tablet server but you won't
have collisions

The example at the TaskQueue man page also mentions the 1/s update by
transaction.

2010/6/24 dflorey <daniel...@gmail.com>:

[dflorey]

Yes, that is what I thought. So I've remodeled the datastore schema
and do not get any collisions any more when executing my concurrent
tasks (using sharded counters etc.).
But still it takes ~5-10 minutes to execute ~1000 tasks.
Is there anything I can do? I've seen some notes about "qps" in the
upcoming preview version, but no idea what this is and if it will help
me.

On Jun 24, 2:05 am, djidjadji <djidja...@gmail.com> wrote:
> It's not a 1 second limitation. But if you hammer with concurrent
> transactions on a single entity group you get a lot of "collisions"
> and retries and that slows the whole process down. A rule of thumb is
> don't try to update a single entity group more then once a second for
> a long period of time. It's not about the kind but the individual
> instances of a kind. If you have 50 request/s and they all want to
> update 50 different objects with transactions it should work. It might
> slow down a bit if they all hit the same tablet server but you won't
> have collisions
>
> The example at the TaskQueue man page also mentions the 1/s update by
> transaction.
>

> 2010/6/24 dflorey <daniel.flo...@gmail.com>:

[djidjadji]

I have written a test application that has 10 sharded counters each
with 10 shards.
It uses the ShardConfig class to keep record of the counters and the
number of shards.

The application generates 1000 tasks. Each task has a parameter that
is the name of a counter, chosen random. The task increments 1 of the
shards of this counter in a transaction.
All tasks where GET requests, then you can see the parameter in the
logs. A second parameter is the tasknumber (0-999). This lets you see
when the added tasks are executed.

It takes about 10-13 seconds to add 1000 tasks in a queue.
Resulting in a fill rate of 75/s. Adding 1 task at a time in the Queue.
If I use batch add for the tasks in a Queue, in batches of 50, it
takes 1.2 seconds to add 1000 tasks.

Queue settings are rate=50/s bucket_size=10.

For sequential add:
It takes 97 sec to process the tasks ( 10.3 tasks/sec).
Looking at the logs I notice that the later tasks are executed in
groups with a size of bucket_size.
The time between groups is 2 seconds, right down to the millisecond.

I increased the bucket_size to 20 (the maximum, says appcfg.py).

Now it took only 58sec for 1000 tasks (17 tasks/sec), sequential fill.
If you plot a graph of the cumulative tasks finishing for every
second, then the first 12 seconds 46 tasks/sec finish. Then until the
end you get 9.6 tasks/sec. That means that half the work is done in
just the first 12 seconds.

I found in the doc page that you can batch add tasks, using the
Queue.add() method.

The fill time is reduced to 1.3 seconds but now it takes 100-105
seconds for 1000 tasks (10 tasks/sec). Many of the early tasks took
1-8 seconds to complete. Later tasks took 100-200ms to complete. Only
in the first second 27 tasks finish, after that 10 tasks/sec finish
(20 every 2 seconds).

It looks like that during the request that inserts tasks you get a
larger tasks/sec that finish. When this request ends you get
bucket_size tasks every 2 seconds until the end.
Is there an explanation for this behavior?

During these experiments I got around 15-30 collisions for every 1000 tasks.
No task needed a rerun.

At the moment all counter shards have the same prefix for the
key_name. There are moments where 60-70 shards got an update in 1
second. Clearly no problem that they all address the same tablet
server.

Djidjadji