Limitation on using App Engine Backends

[Jamil Rzayev]

Dear Sir or Madam,

I am working with Google App Engine now. I need to execute a method
that takes more than 30 seconds.
I decided to try App Engine Backends.

My method should be invoked every hour. It takes around 2 minutes to
be executed.
What happens is that Administration Console gives me a notification
that I reached a limit on using App Engine Backends and now my cron
gives me notification that the invocation is failed every time it
wants to invoke a method.

I use Backed of class B8 with 5 instances.

My question is how to configure app engine backend so that i wouldn't
get any problems with usage limitations.

Thank you

[Gerald Tan]

Each time you shutdown a backend instance, it will continue to be billed for 15 minutes after.
So if your job takes 2 minutes to run on the backend, you will be billed for 17 minutes.
B8 instances cost $0.64 for 60 minutes, and your free daily quota is $0.72, so you will only be able to run for 68 minutes before running out of quota, which will happen while the 5th hourly job is being shutdown.

Try using B1 instances, and you'll be fine as long as your job completes in under 7.5 minutes.
7.5 + 15 = 22.5 minutes billed per job
22.5 * 24 / 60 = 9 hours billed per day
9 * 0.08 = $0.72 billed per day which is your free quota

[Rishi Arora]

To execute a task that takes 2 minutes every hour, why do you need a B8 with 5 max instances?  I don't know what daily quota you're trying to stay within, but if your intention is to stay within 9 hours a day, which costs $0.72, a B5 with 5 instances will exhaust that very quickly.  If your two minutes of processing time is mostly spent in I/O - such as datastore operations, cache read/writes, URL fetches, etc, then it'll be a lot more efficient to use a B1 instance.  A B8 would only speed up CPU intensive operations, or provide more RAM, if you need it.

Anyways, assuming you need the memory and CPU cycles a B8 provides, you still don't need 5 simultaneous instances to execute a task that takes 2 minutes every hour.  I'd recommend tune the instance class to suit your processing needs, but keep the max instance to 1 in bakends.yaml.  And a 2 minute job means 17 minutes of instance hour usage every hour (15 minutes standard overhead + 2 minutes task running time), which equals 6.8 hours per day.  If you use a B1 backend, you can stay within the daily free quota.

The other thing to watch out for is - when you trigger your backend with a cron job, does your task processing spawn other tasks in the task queue?  If so, you'll create multiple backends (if allowed in backends.yaml), and even though those multiple simultaneous backends will execute only for a seconds or minutes, you'll incur the 15 minute overhead cost for starting each backend instance.  The best way to "smoothen" out the load / processing is to queue up the multiple requests and execute them seriallly on a single backend instance.  The best way to accomplish that is to use "Push Queues".  You enqueue requests into push queues, and when you dequeue (or "lease") requests one at a time, and execute them one at a time on your backend.

Hope this helps.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/__eDHo5o8LQJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Simon Knott]

Why don't you just use a cron job against front-end instances?  Cron jobs can last 10 minutes.

[Rishi Arora]

A model where front-end instances are focused on latency sensitive requests and backends are reserved for jobs that have no strict latency requirements will scale better.  Imagine a case where your "max idle front-end instances" is set to 1, and you run a 2-minute long cron job on a front-end.  During those two minutes, every request destined for a front-end could potentially cause a new instance to be created, causing large latency spikes under load.  For apps where initialization times are large, compared with request processing times, this won't scale at all.

On Wed, Nov 16, 2011 at 12:00 PM, Simon Knott <knott...@gmail.com> wrote:

Why don't you just use a cron job against front-end instances?  Cron jobs can last 10 minutes.

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/XIEW58_YyJMJ.

[stevep]

What about running your TaskQueue separately via the "run on default
instance" feature? Leave version-N to optimally handle your on-line
requests.

(Note: I may be misinterpreting how/what/why happens when you enqueue
tasks to Default from another version. Haven't looked into it yet
other than my optimistic response to reading that we can enqueue
tasks to something other than the primary OL handler.)

stevep