Application over quota

[kordi]

Why is my application over quota?

CPU Used 0% 380.76 of 199608.00 Gigacycles (0%)
Data Sent 0% 0.25 of 2048.00 Megabytes (0%)
Data Received 0% 0.00 of 2048.00 Megabytes (0%)
Emails Sent 0% 0.00 of 2000.00 Emails (0%)
Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)

I made 4347 request since 12am PST

[Marzia Niccolai]

Hi,
 Please be aware, that the Google App Engine quota system does not allow you to use up all of your quota at once.  If you made 4,000 requests in a very short time, say over a couple of minutes, it is probable that your application would go over quota.

 The system is designed so that you will be able to serve traffic contiguously over 24 hours, and won't let you use up your daily quota in a short period of time.  From our article:
http://code.google.com/appengine/articles/quotas.html

"If your application experiences sudden spikes in traffic, Google App Engine will attempt to serve as many of the requests as possible while ensuring your application will be able to serve some traffic throughout the rest of the day.

Consider a situation where requests for your application experience a sudden spike in traffic due to being linked from a popular site. The rate at which your application can consume resources will instantly be increase and initially all of the requests for your application will be served as the traffic starts to increase. This "spike rate" will then gradually decrease in order to ensure your app will continue to serve throughout the day while remaining under the 24-hour quota limits. Once the rate of requests pushes your application beyond this decreasing spike rate, the overage requests will be denied with a 403 error."

Thanks,
Marzia

[wave connexion(BQ)]

>Data Sent                    0% 0.25 of 2048.00 Megabytes (0%)
>Data Received              0% 0.00 of 2048.00 Megabytes (0%)

Is that in-out bandwidth limit 10GB?

--
BQ

[kordi]

I made this 4000 Request in about 45 to 60 Minutes. So this shouldnt
be a problem?

On 13 Jun., 23:18, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
> Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once. If you made 4,000 requests in a very
> short time, say over a couple of minutes, it is probable that your
> application would go over quota.
>
> The system is designed so that you will be able to serve traffic
> contiguously over 24 hours, and won't let you use up your daily quota in a
> short period of time. From our article:http://code.google.com/appengine/articles/quotas.html
>
> "If your application experiences sudden spikes in traffic, Google App Engine
> will attempt to serve as many of the requests as possible while ensuring
> your application will be able to serve some traffic throughout the rest of
> the day.
>
> Consider a situation where requests for your application experience a sudden
> spike in traffic due to being linked from a popular site. The rate at which
> your application can consume resources will instantly be increase and
> initially all of the requests for your application will be served as the
> traffic starts to increase. This "spike rate" will then gradually decrease
> in order to ensure your app will continue to serve throughout the day while
> remaining under the 24-hour quota limits. Once the rate of requests pushes
> your application beyond this decreasing spike rate, the overage requests
> will be denied with a 403 error."
> Thanks,
> Marzia
>

[babylon]

It seems like Google engineers always like to give excuses and blaming
developers for their own problem. That is why I think developers can't
rely on this service.

On Jun 14, 5:18 am, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
>  Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once.  If you made 4,000 requests in a very
> short time, say over a couple of minutes, it is probable that your
> application would go over quota.
>
>  The system is designed so that you will be able to serve traffic
> contiguously over 24 hours, and won't let you use up your daily quota in a
> short period of time.  From our article:http://code.google.com/appengine/articles/quotas.html
>
> "If your application experiences sudden spikes in traffic, Google App Engine
> will attempt to serve as many of the requests as possible while ensuring
> your application will be able to serve some traffic throughout the rest of
> the day.
>
> Consider a situation where requests for your application experience a sudden
> spike in traffic due to being linked from a popular site. The rate at which
> your application can consume resources will instantly be increase and
> initially all of the requests for your application will be served as the
> traffic starts to increase. This "spike rate" will then gradually decrease
> in order to ensure your app will continue to serve throughout the day while
> remaining under the 24-hour quota limits. Once the rate of requests pushes
> your application beyond this decreasing spike rate, the overage requests
> will be denied with a 403 error."
> Thanks,
> Marzia
>

[javaDinosaur]

> Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once.

Marzia your explanation is clear and Google's objective with rationing
the daily quotas over small time slices is understandable.

However there does seem to be a common theme of sites going over quota
without explanation. I am hoping in the future a review of the quota
accounting logic will highlight a simple bug like a misplaced decimal
point in the maths.

[Jamie Fraser]

Are you paying for the service? Nope.

I think the explanation given (which I wouldn't classify as an
"excuse") is perfectly reasonable, appropriate, and workable.

[Marzia Niccolai]

Hi,
 Thanks for this feedback.  We are still developing user resource for understanding quota usage, as well as working on giving more insight as to how quotas are calculated with the system.  We are committed to giving as much insight as possible as we continue development, so stay tuned.

 Somewhat tangentially, we do encourage everyone to use the Python profiler to get a sense of how their application is using resources.  Information on how to use the profiler can be found here:
http://code.google.com/appengine/kb/commontasks.html#profiling

 A good strategy for using the profile is to have it run for every nth request to your application (set up a random number generator in your main function to choose when to run profile_main and real_main).  This will give you an idea of how your application performs when subject to real usage.

-Marzia

[javaDinosaur]

> We are still developing user resource for understanding

> quota usage.

I would be interested to hear if elapsed time on Datastore or urlFetch
calls counts as usage against the CPU quota? There was some post
Google I-O forum debate on this subject.

[Brett Morgan]

On Wed, Jun 18, 2008 at 8:58 PM, babylon <babyl...@gmail.com> wrote:
>
> It seems like Google engineers always like to give excuses and blaming
> developers for their own problem. That is why I think developers can't
> rely on this service.
>

That is a little harsh. AppEngine is a preview release, the appengine
team is scaling up their service. As with any new software release,
there are unexpected gotchas that have to be debugged in real time.
I'm impressed with how quickly the team have scaled up.

--

Brett Morgan http://brett.morgan.googlepages.com/

[g-man]

Yes, it can be worrisome...

For instance, in my case, while developing my app, I've got one table
and two objects in the datastore, yet I get warnings!

Here is how I see the situation:

Google looks at the change in USAGE PATTERNS, so when I query the
datastore after a time of inactivity, then yes, compared to the
AVERAGE usage, even a tiny little spike looks like the beginnings of a
'slashdotting'. So, to err on the side of caution, appengine sets up a
yellow flag, saying, in essence, 'Watch out, things could be
increasing rapidly, so let's keep an eye on the situation!'

I'm sure if my average usage was taken up to the level of 1000 or more
hits a day, steadily, there would be no complaints, but if there was a
new higher spike above that level, then the warnings would start
again.

That's my understanding of the quotas from what I have read and heard
on the videos.

On Jun 18, 5:22 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:

[peterk]

So apparently I'm over quota now, and I've no idea how I could be...my
app isn't even live, I'm just using it myself for development and
debugging.

An error has occured. Your application has exceeded its quota. (Error
Code: 20) help
CPU Used
0%
21.20 of 199608.00 Gigacycles (0%)
Megabytes Stored
0%
4.96 of 500.00 Megabytes (0%)

Everything else is at zero. Even with a 24 hour moving window, how can
21.20 out of 199608 be remotely over quota? Maybe I'm not
understanding it right.

[Joel]

I've hit the 403 Over Quote page now as well.
Looking at the dashboard I can't see any indication of why I would be
over quota. I had a single peak of megacycles used/second at 400 and
even if I stayed at that rate it'd only end up being about 34000
Gigacycles for the 24 hour period. Way below the quota.
All the other resources I can see are well below any indication of a
potential problem.

I do have warnings about requests using higher than the average amount
of a request CPU limit. Perhaps it is related to that. It would be
nice to know what that means, especially related to the gigacycles
allowed per 24 hour period. If I'm allowed 200,000 Gigacycles per 24
hour window that means I ought to be able to use 2314 megacycles per
second. I don't have any indication of being near that in my
dashboard.

I'm hypothesizing that the 200,000 gigacycles quote doesn't correlate
with the high amount of CPU used per average request warnings.

Looking in the logs I track my overall request processing time as
well. I have a warning of 1.9 times over the average CPU request in
one request where it took me .28 seconds. I don't have a warning in
one which takes me .37 seconds. In both requests I am doing similar
work.

For me there is a lot of confusion relating
- Gigacycle quotas
- megacycles used per second (on dashboard)
- High CPU warnings in logs

[José Oliver Segura]

On Fri, Jul 18, 2008 at 7:57 PM, Joel <joel.r...@gmail.com> wrote:
>
> I've hit the 403 Over Quote page now as well.

> I do have warnings about requests using higher than the average amount

During my load/deletion of entities in my test app I've also
hit the 403 over quota page; no indications in the Dashboard other
than the "high cpu usage" warnings. I've had them also using my webapp
interactively from a browser, and nothing happened; using my client
api from a batch script (no related to bulkloader), I've observed that
after 10-20 warnings in a short period (10-20 seconds) my app went
overquota.

The high-cpu-usage messages are quite funny, since only tell you
about "x times the cpu usage". From that, combined with app's
profiling, I've internally assumed a limit of 250 ms for anything
related to datastore reading, and no idea about requests making
datastore updates; Trying to obtain a correlation among response time
and cpu warning, in my case, was impossible, since I've had warnings
for requests with 0.3 response time and no warnings for requests with
0.7 response times (read-only requests), and, for writing-datastore
requests, 2.0 seconds didn't make any warning to appear :-/

The problem is that I've not been able either to make that
correlation using the profiler's reported CPU usage time; low cpu
usage time (from the profiler) ocasionally produce cpu usage warnings
and, on the other side, some times those warnings don't show up for a
requests using much time that the first ones (by the profiler logs).

funny...
Jose

[Joel]

Apparently I've been really bad.
It's been 2-3 hours since I started getting the 403 over quota page
and it's still saying over quota.
That's 1/8th of the day.

It's like being grounded with no explanation :-)
Joel

[rchurch]

I am experiencing the same problem. I have been over the quota for
3.5 hours now. My dashboard doesn't show any activity.

CPU Used 0% 0.00 of 199608.00 Gigacycles (0%)
Data Sent 0% 20.48 of 2048.00 Megabytes (0%)
Data Received 0% 20.48 of 2048.00 Megabytes (0%)

Emails Sent 0% 0.00 of 2000.00 Emails (0%)

Megabytes Stored10% 50.35 of 500.00 Megabytes (10%)

How do I get my site back up?

[marc fawzi]

"How do I get my site back up?"

>> Try running v146r4.py once a day

(I'm sure they'll fix or alleviate the quota problem... eventually... But Google is an engineering organization not a service organization. So, speaking long term, I'm not sure how they'll handle the headaches of providing consistent support to millions of developers when this goes out of confined beta...)

On Fri, Jul 18, 2008 at 3:51 PM, rchurch <ni...@rchurch.com> wrote:

I am experiencing the same problem.  I have been over the quota for
3.5 hours now.  My dashboard doesn't show any activity

[Calvin Spealman]

On Fri, Jul 18, 2008 at 7:30 PM, marc fawzi <ma...@onetopia.com> wrote:
> "How do I get my site back up?"
>
>>> Try running v146r4.py once a day
>
> (I'm sure they'll fix or alleviate the quota problem... eventually... But
> Google is an engineering organization not a service organization. So,
> speaking long term, I'm not sure how they'll handle the headaches of
> providing consistent support to millions of developers when this goes out of
> confined beta...)

To google, services and support is just another engineering problem.
Everything can be engineered.

[rchurch]

12 hours later.... still getting a 403. I guess tomorrow is another
day. A bad query and you get shut down for half a day. Something
doesn't seem right.

On Jul 18, 9:30 pm, "Calvin Spealman" <ironfro...@gmail.com> wrote:

[Carsten Neubert]

On Jul 18, 8:06 pm, "José Oliver Segura" <primi...@gmail.com> wrote:
>       The high-cpu-usage messages are quite funny, since only tell you
> about "x times the cpu usage". From that, combined with app's
> profiling, I've internally assumed a limit of 250 ms for anything
> related to datastore reading, and no idea about requests making
> datastore updates; Trying to obtain a correlation among response time
> and cpu warning, in my case, was impossible, since I've had warnings
> for requests with 0.3 response time and no warnings for requests with
> 0.7 response times (read-only requests), and, for writing-datastore
> requests, 2.0 seconds didn't make any warning to appear :-/

Maybe I'm oversimplifying, but after some tests I assume that the "x
times the cpu usage" warning gets triggered by a request over about
250ms-300ms. API calls don't seem to directly count towards that
limit.

The warnings are easy to reproduce consistently by blocking the
request via "sleep" or other means not involving the API.

[Fernando Correia]

2008/7/19 rchurch <ni...@rchurch.com>:

>
> 12 hours later.... still getting a 403. I guess tomorrow is another
> day. A bad query and you get shut down for half a day. Something
> doesn't seem right.

This is a very serious concern for all of us considering trusting
Google with our applications. I expect someone from Google can provide
more information on this.

Hours of service outage caused by a few bad requests seems very outrageous.

[rchurch]

So I think I know what caused the problem on my site, just so no one
makes the same mistake.

I have a type "Church" that has 80,000+ entities.

I wanted to create a simple paginator for the 80,000 churches.

I tried doing the following:

query = Church.all()
query.order('title')
churches = query.fetch(500, 0)

after trying 4 or 5 requests, I was shut out for half a day. I read
some where that I need to filter to smaller result sets and the offset
isn't really reducing what goes into memory. If someone could confirm
this or explain what really happened... I would greatly appreciate it.



On Jul 19, 5:51 am, "Fernando Correia" <fernandoacorr...@gmail.com>
wrote:
> 2008/7/19 rchurch <n...@rchurch.com>:
>
>
>
> > 12 hours later.... still getting a403. I guess tomorrow is another

[David Symonds]

On Sun, Jul 20, 2008 at 3:17 PM, rchurch <ni...@rchurch.com> wrote:
>
> So I think I know what caused the problem on my site, just so no one
> makes the same mistake.
>
> I have a type "Church" that has 80,000+ entities.
>
> I wanted to create a simple paginator for the 80,000 churches.
>
> I tried doing the following:
>
> query = Church.all()
> query.order('title')
> churches = query.fetch(500, 0)
>
> after trying 4 or 5 requests, I was shut out for half a day. I read
> some where that I need to filter to smaller result sets and the offset
> isn't really reducing what goes into memory. If someone could confirm
> this or explain what really happened... I would greatly appreciate it.

That's correct. Don't paginate by number directly, paginate by some
property on your entities:

query = Church.all().order('title')
if 'next_first_key' in self.request.GET:
query.filter('title >=', Church.get(self.request.get('next_first_key')).title)
churches = query.fetch(501) # one extra
churches, next_first_key = churches[:-1], churches[-1].key()

Then your "next page" link will go to something with
"?next_first_key={{ next_first_key }}".

You'll also have to add code to deal with the edge cases, but the
basic principle here (relying on the datastore indexes to do your
offsetting) is how to get the best performance from GAE.

Dave.

[magnamerc]

I just experienced an 8+ hour outtage of my website. It was my first
"over quota" outtage, and my last, because my domain is going back to
my old hosting service.

Google: The whole point of me using your service, was to avoid lengthy
outtages.

I bought into the whole idea that, my application would scale
automatically as traffic grew. And that any "spike" in traffic would
be handled. But instead, I find, that my site is down, and the reason
seems quite arbitrary.

Google needs to be upfront about what constitutes going "over quota",
there shouldn't be any guess work. If the application is at fault,
then by all means shut it down. But if its just traffic, then fullfill
your end of the bargain.

[Barry Hunter]

It does say "This is a PREVIEW RELEASE of Google App Engine--for now,
applications are restricted to the free account limits. NEW! Now
everyone can try Google App Engine without waiting for an account. Try
it now!" right on the homepage.

Its not set to really scale, just a preview of what will (hopefully)
be possible. Hosting 'real world' apps on it at the moment is always
going to be a risk.

--
Barry

- www.nearby.org.uk - www.geograph.org.uk -

[thedude]

i agree with magnamerc, i understand this is a preview release, but
seriously, it is at this point a novel, overly hyped service from what
i gather. if this was anyone "but google" who offered it, we'd all go
back to media temple, php and mysql, where i've happily and quickly
build deployed, scalable apps, that have NEVER been down.

i posted 100 entries into a data model that has 3 properties, and got
"over quota" for 4 hours....and every single admin console tool said i
was using 0% of everything/anything.



On Jul 29, 11:11 am, "Barry Hunter" <barrybhun...@googlemail.com>
wrote:

> It does say "This is a PREVIEW RELEASE of Google App Engine--for now,
> applications are restricted to the free account limits. NEW! Now
> everyone can try Google App Engine without waiting for an account. Try
> it now!" right on the homepage.
>
> Its not set to really scale, just a preview of what will (hopefully)
> be possible. Hosting 'real world' apps on it at the moment is always
> going to be a risk.
>
>
>
> On Tue, Jul 29, 2008 at 4:19 PM, magnamerc <magnam...@yahoo.com> wrote:
>
> > I just experienced an 8+ hour outtage of my website. It was my first

> > "overquota" outtage, and my last, because my domain is going back to

> > my old hosting service.
>
> > Google: The whole point of me using your service, was to avoid lengthy
> > outtages.
>
> > I bought into the whole idea that, my application would scale
> > automatically as traffic grew. And that any "spike" in traffic would
> > be handled. But instead, I find, that my site is down, and the reason
> > seems quite arbitrary.
>

> > Google needs to be upfront about what constitutes going "overquota",

[thedude]

here is what is more fun....my app was running fine all day. i've done
not a single datastore request, nothing. i just changed some of the
code, and did a appcfg.py update [app].

boom, i'm over quota. . . .wow.....php/mysql, will you take me
back ;-)

On Jul 29, 11:11 am, "Barry Hunter" <barrybhun...@googlemail.com>
wrote:

> It does say "This is a PREVIEW RELEASE of Google App Engine--for now,
> applications are restricted to the free account limits. NEW! Now
> everyone can try Google App Engine without waiting for an account. Try
> it now!" right on the homepage.
>
> Its not set to really scale, just a preview of what will (hopefully)
> be possible. Hosting 'real world' apps on it at the moment is always
> going to be a risk.
>
>
>
> On Tue, Jul 29, 2008 at 4:19 PM, magnamerc <magnam...@yahoo.com> wrote:
>
> > I just experienced an 8+ hour outtage of my website. It was my first

> > "overquota" outtage, and my last, because my domain is going back to

> > my old hosting service.
>
> > Google: The whole point of me using your service, was to avoid lengthy
> > outtages.
>
> > I bought into the whole idea that, my application would scale
> > automatically as traffic grew. And that any "spike" in traffic would
> > be handled. But instead, I find, that my site is down, and the reason
> > seems quite arbitrary.
>

> > Google needs to be upfront about what constitutes going "overquota",