CPU Used 0% 380.76 of 199608.00 Gigacycles (0%)
Data Sent 0% 0.25 of 2048.00 Megabytes (0%)
Data Received 0% 0.00 of 2048.00 Megabytes (0%)
Emails Sent 0% 0.00 of 2000.00 Emails (0%)
Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
Hi, Please be aware, that the Google App Engine quota system does not allow you to use up all of your quota at once. If you made 4,000 requests in a very short time, say over a couple of minutes, it is probable that your application would go over quota.
The system is designed so that you will be able to serve traffic contiguously over 24 hours, and won't let you use up your daily quota in a short period of time. From our article: http://code.google.com/appengine/articles/quotas.html
"If your application experiences sudden spikes in traffic, Google App Engine will attempt to serve as many of the requests as possible while ensuring your application will be able to serve some traffic throughout the rest of the day.
Consider a situation where requests for your application experience a sudden spike in traffic due to being linked from a popular site. The rate at which your application can consume resources will instantly be increase and initially all of the requests for your application will be served as the traffic starts to increase. This "spike rate" will then gradually decrease in order to ensure your app will continue to serve throughout the day while remaining under the 24-hour quota limits. Once the rate of requests pushes your application beyond this decreasing spike rate, the overage requests will be denied with a 403 error." Thanks, Marzia
On Fri, Jun 13, 2008 at 3:39 AM, kordi <kordu...@googlemail.com> wrote:
> Why is my application over quota?
> CPU Used 0% 380.76 of 199608.00 Gigacycles (0%) > Data Sent 0% 0.25 of 2048.00 Megabytes (0%) > Data Received 0% 0.00 of 2048.00 Megabytes (0%) > Emails Sent 0% 0.00 of 2000.00 Emails (0%) > Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
On Sat, Jun 14, 2008 at 5:18 AM, Marzia Niccolai <ma...@google.com> wrote: > Hi, > Please be aware, that the Google App Engine quota system does not allow > you to use up all of your quota at once. If you made 4,000 requests in a > very short time, say over a couple of minutes, it is probable that your > application would go over quota.
> The system is designed so that you will be able to serve traffic > contiguously over 24 hours, and won't let you use up your daily quota in a > short period of time. From our article: > http://code.google.com/appengine/articles/quotas.html
> "If your application experiences sudden spikes in traffic, Google App > Engine will attempt to serve as many of the requests as possible while > ensuring your application will be able to serve some traffic throughout the > rest of the day.
> Consider a situation where requests for your application experience a > sudden spike in traffic due to being linked from a popular site. The rate at > which your application can consume resources will instantly be increase and > initially all of the requests for your application will be served as the > traffic starts to increase. This "spike rate" will then gradually decrease > in order to ensure your app will continue to serve throughout the day while > remaining under the 24-hour quota limits. Once the rate of requests pushes > your application beyond this decreasing spike rate, the overage requests > will be denied with a 403 error." > Thanks, > Marzia
> On Fri, Jun 13, 2008 at 3:39 AM, kordi <kordu...@googlemail.com> wrote:
>> Why is my application over quota?
>> CPU Used 0% 380.76 of 199608.00 Gigacycles (0%) >> Data Sent 0% 0.25 of 2048.00 Megabytes (0%) >> Data Received 0% 0.00 of 2048.00 Megabytes (0%) >> Emails Sent 0% 0.00 of 2000.00 Emails (0%) >> Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
> Hi,
> Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once. If you made 4,000 requests in a very
> short time, say over a couple of minutes, it is probable that your
> application would go over quota.
> The system is designed so that you will be able to serve traffic
> contiguously over 24 hours, and won't let you use up your daily quota in a
> short period of time. From our article:http://code.google.com/appengine/articles/quotas.html
> "If your application experiences sudden spikes in traffic, Google App Engine
> will attempt to serve as many of the requests as possible while ensuring
> your application will be able to serve some traffic throughout the rest of
> the day.
> Consider a situation where requests for your application experience a sudden
> spike in traffic due to being linked from a popular site. The rate at which
> your application can consume resources will instantly be increase and
> initially all of the requests for your application will be served as the
> traffic starts to increase. This "spike rate" will then gradually decrease
> in order to ensure your app will continue to serve throughout the day while
> remaining under the 24-hour quota limits. Once the rate of requests pushes
> your application beyond this decreasing spike rate, the overage requests
> will be denied with a 403 error."
> Thanks,
> Marzia
> On Fri, Jun 13, 2008 at 3:39 AM, kordi <kordu...@googlemail.com> wrote:
> > Why is my application over quota?
> > CPU Used 0% 380.76 of 199608.00 Gigacycles (0%)
> > Data Sent 0% 0.25 of 2048.00 Megabytes (0%)
> > Data Received 0% 0.00 of 2048.00 Megabytes (0%)
> > Emails Sent 0% 0.00 of 2000.00 Emails (0%)
> > Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
It seems like Google engineers always like to give excuses and blaming
developers for their own problem. That is why I think developers can't
rely on this service.
On Jun 14, 5:18 am, "Marzia Niccolai" <ma...@google.com> wrote:
> Hi,
> Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once. If you made 4,000 requests in a very
> short time, say over a couple of minutes, it is probable that your
> application would go over quota.
> The system is designed so that you will be able to serve traffic
> contiguously over 24 hours, and won't let you use up your daily quota in a
> short period of time. From our article:http://code.google.com/appengine/articles/quotas.html
> "If your application experiences sudden spikes in traffic, Google App Engine
> will attempt to serve as many of the requests as possible while ensuring
> your application will be able to serve some traffic throughout the rest of
> the day.
> Consider a situation where requests for your application experience a sudden
> spike in traffic due to being linked from a popular site. The rate at which
> your application can consume resources will instantly be increase and
> initially all of the requests for your application will be served as the
> traffic starts to increase. This "spike rate" will then gradually decrease
> in order to ensure your app will continue to serve throughout the day while
> remaining under the 24-hour quota limits. Once the rate of requests pushes
> your application beyond this decreasing spike rate, the overage requests
> will be denied with a 403 error."
> Thanks,
> Marzia
> On Fri, Jun 13, 2008 at 3:39 AM, kordi <kordu...@googlemail.com> wrote:
> > Why is my application over quota?
> > CPU Used 0% 380.76 of 199608.00 Gigacycles (0%)
> > Data Sent 0% 0.25 of 2048.00 Megabytes (0%)
> > Data Received 0% 0.00 of 2048.00 Megabytes (0%)
> > Emails Sent 0% 0.00 of 2000.00 Emails (0%)
> > Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
> Please be aware, that the Google App Engine quota system does not allow you
> to use up all of your quota at once.
Marzia your explanation is clear and Google's objective with rationing
the daily quotas over small time slices is understandable.
However there does seem to be a common theme of sites going over quota without explanation. I am hoping in the future a review of the quota accounting logic will highlight a simple bug like a misplaced decimal
point in the maths.
On Wed, Jun 18, 2008 at 11:58 AM, babylon <babylon2...@gmail.com> wrote:
> It seems like Google engineers always like to give excuses and blaming > developers for their own problem. That is why I think developers can't > rely on this service.
> On Jun 14, 5:18 am, "Marzia Niccolai" <ma...@google.com> wrote: >> Hi, >> Please be aware, that the Google App Engine quota system does not allow you >> to use up all of your quota at once. If you made 4,000 requests in a very >> short time, say over a couple of minutes, it is probable that your >> application would go over quota.
>> The system is designed so that you will be able to serve traffic >> contiguously over 24 hours, and won't let you use up your daily quota in a >> short period of time. From our article:http://code.google.com/appengine/articles/quotas.html
>> "If your application experiences sudden spikes in traffic, Google App Engine >> will attempt to serve as many of the requests as possible while ensuring >> your application will be able to serve some traffic throughout the rest of >> the day.
>> Consider a situation where requests for your application experience a sudden >> spike in traffic due to being linked from a popular site. The rate at which >> your application can consume resources will instantly be increase and >> initially all of the requests for your application will be served as the >> traffic starts to increase. This "spike rate" will then gradually decrease >> in order to ensure your app will continue to serve throughout the day while >> remaining under the 24-hour quota limits. Once the rate of requests pushes >> your application beyond this decreasing spike rate, the overage requests >> will be denied with a 403 error." >> Thanks, >> Marzia
>> On Fri, Jun 13, 2008 at 3:39 AM, kordi <kordu...@googlemail.com> wrote:
>> > Why is my application over quota?
>> > CPU Used 0% 380.76 of 199608.00 Gigacycles (0%) >> > Data Sent 0% 0.25 of 2048.00 Megabytes (0%) >> > Data Received 0% 0.00 of 2048.00 Megabytes (0%) >> > Emails Sent 0% 0.00 of 2000.00 Emails (0%) >> > Megabytes Stored 0% 5.00 of 500.00 Megabytes (0%)
Hi, Thanks for this feedback. We are still developing user resource for understanding quota usage, as well as working on giving more insight as to how quotas are calculated with the system. We are committed to giving as much insight as possible as we continue development, so stay tuned.
Somewhat tangentially, we do encourage everyone to use the Python profiler to get a sense of how their application is using resources. Information on how to use the profiler can be found here: http://code.google.com/appengine/kb/commontasks.html#profiling
A good strategy for using the profile is to have it run for every nth request to your application (set up a random number generator in your main function to choose when to run profile_main and real_main). This will give you an idea of how your application performs when subject to real usage.
-Marzia
On Wed, Jun 18, 2008 at 5:09 AM, javaDinosaur <jonathan...@hotmail.co.uk> wrote:
> > Please be aware, that the Google App Engine quota system does not allow > you > > to use up all of your quota at once.
> Marzia your explanation is clear and Google's objective with rationing > the daily quotas over small time slices is understandable.
> However there does seem to be a common theme of sites going over quota > without explanation. I am hoping in the future a review of the quota > accounting logic will highlight a simple bug like a misplaced decimal > point in the maths.
> We are still developing user resource for understanding
> quota usage.
I would be interested to hear if elapsed time on Datastore or urlFetch
calls counts as usage against the CPU quota? There was some post
Google I-O forum debate on this subject.
On Wed, Jun 18, 2008 at 8:58 PM, babylon <babylon2...@gmail.com> wrote:
> It seems like Google engineers always like to give excuses and blaming > developers for their own problem. That is why I think developers can't > rely on this service.
That is a little harsh. AppEngine is a preview release, the appengine team is scaling up their service. As with any new software release, there are unexpected gotchas that have to be debugged in real time. I'm impressed with how quickly the team have scaled up.
For instance, in my case, while developing my app, I've got one table
and two objects in the datastore, yet I get warnings!
Here is how I see the situation:
Google looks at the change in USAGE PATTERNS, so when I query the
datastore after a time of inactivity, then yes, compared to the
AVERAGE usage, even a tiny little spike looks like the beginnings of a
'slashdotting'. So, to err on the side of caution, appengine sets up a
yellow flag, saying, in essence, 'Watch out, things could be
increasing rapidly, so let's keep an eye on the situation!'
I'm sure if my average usage was taken up to the level of 1000 or more
hits a day, steadily, there would be no complaints, but if there was a
new higher spike above that level, then the warnings would start
again.
That's my understanding of the quotas from what I have read and heard
on the videos.
On Jun 18, 5:22 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:
> On Wed, Jun 18, 2008 at 8:58 PM, babylon <babylon2...@gmail.com> wrote:
> > It seems like Google engineers always like to give excuses and blaming
> > developers for their own problem. That is why I think developers can't
> > rely on this service.
> That is a little harsh. AppEngine is a preview release, the appengine
> team is scaling up their service. As with any new software release,
> there are unexpected gotchas that have to be debugged in real time.
> I'm impressed with how quickly the team have scaled up.
So apparently I'm over quota now, and I've no idea how I could be...my
app isn't even live, I'm just using it myself for development and
debugging.
An error has occured. Your application has exceeded its quota. (Error
Code: 20) help
CPU Used
0%
21.20 of 199608.00 Gigacycles (0%)
Megabytes Stored
0%
4.96 of 500.00 Megabytes (0%)
Everything else is at zero. Even with a 24 hour moving window, how can
21.20 out of 199608 be remotely over quota? Maybe I'm not
understanding it right.
On Jun 19, 8:57 pm, g-man <gregor...@gmail.com> wrote:
> For instance, in my case, while developing my app, I've got one table
> and two objects in the datastore, yet I get warnings!
> Here is how I see the situation:
> Google looks at the change in USAGE PATTERNS, so when I query the
> datastore after a time of inactivity, then yes, compared to the
> AVERAGE usage, even a tiny little spike looks like the beginnings of a
> 'slashdotting'. So, to err on the side of caution, appengine sets up a
> yellow flag, saying, in essence, 'Watch out, things could be
> increasing rapidly, so let's keep an eye on the situation!'
> I'm sure if my average usage was taken up to the level of 1000 or more
> hits a day, steadily, there would be no complaints, but if there was a
> new higher spike above that level, then the warnings would start
> again.
> That's my understanding of the quotas from what I have read and heard
> on the videos.
> On Jun 18, 5:22 pm, "Brett Morgan" <brett.mor...@gmail.com> wrote:
> > On Wed, Jun 18, 2008 at 8:58 PM, babylon <babylon2...@gmail.com> wrote:
> > > It seems like Google engineers always like to give excuses and blaming
> > > developers for their own problem. That is why I think developers can't
> > > rely on this service.
> > That is a little harsh. AppEngine is a preview release, the appengine
> > team is scaling up their service. As with any new software release,
> > there are unexpected gotchas that have to be debugged in real time.
> > I'm impressed with how quickly the team have scaled up.
I've hit the 403 Over Quote page now as well.
Looking at the dashboard I can't see any indication of why I would be
over quota. I had a single peak of megacycles used/second at 400 and
even if I stayed at that rate it'd only end up being about 34000
Gigacycles for the 24 hour period. Way below the quota.
All the other resources I can see are well below any indication of a
potential problem.
I do have warnings about requests using higher than the average amount
of a request CPU limit. Perhaps it is related to that. It would be
nice to know what that means, especially related to the gigacycles
allowed per 24 hour period. If I'm allowed 200,000 Gigacycles per 24
hour window that means I ought to be able to use 2314 megacycles per
second. I don't have any indication of being near that in my
dashboard.
I'm hypothesizing that the 200,000 gigacycles quote doesn't correlate
with the high amount of CPU used per average request warnings.
Looking in the logs I track my overall request processing time as
well. I have a warning of 1.9 times over the average CPU request in
one request where it took me .28 seconds. I don't have a warning in
one which takes me .37 seconds. In both requests I am doing similar
work.
For me there is a lot of confusion relating
- Gigacycle quotas
- megacycles used per second (on dashboard)
- High CPU warnings in logs
On Fri, Jul 18, 2008 at 7:57 PM, Joel <joel.riede...@gmail.com> wrote:
> I've hit the 403 Over Quote page now as well. > I do have warnings about requests using higher than the average amount
During my load/deletion of entities in my test app I've also hit the 403 over quota page; no indications in the Dashboard other than the "high cpu usage" warnings. I've had them also using my webapp interactively from a browser, and nothing happened; using my client api from a batch script (no related to bulkloader), I've observed that after 10-20 warnings in a short period (10-20 seconds) my app went overquota.
The high-cpu-usage messages are quite funny, since only tell you about "x times the cpu usage". From that, combined with app's profiling, I've internally assumed a limit of 250 ms for anything related to datastore reading, and no idea about requests making datastore updates; Trying to obtain a correlation among response time and cpu warning, in my case, was impossible, since I've had warnings for requests with 0.3 response time and no warnings for requests with 0.7 response times (read-only requests), and, for writing-datastore requests, 2.0 seconds didn't make any warning to appear :-/
The problem is that I've not been able either to make that correlation using the profiler's reported CPU usage time; low cpu usage time (from the profiler) ocasionally produce cpu usage warnings and, on the other side, some times those warnings don't show up for a requests using much time that the first ones (by the profiler logs).
Apparently I've been really bad.
It's been 2-3 hours since I started getting the 403 over quota page
and it's still saying over quota.
That's 1/8th of the day.
It's like being grounded with no explanation :-)
Joel
I am experiencing the same problem. I have been over the quota for
3.5 hours now. My dashboard doesn't show any activity.
CPU Used 0% 0.00 of 199608.00 Gigacycles (0%)
Data Sent 0% 20.48 of 2048.00 Megabytes (0%)
Data Received 0% 20.48 of 2048.00 Megabytes (0%)
Emails Sent 0% 0.00 of 2000.00 Emails (0%)
Megabytes Stored10% 50.35 of 500.00 Megabytes (10%)
How do I get my site back up?
On Jul 18, 12:57 pm, Joel <joel.riede...@gmail.com> wrote:
> Apparently I've been really bad.
> It's been 2-3 hours since I started getting the 403 overquotapage
> and it's still saying overquota.
> That's 1/8th of the day.
> It's like being grounded with no explanation :-)
> Joel
(I'm sure they'll fix or alleviate the quota problem... eventually... But Google is an engineering organization not a service organization. So, speaking long term, I'm not sure how they'll handle the headaches of providing consistent support to millions of developers when this goes out of confined beta...)
On Fri, Jul 18, 2008 at 3:51 PM, rchurch <n...@rchurch.com> wrote:
> I am experiencing the same problem. I have been over the quota for > 3.5 hours now. My dashboard doesn't show any activity
> CPU Used 0% 0.00 of 199608.00 Gigacycles (0%) > Data Sent 0% 20.48 of 2048.00 Megabytes (0%) > Data Received 0% 20.48 of 2048.00 Megabytes (0%) > Emails Sent 0% 0.00 of 2000.00 Emails (0%) > Megabytes Stored10% 50.35 of 500.00 Megabytes (10%)
> How do I get my site back up?
> On Jul 18, 12:57 pm, Joel <joel.riede...@gmail.com> wrote: > > Apparently I've been really bad. > > It's been 2-3 hours since I started getting the 403 overquotapage > > and it's still saying overquota. > > That's 1/8th of the day.
> > It's like being grounded with no explanation :-) > > Joel
On Fri, Jul 18, 2008 at 7:30 PM, marc fawzi <m...@onetopia.com> wrote: > "How do I get my site back up?"
>>> Try running v146r4.py once a day
> (I'm sure they'll fix or alleviate the quota problem... eventually... But > Google is an engineering organization not a service organization. So, > speaking long term, I'm not sure how they'll handle the headaches of > providing consistent support to millions of developers when this goes out of > confined beta...)
To google, services and support is just another engineering problem. Everything can be engineered.
12 hours later.... still getting a 403. I guess tomorrow is another
day. A bad query and you get shut down for half a day. Something
doesn't seem right.
On Jul 18, 9:30 pm, "Calvin Spealman" <ironfro...@gmail.com> wrote:
> On Fri, Jul 18, 2008 at 7:30 PM, marc fawzi <m...@onetopia.com> wrote:
> > "How do I get my site back up?"
> >>> Try running v146r4.py once a day
> > (I'm sure they'll fix or alleviate the quota problem... eventually... But
> > Google is an engineering organization not a service organization. So,
> > speaking long term, I'm not sure how they'll handle the headaches of
> > providing consistent support to millions of developers when this goes out of
> > confined beta...)
> To google, services and support is just another engineering problem.
> Everything can be engineered.
On Jul 18, 8:06 pm, "José Oliver Segura" <primi...@gmail.com> wrote:
> The high-cpu-usage messages are quite funny, since only tell you
> about "x times the cpu usage". From that, combined with app's
> profiling, I've internally assumed a limit of 250 ms for anything
> related to datastore reading, and no idea about requests making
> datastore updates; Trying to obtain a correlation among response time
> and cpu warning, in my case, was impossible, since I've had warnings
> for requests with 0.3 response time and no warnings for requests with
> 0.7 response times (read-only requests), and, for writing-datastore
> requests, 2.0 seconds didn't make any warning to appear :-/
Maybe I'm oversimplifying, but after some tests I assume that the "x
times the cpu usage" warning gets triggered by a request over about
250ms-300ms. API calls don't seem to directly count towards that
limit.
The warnings are easy to reproduce consistently by blocking the
request via "sleep" or other means not involving the API.
> 12 hours later.... still getting a 403. I guess tomorrow is another > day. A bad query and you get shut down for half a day. Something > doesn't seem right.
This is a very serious concern for all of us considering trusting Google with our applications. I expect someone from Google can provide more information on this.
Hours of service outage caused by a few bad requests seems very outrageous.
after trying 4 or 5 requests, I was shut out for half a day. I read
some where that I need to filter to smaller result sets and the offset
isn't really reducing what goes into memory. If someone could confirm
this or explain what really happened... I would greatly appreciate it.
On Jul 19, 5:51 am, "Fernando Correia" <fernandoacorr...@gmail.com>
wrote:
> > 12 hours later.... still getting a403. I guess tomorrow is another
> > day. A bad query and you get shut down for half a day. Something
> > doesn't seem right.
> This is a very serious concern for all of us considering trusting
> Google with our applications. I expect someone from Google can provide
> more information on this.
> Hours of service outage caused by a few bad requests seems very outrageous.
> after trying 4 or 5 requests, I was shut out for half a day. I read > some where that I need to filter to smaller result sets and the offset > isn't really reducing what goes into memory. If someone could confirm > this or explain what really happened... I would greatly appreciate it.
That's correct. Don't paginate by number directly, paginate by some property on your entities:
query = Church.all().order('title') if 'next_first_key' in self.request.GET: query.filter('title >=', Church.get(self.request.get('next_first_key')).title) churches = query.fetch(501) # one extra churches, next_first_key = churches[:-1], churches[-1].key()
Then your "next page" link will go to something with "?next_first_key={{ next_first_key }}".
You'll also have to add code to deal with the edge cases, but the basic principle here (relying on the datastore indexes to do your offsetting) is how to get the best performance from GAE.
I just experienced an 8+ hour outtage of my website. It was my first
"over quota" outtage, and my last, because my domain is going back to
my old hosting service.
Google: The whole point of me using your service, was to avoid lengthy
outtages.
I bought into the whole idea that, my application would scale
automatically as traffic grew. And that any "spike" in traffic would
be handled. But instead, I find, that my site is down, and the reason
seems quite arbitrary.
Google needs to be upfront about what constitutes going "over quota",
there shouldn't be any guess work. If the application is at fault,
then by all means shut it down. But if its just traffic, then fullfill
your end of the bargain.
It does say "This is a PREVIEW RELEASE of Google App Engine--for now, applications are restricted to the free account limits. NEW! Now everyone can try Google App Engine without waiting for an account. Try it now!" right on the homepage.
Its not set to really scale, just a preview of what will (hopefully) be possible. Hosting 'real world' apps on it at the moment is always going to be a risk.
On Tue, Jul 29, 2008 at 4:19 PM, magnamerc <magnam...@yahoo.com> wrote:
> I just experienced an 8+ hour outtage of my website. It was my first > "over quota" outtage, and my last, because my domain is going back to > my old hosting service.
> Google: The whole point of me using your service, was to avoid lengthy > outtages.
> I bought into the whole idea that, my application would scale > automatically as traffic grew. And that any "spike" in traffic would > be handled. But instead, I find, that my site is down, and the reason > seems quite arbitrary.
> Google needs to be upfront about what constitutes going "over quota", > there shouldn't be any guess work. If the application is at fault, > then by all means shut it down. But if its just traffic, then fullfill > your end of the bargain.
|
