Was I scared too much with high cpu quota ?
14 views
Skip to first unread message
Matija
Dec 18, 2008, 6:58:16 AM12/18/08
to Google App Engine
This morning I did some final testing and because app engine team
introduced 'quota details' page with some more explanation in
http://code.google.com/appengine/docs/quotas.html link now I am
thinking that I created a little bit more complicated code than I
needed. Just because they introduced suggestion warnings in admin
console log with yellow and red colors plus some warning sign from the
beginning.
I am using app engine last half year and I have read almost every
article, watched every interesting video and constantly read app
engine group discussions. In the last half year every now and then
they explained more and more about high cpu quota, difference between
api cpu and our code cpu usage and difference in its presentation over
the admin console.
My final test was simple:
class Test(webapp.RequestHandler):
def get(self):
for i in range(200):
t = Testna(podatak = str(i))
t.put()
self.response.out.write('radi')
Results were like some religious revelation for me:
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281097404943213970
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281097405750931410
Although they stated over and over (not at the beginning but lately)
these results, I needed proof. And now I have it.
I will not be punished (in quota sense) for more than one datastore
write operation. During my test I had no high cpu quota usage. There
was some warning but I think that they are there when we are near 8s
limit per request, and when we break this limit, but even then this
was not counted as high cpu quota.
I understand contention problem and even user experience with too slow
response, but I will not be punished, temporary baned or something
like that.
This is so important for me because I mostly need more write operation
only for administrative purpose. Not 200 writes but let's say 10 or
20. With bigtable system we are creating super redundant datastore,
but when I need to correct (for some reason) that data, now I know
that I don't have to do this in thousand 'one write' operation step,
than in a 50 (or less) '20 write' operation request. If I do this when
there is low usage (night for example) I could even bypass some
contention problem.
Second reason was lack of model count ability. So when I needed to use
additional counter model with shard principle for avoiding contention
problem. But I needed two write operation. In my fear of 'punishment'
i did this in two connected ajax requests. If first request (write
operation) succeeded I will send second request to increment count.
With this approach I was under 1000 mcycles cpu usage (with datastore
api), but this also introduced new problems. What if second request
has not succeeded. What now.
Now I have wrong count data. With this more complicated approach I had
problem and even I have thought that app engine could not be used for
serious purpose where this discrepancy in data is not an option. Last
five years I was lead developer for hospital information system so you
can understand my fear for error data. Counting comments on web page,
poll data or something like data can have 99% precision but something
else can't. Of course I am not developing now something serious like
hospital information system but it takes time to switch from 100%
desktop application data to 99% web application data.
And now I have found out that this was not necessary and that I can
have 100% data precision in connection with user perception of
succeeded or not succeeded operation. I can have two, four, six or
more write operation in one request only if I stay under 8(or 9?)
seconds time request limit because contention can impact only on
request time limit.
All these high cpu fuss (in connection with write operations, of
course) were just design warnings and not some forcing statement.
Or am I wrong again ?
introduced 'quota details' page with some more explanation in
http://code.google.com/appengine/docs/quotas.html link now I am
thinking that I created a little bit more complicated code than I
needed. Just because they introduced suggestion warnings in admin
console log with yellow and red colors plus some warning sign from the
beginning.
I am using app engine last half year and I have read almost every
article, watched every interesting video and constantly read app
engine group discussions. In the last half year every now and then
they explained more and more about high cpu quota, difference between
api cpu and our code cpu usage and difference in its presentation over
the admin console.
My final test was simple:
class Test(webapp.RequestHandler):
def get(self):
for i in range(200):
t = Testna(podatak = str(i))
t.put()
self.response.out.write('radi')
Results were like some religious revelation for me:
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281097404943213970
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281097405750931410
Although they stated over and over (not at the beginning but lately)
these results, I needed proof. And now I have it.
I will not be punished (in quota sense) for more than one datastore
write operation. During my test I had no high cpu quota usage. There
was some warning but I think that they are there when we are near 8s
limit per request, and when we break this limit, but even then this
was not counted as high cpu quota.
I understand contention problem and even user experience with too slow
response, but I will not be punished, temporary baned or something
like that.
This is so important for me because I mostly need more write operation
only for administrative purpose. Not 200 writes but let's say 10 or
20. With bigtable system we are creating super redundant datastore,
but when I need to correct (for some reason) that data, now I know
that I don't have to do this in thousand 'one write' operation step,
than in a 50 (or less) '20 write' operation request. If I do this when
there is low usage (night for example) I could even bypass some
contention problem.
Second reason was lack of model count ability. So when I needed to use
additional counter model with shard principle for avoiding contention
problem. But I needed two write operation. In my fear of 'punishment'
i did this in two connected ajax requests. If first request (write
operation) succeeded I will send second request to increment count.
With this approach I was under 1000 mcycles cpu usage (with datastore
api), but this also introduced new problems. What if second request
has not succeeded. What now.
Now I have wrong count data. With this more complicated approach I had
problem and even I have thought that app engine could not be used for
serious purpose where this discrepancy in data is not an option. Last
five years I was lead developer for hospital information system so you
can understand my fear for error data. Counting comments on web page,
poll data or something like data can have 99% precision but something
else can't. Of course I am not developing now something serious like
hospital information system but it takes time to switch from 100%
desktop application data to 99% web application data.
And now I have found out that this was not necessary and that I can
have 100% data precision in connection with user perception of
succeeded or not succeeded operation. I can have two, four, six or
more write operation in one request only if I stay under 8(or 9?)
seconds time request limit because contention can impact only on
request time limit.
All these high cpu fuss (in connection with write operations, of
course) were just design warnings and not some forcing statement.
Or am I wrong again ?
Jeff S
Dec 18, 2008, 5:33:59 PM12/18/08
to Google App Engine
Hi Matija,
Glad that we could be of help in simplifying your app design (I recall
our conversation on IRC). It does seem like some misunderstandings
have cropped up about CPU utilization, measurement, etc. and we've
tried to make the CPU usage measurements more clear.
There were a couple of points I thought I should chime in on below, to
try to reduce confusion for those that are new to these design points.
On Dec 18, 3:58 am, Matija <matija.jerko...@gmail.com> wrote:
> This morning I did some final testing and because app engine team
> introduced 'quota details' page with some more explanation inhttp://code.google.com/appengine/docs/quotas.htmllink now I am
simple to accurately measure performance for many people's apps (might
be accurate for the simple counter case). The inserted entities here
are very simple. Things like the size and structure of the data,
number and complexity of indexes, entity group relationships, etc. can
all effect the datastore CPU usage, response time, and/or likelyhood
of contention. I'd go as far as to say it is impossible to make a
blanket statement about how many writes can be performed per request
without understanding the "shape" of the data.
>
> Results were like some religious revelation for me:
>
> http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9...http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9...
was using too much runtime CPU ("This request used a high amount of
CPU, and was roughly 2.9 times over the average request CPU
limit...."). At the moment there is a limit of two high CPU requests
per minute, and you can bank up to sixty "credits" for high runtime
CPU. If you had run this test continually I suspect you would have
begun to see quota denials. (See http://code.google.com/appengine/kb/general.html#highcpu
for more information.)
>
> I understand contention problem and even user experience with too slow
> response, but I will not be punished, temporary baned or something
> like that.
Contention will actually effect more than just the response time.
Failed transactions are automatically retried, consuming datastore
CPU, so this is another limit which you could hit. Eventually, a
transaction or write may fail (and raise an exception) after
unsuccessful retries.
>
> This is so important for me because I mostly need more write operation
> only for administrative purpose. Not 200 writes but let's say 10 or
> 20. With bigtable system we are creating super redundant datastore,
> but when I need to correct (for some reason) that data, now I know
> that I don't have to do this in thousand 'one write' operation step,
> than in a 50 (or less) '20 write' operation request. If I do this when
> there is low usage (night for example) I could even bypass some
> contention problem.
In general, the idea is to minimize the number of concurrent writes to
a particular entity or entity group. I wasn't completely sure about
what you meant by low usage in the above. Low usage of your app by
users?
While in some cases these warnings do not indicate that your app will
be immediately blocked on quotas, they can be an indicator of a design
problem. The CPU threshold of 1000 megacycles applies to runtime CPU,
so the colored warning numbers which include API CPU usage do not
always need to be under 1000, but the combined numbers are an
indicator of the overall CPU utilization in a request.
Happy coding,
Jeff
Glad that we could be of help in simplifying your app design (I recall
our conversation on IRC). It does seem like some misunderstandings
have cropped up about CPU utilization, measurement, etc. and we've
tried to make the CPU usage measurements more clear.
There were a couple of points I thought I should chime in on below, to
try to reduce confusion for those that are new to these design points.
On Dec 18, 3:58 am, Matija <matija.jerko...@gmail.com> wrote:
> This morning I did some final testing and because app engine team
> thinking that I created a little bit more complicated code than I
> needed. Just because they introduced suggestion warnings in admin
> console log with yellow and red colors plus some warning sign from the
> beginning.
>
> I am using app engine last half year and I have read almost every
> article, watched every interesting video and constantly read app
> engine group discussions. In the last half year every now and then
> they explained more and more about high cpu quota, difference between
> api cpu and our code cpu usage and difference in its presentation over
> the admin console.
>
> My final test was simple:
>
> class Test(webapp.RequestHandler):
>
> def get(self):
> for i in range(200):
> t = Testna(podatak = str(i))
> t.put()
>
> self.response.out.write('radi')
It's great that you ran a test, but I think the above may be too
> needed. Just because they introduced suggestion warnings in admin
> console log with yellow and red colors plus some warning sign from the
> beginning.
>
> I am using app engine last half year and I have read almost every
> article, watched every interesting video and constantly read app
> engine group discussions. In the last half year every now and then
> they explained more and more about high cpu quota, difference between
> api cpu and our code cpu usage and difference in its presentation over
> the admin console.
>
> My final test was simple:
>
> class Test(webapp.RequestHandler):
>
> def get(self):
> for i in range(200):
> t = Testna(podatak = str(i))
> t.put()
>
> self.response.out.write('radi')
simple to accurately measure performance for many people's apps (might
be accurate for the simple counter case). The inserted entities here
are very simple. Things like the size and structure of the data,
number and complexity of indexes, entity group relationships, etc. can
all effect the datastore CPU usage, response time, and/or likelyhood
of contention. I'd go as far as to say it is impossible to make a
blanket statement about how many writes can be performed per request
without understanding the "shape" of the data.
>
> Results were like some religious revelation for me:
>
>
> Although they stated over and over (not at the beginning but lately)
> these results, I needed proof. And now I have it.
>
> I will not be punished (in quota sense) for more than one datastore
> write operation. During my test I had no high cpu quota usage. There
> was some warning but I think that they are there when we are near 8s
> limit per request, and when we break this limit, but even then this
> was not counted as high cpu quota.
The picture of the logs which you posted did indicate that the request
> Although they stated over and over (not at the beginning but lately)
> these results, I needed proof. And now I have it.
>
> I will not be punished (in quota sense) for more than one datastore
> write operation. During my test I had no high cpu quota usage. There
> was some warning but I think that they are there when we are near 8s
> limit per request, and when we break this limit, but even then this
> was not counted as high cpu quota.
was using too much runtime CPU ("This request used a high amount of
CPU, and was roughly 2.9 times over the average request CPU
limit...."). At the moment there is a limit of two high CPU requests
per minute, and you can bank up to sixty "credits" for high runtime
CPU. If you had run this test continually I suspect you would have
begun to see quota denials. (See http://code.google.com/appengine/kb/general.html#highcpu
for more information.)
>
> I understand contention problem and even user experience with too slow
> response, but I will not be punished, temporary baned or something
> like that.
Failed transactions are automatically retried, consuming datastore
CPU, so this is another limit which you could hit. Eventually, a
transaction or write may fail (and raise an exception) after
unsuccessful retries.
>
> This is so important for me because I mostly need more write operation
> only for administrative purpose. Not 200 writes but let's say 10 or
> 20. With bigtable system we are creating super redundant datastore,
> but when I need to correct (for some reason) that data, now I know
> that I don't have to do this in thousand 'one write' operation step,
> than in a 50 (or less) '20 write' operation request. If I do this when
> there is low usage (night for example) I could even bypass some
> contention problem.
a particular entity or entity group. I wasn't completely sure about
what you meant by low usage in the above. Low usage of your app by
users?
be immediately blocked on quotas, they can be an indicator of a design
problem. The CPU threshold of 1000 megacycles applies to runtime CPU,
so the colored warning numbers which include API CPU usage do not
always need to be under 1000, but the combined numbers are an
indicator of the overall CPU utilization in a request.
Happy coding,
Jeff
Matija
Dec 19, 2008, 5:25:43 AM12/19/08
to Google App Engine
Hi Jeff
On Dec 18, 11:33 pm, Jeff S <j...@google.com> wrote:
> Hi Matija,
>
> Glad that we could be of help in simplifying your app design (I recall
> our conversation on IRC). It does seem like some misunderstandings
> have cropped up about CPU utilization, measurement, etc. and we've
> tried to make the CPU usage measurements more clear.
>
> There were a couple of points I thought I should chime in on below, to
> try to reduce confusion for those that are new to these design points.
>
> On Dec 18, 3:58 am, Matija <matija.jerko...@gmail.com> wrote:
>
>
>
> > This morning I did some final testing and because app engine team
> > introduced 'quota details' page with some more explanation inhttp://code.google.com/appengine/docs/quotas.htmllinknow I am
I understand connection between size and structure of the data,
indexes, etc. with overall CPU usage. Actually I did start test with
one write, two, four, eight, 100 and finally with 12 times 200 write
operations. Simple one write operation (no indexes, low size table
etc.) cost around 470 ms-cpu. Two 940 ms-cpu, 4 1880 ms-cpu, 8 3750 ms-
cpu, 100 23400 ms-cpu and 200 write operation request 46600 ms-cpu.
And as I noticed under 10 write operations there was proportional cpu
time with write operation number at about 470 ms per write operation.
With more data in table, few indexes and of course some more code than
a simple datastore api we couldn't have more than one datastore write
operation per request to stay under 1000 ms-cpu limit, if datastore
api isn't directly excluded from high cpu quota.
Nobody should have too many write operation because multiuser
environment (which web applications are) have extreme contention
problem and actually great scalability solutions are in a way
contention bypass solutions. I am sure that somebodies one write
operation request could have big contention problem in a poorly
designed system, not to mention how it would be in more write
operations request. If this is not true why do we even bother with
shard counter principle!
> > Results were like some religious revelation for me:
>
> >http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9......
with 200 write operation request. Six were with no warning message,
one with warning message(..was roughly 1.1 times over..) and five were
DeadlineExceededError error (i did some parallel request to test
contention in combination with near request time limit). Two out of
this five had also warning message (...roughly 2.9 times
over..., ...roughly 1.0 times over...). But during this test I never
saw any high cpu quota usage in quota details (other quota data did
change).
And here I went to do more tests.
23 requests with 200 write operation.
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281440712941963666
Indeed, when ever I noticed 'This request used a high amount of
CPU...' message there was high cpu quota usage. I didn't noticed first
time because high cpu quota is replenished pretty fast for one web
site user. 11 request out of this 23 have had high cpu warning
(...roughly 1.0 times...). Big question is why other 12 request
didn't. This time I didn't get any DeadlineExceededError error.
Then I did 20 requests with 100 write operation. There was two high
cpu usage warning and even one 'InternalError: internal error' (sorry
for that, but that message did bring me memory on ms sql server :)).
And finally I did 100 requests with 10 write operation. Ctrl+tab, Ctrl
+t, ctrl+v, enter and F5 can do miracle in no time. No high cpu usage
warning at all.
Funny but now I noticed also that overall cpu time per write operation
is now 235 ms. Is this result of some positive scalability growth or
simple small datastore usage during USA night time. Interesting time
zone difference impact between USA and Europe day time.
Conclusion is that I can be safe with more than one datastore write
operation request, but I need to take care for possible contention
situations, that can evolve even in one datastore write operation
requests.
Tnx, Matija.
On Dec 18, 11:33 pm, Jeff S <j...@google.com> wrote:
> Hi Matija,
>
> Glad that we could be of help in simplifying your app design (I recall
> our conversation on IRC). It does seem like some misunderstandings
> have cropped up about CPU utilization, measurement, etc. and we've
> tried to make the CPU usage measurements more clear.
>
> There were a couple of points I thought I should chime in on below, to
> try to reduce confusion for those that are new to these design points.
>
> On Dec 18, 3:58 am, Matija <matija.jerko...@gmail.com> wrote:
>
>
>
> > This morning I did some final testing and because app engine team
indexes, etc. with overall CPU usage. Actually I did start test with
one write, two, four, eight, 100 and finally with 12 times 200 write
operations. Simple one write operation (no indexes, low size table
etc.) cost around 470 ms-cpu. Two 940 ms-cpu, 4 1880 ms-cpu, 8 3750 ms-
cpu, 100 23400 ms-cpu and 200 write operation request 46600 ms-cpu.
And as I noticed under 10 write operations there was proportional cpu
time with write operation number at about 470 ms per write operation.
With more data in table, few indexes and of course some more code than
a simple datastore api we couldn't have more than one datastore write
operation per request to stay under 1000 ms-cpu limit, if datastore
api isn't directly excluded from high cpu quota.
Nobody should have too many write operation because multiuser
environment (which web applications are) have extreme contention
problem and actually great scalability solutions are in a way
contention bypass solutions. I am sure that somebodies one write
operation request could have big contention problem in a poorly
designed system, not to mention how it would be in more write
operations request. If this is not true why do we even bother with
shard counter principle!
> > Results were like some religious revelation for me:
>
>
> > Although they stated over and over (not at the beginning but lately)
> > these results, I needed proof. And now I have it.
>
> > I will not be punished (in quota sense) for more than one datastore
> > write operation. During my test I had no high cpu quota usage. There
> > was some warning but I think that they are there when we are near 8s
> > limit per request, and when we break this limit, but even then this
> > was not counted as high cpu quota.
>
> The picture of the logs which you posted did indicate that the request
> was using too much runtime CPU ("This request used a high amount of
> CPU, and was roughly 2.9 times over the average request CPU
> limit...."). At the moment there is a limit of two high CPU requests
> per minute, and you can bank up to sixty "credits" for high runtime
> CPU. If you had run this test continually I suspect you would have
> begun to see quota denials. (Seehttp://code.google.com/appengine/kb/general.html#highcpu
> for more information.)
Actually I think that here is something else involved. I did 12 test
> > Although they stated over and over (not at the beginning but lately)
> > these results, I needed proof. And now I have it.
>
> > I will not be punished (in quota sense) for more than one datastore
> > write operation. During my test I had no high cpu quota usage. There
> > was some warning but I think that they are there when we are near 8s
> > limit per request, and when we break this limit, but even then this
> > was not counted as high cpu quota.
>
> The picture of the logs which you posted did indicate that the request
> was using too much runtime CPU ("This request used a high amount of
> CPU, and was roughly 2.9 times over the average request CPU
> limit...."). At the moment there is a limit of two high CPU requests
> per minute, and you can bank up to sixty "credits" for high runtime
> CPU. If you had run this test continually I suspect you would have
> begun to see quota denials. (Seehttp://code.google.com/appengine/kb/general.html#highcpu
> for more information.)
with 200 write operation request. Six were with no warning message,
one with warning message(..was roughly 1.1 times over..) and five were
DeadlineExceededError error (i did some parallel request to test
contention in combination with near request time limit). Two out of
this five had also warning message (...roughly 2.9 times
over..., ...roughly 1.0 times over...). But during this test I never
saw any high cpu quota usage in quota details (other quota data did
change).
And here I went to do more tests.
23 requests with 200 write operation.
http://picasaweb.google.com/matija.jerkovic/GoogleAppEngine?authkey=9MVoK12s0SA#5281440712941963666
Indeed, when ever I noticed 'This request used a high amount of
CPU...' message there was high cpu quota usage. I didn't noticed first
time because high cpu quota is replenished pretty fast for one web
site user. 11 request out of this 23 have had high cpu warning
(...roughly 1.0 times...). Big question is why other 12 request
didn't. This time I didn't get any DeadlineExceededError error.
Then I did 20 requests with 100 write operation. There was two high
cpu usage warning and even one 'InternalError: internal error' (sorry
for that, but that message did bring me memory on ms sql server :)).
And finally I did 100 requests with 10 write operation. Ctrl+tab, Ctrl
+t, ctrl+v, enter and F5 can do miracle in no time. No high cpu usage
warning at all.
Funny but now I noticed also that overall cpu time per write operation
is now 235 ms. Is this result of some positive scalability growth or
simple small datastore usage during USA night time. Interesting time
zone difference impact between USA and Europe day time.
Conclusion is that I can be safe with more than one datastore write
operation request, but I need to take care for possible contention
situations, that can evolve even in one datastore write operation
requests.
Tnx, Matija.
Reply all
Reply to author
Forward
0 new messages