DeadlineExceededError on HRD and sky-high latency (python27+threadsafe)
sergio...@gmail.com
I've successfully migrated from master-slave to hdd with python 27 and
threadsafe however I've been getting sporadically a few deadline
exceeded errors (bellow stack trace excerpt).
I also have configured min idle requests to automatic and maximum to 1
and pending latency to 800ms, I've observed on occasion latencies of
up to 63000(!!!)ms and without app engine spinning extra instances to
serve requests.
Any thoughts? I can provide app-id off-list if someone from google
wants to have a look.
Thanks!
File "/base/python27_runtime/python27_lib/versions/1/google/appengine/
ext/db/__init__.py", line 3588, in __get__
instance = get(reference_id)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 1515, in get
return get_async(keys, **kwargs).get_result()
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/api/apiproxy_stub_map.py", line 592, in get_result
return self.__get_result_hook(self)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/datastore/datastore_rpc.py", line 1437, in __get_hook
entities = rpc.user_data(entities)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/api/datastore.py", line 623, in local_extra_hook
return extra_hook(result)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 1485, in extra_hook
model = cls1.from_entity(entity)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 1419, in from_entity
return cls(None, _from_entity=entity, **entity_values)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 1800, in __init__
super(Expando, self).__init__(parent, key_name, _app, **kwds)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 945, in __init__
prop.__set__(self, value)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 600, in __set__
setattr(model_instance, self._attr_name(), value)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 1826, in __setattr__
check_reserved_word(key)
File "/base/python27_runtime/python27_lib/versions/1/google/
appengine/ext/db/__init__.py", line 300, in check_reserved_word
if datastore_types.RESERVED_PROPERTY_NAME.match(attr_name):
DeadlineExceededError
Brandon Wirtz
simultaneous requests and each request takes 1 second, and your spin up time
is 5 seconds, then you will have time outs with the settings you describe.
Consider running full auto for 24 hours and then clamping down. Also
consider reading the tuning guides published on other threads.
Thanks!
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to
google-appengi...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
sergio...@gmail.com
The issue is that I've seen sky-high latencies *without* the app
engine even trying to spin other instances!
Just right now I captured this screenshot from the management console:
http://i42.tinypic.com/bhzyf7.png
The above screenshot is not normal! BTW, around the same time I took
the screenshot I got a 500-Internel server error accessing the app
engine dashboard at appspot.com.
Most of the time I've got ~ 0.2 QPS and 100-200ms latencies that's why
I have just one idle instance.
On Dec 5, 9:43 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
> Allow more than one idle instance. How long is your spin-up? If you get 8
> simultaneous requests and each request takes 1 second, and your spin up time
> is 5 seconds, then you will have time outs with the settings you describe.
>
> Consider running full auto for 24 hours and then clamping down. Also
> consider reading the tuning guides published on other threads.
>
>
>
>
>
>
>
> -----Original Message-----
> From: google-a...@googlegroups.com
>
> [mailto:google-a...@googlegroups.com] On Behalf Of
Andrius A
Alex Epshteyn
datastore. Didn't see this thread, so started my own:
http://groups.google.com/group/google-appengine/browse_thread/thread/31d1c228070ac09b#
Also filed an issue:
http://code.google.com/p/googleappengine/issues/detail?id=6447
I also don't think it's a matter of having more instances. My app
seems to have tons of idle instances sitting around (despite my max
idle instance setting being 1), and still I'm seeing tons of pending
latency.
On Dec 5, 5:37 pm, Andrius A <andriu...@gmail.com> wrote:
> I can confirm that as well, don't think its related with python 2.7 or HRD.
> I am seeing high latencies similar to yours with python 2.5 and MS just
> right now, even GAE console takes ages to load and some times throws
> errors. I noticed that similar slowness happens few times per week, I
> wonder will it ever become stable?
>
> On 5 December 2011 22:32, Sergio.Jar...@gmail.com
> <sergio.jar...@gmail.com>wrote:
Sarang
Andrius A
--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/-JsYHq24EyMJ.
Brian Quinlan
<sergio...@gmail.com> wrote:
> Hi!
> I've successfully migrated from master-slave to hdd with python 27 and
> threadsafe however I've been getting sporadically a few deadline
> exceeded errors (bellow stack trace excerpt).
>
> I also have configured min idle requests to automatic and maximum to 1
> and pending latency to 800ms, I've observed on occasion latencies of
> up to 63000(!!!)ms and without app engine spinning extra instances to
> serve requests.
>
> Any thoughts? I can provide app-id off-list if someone from google
> wants to have a look.
You are likely encountering this issue:
http://code.google.com/p/googleappengine/issues/detail?id=6323
Could you try setting threadsafe to "no" and seeing if that fixes the problem?
Cheers,
Brian
> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
Brian Quinlan
On Wed, Dec 7, 2011 at 9:14 AM, Andrius A <andr...@gmail.com> wrote:
> You are not the only one, I don't like the fact that google puts a blame on
> MS, there is something wrong with GAE constantly going on and they should
> admit that and explain us in more details what's happening.
There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.
There are also known issues where accessing the MS datastore can
result in highly variable latencies and possibly errors. This is the
result of the design of the MS datastore and is why we promote
migrating to the HR datastore.
The App Engine Administration Console uses the MS datastore (for a
variety of esoteric reasons) so its latency spikes will often
correlate with MS latency spikes.
> We have a form for issue like this to be noticed, please fill in to get more
> attention:
> https://docs.google.com/spreadsheet/viewform?hl=en_US&formkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0
Can I also suggest that people file bugs using the public issue tracker at:
http://code.google.com/p/googleappengine/issues ?
Also, if you are using an experimental feature, it would be great if
you do a quick search to see if the issue has already been filed and
make a note or star it yourself it is has. The more information we
have the easier it is to prioritize and fix bugs.
> I really believe into GAE and still think this is a great a platform, but it
> has issues which need be admitted and taken care of.
No one is more aware of App Engine's imperfections than the App Engine
team and we have a public issue tracker so that people can make us
aware of their particular issues and see what problems other people
are having.
Cheers,
Brian
Andrius A
Brian Quinlan
On Wed, Dec 7, 2011 at 11:15 AM, Andrius A <andr...@gmail.com> wrote:
> Hey Brian,
>
> thank you for your reply. I am still on 2.5 since 2.7 is still experimental,
> and I don't really believe that MS is everything to blame, there are issues
> with requests latencies, which I have monitored and there was nothing wrong
> with datastore. I have been posting about that here and opened a ticket but
> have not received an answer unfortunately.
I couldn't see your name attached to an issue in the public issue
tracker. Could you provide a link?
> I am moaning here all the time because we shouldn't be charged so much as
> you confirmed already many times that there are issues and 2.7 is still
> faulty.
> The main point here is that we should not pay more just because GAE is now
> called production ready. I will be repeating that until you go back to
> preview release or admit that by dropping the prices or improve the
> reliability.
You can send pricing-related issues to Peter at p...@google.com.
Cheers,
Brian
Andrius A
Felippe Bueno
There is a known bug where the combination of Python 2.7 and enabling
concurrent requests causes large latency increases. The Python 2.7
runtime is still experimental and bugs like this are expected.
code changes?
Python concurrency will be handled by our release of Python 2.7 on App
Engine. We’ve heard a lot of feedback from our Python users who are worried
that the incentive is to move to Java because of its support for concurrent
requests, so we’ve made a change to the new pricing to account for
that. *While
Python 2.7 support is currently in progress it is not yet done so we will be
providing a half-sized instance for Python (at half the price) until Python
2.7 is released.*
Brian Quinlan
>
> Thanks Brian for sending this info, as it is really tricky to contact you guys.
If you need technical help from the App Engine team, you should
consider a Premier Account:
http://code.google.com/appengine/docs/premier/index.html
> This is the issue I am mostly concerned:
> http://code.google.com/p/googleappengine/issues/detail?id=6309&can=1&q=175ms&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log
I had a look at your issue and it isn't something that I can explain
without some significant digging.
Cheers,
Brian
Gregory D'alesandre
sergio...@gmail.com
Thanks for your valuable insights!
I did know about the issue 6323 and had stared it. However I was not
sure it was the same as the issue mentioned CPU bound requests.
My app is very much RPC bound. I've added a comment to this thread to
the issue and I've set threadsafe: no and will monitor. So far latency
seems Ok and the number of instances is 5x-6x of when I had
threadsafe: yes.
I do agree with Felippe, we should be able to maintain the 50%
discount until python27 and threadsafe are fully supported.
This is also making things worse - it seems the minimum latency is not
being enforced correctly, I also have this.
http://code.google.com/p/googleappengine/issues/detail?id=5765&can=5&sort=-status&colspec=ID%20Type%20Component%20Status%20Stars%20Summary%20Language%20Priority%20Owner%20Log
Sérgio
On Dec 7, 5:12 am, "Gregory D'alesandre" <gr...@google.com> wrote:
> Hi Felippe,
>
> As you noted Python 2.7 is released. While Python 2.7 is still
> experimental it is working well for some but indeed there are still issues
> before it will be fully GA. There was a concern that we were abandoning
> Python because there was no solution for concurrent requests. Python 2.7,
> even in its current state, shows that we are not abandoning Python. We
> tried our best to make appropriate trade-offs with the new pricing model
> but at some point we needed to roll it out for the reasons we've discussed
> at length in the past. For instance, when this was written we were also
> going to launch the new pricing 1.5 months earlier than we actually
> released it, which likely saved you quite a bit of money, as opposed to
> this which is costing you more.
>
> While my guess is you'd still rather that the 50% discount persisted I hope
> that answers your question,
>
> Greg
>
> On Tue, Dec 6, 2011 at 5:29 PM, Felippe Bueno <felippe.bu...@gmail.com>wrote:
>
>
>
>
>
>
>
>
>
> > On Tue, Dec 6, 2011 at 8:56 PM, Brian Quinlan <bquin...@google.com> wrote:
>
> >> There is a known bug where the combination of Python 2.7 and enabling
> >> concurrent requests causes large latency increases. The Python 2.7
> >> runtime is still experimental and bugs like this are expected.
>
> > Ok, I understand that.
>
> > Greg, could you please let us know why we have the new full-sized instance
> > hour for python and still not have python concurrent requests ?
>
> > I'm asking because you wrote<http://groups.google.com/group/google-appengine/browse_thread/thread/...>
> > :
>
> > Q: Will there be a solution for Python concurrency? Will this require any
> > code changes?
> > Python concurrency will be handled by our release of Python 2.7 on App
> > Engine. We’ve heard a lot of feedback from our Python users who are
> > worried
> > that the incentive is to move to Java because of its support for
> > concurrent
> > requests, so we’ve made a change to the new pricing to account for
> > that. *While
> > Python 2.7 support is currently in progress it is not yet done *so we
> > will be **
> > providing a half-sized instance for Python (at half the price) until
> > Python
> > 2.7 is released.* *
Felippe Bueno
Brian Quinlan
We have been treating
http://code.google.com/p/googleappengine/issues/detail?id=6323 as a
high-priority issue and I've marked it as such.
That being said, the Python 2.7 runtime is experimental and we can't
offer any support for it or provide any guarantees on when this issue
will be fixed.
Cheers,
Brian