DeadlineExceeded errors haunting every now and then

[Sarang]

I have started getting DeadlineExceeded errors every once in a while and they are annoying and reflect poorly on the GAE and hence my app. Here is what is happening today:

http://gyazo.com/364eed93a907bbf3a80b5cd3d22ec066

Is there any documented method to resolve these? As these comes and go, I assume these are nothing to do with my setup, and everything to do with GAE. Is anyone monitoring these at all? For me GAE has not yet gotten out of beta.. it is very much in beta and I have to pay for a beta product now :(

Can anyone from Google please reply.

Regards,

Sarang

[voscausa]

I have never seen this before. I think something is wrong with your code. Warmup will load a new instance and when this load fails, you can receive all kinds of errors. For instance when an import fails during the load. 

What is in the details of the logs.

[Sharp-Developer.Net]

We started to get same today. Seems to be an isssue with GAE.

[Sarang]

Here are more examples. Can anyone from Google bother to respond? I do not understand sometimes why I am wasting my money, time and efforts with GAE. Makes me feel helpless. I am now going to start looking for alternative hosting.

http://gyazo.com/5a1445b8290e0f5dfd7c54233b27c4b1

@voscausa These errors come every once in a while, so could not be my code. Even if it is my code, I expect "service" from GAE folks. If they cannot provide service, then they should stop building software for end customers. My respect for Google has just gone down a notch.

Sarang

[Sarang]

Just wanted to point that I opened a ticket on Nov 18th for this issue, and there is still no reply or anything else for that matter from Google.

http://code.google.com/p/googleappengine/issues/list?can=4&cursor=googleappengine%3A6359&ts=1323189503&updated=6359

[Johan Euphrosine (Google)]

Hi Sarang,

Sorry for the late followup, but please keep in mind that there is no guaranteed response time for community support on the public issue tracker and the groups.

Your issue is likely to be due to temporary M/S latency, I would recommend switching to the High Replication Datastore using the new migration tool which is way more far reliable.

Hope that helps.

[Sarang]

While I would really like to take your word on this, a simple "google" search completely contradicts what you are saying:

https://groups.google.com/forum/#!topic/google-appengine/ZXX_SgaDu2s

I am paying unnecessarily 8 cents/hr for DeadLineExceeded errors and I do not intend to pay even more using HRD and getting to the same place. I also read that blobstore data would be gone.. this is crazy. How can apps migrate then?

Also, regarding response time, if Google can deduct my credit card on time, then I also expect you to respond on time. 20 days is not acceptable at all when so many of your users are experiencing the same issues and your "status" shows everything is OK!

Sarang

[Rishi Arora]

Hi Johan,

I still can't get down to the bottom of why HRD is always the answer to high latencies.  M/S versus HRD should be an entirely application-dependent decision.  API timeouts can be used to gracefully degrade during a downtime, but that's a conscious choice on part of the application's author.  So, why the push towards HRD?  Also, if you keep pushing HRD, costs are a better incentive than words :)  Basically, I'm saying that if Google has a strong technical reason to move users towards HRD (such as: why bother maintaining two options), then lower HRD costs to reflect that.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[sergio...@gmail.com]

I'm seeing this on HRD!
Please see:

http://groups.google.com/group/google-appengine/browse_thread/thread/6575ff4a0683bb6b

On Dec 6, 5:32 pm, "Johan Euphrosine (Google)" <pro...@google.com>
wrote:

[Johan Euphrosine]

Hi Saraang,

DeadlineExceededError just means that the response has taken more time
that the authorized deadline (60s), this could be due to several
reason ranging from slow application code loading, slow application
code execution, slow datastore queries to M/S latency issues.

I only suggered the latter because it is the most common, and
migrating to HRD would eliminate this possible cause.

If you are looking more operational support with a response SLA, we
are offering this as part of the premier accounts:
http://www.google.com/enterprise/cloud/appengine/pricing.html

Hope that helps.

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/UrfiFXjdiTcJ.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

--
Johan Euphrosine (proppy)
Developer Programs Engineer
Google Developer Relations

[Andrius A]

Guys, we have a form where we collect issues you are experiencing and will pass it to Google to get an attention and hopefully have things getting better. So please fill in:

https://docs.google.com/spreadsheet/viewform?hl=en_US&formkey=dC1DMmpPSU1WZnk0d1FMa3JmNXIwaGc6MQ#gid=0

Johan, thank you for your attention, but I don't understand what you mean by slow code loading and slow execution. Code would normally load and execute fast, but in some cases it will take tens of seconds to load and/or throw deadline exceeded errors at the high rate - how can you explain that? Is this going to happen all the time and we should not complain?

If this is the case only with MS you should publicly acknowledge that by clearly stating that in documentation, at the moment we are confused! MS works for us because it's much faster and cheaper and we are ok with scheduled maintenance.

Thank you,

Andrius

[Rishi Arora]

Andrius,

I agree with your point about MS too.  It sounds really suspicious that Google keeps pushing HRD.  From the documentation, MS and HRD have clearly stated cost-benefit trade-offs - but its too bad that in practice that isn't the case.  It should be the app author's choice.  And if Google has a hidden agenda to push HRD, then they should incentivize us by lowering its cost, since its more of an advantage to them than to people like us.

Rishi

[Gregory D'alesandre]

Hello All,

The reason that Johan is encouraging you to migrate to HRD is because it is more reliable.  M/S is wholly dependent on a single piece of storage infrastructure, if that happens to be slow your app will likely slow down and you can then end up getting DeadlineExceeded errors.  When you ask about "fixing" the latency problems with M/S, we have done so by providing an option that has a much more consistent latency profile (called HRD).  Beyond that there just isn't that much we can do to fix one-off latency issues, they are typically not systemwide issues which is why a few people at a time are seeing them.  

Sergio's comment that he is seeing this on HRD is most likely because he is using Python 2.7 which is still experimental (meaning you will periodically run into issues).

To be clear, HRD and M/S are the same price, there is no price differential.  I'm not sure why you think it is suspicious that we are encouraging everyone to move to a more stable product.  You are completely correct that, today, it is your choice as to whether you want to use M/S or HRD.  But when you are choosing M/S you are choosing an inherently less stable option and this means that when you run into issues you are taking on that responsibility as well.  To Andrius' point, you should feel free to complain but the basic response will be: if you want more consistent latency profiles, you should move to HRD.  We have publicly stated this by not providing any SLA for M/S but only for HRD.

I hope that helps clarify,

Greg D'Alesandre

Senior Product Manager, Google App Engine

[Kenneth]

Hi Greg,

Can you give us some indication if the migration tool is ever going to include blobstore migration? That's the only thing holding me back. I know I can do it myself but I'd rather not. I see now that it has gone ga, have you stuck a fork in it and called it done?

Thanks

[Gregory D'alesandre]

Hi Kenneth, 

We are planning to add blobstore migration at some point but it won't likely be until the middle of Q1 at the earliest.

Greg

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/f7KzNP-pCuwJ.

[Andrius A]

In docs it says that HRD is more costly "The High Replication Datastore costs more due to the additional replication (see billing page for pricing details). Due to the higher cost, we recommend High Replication Datastore primarily for those developers building critical App Engine applications that require the highest availability". 

It also says "queries on a single guestbook to be strongly consistent, but also limits changes to the guestbook to 1 write per second (the supported limit for entity groups). Therefore, writing to a single entity group per guestbook is not ideal when high usage is expected. If your app is likely to encounter heavy write usage, consider using another means. For example, you can put recent posts in memcache with an expiration, and then display a mix of recent posts from memcache and posts retrieved from the datastore." 

How could we trust by putting data to memcache if we know it can be evicted any time? To use HRD is not viable for applications which need strong consistency for high rate of puts. At the moment MS is perfect what it does but bloody thing keeps dying.. I can't believe there is no way to improve MS and you are switching to HRD. In long term HRD can cause more problems for applications parts were it wasn't aticipating to receive higher rate of inserts, and having a limit of 1 write per second is a disastrous.

[Brian Quinlan]

On Wed, Dec 7, 2011 at 10:44 AM, Andrius A <andr...@gmail.com> wrote:
> In docs it says that HRD is more costly "The High Replication Datastore
> costs more due to the additional replication (see billing page for pricing
> details). Due to the higher cost, we recommend High Replication Datastore
> primarily for those developers building critical App Engine applications
> that require the highest availability".

I've filed a bug against this documentation.

> It also says "queries on a single guestbook to be strongly consistent, but
> also limits changes to the guestbook to 1 write per second (the supported
> limit for entity groups). Therefore, writing to a single entity group per
> guestbook is not ideal when high usage is expected. If your app is likely to
> encounter heavy write usage, consider using another means. For example, you
> can put recent posts in memcache with an expiration, and then display a mix
> of recent posts from memcache and posts retrieved from the datastore."
> How could we trust by putting data to memcache if we know it can be evicted
> any time?

You wouldn't as definitive storage but it is reasonable to cache
recent posts there.

> To use HRD is not viable for applications which need strong
> consistency for high rate of puts.

You may have to think about how your organize your data in order to
satisfy your consistency and throughput requirements but I am very
skeptical when you say that it is simply not viable.

Google, for example, has been successful using the same technology for
many of its own high-volume properties.

> At the moment MS is perfect what it does
> but bloody thing keeps dying..

That is an architectural problem that is very hard to fix with making
the same kinds of trade-offs that we made when implementing HRD .

Cheers,
Brian

[Andrius A]

Hi Brian,

just an idea, do you think you could build highly reliable simple but fast hash table similar to memcache but that could guarantee limited number of single entities permanent existence per application?

Having that and HRD would make things work brilliantly!

[Brian Quinlan]

On Wed, Dec 7, 2011 at 11:34 AM, Andrius A <andr...@gmail.com> wrote:
> Hi Brian,
>
> just an idea, do you think you could build highly reliable simple but fast
> hash table similar to memcache but that could guarantee limited number of
> single entities permanent existence per application?

Probably not.

"Highly reliable" suggests that you synchronously replicate your data
across several machines and data centers (assuming that you want
consistency) but it is hard to make that "fast".

Cheers,
Brian

[blackpawn]

I've been using app engine for a year and a half or so all on M/S and regularly see this DeadlineExceeded error trouble as well.  It's definitely not an app side issue but more like bad weather in the infrastructure that comes as goes.  Things can go for a couple months running smoothly with zero errors then hit a rough patch of a few days to a week of random DeadlineExceeded errors every few seconds or minutes.  It can be very frustrating since there's no predicting when this will strike you and especially bad if there are important things going on with your app at that time!

I also really hope this is a M/S issue only and do plan to move to HRD but also need to migrate tons of blobstore data.

[c h]

I just started a *ton* of these errors today.  Looking at the GAE status page (http://code.google.com/status/appengine/detail/datastore/2011/12/07#ae-trust-detail-datastore-get-latency) the latency has more than doubled since yesterday's maintenance.  Is there any chance that this will return to pre-maintenance levels soon?

Is the migration tool considered production ready yet?  i'm hesitant to move gigs of data with a "beta" tool....

thanks,

christian

[blackpawn]

I'm getting tons of DeadlineExceededError since the maintenance last night too.  It'll say in the log the requests are like 60 to 85 seconds and the same requests at other times go through in 200 milliseconds.

Today I'm also getting a lot of "InternalError: internal error."

[Jeff Schnitzer]

Pretend the M/S datastore is a sinking ship.  The mermaids will not save you.  Get off now.

Jeff

On Wed, Dec 7, 2011 at 9:12 PM, blackpawn <pharmap...@gmail.com> wrote:

I'm getting tons of DeadlineExceededError since the maintenance last night too.  It'll say in the log the requests are like 60 to 85 seconds and the same requests at other times go through in 200 milliseconds.

Today I'm also getting a lot of "InternalError: internal error."

--

You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/c0dV7sBY9jEJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--

We are the 20%

[Will]

I've seen about 10 DDEs on HDR, Python25, in the past hour. There hadn't been any problems for the past 3 days up to the point.

 

Will

[daniel_ge_smith]

Currently, seeing the same high number of DDEs - requests taking up to 70seconds. Consequently, instance usage has gone through the roof, and we're burning through our daily budget at a rate way higher than normal. Not very happy right now!

[Alexis]

Seeing spikes or DEE errors too with my HRD app recently,
while it has been right for several months.

Reminds me of the symptoms seen when we used M/S: suddenly a burst of
DEE and many instances gets killed, with these two logging messages:

I 2011-12-08 13:03:35.164
This request caused a new process to be started for your application,
and thus caused your application code to be loaded for the first time.
This request may thus take longer and use more CPU than a typical
request for your application.

W 2011-12-08 13:03:35.164
A serious problem was encountered with the process that handled this
request, causing it to exit. This is likely to cause a new process to
be used for the next request to your application. If you see this
message frequently, you may be throwing exceptions during the
initialization of your application. (Error code 104)

[bFlood]

using an HR app and getting sporadic DDE as well

"The API call datastore_v3.Get() took too long to respond and was
cancelled."

so much for HR solving everything

and fwiw, I've also run up $100 in charges on an older M/S app (no
code changes in months) that gets maybe 1000 requests/day. Nearly
every other request is a DDE

[Abhishek Mathur]

I am getting the same error as well.

[infoDiagram office]

Same errors for couple of days here as well. We are really disappointed with this. It is already paid service so I would expect some sort of communication from Google on it. Looking at the amount of comments above it is definitively platform issue. Dear Google team please share some more info. 

Thanks

[thstart]

I got the same error today, usually my app is very fast

[blackpawn]

Currently seeing this DeadlineExceededError bad weather again on app
id sketch-club if it's of any use to you Googlers for diagnosing this
problem.

[Ikai Lan (Google)]

Do you guys have any application IDs of High Replication applications that are experiencing datastore timeouts? What are you doing with these calls? Could these be contention related?

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

[Dennis]

I'm looking into converting to HRD.
But as part of the process to explore that option,
I've problems with the datastore admin page:

http://groups.google.com/group/google-appengine/browse_frm/thread/640d7f7392e22f22#

On Dec 22, 3:44 am, "Ikai Lan (Google)" <ika...@google.com> wrote:
> Do you guys have any application IDs of High Replication applications that
> are experiencing datastore timeouts? What are you doing with these calls?
> Could these be contention related?
>
> --
> Ikai Lan
> Developer Programs Engineer, Google App Engine
> plus.ikailan.com | twitter.com/ikai
>

[Ikai Lan (Google)]

Don't use that tool. Use the migration tool that's available when you click "Application Settings" and "View Migration Tool" on that page.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Dennis Fogg]

Since the migration tool is still experimental, I want to try making copies of my data before migrating.

Thus, I need to enable the datastore admin.

[Ikai Lan (Google)]

You can run the first step that won't do the full migration. The way the tool works is like this:

1. Copy all your data without putting the source app into read-only mode so you can continue serving

2. Ask you if you want to put the app into read-only mode (just don't do this)

3. Copy the delta (changed data) to the destination app

4. Ask you if you want to alias the app ID to the new application. When you say yes, you'll be serving from your new, shiny HRD application.

The migration tool only does reads against your source datastore. It does not use quota or impact serving. It's marked "experimental" because there are a few cases where the migration will get stuck and require our manual intervention to unfreeze the migration, but from what we've seen, this only happens during step 1. It's never occurred yet on the critical, read-only period delta migration. You should also be able to run the delta migration as many times as you need, so it never hurts to start this process.

Short answer: there's no impact on your serving app. The worst thing that might happen is you get charged for data storage on the destination app, which, if you are planning on using the datastore admin anyway, is something you were already planning on doing.

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai