URLFetch timout

[Kevin Hoffman]

I am experiencing URLFetch timeouts only when deployed live.  Local dev server is able to connect to outside services no problem. Same exact code that has been working fine now times out.  No code or config changes in dev or prod.  Anyone else seeing this?

[Joshua Smith]

Happens all the time. Has forever. You have to program defensively against URL Fetch timeouts.

On Jul 12, 2016, at 2:27 PM, Kevin Hoffman <ke...@erainmaker.com> wrote:

I am experiencing URLFetch timeouts only when deployed live.  Local dev server is able to connect to outside services no problem. Same exact code that has been working fine now times out.  No code or config changes in dev or prod.  Anyone else seeing this?

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/3d52445d-db47-420b-8e39-9dda321dc321%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Hoffman]

Joshua, thank you for the quick reply.  I clearly need to handle that better in the code.

However, the code has been working flawlessly for nearly a year, seems odd to have such a major spike and for it to run for several hours now.

Have you seen that before?

Best!

[Adam (Cloud Platform Support)]

Are you seeing constant timeouts or is this intermittent? Is this happening generally or to a specific host / domain? What are you setting as a URL Fetch deadline? 

[Kevin Hoffman]

I was seeing constant timeouts for about six hours.  URLFetch started working about ten minutes ago.  I was seeing the error across the three integrations I have in place, twillio, authorize.net and sendgrid.

Previously I had not set any timeout, when diagnosing I set it to 45 seconds with no change in behavior.

Clearly I need to handle the timeouts better in my code and will be doing that immediately.

Thanks for any insights.

Best,

Kevin

[Max Loren]

Hi

I am experiencing this issue until now. My apps tried to send GET request to https://graph.facebook.com/.., all requests failure with timeout reason. I have 2 different apps on same account and both have the same issue,

Thanks

Max

[Yuuki T]

Me too, GAE URL Fetch returns timeout...

Get https://graph.facebook.com/***: API error 5 (urlfetch: DEADLINE_EXCEEDED)

2016年7月13日水曜日 3時27分21秒 UTC+9 Kevin Hoffman:

[Yuuki T]

Just now, this timeout issue has been gone/disappeared on my web service.

2016年7月13日水曜日 18時59分35秒 UTC+9 Yuuki T:

[Nathan Trevivian]

We're seeing the same issue on calls to third party RESTful APIs. Could program defensively, but where one of those APIs is your payment/checkout service, and you're losing money over it, and people are complaining all over your company Facebook page about it, it starts to get a little bit upsetting.

Usually it's very intermitted - say 1 per week. Today it's been happening ALL morning to 100% of attempts to call out to third party RESTful APIs. What's going on?

[Nathan Trevivian]

Having just posted this, we just had one successful one. Hopefully Yuuki T is right and that's the end of it for now. I'd still like to know what happened though?

[Kevin Hoffman]

Agreed, when payment services are involved it's a matter of losing money. I've spent a year building my company's website and service offering on GAE, can't have the ability to communicate with my payment gateway be anything less than 100% reliable.

[Hitoshi Ichikawa]

We were expeiencing the same issue, but recovered now.
There is no service desruption on google service dashboard.

There are so many reports here, I think this is caused by google's
system failure, isn't it?

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to google-appengi...@googlegroups.com.
> To post to this group, send email to google-a...@googlegroups.com.
> Visit this group at https://groups.google.com/group/google-appengine.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/google-appengine/b056dd46-8b3a-43c7-b074-e60a6ee7c713%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

--
株式会社アイキューブドシステムズ
製品開発運用本部
市川　仁
Mail : ichi...@i3-systems.com
Address : 〒815-0033 福岡県福岡市南区大橋2丁目1-1 花村ビル
TEL : 092-552-4358
WEB : http://www.i3-systems.com/

[Joshua Smith]

On Jul 13, 2016, at 7:33 AM, Kevin Hoffman <ke...@erainmaker.com> wrote:

Agreed, when payment services are involved it's a matter of losing money. I've spent a year building my company's website and service offering on GAE, can't have the ability to communicate with my payment gateway be anything less than 100% reliable.

Then you picked the wrong horse.

GAE URLFetch has never, and will never, be anywhere close to 100% reliable.

Nothing is 100% but if you think you need “five nines” (99.99999%) then you need to change infrastructure.

I’d guess that it’s much closer to 99% over the long term (given that they regularly have such massive downtime, plus the chronic random failures).

I’m not talking about GAE in general. Just URLFetch.

-Joshua

[Nicholas (Google Cloud Support)]

An intermittent issue was found with the URLFetch API service.  Fetch requests to non-Google related services were failing with deadline exceeded errors like the ones reported on this thread.  The error rate started increasing around 22:00 US/Pacific on Tuesday, July 12.  The issue should have been resolved as of 2:50 US/Pacific today (July 13).  We will conduct an internal investigation of this issue and make appropriate improvements to our systems to prevent or minimize future recurrence.

Is anyone else still experiencing this elevated rate of deadline exceeded errors?

[Nathan Trevivian]

We're still getting it - last occurrence was at 15:35 BST. It's happening less regularly than before (we were getting one every 5 minutes - AKA all the time), but it's still happening more than it should do/used to.

[Kevin Hoffman]

Nicholas, thank you.  Appreciate the responsiveness. I'm not seeing any URLFetch errors at the moment.

I don't agree with Joshua that I picked the wrong horse, love GAE, but he is right in that many of us do have mission-critical integrations that need a reliable SLA. Especially for billing-enabled accounts.

Thanks again.

Kevin

[Joshua Smith]

Good luck with getting any company to give you a SLA that promises 100% uptime, or which compensates you for lost e-commerce transactions.

Google’s SLA, like pretty much everyone else’s, promises to not charge you for THEIR SERVICES when they are down. Nobody promises to make your business whole for the losses you endured from that downtime. You might be able to buy an insurance policy that does that, I suppose. But that’s not what SLAs are for.

Anyway, if you need 100% uptime, the only way you could possibly get close to that would be to build two parallel systems (say, one on GAE and another on AWS or Azure) and fail-over with clever DNS.

-Joshua

-- 
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ed5eeb49-c057-419e-9d44-765a753c3f12%40googlegroups.com.