Lots of 502 bad gateway errors

Ashley Xu

unread,

Jun 1, 2015, 8:41:56 PM6/1/15

to adwor...@googlegroups.com

Hi,

We have seen a lot of 502 bad gateway errors recently. Please find below the stack trace and the number of failures per day. Could you please help take a look?

Thanks

Ashley

Sync error: WebExceptionProtocolError ---> System.Net.WebException: The request failed with HTTP status 502: Bad Gateway.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Google.Api.Ads.Common.Lib.AdsSoapClient.MakeApiCall(String methodName, Object[] parameters)
   at Google.Api.Ads.AdWords.v201409.ManagedCustomerService.get(Selector serviceSelector)

LogDate	NumFailures
5/1/2015	2
5/2/2015	1
5/3/2015	1
5/4/2015	3
5/4/2015	6
5/4/2015	16
5/12/2015	4
5/12/2015	9
5/12/2015	23
5/13/2015	1
5/13/2015	10
5/14/2015	4
5/18/2015	1
5/18/2015	2
5/19/2015	1
5/19/2015	3
5/19/2015	3
5/20/2015	1
5/20/2015	3
5/21/2015	2
5/26/2015	1
5/26/2015	1
5/27/2015	1
5/27/2015	4
5/28/2015	6
5/28/2015	7
5/29/2015	35
5/29/2015	38
5/30/2015	6
5/30/2015	5

Ashley Xu

unread,

Jun 2, 2015, 5:26:00 PM6/2/15

to adwor...@googlegroups.com

Could someone please help to investigate?

Nadine Sundquist (AdWords API Team)

unread,

Jun 3, 2015, 1:06:40 PM6/3/15

to adwor...@googlegroups.com, ashle...@gmail.com

Hello Ashley,

I took a look at the errors on our servers. I'm not seeing anything unusual that could cause gateway issues. If your SOAP response does not have a requestId, then the request may not have even made it to our servers. I would recommend checking with your network administrator to see if you can trace the route of the request.

Best,

Nadine, AdWords API Team

Zweitze

unread,

Jun 4, 2015, 2:34:33 AM6/4/15

to adwor...@googlegroups.com, ashle...@gmail.com

>If your SOAP response does not have a requestId, then the request may not have even made it to our servers.

I am surprised to learn this. In ten years (!) I have never seen AdWords servers return a RequestId with a HTTP 5xx response.

Anyway, those errors started to pop up here too. When an error appears we also log the headers.

Example 1 - Request headers:

Content-Type	application/x-www-form-urlencoded
Authorization	Bearer ya29.h...uX7W1b
clientCustomerId	8077325588
developerToken	kNkfox...m9qg_g
skipReportHeader	True
skipReportSummary	True
User-Agent	Adwords API by Maxlead
Host	adwords.google.com
Content-Length	1865
Accept-Encoding	gzip, deflate

Request (a report):

<?xml version="1.0"?>
<reportDefinition>
  <selector xmlns="https://adwords.google.com/api/adwords/cm/v201409">
    <fields>CampaignId</fields>
    <fields>Month</fields>
    <fields>AdNetworkType1</fields>
    <fields>AveragePosition</fields>
    <fields>Clicks</fields>
    <fields>Cost</fields>
    <fields>Impressions</fields>
    <fields>ContentBudgetLostImpressionShare</fields>
    <fields>ContentImpressionShare</fields>
    <fields>ContentRankLostImpressionShare</fields>
    <fields>SearchBudgetLostImpressionShare</fields>
    <fields>SearchExactMatchImpressionShare</fields>
    <fields>SearchImpressionShare</fields>
    <fields>SearchRankLostImpressionShare</fields>
    <fields>RelativeCtr</fields>
    <predicates>
      <field>Status</field>
      <operator>IN</operator>
      <values>ENABLED</values>
      <values>PAUSED</values>
      <values>REMOVED</values>
    </predicates>
    <dateRange>
      <min>20150401</min>
      <max>20150603</max>
    </dateRange>
  </selector>
  <reportName xmlns="https://adwords.google.com/api/adwords/cm/v201409">CampaignImpressionShare_20150401_20150603</reportName>
  <reportType xmlns="https://adwords.google.com/api/adwords/cm/v201409">CAMPAIGN_PERFORMANCE_REPORT</reportType>
  <dateRangeType xmlns="https://adwords.google.com/api/adwords/cm/v201409">CUSTOM_DATE</dateRangeType>
  <downloadFormat xmlns="https://adwords.google.com/api/adwords/cm/v201409">GZIPPED_XML</downloadFormat>
  <includeZeroImpressions xmlns="https://adwords.google.com/api/adwords/cm/v201409">false</includeZeroImpressions>
</reportDefinition>

Response headers:

Content-Type	text/html; charset=UTF-8
Content-Length	1477
Date	Wed, 03 Jun 2015 16:16:06 GMT
Server	GFE/2.0

Unfortunately we don't log the content of the error response.
Also note that we don't have proxies or anything like that, and having seen many AdWords API responses, those headers look like something that was generated by Google.

Example 2 - request headers

Content-Type	application/x-www-form-urlencoded
Authorization	Bearer ya29.h...uX7W1b
clientCustomerId	9121915867
developerToken	kNkfox...m9qg_g
skipReportHeader	True
skipReportSummary	True
User-Agent	Adwords API by Maxlead
Host	adwords.google.com
Content-Length	1494
Accept-Encoding	gzip, deflate

Request (a different report):

<?xml version="1.0"?>
<reportDefinition>
  <selector xmlns="https://adwords.google.com/api/adwords/cm/v201409">
    <fields>CampaignId</fields>
    <fields>AdGroupId</fields>
    <fields>AdGroupName</fields>
    <fields>AdGroupStatus</fields>
    <fields>CpcBid</fields>
    <fields>CpmBid</fields>
    <predicates>
      <field>CampaignStatus</field>
      <operator>IN</operator>
      <values>ENABLED</values>
      <values>PAUSED</values>
      <values>REMOVED</values>
    </predicates>
    <predicates>
      <field>Status</field>
      <operator>IN</operator>
      <values>ENABLED</values>
      <values>PAUSED</values>
      <values>REMOVED</values>
    </predicates>
  </selector>
  <reportName xmlns="https://adwords.google.com/api/adwords/cm/v201409">GroupInfo_All_Time</reportName> <reportType xmlns="https://adwords.google.com/api/adwords/cm/v201409">ADGROUP_PERFORMANCE_REPORT</reportType>
  <dateRangeType xmlns="https://adwords.google.com/api/adwords/cm/v201409">ALL_TIME</dateRangeType>
  <downloadFormat xmlns="https://adwords.google.com/api/adwords/cm/v201409">GZIPPED_XML</downloadFormat>
  <includeZeroImpressions xmlns="https://adwords.google.com/api/adwords/cm/v201409">true</includeZeroImpressions>
</reportDefinition>

Response headers:

Content-Type	text/html; charset=UTF-8
Content-Length	1477
Date	Wed, 03 Jun 2015 16:30:43 GMT
Server	GFE/2.0

Note: in case of HTTP 502 we always retry the request three times before giving up. In other words, we experienced at least 8 errors. Unfortunately we don't log the older requests and responses.

At the same time (approx 16.30h GMT) we also experienced a timeout, which could be related.
Headers:

Content-Type	application/x-www-form-urlencoded
Authorization	Bearer ya29.h...uX7W1b
clientCustomerId	9861198322
developerToken	kNkfox...m9qg_g
skipReportHeader	True
skipReportSummary	True
User-Agent	Adwords API by Maxlead
Host	adwords.google.com
Content-Length	1106
Accept-Encoding	gzip, deflate

Request:

<?xml version="1.0"?>
<reportDefinition>
  <selector xmlns="https://adwords.google.com/api/adwords/cm/v201409">
    <fields>AccountDescriptiveName</fields>
    <fields>AccountCurrencyCode</fields>
    <fields>AccountTimeZoneId</fields>
    <fields>Impressions</fields>
    <fields>ExternalCustomerId</fields>
  </selector>
  <reportName xmlns="https://adwords.google.com/api/adwords/cm/v201409">AdwordsReportAccountInfo_All_Time</reportName>
  <reportType xmlns="https://adwords.google.com/api/adwords/cm/v201409">ACCOUNT_PERFORMANCE_REPORT</reportType>
  <dateRangeType xmlns="https://adwords.google.com/api/adwords/cm/v201409">ALL_TIME</dateRangeType>
  <downloadFormat xmlns="https://adwords.google.com/api/adwords/cm/v201409">GZIPPED_XML</downloadFormat>
  <includeZeroImpressions xmlns="https://adwords.google.com/api/adwords/cm/v201409">true</includeZeroImpressions>
</reportDefinition>

Again, a timeout will only stop our processes when it appears four times in a row.

I hope this data helps!

Andrey Shetov

unread,

Jun 4, 2015, 10:04:25 AM6/4/15

to adwor...@googlegroups.com

We have the same situation here in Russia, a lot of 502 bad gateway exceptions from googleads library.

File "/var/app/.env/local/lib/python2.7/site-packages/googleads/common.py", line 296, in MakeSoapRequest

for arg in args])

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 521, in __call__

return client.invoke(args, kwargs)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 581, in invoke

result = self.send(soapenv)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 619, in send

description=tostr(e), original_soapenv=original_soapenv)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 677, in process_reply

raise Exception((status, description))

Exception: (502, u'Bad Gateway')

Andrey Shetov

unread,

Jun 4, 2015, 10:04:45 AM6/4/15

to adwor...@googlegroups.com

We have the same situation here in Russia. A lot of 502 exceptions recently from googleads python library.

File "/var/app/.env/local/lib/python2.7/site-packages/googleads/common.py", line 296, in MakeSoapRequest

for arg in args])

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 521, in __call__

return client.invoke(args, kwargs)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 581, in invoke

result = self.send(soapenv)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 619, in send

description=tostr(e), original_soapenv=original_soapenv)

File "/var/app/.env/local/lib/python2.7/site-packages/suds/client.py", line 677, in process_reply

raise Exception((status, description))

Exception: (502, u'Bad Gateway')

Nadine Sundquist (AdWords API Team)

unread,

Jun 4, 2015, 11:21:19 AM6/4/15

to adwor...@googlegroups.com, zwe...@gmail.com, ashle...@gmail.com

Hello Ashley,

Thanks for the additional information. I'm having some of our engineers take a look into it. I'll contact you as soon as they get back to me.

Cheers,

Nadine, AdWords API Team

...

Message has been deleted

Андрей Шитов

unread,

Jun 4, 2015, 3:51:04 PM6/4/15

to adwor...@googlegroups.com

# traceroute adwords.google.com
traceroute to adwords.google.com (173.194.67.101), 30 hops max, 60 byte packets
 1  static.129.109.4.46.clients.your-server.de (46.4.109.129)  1.137 ms  1.199 ms  1.219 ms
 2  hos-tr4.juniper2.rz14.hetzner.de (213.239.224.225)  0.173 ms hos-tr2.juniper1.rz14.hetzner.de (213.239.224.161)  0.781 ms hos-tr4.juniper2.rz14.hetzner.de (213.239.224.225)  0.179 ms
 3  core22.hetzner.de (213.239.245.125)  0.206 ms  0.212 ms core21.hetzner.de (213.239.245.85)  1.223 ms
 4  core1.hetzner.de (213.239.245.218)  4.831 ms core4.hetzner.de (213.239.245.14)  4.886 ms  4.871 ms
 5  juniper1.ffm.hetzner.de (213.239.245.5)  4.954 ms  4.981 ms juniper4.ffm.hetzner.de (213.239.245.1)  4.885 ms
 6  de-cix20.net.google.com (80.81.193.108)  5.231 ms  5.230 ms  5.270 ms
 7  209.85.246.66 (209.85.246.66)  5.791 ms  5.776 ms  6.028 ms
 8  72.14.234.231 (72.14.234.231)  5.639 ms 209.85.251.246 (209.85.251.246)  5.851 ms 72.14.234.231 (72.14.234.231)  5.612 ms
 9  216.239.41.135 (216.239.41.135)  14.364 ms 216.239.41.137 (216.239.41.137)  14.160 ms 209.85.248.128 (209.85.248.128)  9.204 ms
10  74.125.37.156 (74.125.37.156)  13.825 ms 74.125.37.152 (74.125.37.152)  14.559 ms 74.125.37.97 (74.125.37.97)  13.851 ms
11  216.239.51.153 (216.239.51.153)  13.688 ms 72.14.239.96 (72.14.239.96)  14.034 ms 209.85.248.101 (209.85.248.101)  13.943 ms
12  * * *
13  wi-in-f101.1e100.net (173.194.67.101)  14.012 ms  13.850 ms  13.941 ms

# ping  adwords.google.com
PING adwords.l.google.com (173.194.67.101) 56(84) bytes of data.
64 bytes from wi-in-f101.1e100.net (173.194.67.101): icmp_req=1 ttl=48 time=14.2 ms
64 bytes from wi-in-f101.1e100.net (173.194.67.101): icmp_req=2 ttl=48 time=14.0 ms
64 bytes from wi-in-f101.1e100.net (173.194.67.101): icmp_req=3 ttl=48 time=14.0 ms
64 bytes from wi-in-f101.1e100.net (173.194.67.101): icmp_req=4 ttl=48 time=14.0 ms

On Thu, Jun 4, 2015 at 7:03 PM, <adwor...@googlegroups.com> wrote:

Hello,

I want to to make sure our engineers have all the information they need to track this down.
Are you also seeing 502 errors from the AdWords UI as well, or are these errors only coming from the API?
Could you please run traceroute adwords.google.com and send me the output?
Could you please run ping adwords.google.com and send me the output?
Thanks,
Nadine, AdWords API Team
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and discussion group:
http://googleadsdeveloper.blogspot.com/search/label/adwords_api
https://developers.google.com/adwords/api/community/
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~

You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwor...@googlegroups.com
To unsubscribe from this group, send email to
adwords-api...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to the Google Groups "AdWords API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to adwords-api...@googlegroups.com.
Visit this group at http://groups.google.com/group/adwords-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/fb218080-7569-43e2-9a9c-c867dcafea7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Ashley Xu

unread,

Jun 6, 2015, 12:50:23 PM6/6/15

to adwor...@googlegroups.com

Hi,

Did we have any update on this? We are still seeing the issue but it just happened less frequently.

Thanks

Ashley

On Monday, June 1, 2015 at 5:41:56 PM UTC-7, Ashley Xu wrote:

Nadine Sundquist (AdWords API Team)

unread,

Jun 8, 2015, 1:46:47 PM6/8/15

to adwor...@googlegroups.com, ashle...@gmail.com

Hi Ashley,

There are no updates, yet. I'll let you know as soon as I hear anything.

Thanks,

Nadine, AdWords API Team

Kristopher Windsor

unread,

Jun 9, 2015, 2:10:02 PM6/9/15

to adwor...@googlegroups.com, ashle...@gmail.com

Hi, I am seeing this error much more than usual, starting sometime in the last 2-3 weeks:

SOAP-ERROR: Parsing WSDL: Couldn't load from 'https:\/\/adwords.google.com\/api\/adwords\/cm\/v201502\/AdGroupService?wsdl' : failed to load external entity \"https:\/\/adwords.google.com\/api\/adwords\/cm\/v201502\/AdGroupService?wsdl

Is this error possibly related?

Thanks,

Nadine Sundquist (AdWords API Team)

unread,

Jun 9, 2015, 2:49:30 PM6/9/15

to adwor...@googlegroups.com, kristoph...@gmail.com, ashle...@gmail.com

Hi Kristopher,

That error usually has a different root cause than what we're seeing here in this thread. If the problem persists, please start a new thread with information on your issue.

Thanks,

Nadine, AdWords API Team

Nadine Sundquist (AdWords API Team)

unread,

Jun 12, 2015, 9:24:15 AM6/12/15

to adwor...@googlegroups.com, adwordsapiad...@google.com, ashle...@gmail.com

Hello Ashley,

I've been monitoring the progress on this, and you should have seen a significant improvement throughout the week.