Read timed out on API calls

[Stefan Kleine Stegemann]

Hi,

over the past days, we experienced an increasing number of read-timeouts when making API calls.

The timeouts are not restricted to a specific service but we have observed them for CampaignService, 

TrafficEstimatorService, Reporting and some others . I seems that our network is not the problem

because other (concurrent) calls are processed with out problems at the same time. It just seems

that, once in a while, an API call gets "hit" by a timeout. Actually, it looks like that making the request

is not the problem but rather reading the response. Most of the time, we our retry mechanism

catches the timeout and re-runs the API call. As far we have examined it, all calls got processed

without problems when retried.

Our main problem is that the timeout occurs after an exceptionally long time (ranging from minutes to

up to an hour). This slows down some of our backend processes significantly because some calls 

hang for an hour before the retry mechanism applies. We have not set any explicit timeout on the

adwords session or the API in general. Also, we did not experience this behavior before, it just

started to appear some days ago.

We don't really know if we can do anything to improve the situation but as a first step, we'd like to 

decrease the timeout for the adwords session. However, we don't know how to this and did not

manage to find instructions. So is there any way of explicitly setting a timeout for an adwords session?

I've attached an example request that failed with a timeout (to CampaignService) and response (well,

the response is empty) as well as the stack trace of the root cause exception.

Cheers,

Stefan

---------------------------

Here is the exception (I cropped the stacktrace where the call "leaves" our code):

java.net.SocketTimeoutException: Read timed out

        at java.net.SocketInputStream.socketRead0(Native Method) ~[na:1.8.0_25]

        at java.net.SocketInputStream.read(SocketInputStream.java:150) ~[na:1.8.0_25]

        at java.net.SocketInputStream.read(SocketInputStream.java:121) ~[na:1.8.0_25]

        at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) ~[na:1.8.0_25]

        at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593) ~[na:1.8.0_25]

        at sun.security.ssl.InputRecord.read(InputRecord.java:532) ~[na:1.8.0_25]

        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:954) ~[na:1.8.0_25]

        at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:911) ~[na:1.8.0_25]

        at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ~[na:1.8.0_25]

        at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) ~[na:1.8.0_25]

        at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) ~[na:1.8.0_25]

        at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[na:1.8.0_25]

        at java.io.FilterInputStream.read(FilterInputStream.java:133) ~[na:1.8.0_25]

        at org.apache.axis.SOAPPart.getAsBytes(SOAPPart.java:453) [axis-1.4.jar:na]

        at org.apache.axis.SOAPPart.getAsString(SOAPPart.java:559) [axis-1.4.jar:na]

        at org.apache.axis.Message.getSOAPPartAsString(Message.java:412) [axis-1.4.jar:na]

        at com.google.api.ads.common.lib.soap.axis.AxisHandler.invokeSoapCall(AxisHandler.java:281) [ads-lib-axis-2.17.0.jar:na]

        at com.google.api.ads.common.lib.soap.SoapServiceClient.callSoapClient(SoapServiceClient.java:62) [ads-lib-2.17.0.jar:na]

        at com.google.api.ads.common.lib.soap.SoapServiceClient.invoke(SoapServiceClient.java:92) [ads-lib-2.17.0.jar:na]

        at com.sun.proxy.$Proxy122.get(Unknown Source) [na:na]

        at com.adference.connector.google.CampaignServiceGET.execute(CampaignServiceGET.java:23) [google-adwords-api-connector-v21_2.jar:na]

        ...

The SOAP request:

---------------------------

<?xml version="1.0" encoding="UTF-8" standalone="no"?>

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

    <soapenv:Header>

        <ns1:RequestHeader xmlns:ns1="https://adwords.google.com/api/adwords/cm/v201607" soapenv:mustUnderstand="0">

            <ns1:clientCustomerId>xxxxxxxxxxx</ns1:clientCustomerId>

            <ns1:developerToken>REDACTED</ns1:developerToken>

            <ns1:userAgent>xxxxxxxxxxx</ns1:userAgent>

            <ns1:validateOnly>false</ns1:validateOnly>

            <ns1:partialFailure>false</ns1:partialFailure>

        </ns1:RequestHeader>

    </soapenv:Header>

    <soapenv:Body>

        <get xmlns="https://adwords.google.com/api/adwords/cm/v201607">

            <serviceSelector>

                <fields>Id</fields>

                <fields>Name</fields>

                <fields>Status</fields>

                <fields>Amount</fields>

                <fields>Settings</fields>

                <fields>AdvertisingChannelType</fields>

                <predicates>

                    <field>AdvertisingChannelType</field>

                    <operator>IN</operator>

                    <values>SEARCH</values>

                    <values>SHOPPING</values>

                </predicates>

                <predicates>

                    <field>Status</field>

                    <operator>IN</operator>

                    <values>PAUSED</values>

                    <values>ENABLED</values>

                    <values>REMOVED</values>

                </predicates>

                <paging>

                    <startIndex>0</startIndex>

                    <numberResults>9999</numberResults>

                </paging>

            </serviceSelector>

        </get>

    </soapenv:Body>

</soapenv:Envelope>

---------------------------

And here the SOAP response

2016-09-08 08:40:46,725 1366360824 [qtp1501587365-37324] WARN  soapXmlLogger  SOAP Response:

null

[Christian]

Hi,

we got similar problems starting September 6, 3pm UTC. Since then an increased number of API calls hang when reading from a socket (we read in blocking mode from sockets). For now the connections seem to be held open indefinitely. I would expect it to be closed by some Google server sooner or later -- that would at least make it easier on our side to deal with these problems.

We had no application or infrastructure change on our side in the last days. Also, we do not see these problems on other network connections (to other services than AdWords).

Could you please investigate if there possibly related network changes or issues at Google/AdWords?

Thanks for your support!

Best regards,

Christian

[Josh Radcliff (AdWords API Team)]

Hi,

I'll look into the increased occurrences of timeouts, but in the meantime, if you are using the Java client library, you can reduce the SOAP request timeout from the default of 20 minutes (1200000 milliseconds) by setting the api.adwords.soapRequestTimeout value in your ads.properties file.

Thanks,

Josh, AdWords API Team

[Josh Radcliff (AdWords API Team)]

Hi,

One other tip: if you're encountering this issue with reporting requests and using the Java client library, you can modify the api.adwords.reportDownloadTimeout value as well in your ads.properties file.

Thanks,

Josh, AdWords API Team

[Josh Radcliff (AdWords API Team)]

Hi,

Would either of you happen to have a particularly unique example request where this failed? It looks like you're not getting back a requestId in this case, but if your request contained a string or ID that was unique to that request, that would help me track down what happened.

If you would prefer, you can send the information only to me by clicking Reply privately to author on this message.

Thanks,

Josh, AdWords API Team

[Stefan Kleine Stegemann]

Hi Josh,

thanks for your reply. First, the situation got much worse this morning with the number of timeouts increasing significantly. About setting the read timeout, is there a away of doing that in pure Java without ads.properties because we have our own "setup code" and configuration mechanism.

For the request-id, I'll check our log-files today and see if I can find an id.

Best wishes,

Stefan

[Christian Bünnig]

Hi,

we are using the Python client library. Looks like there is a default timeout of 1 hour for SOAP (suds) requests, but in practice the hanging reads on connections do not time out. I'll check for a unique request identifier as well and will forward it to you, Josh.

Best,

Christian

---

Christian Bünnig
Head of Software Development

Bidmanagement GmbH | Chausseestraße 22 | D-10115 Berlin
T +49 30 4081715-500 | F +49 30 4081715-555
christia...@adspert.de | http://www.adspert.de

Amtsgericht: Berlin-Charlottenburg 133120 B
Geschäftsführer: Marcel Pirlich, Stephanie Richter

--
--
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
Also find us on our blog and Google+:
https://googleadsdeveloper.blogspot.com/
https://plus.google.com/+GoogleAdsDevelopers/posts
=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~=~
 
You received this message because you are subscribed to the Google
Groups "AdWords API Forum" group.
To post to this group, send email to adwor...@googlegroups.com
To unsubscribe from this group, send email to
adwords-api+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/adwords-api?hl=en
---
You received this message because you are subscribed to a topic in the Google Groups "AdWords API Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/adwords-api/OWusHGc6xaY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to adwords-api+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/adwords-api.
To view this discussion on the web visit https://groups.google.com/d/msgid/adwords-api/b31e041c-b8e8-4f6d-a86d-46d952921d65%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tomas Rimkus]

Hi,

We are experiencing exactly the same problem since 7th of September when using Reporting API. It seems the time-outs occur for some accounts more often than the others.

It's also taking way too long for the connection to be dropped because of "time-out" with default settings. How can we set max time-out time in PHP client library?

[Stefan Kleine Stegemann]

Hi Josh,

i'm sorry but I wasn't able to find a request with a request-id in our logs. I inspected the soap dumps from failed requests and found none with a request id. I found some other failed requests where the id was displayed but these requests failed because of "normal" errors (such as user permission denied). I wonder what's the rule for request having an id or not?

Cheers,

Stefan

[Josh Radcliff (AdWords API Team)]

Hi Stefan,

Thanks for looking. Since you aren't getting a response back due to the timeout, you won't have a requestId (which is part of the response payload). However, if you have a specific ad group ID or campaign ID where you've encountered timeouts, along with the timestamp (including timezone) of the timed out request, I may be able to find what went wrong with that particular request.

Thanks,

Josh, AdWords API Team

[Stefan Kleine Stegemann]

Hi Josh,

I sent you a private mail with some information.

I hope that helps.

Thanks,

Stefan

[Josh Radcliff (AdWords API Team)]

Hi Stefan,

Thanks for sending over that info. I looked at those requests and didn't see any errors in terms of AdWords API handling, which suggests this may be an issue at a different layer on our side. I'll keep digging and let you know what I find.

Cheers,

Josh, AdWords API Team

[Stefan Kleine Stegemann]

Hi Josh,

is there any update on this issue? The situation is still the same and we haven't noticed an improvement over the past days (it also did not get worse, though).

Thanks,

Stefan

[Tomas Rimkus]

I can confirm that the issue is still present. At least one report download failed after multiple retries due to read time-out on our end.

[Stefan Kleine Stegemann]

Hi Josh,

as an additional information, our main problem seems to be the AdWords management API, not the reporting API. The latter is giving us timeouts as well but we are usually talking "only" about minutes here. The main problem are the services (TrafficEstimatorService, CampaignService, AdGroupService) which have very long timeouts (up to an hour, as I've mentioned in my first post).

Cheers,

Stefan

[Josh Radcliff (AdWords API Team)]

Hi Stefan,

Thanks for sending over the details of your reporting request. What's odd is that I ran that same request and it returned in ~4 minutes.

Everyone who is watching this thread: Could you let me know which region or country your hosts reside in, provided you are comfortable sharing that information? I'd like to know if this is limited to a specific region, or if region is not a factor. If you'd prefer to share that information only with me, you can click Reply privately to author on this message.

Thanks,

Josh, AdWords API Team

[Tomas Rimkus]

We are running Amazon EC2 instance located in Frankfurt, Germany.

[Stefan Kleine Stegemann]

Hi Josh,

we're also running on AWS services located in Frankfurt, Germany.

Stefan

[Stefan Kleine Stegemann]

Hi Josh,

sorry for the second post but I did not read it completely first.

The failed request was also retried on our side and went through without problems (definitely less than 4 minutes).

Thanks,

Stefan

[Christian]

Hi,

we are in AWS Frankfurt as well. Could be relevant: we are use AWS' managed NAT service.

Best,

Christian

[Thanet Knack Praneenararat (AdWords API Team)]

Hello Christian, Stefan,

Thanks for information.

I'll pass this to engineering for more investigation.

Cheers,

Thanet, AdWords API Team

[Stefan Kleine Stegemann]

Hi Thanet,

just wanted to report that the timeout problems have gone now. Timeouts started to drop significantly on friday morning and since saturday, we don't any have timeout at all. Just out of interest: did you change anything on your side?

Anyway, thanks for your support.

Stefan

[Tomas Rimkus]

It seems to be working well for us too since Friday.

It would be nice if google developers could share what was the problem on their end which caused the timeouts.

[Thanet Knack Praneenararat (AdWords API Team)]

Hi Thomas, Stefan,

Glad to hear that the problem has gone.

I'm still following up with engineering and also doesn't have clue what happened.

If I get more information, I will keep you posted.

Best,

Thanet, AdWords API team