Intermittent server timeout when accessing API

[Danilo B.]

I work on a project where we have a lot of outgoing HTTP requests to the Adwords API. The system worked good until last week, where a lot of the outgoing calls to Adwords started getting timeouts without any response from the Server.

I managed to find a request to the reporting URL that seemed to trigger the problem every 5-10 requests (using the python-requests library):

    requests.post('https://adwords.google.com/api/adwords/reportdownload/v201406',

        data='__rdxml=XXX',

        headers={

            'clientCustomerID': 1234,

            'developerToken': 'XXX',

            'Accept-Encoding': 'gzip',

            'User-Agent': 'XXX',

            'Content-Type': 'application/x-www-form-urlencoded',

            'Authorization': u'Bearer XXX'

        },

        stream=False)

With this, every few requests the following exception occured:

> ConnectionError: HTTPSConnectionPool(host='adwords.google.com', port=443): Max retries exceeded with url: /api/adwords/reportdownload/v201406 (Caused by <class 'httplib.BadStatusLine'>: '')

This only happens on our Amazon EC2 instance though. If I try to do the request from my local host, everything runs fine for hundreds of requests.

The problem not only occurs when querying the reports, but also when using the SOAP API via the googleads library / suds.

Unauthenticated GET requests on the same resource are not affected:

    >>> for i in range(500):

    ...     url = 'https://adwords.google.com/api/adwords/reportdownload/v201406'

    ...     requests.get(url)

    ...

    <Response [400]>

    <Response [400]>

    (...)

I tried to debug the call using strace. The file descriptor is created, then a connection is established:

    fcntl(8, F_GETFL)                       = 0x2 (flags O_RDWR)

    fcntl(8, F_SETFL, O_RDWR)               = 0

    connect(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("74.125.24.113")}, 16) = 0

    setsockopt(8, SOL_TCP, TCP_NODELAY, [1], 4) = 0

    getpeername(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("74.125.24.113")}, [16]) = 0

    open("/var/venv/radar/local/lib/python2.7/site-packages/requests/cacert.pem", O_RDONLY) = 9

    read(9, "XXX"..., 4096) = 4096

    (...)

    read(9, "XXX"..., 4096) = 1234

    read(9, "", 4096)                       = 0

    close(9)                                = 0

    munmap(0x7f63311fe000, 4096)            = 0

Then the TLS handshake takes place and the request is sent. But a response is never returned.

    write(8, "\26\3\1\1\37\1\0\1\33\3\3T+\343\2175\21\361\221X\224P\371uf\362\313O\272\353;\21"..., 292) = 292

    read(8, "\26\3\3\0=\2\0", 7)            = 7

    read(8, "\0009\3\3T+\343\217\277\377`\370\220\177\350\nh\214\10\305W\247\37\267e\252\200\235\32\331g\276"..., 59) = 59

    read(8, "\26\3\3\16\366", 5)            = 5

    read(8, "\v\0\16\362\0\16\357\0\7q0\202\7m0\202\6U\240\3\2\1\2\2\10%\323H%\200`\276"..., 3830) = 3830

    read(8, "\26\3\3\1M", 5)                = 5

    read(8, "\f\0\1I\3\0\27A\4]\255\356\356\35F\340UW/w\361\323\334\244\212\242k\223\244\377\2167"..., 333) = 333

    read(8, "\26\3\3\0\4", 5)               = 5

    read(8, "\16\0\0\0", 4)                 = 4

    write(8, "\26\3\3\0F\20\0\0BA\4\325\306k,\300\244*\246\34\32\232\322/,\21F\337\230\254\270\332"..., 126) = 126

    read(8, "\26\3\3\0\256", 5)             = 5

    read(8, "\4\0\0\252\0\1\211\300\0\244\03003\177/St\230G\250ov\210&P\376 uI?k\306"..., 174) = 174

    read(8, "\24\3\3\0\1", 5)               = 5

    read(8, "\1", 1)                        = 1

    read(8, "\26\3\3\0(", 5)                = 5

    read(8, "\0\0\0\0\0\0\0\0\204\251\275\247/L\205v\273R\253\207\255\215v\227\263P\r%\200r\357\7"..., 40) = 40

    write(8, "\27\3\3\6\2\234\35\255\207D\357\357\324\222\nZ\267\377LgY\7Zm\270\344q\20d\3406\305"..., 1543) = 1543

    fcntl(8, F_GETFL)                       = 0x2 (flags O_RDWR)

    fcntl(8, F_SETFL, O_RDWR)               = 0

    read(8, <unfinished ...>

After a few minutes of no response data, a timeout aborts the read call. As there is no response data, there is also no valid HTTP header and status code, which causes the httplib.BadStatusLine error.

The connection to the server appears to be established successfully, the server responds with data to the TLS handshake. Also, in about 90% of the cases the request seems to work successfully. Therefore I suspect that the problem is on the server side.

Any pointers on solving this problem?

[Danilo B.]

The Adwords API appears to be behind some load balancer. Using netstat, I could trace back the "hanging" to a specific IP. This is the output during a "successful" request:

    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

    tcp        0      0 172.31.42.112:49312     74.125.24.101:443       ESTABLISHED 3939/python     

    tcp6       0      0 ::1:37156               ::1:5432                ESTABLISHED 3939/python     

And this is the "hanging" request:

    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name

    tcp        0   1543 172.31.42.112:57350     74.125.24.113:443       ESTABLISHED 3939/python     

    tcp6       0      0 ::1:37156               ::1:5432                ESTABLISHED 3939/python     

So it appears that the server at 74.125.24.113 is having problems.

Could anyone confirm this?

Best regards,

Danilo

[Anash P. Oommen (AdWords API Team)]

Hi Danilo,

Could you share your API MCC customer id by using the "Reply to Author" option?

Cheers,
Anash P. Oommen,

AdWords API Advisor.