Our AWQL response latency has increased 3x recently. Is it just us?

[Axel Vermeil]

Hi!

Since the 7th of June, we've noticed that our AWQL requests are taking a lot longer than we're used to. 

Here's our AWQL request time over the last month: https://www.dropbox.com/s/r4v3uqlj6b7l5m9/Screenshot%202017-06-12%2017.23.38.png (the y-axis is milliseconds)

We usually expect responses back in about 1000ms. Lately, the average has been closer to 3500ms, with high variability in individual queries. Both the initial HEAD time and the total transfer time have increased.

This is using a single developer token across thousands of different customer IDs.

We use AWQL heavily in our customer facing app as well as our backend jobs. Both have been suffering due to this increased latency. Usually, we'd run around 100 AWQL queries per second (close to our rate limit).

I've triple checked that this isn't an issue on our end -- even when doing AWQL requests completely manually in curl/postman, the latency is abnormally high: https://www.dropbox.com/s/uz10bkf3wfdbeqa/Screenshot%202017-06-12%2017.30.31.png?dl=0 (request time is bottom right). I've also tried running queries from servers in two continents (UK and US), in case it's a datacenter-specific issue, but no luck.

Has anybody else noticed this? 

I can supply tokens/IDs if that helps debug this issue from the AW end.

Thanks in advance :)

-- Axel

[Shwetha Vastrad (AdWords API Team)]

Hi Axel, 

We are aware of the issue. We are investigating the latency in report downloads in AdWords API. I'll update this thread once I have more information. 

Regards,

Shwetha, AdWords API Team.

[Axel Vermeil]

Alright, thanks for the update :)

Roughly when can we expect a resolution for this? Are we talking days, weeks, or months? I'm asking because we need to know whether it's worth our time to invest into some technical infrastructure to support these higher latencies.

Axel

[Shwetha Vastrad (AdWords API Team)]

Hi Axel, 

I'm afraid I don't have a timeline for when this will be resolved. I'll keep you notified when I have more information. 

[Tomas R]

Do you know if it's region specific issue or for all requests?

[dev eloper]

Hi, 

We too have been suffering from a major lag when requesting reports.  In most cases its 300-450% slower than usual. 

This has been happening since last Friday.  

This thread is the only indication that there is a problem with the google servers.

I cannot find any information about this problem anywhere else.  How do I get an update, estimated time to fix etc?  I need to update customers / other parts of the business.

Thanks

[Anash P. Oommen (AdWords API Team)]

Folks,

We are working on it, we don't have an ETA yet. I'll update this thread as I hear from the team.

Cheers

Anash P. Oommen,
AdWords API Advisor.

[Anash P. Oommen (AdWords API Team)]

Folks,

We are pushing an update today that improves the report download latency. We will continue the investigations next week with a goal to bring it down even further, ideally to original levels. I'll update this thread as I hear from the team.

Thanks everyone for your patience!

Cheers

Anash P. Oommen,
AdWords API Advisor.

[Anash P. Oommen (AdWords API Team)]

Folks,

The first stage of fixes are live on production. The latency should improve significantly, though there's still room for improvement. We are exploring more changes that should improve the latency further, bringing it closer to pre-June-6 values. I'll update this thread as more fixes are rolled out.

Cheers

Anash P. Oommen,
AdWords API Advisor.

[Dorian Kind]

Hi,

we're still seeing lots of 500 error response codes from the reporting endpoint with the reason ERROR_GETTING_RESPONSE_FROM_BACKEND and the trigger "Unable to read report data". These errors persist when retrying using exponential backoff, as well. Is there anything we can do on our end to alleviate the problem?

Thanks and best regards,
Dorian

[Anash P. Oommen (AdWords API Team)]

Hi Dorian,

If you can use the "Reply to Author" share your reporting HTTP logs, it would be most helpful. I want to make sure if this is related to the same issue or not before confirming.

Cheers

Anash P. Oommen,
AdWords API Advisor. 

[Axel Vermeil]

Hi Anash,

The latency has improved significantly for us, down to roughly 150% of what it was pre-June-6. Thanks for your team's work so far!

Is this the new normal, or should we expect further improvements?

Cheers,

Axel