AWQL Reporting slowdown?

[AdWordsApiUser]

Hi Team,

We're noticing a significant slowdown in the throughput of the AWQL Reporting using the API since last week, where we're spending about 30%-40% more time than usual. Is this a known issue, or something that is being changed? Any insights will be appreciated.

Cheers, and have a great week ahead!

MG 

[Shwetha Vastrad (AdWords API Team)]

Hi, 

Could you provide the report definitions and the clientCustomerId for which you observed reduced throughput for AdWords API report download? Please use Reply privately to author when responding

Thanks,

Shwetha, AdWords API Team.

[Tomas R]

Hi,

I am seeing it as well. Can you tell us if there is an actual issue on your end?

[Shwetha Vastrad (AdWords API Team)]

Hi, 

There isn't an ongoing issue that should be affecting report download latency via AdWords API. Could you provide the report definitions and the clientCustomerId where you observe the report download latency has increased? Could you also provide an approximate time when you started seeing this increase?

Thanks,

Shwetha Vastrad

AdWords API Team.

[AdWordsApiUser]

Hi Shwetha,

I sent you the details in a private message. Does the version of API we use affect the speed? 

Thanks,

MG

[Michael Cloonan (AdWords API Team)]

Hello,

Are you only seeing this issue with AWQL reports, or are you seeing the same issue if you run reports with the definition-style selector as well? I'm curious if it's AWQL-specific or if the slowdown is general to all reporting.

Regards,

Mike, AdWords API Team

[AdWordsApiUser]

Hi Michael,

Thanks for your reply. We only use AWQL styled queries, so I don't know right now if that could be the reason. Do you suggest that we try using the definition styled reports and compare performance? As I mentioned earlier, my job that used to get over in 8 hours now takes 16.

Cheers,
MG

[Michael Cloonan (AdWords API Team)]

Hello,

No, that will not be necessary; you specifically specified AWQL so I wanted to know if you had done any tests narrowing that down.

When you say your "job", you mean that your job runs multiple reports that, in aggregate, used to take 8 hours and now take 16, right? It's not a single report taking that long?

I will pass along the details you sent privately to our engineering team to investigate in the meanwhile.

Regards,

Mike, AdWords API Team

[AdWordsApiUser]

I saw this new thread mentioning the slowdown in TSV reports - we use TSV reports exclusively - probably that is the issue - https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/adwords-api/lnuRYUwohNw

A swift resolution will be appreciated :-)

MG

[AdWordsApiUser]

Any updates on this? Thanks!

[Michael Cloonan (AdWords API Team)]

Hello,

No updates yet. I will follow up with the team to see what they've found so far. I will post here when I have any information to share.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

No changes were made that should affect reporting speed, so I have some followup questions to see if we can determine the root cause here.

- Is this across all report types, or is there a specific report type where you're seeing the slower behavior?

- Do you see more rows get returned or anything else changed in the content of the reports?

- For a single report, how long did it take previously and how long does it take now?

Regards,

Mike, AdWords API Team

[AdWordsApiUser]

Let me run some tests and gather more data over the weekend to support my case. I'll update with my findings.

[Dorian Kind]

Hello,

FWIW, we also have experienced somewhat of a performance degradation in the reporting infrastructure.

We have one job in particular that collects a number of KPIs and structural data for a few hundred accounts using non-AWQL report definitions, and up until September 7th its run time was always in the order of ~200 minutes. Starting from September 8th however, this job now takes more than double that time, usually between 400 and 500 minutes for the same number of accounts. It's not a problem for us as the functionality is not time-critical, but it definitely became slower.

The timing suggests that it could be the same issue as described by AdWordsApiUser in the original post.

Best regards,
Dorian

[AdWordsApiUser]

I have been running tests the last couple of weeks. 

Unfortunately, as I do not have a snapshot of what throughput we used to get earlier, all I have been trying with is different settings: GZIP, no-GIP, different download formats (as reported in another thread), but nothing seems to affect performance. Our reports have become slower significantly since around September 7th. I would highly appreciate if someone could help us with this - we can keep increasing the parallelism, but that is not the way we want to go - we'll stress the API system, and likely run into rate issues. I am happy to share any details that might help with the technical investigation. At a scale of thousands of accounts, the speed difference really makes a difference.

@Dorian, do you use TSV format by any chance? I keep suspecting that it is something to do with TSV format reports, but I am not able to prove it in experiments with a hundred or so reports. 

Regards,

MG

[AdWordsApiUser]

Some more info:

- Is this across all report types, or is there a specific report type where you're seeing the slower behavior?

It appears across all report types

- Do you see more rows get returned or anything else changed in the content of the reports?

No Change

- For a single report, how long did it take previously and how long does it take now?

When pulling reports in bulk, we are roughly getting half the throughput compared to earlier. I'm talking about thousands of accounts.

[Tomas R]

We are currently using AdWords reporting, Analytics Reporting API v4 and Facebook Ads Insights reporting API.

On average AdWords reporting is approximately at least twice as slow as Analytics Reporting (usually more) and usually three times as slow as Facebook Ads Insights API. And yes, we have standard access.

I am pretty sure there should be a way for you to improve the situation.

[Dorian Kind]

On Monday, 16 October 2017 20:28:44 UTC+2, AdWordsApiUser wrote:

@Dorian, do you use TSV format by any chance? I keep suspecting that it is something to do with TSV format reports, but I am not able to prove it in experiments with a hundred or so reports. 

No, we use CSV reports exclusively.

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for the answers to my questions, AdWordsApiUser. For the last question, can you give the real-time it takes to run reports previously and now, rather than just how much slower it is relatively to before?

Regards,

Mike, AdWords API Team

[AdWordsApiUser]

I don't have numbers prior to September 7, because we did not do per call timing tests before the apparent slowdown.

A nightly job, that pulled the same query for many thousand accounts, would complete by 11AM. Starting September 7, it now runs till 5 PM. Something happened between Sep 6 and Sep 7. This is the second slowdown after the one that happened about 6 months ago (which was discussed in detail in this forum as well)

Cheers,

MG

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for the extra context. I will relay this to the team and I will share here when I have any updates.

Regards,

Mike, Adwords API Team

[AdWordsApiUser]

Hi Michael,

Any updates on this? It appears that some particular accounts are affected more badly than others. If I find something specific I will send it across. Thanks.

MG

[Tomas R]

It's a bit like a closed loop:

1) Reporting API does not support sorters and paging, so you end up getting data you don't really need which increases response time.

2) All the requests are super slow anyway, so you have to implement caching with multi-threading. The slower it gets, the more threads you use. Multiple companies hammering reporting API this way, obviously doesn't improve API performance.

3) Finally from time-to-time the API stops responding in timely fashion at all or starts throwing 500 errors (happens relatively often these days) and the engineers apply some temporary work-around which doesn't solve the real issue and we are back at point 1.

[Dorian Kind]

Hi all,

I was just wondering whether the situation is still the same for you? On our end, the reporting slowdown first encountered in September seems to be permanent.

Thanks for your feedback and best regards,

Dorian

[Tomas R]

Yes, the reporting API is still extremely slow and I doubt anything will change in near future.

[chris.s...@outlook.com]

The response times for various parts of the API are in constant fluctuation, with some responses taking around 200ms on some days, and over 79 seconds other days. What is going on? Why is it so unstable?

[AdWordsApiUser]

Starting December 23rd/24th, I have seen a significant speed up in the reporting API responses. Anyone else noticed this? It is still not back to the pre-September 7 levels, but there is a noticeable 20% or more improvement. Could the API team do more of this magic that happened? Thanks :)