StreamingResponseIterator from search_stream gets stuck

[Tjadi Peeters]

Hi,

I am wondering regarding what to do when a StreamingResponseIterator from a ga_service.search_stream(search_request) seems to get stuck at a certain 'row' and whether this is a bug or expected. 

Example (with printing to show that it gets stuck):

It doesn't matter how many times I redo the query, it gets stuck at some point at a specific point (batch 17 after row 7000). Before yesterday evening the exact same code/ data used to run fine within a minute. One of our jobs continued in this state for 6 hours, before it got killed by its time out (while yesterday it ran in 5 min). Not sure what I can do about this?

Kind regards,

[Google Ads API Forum Advisor]

Hi Tjadi,

Thank you for reaching out to us.

For our team to further investigate this, could you provide us with the complete API logs(request and response logs with request ID) generated on your end? Note that this can be requested or provided to the developer handling the Google Ads API transactions when logging of the API requests has been enabled.

These are the specific guidelines to enable it.

• Java - https://developers.google.com/google-ads/api/docs/client-libs/java/logging
• .Net - https://developers.google.com/google-ads/api/docs/client-libs/dotnet/logging
• PHP - https://developers.google.com/google-ads/api/docs/client-libs/php/logging
• Python - https://developers.google.com/google-ads/api/docs/client-libs/python/logging
• Ruby - https://developers.google.com/google-ads/api/docs/client-libs/ruby/logging
• Perl - https://developers.google.com/google-ads/api/docs/client-libs/perl/logging

You may then send the requested information via the Reply privately to author option. If this option is not available, you may send the details directly to our googleadsa...@google.com alias instead.

Best regards,

Heidi Google Ads API Team

ref:_00D1U1174p._5004Q2egwqd:ref

[Robert Heise]

Hi,

we're having the same problem, though we're using the official ruby gem from Google Ads.  The exact same ruby code worked fine until today. Since today most jobs are getting stuck while iterating through the result. Since there are a few forum feeds reporting issues with search_stream behavior today (performance, timeout, getting stuck) it feels like an issue on the Google Ads API side. Can you confirm and give guidance how to overcome this behavior of the Google Ads API?

Thanks,

[Tjadi Peeters]

Hi,

Not sure if I am doing anything wrong, but after running the following on databricks (I run it before initialising client, as the documentation says is needed):

import logging
logging.basicConfig(level=logging.INFO, format='[%(asctime)s - %(levelname)s] %(message).5000s')
logging.getLogger("py4j").setLevel(logging.ERROR)
logging.getLogger('google.ads.googleads.client').setLevel(logging.DEBUG)

I don't get any logs before it gets stuck.

Kind regards,

Op dinsdag 27 september 2022 om 17:32:23 UTC+2 schreef Robert Heise:

[Google Ads API Forum Advisor]

Hi Tjadi and Robert,

Allow me to raise this issue to my rest of the team to look into further investigations. We will respond back to you ASAP once we have any updates.

Regards,

Yasar Google Ads API Team

ref:_00D1U1174p._5004Q2egwqd:ref

[Google Ads API Forum Advisor]

Hello Everyone,

Thank you so much for providing all this information! I believe the issue that was causing this was cleared up a few hours ago. I see our monitoring is showing that traffic is back to normal. If you continue to have the same issue, please respond back to me.

Thanks,

Nadine Wang Google Ads API Team

ref:_00D1U1174p._5004Q2egwqd:ref

[Mario García]

We're still seeing this problem when streaming a large set of keywords from a customer account. It appears the request gets stuck and eventually the process times out. We have a query as follows:

  SELECT 

  ad_group_criterion.criterion_id, 

  ad_group_criterion.status,

  ad_group_criterion.keyword.match_type,

  ad_group_criterion.keyword.text, 

   ...

  FROM ad_group_criterion WHERE  ad_group_criterion.type = KEYWORD

and we use the searchStream API to process the results. The result set is very large, and could exceed  100,000 keywords but up until today we never had a problem with it.

[Andrew Repp]

Hi,

We are also continuing to see the same issue even after the all clear message above. It appears to be related to the number of records involved.

This query

        SELECT
            segments.date,customer.id,campaign.name,campaign.id,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
        FROM
            campaign_audience_view
    WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-23' AND '2022-09-27'

fails with 

Response
-------
Headers: {}
Fault: {
  "created": "@1664321827.318637000",
  "description": "Error received from peer ipv4:142.250.217.74:443",
  "file": "src/core/lib/surface/call.cc",
  "file_line": 904,
  "grpc_message": "Deadline Exceeded",
  "grpc_status": 4
}

But this query

        SELECT
            segments.date,customer.id,campaign.name,campaign.id,campaign_criterion.bid_modifier,campaign_criterion.criterion_id,segments.device,metrics.impressions,metrics.clicks,metrics.cost_micros,metrics.gmail_forwards,metrics.gmail_saves,metrics.gmail_secondary_clicks
        FROM
            campaign_audience_view
    WHERE metrics.impressions > 0 AND segments.date BETWEEN '2022-09-26' AND '2022-09-27'

succeeds. Notice that by only selecting 2 days of data instead of 5, the query is able to complete. Before today, the query for 5 days completed successfully.

Any help from the Google folks on a resolution?

Thanks,

Andrew

[Robert Heise]

Hi,

same here. Some reports succeed now. But some reports don't, most likely related to the data size. We didn't change anything on our side, all report fetches worked fine with their defined date ranges before Sept 27, when the issue started.

The issue hasn't been fully resolved and we're still unable to fetch all reports via the Google Ads API. The behavior for the failing fetches is still the same. The iterator in the search_stream fetch gets stuck, is never finishing and at some point our wrapping timeout kills the task. Since there was no code change on our side and many developers are reporting the same issue here, using different programming languages, we assume the issue is on the Google Ads API side. My guess would be that the GRPC call for search_stream is not properly ended. To the best of our knowledge and our inspection so far, this is far too low level from the perspective of users of the Google Ads API (developers) and needs to be fixed on the Google side.

Ideally please provide an ETA when the issue will be fixed on Google side. Also I wonder where we can best track the status of the API issue. https://ads.google.com/status/publisher/ continuously shows Google Ads operating fine, though the API clearly has severe issues for search_stream requests.

Thanks,

Robert

[Tjadi Peeters]

Hi,

Unfortunately the issue does not seem to have been resolved as of yet. The query shown in the original post still gets stuck at exactly the same spot. What could we do to investigate this further? And I think, as Robert suggests, that a location where the status of this API issue can be tracked would be great, as we are currently in a situation where we have to explain this to our customers. An ETA would also be much appreciated.

Kind regards,

Op woensdag 28 september 2022 om 06:23:35 UTC+2 schreef Robert Heise:

[Google Ads API Forum Advisor]

Hello,

The issue was resolved for a few hours, but then an additional issue popped up. We're working on resolving it. It's our top priority.

Regards,

[Google Ads API Forum Advisor]

Greetings!

We were able to make some changes. It looks like the requests are returning to normal. We'll continue to monitor the situation. Please reach out on this thread if you continue to see this issue.

Thanks,