Performance of SearchStream

[Zweitze]

Looking at the API dashboard of the Google Project (the project where the Google API is enabled), I notice that on Sep 27, method google.ads.googleads.v11.services.GoogleAdsService.SearchStream is suddenly very slow:
Avg latency 3.185 seconds

99th percentile: 9.018 seconds

Which is multitudes slower than previous days.

Also my software -using .NET library- caught this error

Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1664238664.220000000","description":"Error received from peer ipv4:216.58.214.10:443","file":"..\..\..\src\core\lib\surface\call.cc","file_line":1070,"grpc_message":"Stream removed","grpc_status":2}")

Unfortunately I have no RequestId and no logs. My log was dated Sep 27, 02:31:04 CEST.

Anything going on?

The Google Ads Status Dashboard only mentions problems with Google Ad Manager, not Google Ads.

[Ales Sturala]

We can see the same issues with the performance today morning, loads of DeadlineExceeded errors.

[Google Ads API Forum Advisor]

Hello,

Thanks for reaching out. If you are noticing increased latency, please privately provide us with complete request and response logs from a request showing this behavior. 

Thanks,

Matt Google Ads API Team

ref:_00D1U1174p._5004Q2egxrU:ref

[Sean Dorsman]

Hi,

Similar issues on our end:

ResponseCode: DEADLINE_EXCEEDED, Fault: Deadline expired before operation could complete..

processes that ran in 10-15 minutes now get killed at an hour.

[Tjadi Peeters]

Hi,

We have the same issue.

Op dinsdag 27 september 2022 om 16:04:01 UTC+2 schreef sean.d...@coolblue.nl:

[Zweitze]

Thanks for reaching out. If you are noticing increased latency, please privately provide us with complete request and response logs from a request showing this behavior.

Sorry, but no can do. The .NET library assumes one can write logs to disk and offers no alternatives - not possible in our situation.

Instead, why don't you check the dashboard of Google Cloud APIs and Services, for project ID 466194048264?

That contains logs for a few thousand API calls.

Here a screenshot of the dashboard:

On Tuesday, September 27, 2022 at 3:49:03 PM UTC+2 adsapi wrote:

[bar...@ppcbee.com]

Hello,

I see same problem with my application. 

This is log from New Relic for past 3 days:

As you can see search stream is super slow. There are no changes in my source code.

Can you look into this? Our clients are starting to be unhappy that they are not able to update theri Google Ads campaigns.

Thanks,

Ondrej Bartas

[bar...@ppcbee.com]

Hi, 

also I am receiving termination of search stream:

[Thomas Vincent-Sweet]

We're also affected by this. Unable to download large reports, small ones are much slower too. Latency is off the charts so to speak.

Hope this gets resolved very soon!

[Google Ads API Forum Advisor]

Hello,

Thanks for providing feedback on this issue. Please note that this issue is being handled internally. As soon as we have more information, we will provide an update on this thread.

[Google Ads API Forum Advisor]

Greetings!

Thanks for providing the information. You should see traffic coming back to normal with your errors returning to normal levels. The issue that caused this has been resolved. Please get back to me if you continue to see issues.

Thanks,

Nadine Wang Google Ads API Team

ref:_00D1U1174p._5004Q2egxrU:ref

[Robert Heise]

Hi,

still having the same issue. Some reports finish, most of them don't, search_stream requests not finishing and getting stuck.

Please provide an ETA when the issue will be resolved on Google side.

Thanks,

Robert

[Sean Dorsman]

Hello,

Also for us, the large reports still fail. With HTTP requests or with the client library in the ads to bq dataflow pipeline.

Smaller reports do work, but they never stopped working on our end.

[Tjadi Peeters]

Hi,

We are still running into the same issue for large reports (our small ones never stopped working either).

Kind regards,

Op woensdag 28 september 2022 om 07:51:40 UTC+2 schreef sean.d...@coolblue.nl:

[Zweitze]

Same here

I did notice that, when the server-side timeout occurs, the error changed.

 

Sep 27, 02:31 CEST:

Status(StatusCode="Unknown", Detail="Stream removed", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1664238664.220000000","description":"Error received from peer ipv4:216.58.214.10:443","file":"..\..\..\src\core\lib\surface\call.cc","file_line":1070,"grpc_message":"Stream removed","grpc_status":2}")

Sep 28, 02:30 CEST:

Status(StatusCode="Internal", Detail="Received RST_STREAM with error code 2", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"@1664325053.844000000","description":"Error received from peer ipv4:142.250.179.170:443","file":"..\..\..\src\core\lib\surface\call.cc","file_line":1070,"grpc_message":"Received RST_STREAM with error code 2","grpc_status":13}")

This was from the same GAQL query, on the same CustomerId.

[Filipe]

Same here. SearchStream stucks in large reports

[Ronald Findling]

We also experience these problems the last days. Massive slowdown + RST_STREAM errors and some more :/

[Google Ads API Forum Advisor]

Greetings!

Thanks for the additional information. The issue was resolved for a few hours, but then an additional issue popped up. We're working on resolving it. It's our top priority.

Regards,

[Google Ads API Forum Advisor]

Hello Google Ads API Community!

We were able to make some changes. It looks like the requests are returning to normal. We'll continue to monitor the situation. Please reach out on this thread if you continue to see this issue.

Thanks,

[Thomas Vincent-Sweet]

I can confirm the issue was solved from our perspective. Thanks for fixing it.

[Google Ads API Forum Advisor]

Greetings!

Thanks for confirming. 

Take care,

[Robert Heise]

Can confirm as well, fetches are back to normal. Thanks for fixing.

[Google Ads API Forum Advisor]

Hello Robert,

Excellent!

[Google Ads API Forum Advisor]

Hello,

I just checked our monitoring, and our system is back to normal. If you're experiencing any issues now, then it would be different than what was happening before. So, the log you sent us before would not be the same thing. Since that's the case, I would need to see a log from the last day to triage the new issue.

[Top Dev]

Hello 

I am working on google ads api integration using ASP.NET. Sometimes, I just get this error message like before.

System.AggregateException
  HResult=0x80131500
  Message=One or more errors occurred.
  Source=mscorlib
  StackTrace:
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at Google.Ads.GoogleAds.V15.Services.GoogleAdsServiceClient.SearchStream(SearchGoogleAdsStreamRequest request, Action`1 responseCallback, CallSettings callSettings)
   at Google.Ads.GoogleAds.V15.Services.GoogleAdsServiceClient.SearchStream(String customerId, String query, Action`1 responseCallback, SummaryRowSetting summaryRowSetting, CallSettings callSettings)
   at GoogleLIA.Services.GoogleService.GetGoolgeCampaigns() in E:\Client_task\google Ads\asp.net-mvc-googleLIA__\GoogleLIA\Services\GoogleService.cs:line 82
   at GoogleLIA.Controllers.CampaignController.List() in E:\Client_task\google Ads\asp.net-mvc-googleLIA__\GoogleLIA\Controllers\CampaignController.cs:line 27
   at System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters)
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c.<BeginInvokeSynchronousActionMethod>b__9_0(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__11_0()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass11_1.<InvokeActionMethodFilterAsynchronouslyRecursive>b__2()

Inner Exception 1:
RpcException: Status(StatusCode="Unknown", Detail="d

Inner Exception 2:
CoreErrorDetailException: {"created":"@1701081232.617000000","description":"Error received from peer ipv4:64.233.162.95:443","file":"..\..\..\src\core\lib\surface\call.cc","file_line":953,"grpc_message":"Stream removed","grpc_status":2}

I am not sure how to fix this error.

So please help me.

Looking forward to hearing from you soon.

[Google Ads API Forum Advisor]

Greetings!

While I appreciate that you tried to keep the forum threads condensed, the issue here is so old that it's likely no longer related to what you're seeing. That means that this is a new issues that would require a new forum thread. Otherwise, it could confuse people in the future.

Before you start a new forum thread with us, I'll throw out some ideas of the source of this. In the past, I've seen this for 2 reasons when it's not systemic of which I haven't heard of recent reports that it is systemic:

1. The data set is too large, which means the stream can close. In that case, the solution is to add filters to the query so it's retrieving less data in a single query.
2. The other option is that there is a network glitch.

If you happen to be using our .NET client library, then you can also ask about this at https://github.com/googleads/google-ads-dotnet/issues. If you aren't using the client library and these two possibilities don't apply to you, then I recommend starting a new forum thread with us so we can start a deeper investigation for you.

Thanks,
 

This message is in relation to case "ref:!00D1U01174p.!5004Q02egxrU:ref"

 

Nadine Wang Google Ads API Team