Internal (500) Error when requesting change data

[Scott Vaillancourt]

I'm trying to pull change data via the API and have been routinely encountering '500 Internal errors' for the past three weeks. This exact query has been running without issue for over 2 years. I've included the debug output for the request and response below. I realize there have been other similar issues logged and fixes were deployed, but they don't seem to have fully addressed the problem. Thanks.

Scott

[2024-02-27 08:59:32,396 - INFO] Request
-------
Method: /google.ads.googleads.v15.services.GoogleAdsService/Search
Host: googleads.googleapis.com
Headers: {
  "developer-token": "REDACTED",
  "login-customer-id": "REDACTED",
  "x-goog-api-client": "gl-python/3.9.2 grpc/1.62.0 gax/2.17.1 gccl/23.0.0 pb/4.25.3",
  "x-goog-request-params": "customer_id= REDACTED"
}
Request: customer_id: "REDACTED"
query: "\n    SELECT\n      change_event.change_date_time,\n      change_event.change_resource_name,\n      change_event.change_resource_type,\n      change_event.resource_change_operation,\n      change_event.user_email,\n      change_event.changed_fields,\n      campaign.name\n    FROM change_event\n    WHERE\n      change_event.change_date_time BETWEEN \"2024-01-30\" AND \"2024-02-26\"\n    LIMIT 2000"


Response
-------
Headers: {
  "google.ads.googleads.v15.errors.googleadsfailure-bin": "\u0012\u0016fn2plGzwwkQEAGLMv5tx6g",
  "grpc-status-details-bin": "\b\r\u0012\u001bInternal error encountered.\u001a`\nDtype.googleapis.com/google.ads.googleads.v15.errors.GoogleAdsFailure\u0012\u0018\u0012\u0016fn2plGzwwkQEAGLMv5tx6g",
  "request-id": "fn2plGzwwkQEAGLMv5tx6g"
}
Fault: {}

[2024-02-27 08:59:32,397 - WARNING] Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com, Method: /google.ads.googleads.v15.services.GoogleAdsService/Search, RequestId: fn2plGzwwkQEAGLMv5tx6g, IsFault: True, FaultMessage: Internal error encountered.
---------------------------------------------------------------------------
_InactiveRpcError                         Traceback (most recent call last)
File ~/.local/lib/python3.9/site-packages/google/api_core/grpc_helpers.py:76, in _wrap_unary_errors.<locals>.error_remapped_callable(*args, **kwargs)
     75 try:
---> 76     return callable_(*args, **kwargs)
     77 except grpc.RpcError as exc:

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:277, in _UnaryUnaryMultiCallable.__call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    268 def __call__(
    269     self,
    270     request: Any,
   (...)
    275     compression: Optional[grpc.Compression] = None,
    276 ) -> Any:
--> 277     response, ignored_call = self._with_call(
    278         request,
    279         timeout=timeout,
    280         metadata=metadata,
    281         credentials=credentials,
    282         wait_for_ready=wait_for_ready,
    283         compression=compression,
    284     )
    285     return response

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:332, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    329 call = self._interceptor.intercept_unary_unary(
    330     continuation, client_call_details, request
    331 )
--> 332 return call.result(), call

File ~/.local/lib/python3.9/site-packages/grpc/_channel.py:439, in _InactiveRpcError.result(self, timeout)
    438 """See grpc.Future.result."""
--> 439 raise self

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call.<locals>.continuation(new_details, request)
    314 try:
--> 315     response, call = self._thunk(new_method).with_call(
    316         request,
    317         timeout=new_timeout,
    318         metadata=new_metadata,
    319         credentials=new_credentials,
    320         wait_for_ready=new_wait_for_ready,
    321         compression=new_compression,
    322     )
    323     return _UnaryOutcome(response, call)

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:343, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    334 def with_call(
    335     self,
    336     request: Any,
   (...)
    341     compression: Optional[grpc.Compression] = None,
    342 ) -> Tuple[Any, grpc.Call]:
--> 343     return self._with_call(
    344         request,
    345         timeout=timeout,
    346         metadata=metadata,
    347         credentials=credentials,
    348         wait_for_ready=wait_for_ready,
    349         compression=compression,
    350     )

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:332, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    329 call = self._interceptor.intercept_unary_unary(
    330     continuation, client_call_details, request
    331 )
--> 332 return call.result(), call

File ~/.local/lib/python3.9/site-packages/grpc/_channel.py:439, in _InactiveRpcError.result(self, timeout)
    438 """See grpc.Future.result."""
--> 439 raise self

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call.<locals>.continuation(new_details, request)
    314 try:
--> 315     response, call = self._thunk(new_method).with_call(
    316         request,
    317         timeout=new_timeout,
    318         metadata=new_metadata,
    319         credentials=new_credentials,
    320         wait_for_ready=new_wait_for_ready,
    321         compression=new_compression,
    322     )
    323     return _UnaryOutcome(response, call)

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:343, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    334 def with_call(
    335     self,
    336     request: Any,
   (...)
    341     compression: Optional[grpc.Compression] = None,
    342 ) -> Tuple[Any, grpc.Call]:
--> 343     return self._with_call(
    344         request,
    345         timeout=timeout,
    346         metadata=metadata,
    347         credentials=credentials,
    348         wait_for_ready=wait_for_ready,
    349         compression=compression,
    350     )

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:329, in _UnaryUnaryMultiCallable._with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
    327         return _FailureOutcome(exception, sys.exc_info()[2])
--> 329 call = self._interceptor.intercept_unary_unary(
    330     continuation, client_call_details, request
    331 )
    332 return call.result(), call

File ~/.local/lib/python3.9/site-packages/google/ads/googleads/interceptors/exception_interceptor.py:99, in ExceptionInterceptor.intercept_unary_unary(self, continuation, client_call_details, request)
     98 if exception:
---> 99     self._handle_grpc_failure(response)
    100 else:

File ~/.local/lib/python3.9/site-packages/google/ads/googleads/interceptors/exception_interceptor.py:71, in ExceptionInterceptor._handle_grpc_failure(self, response)
     48 """Attempts to convert failed responses to a GoogleAdsException object.
     49
     50 Handles failed gRPC responses of by attempting to convert them
   (...)
     69         will be raised as-is.
     70 """
---> 71 raise self._get_error_from_response(response)

File ~/.local/lib/python3.9/site-packages/grpc/_interceptor.py:315, in _UnaryUnaryMultiCallable._with_call.<locals>.continuation(new_details, request)
    314 try:
--> 315     response, call = self._thunk(new_method).with_call(
    316         request,
    317         timeout=new_timeout,
    318         metadata=new_metadata,
    319         credentials=new_credentials,
    320         wait_for_ready=new_wait_for_ready,
    321         compression=new_compression,
    322     )
    323     return _UnaryOutcome(response, call)

File ~/.local/lib/python3.9/site-packages/grpc/_channel.py:1193, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
   1187 (
   1188     state,
   1189     call,
   1190 ) = self._blocking(
   1191     request, timeout, metadata, credentials, wait_for_ready, compression
   1192 )
-> 1193 return _end_unary_response_blocking(state, call, True, None)

File ~/.local/lib/python3.9/site-packages/grpc/_channel.py:1005, in _end_unary_response_blocking(state, call, with_call, deadline)
   1004 else:
-> 1005     raise _InactiveRpcError(state)

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.INTERNAL
    details = "Internal error encountered."
    debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B2607:f8b0:4023:1000::5f%5D:443 {grpc_message:"Internal error encountered.", grpc_status:13, created_time:"2024-02-27T08:59:32.391236796-06:00"}"
>

The above exception was the direct cause of the following exception:

InternalServerError                       Traceback (most recent call last)
Cell In[10], line 1
----> 1 resp = list(gas.search(customer_id=cid, query=query))

File ~/.local/lib/python3.9/site-packages/google/ads/googleads/v15/services/services/google_ads_service/client.py:4171, in GoogleAdsServiceClient.search(self, request, customer_id, query, retry, timeout, metadata)
   4164 metadata = tuple(metadata) + (
   4165     gapic_v1.routing_header.to_grpc_metadata(
   4166         (("customer_id", request.customer_id),)
   4167     ),
   4168 )
   4170 # Send the request.
-> 4171 response = rpc(
   4172     request,
   4173     retry=retry,
   4174     timeout=timeout,
   4175     metadata=metadata,
   4176 )
   4178 # This method is paged; wrap the response in a pager, which provides
   4179 # an `__iter__` convenience method.
   4180 response = pagers.SearchPager(
   4181     method=rpc,
   4182     request=request,
   4183     response=response,
   4184     metadata=metadata,
   4185 )

File ~/.local/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py:131, in _GapicCallable.__call__(self, timeout, retry, compression, *args, **kwargs)
    128 if self._compression is not None:
    129     kwargs["compression"] = compression
--> 131 return wrapped_func(*args, **kwargs)

File ~/.local/lib/python3.9/site-packages/google/api_core/grpc_helpers.py:78, in _wrap_unary_errors.<locals>.error_remapped_callable(*args, **kwargs)
     76     return callable_(*args, **kwargs)
     77 except grpc.RpcError as exc:
---> 78     raise exceptions.from_grpc_error(exc) from exc

InternalServerError: 500 Internal error encountered. [type_url: "type.googleapis.com/google.ads.googleads.v15.errors.GoogleAdsFailure"
value: "\022\026fn2plGzwwkQEAGLMv5tx6g"
]

[Google Ads API Forum Advisor]

Hi,

Thank you for reaching out to the Google Ads API support team.

After reviewing your concern, I understand that you are encountering the "INTERNAL_ERROR" while trying to fetch the change_event data via the Google Ads API.

Kindly note that the INTERNAL_ERROR occurs when something unexpected happened while processing the request. The common cause could be that the API isn't functioning correctly due to a bug. However, this is transient in nature and the issue may be resolved by retrying the request after a short pause. You may retry any requests that failed with this error, using an exponential backoff schedule for the retries. Here is a code example demonstrating how to implement an exponential back off strategy.

Also, in order to assist you further, kindly provide us with the complete API logs (request and response with request-id and request header) generated at your end without redacting the customer-id and login-customer-id.

Since you are using the Python client library, if you haven't enabled the logging yet, I would request you to enable it by referring to this guide Python.

You can send the details via Reply privately to the author option, or direct private reply to this email.

 

This message is in relation to case "ref:!00D1U01174p.!5004Q02rzUBt:ref"

Thanks,
 

Google Ads API Team