Retrieving changed_fields causes 500 Internal Error

[Scott Vaillancourt]

I'm trying to retrieve changed_fields from the change_event resource (something I've done on several occasions before) and consistently encounter an InternalServerError. I can pull other fields from this resource, but specifying changed_fields appears to trigger the error. The query and output are shown below. Any help would be appreciated. Thanks.

Scott

##########################

gas = client.get_service('GoogleAdsService', version='v6')

query = '''

    SELECT

        change_event.changed_fields 

    FROM change_event

    WHERE

        change_event.change_date_time BETWEEN "2021-02-08" AND "2021-03-08"

    LIMIT 100'''

rl = list(gas.search(cid, query))

#####################

Request made: ClientCustomerId: REDACTED, Host: googleads.googleapis.com:443, Method: /google.ads.googleads.v6.services.GoogleAdsService/Search, RequestId: -dz65qORUAIjdisRc_d83w, IsFault: True, FaultMessage: Internal error encountered.

---------------------------------------------------------------------------

_InactiveRpcError                         Traceback (most recent call last)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)

     72         try:

---> 73             return callable_(*args, **kwargs)

     74         except grpc.RpcError as exc:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in __call__(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    215                  compression=None):

--> 216         response, ignored_call = self._with_call(request,

    217                                                  timeout=timeout,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in _with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    256                                                        request)

--> 257         return call.result(), call

    258 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_channel.py in result(self, timeout)

    332         """See grpc.Future.result."""

--> 333         raise self

    334 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in continuation(new_details, request)

    240             try:

--> 241                 response, call = self._thunk(new_method).with_call(

    242                     request,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    265                   compression=None):

--> 266         return self._with_call(request,

    267                                timeout=timeout,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in _with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    256                                                        request)

--> 257         return call.result(), call

    258 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_channel.py in result(self, timeout)

    332         """See grpc.Future.result."""

--> 333         raise self

    334 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in continuation(new_details, request)

    240             try:

--> 241                 response, call = self._thunk(new_method).with_call(

    242                     request,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    265                   compression=None):

--> 266         return self._with_call(request,

    267                                timeout=timeout,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in _with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    253 

--> 254         call = self._interceptor.intercept_unary_unary(continuation,

    255                                                        client_call_details,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/ads/google_ads/interceptors/exception_interceptor.py in intercept_unary_unary(self, continuation, client_call_details, request)

    168         if exception:

--> 169             self._handle_grpc_failure(response)

    170         else:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/ads/google_ads/interceptors/exception_interceptor.py in _handle_grpc_failure(self, response)

    140         """

--> 141         raise self._get_error_from_response(response)

    142 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_interceptor.py in continuation(new_details, request)

    240             try:

--> 241                 response, call = self._thunk(new_method).with_call(

    242                     request,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_channel.py in with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)

    933                                       wait_for_ready, compression)

--> 934         return _end_unary_response_blocking(state, call, True, None)

    935 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)

    825     else:

--> 826         raise _InactiveRpcError(state)

    827 

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "Internal error encountered."

debug_error_string = "{"created":"@1615322116.438774000","description":"Error received from peer ipv6:[2607:f8b0:4009:813::200a]:443","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Internal error encountered.","grpc_status":13}"

>

The above exception was the direct cause of the following exception:

InternalServerError                       Traceback (most recent call last)

<ipython-input-3-93c9f8c36734> in <module>

----> 1 rl = list(gas.search(cid, query))

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/page_iterator.py in _items_iter(self)

    210     def _items_iter(self):

    211         """Iterator for each item returned."""

--> 212         for page in self._page_iter(increment=False):

    213             for item in page:

    214                 self.num_results += 1

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/page_iterator.py in _page_iter(self, increment)

    241             Page: each page of items from the API.

    242         """

--> 243         page = self._next_page()

    244         while page is not None:

    245             self.page_number += 1

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/page_iterator.py in _next_page(self)

    532             setattr(self._request, self._request_token_field, self.next_page_token)

    533 

--> 534         response = self._method(self._request)

    535 

    536         self.next_page_token = getattr(response, self._response_token_field)

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py in __call__(self, *args, **kwargs)

    143             kwargs["metadata"] = metadata

    144 

--> 145         return wrapped_func(*args, **kwargs)

    146 

    147 

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/retry.py in retry_wrapped_func(*args, **kwargs)

    279                 self._initial, self._maximum, multiplier=self._multiplier

    280             )

--> 281             return retry_target(

    282                 target,

    283                 self._predicate,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/retry.py in retry_target(target, predicate, sleep_generator, deadline, on_error)

    182     for sleep in sleep_generator:

    183         try:

--> 184             return target()

    185 

    186         # pylint: disable=broad-except

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/timeout.py in func_with_timeout(*args, **kwargs)

    212             """Wrapped function that adds timeout."""

    213             kwargs["timeout"] = next(timeouts)

--> 214             return func(*args, **kwargs)

    215 

    216         return func_with_timeout

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)

     73             return callable_(*args, **kwargs)

     74         except grpc.RpcError as exc:

---> 75             six.raise_from(exceptions.from_grpc_error(exc), exc)

     76 

     77     return error_remapped_callable

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/six.py in raise_from(value, from_value)

InternalServerError: 500 Internal error encountered.

[Mat]

Hi there,

we are facing similar "internal errors" since yesterday while fetching change_event reports. I've already sent the request and response logs via private mail (twice actually).

Regards

Mat

[Google Ads API Forum Advisor Prod]

Hi All,

Thanks for bringing up this issue with us. I have shared all the information you have provided with our team. We will examine the issue and provide an update as soon as we can. Thank you for your patience.

Regards,
Matt
Google Ads API Team

Matt Google Ads API Team

ref:_00D1U1174p._5004Q2Du7a3:ref

[kol...@exito.de]

We get also tons of 500 errors since we switched to API v6.1   We didn't have any such problems with v6

It does not seem be restricted to        change_event.changed_fields
I just left this field out of a request and still get 500 errors.


Here's some error info:


RequestId: qjX8ZXQsgwLcDN9CFXcImw

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "Internal error encountered."

debug_error_string = "{"created":"@1615570754.257000000","description":"Error received from peer ipv4:172.217.16.138:443","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Internal error encountered.","grpc_status":13}"



RequestId: Ytzj0SsbK3Yq07KfKlSFZA

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

status = StatusCode.INTERNAL

details = "Internal error encountered."

debug_error_string = "{"created":"@1615570687.872000000","description":"Error received from peer ipv4:172.217.16.138:443","file":"src/core/lib/surface/call.cc","file_line":1068,"grpc_message":"Internal error encountered.","grpc_status":13}"

[Google Ads API Forum Advisor Prod]

Hello - 

It seems this issue was caused by a bug on our end. It appears as though a fix is already in the process of being rolled out, I'll follow up here to let you know when I expect it should be fixed.

Thanks,
Ben, Google Ads API Team

ref:_00D1U1174p._5004Q2Du7a3:ref

[kol...@exito.de]

Do you have any timeline when the bug will finally be fixed? We still get many many 500 errors during the day.

TBH: This is quite annoying because the problem has been dragging on for more than two weeks. As it's still not possible to retrieve older change event data ("The requested start date is too old. It cannot be older than 30 days."), it means that a lot of historic change event data will be lost. We run a daily script which builds up a change event database with data older than 30 days. With all those 500 errors a lot of data will be lost and cannot be recovered at a later point.

Thanks,

Thomas

[Mat]

Hi Thomas,

we still are in the same boat ...

Regards

Mat

[Google Ads API Forum Advisor Prod]

Hi All - 

It appears as though this fix has been rolled out all the way to prod. Would you mind checking again whether you're continuing to receive these errors?

[Scott Vaillancourt]

Based on the queries I'm doing, this appears to be resolved. Thanks.

Scott

[kol...@exito.de]

Seems to work again without problems, we didn't have a single 500 error this morning. Thanks!

[Mat]

Hi Ben,

this morning we were able to pull about 2 million chance events across a 3 digit number of client customers without an error. So this problem seems to be fixed, ty.

However: we still did not receive a single change_event that was made through the Google Ads Editor. But that seems to be a different ongoing issue.

Regards

Mat

[Google Ads API Forum Advisor Prod]

Hi Mat - 

I'll respond on that other post with an update so that the topics stay consolidated there.

Closing this one out as fixed.