UNEXPECTED_INTERNAL_API_ERROR at ConversionTrackerService.get()

[Zweitze]

Hi all,

This morning we received an UNEXPECTED_INTERNAL_API_ERROR while calling ConversionTrackerService.get(). We retried the call three more times, but the error persisted.

What's going on?

The exception we received:
<detail>
  <ApiExceptionFault xmlns="https://adwords.google.com/api/adwords/cm/v201502">
    <message>[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.&lt;init&gt;(InternalApiErro]</message>
    <ApplicationException.Type>ApiException</ApplicationException.Type>
    <errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="InternalApiError">
      <fieldPath>
      </fieldPath>
      <trigger>
      </trigger>
      <errorString>InternalApiError.UNEXPECTED_INTERNAL_API_ERROR</errorString>
      <ApiError.Type>InternalApiError</ApiError.Type>
      <reason>UNEXPECTED_INTERNAL_API_ERROR</reason>
    </errors>
  </ApiExceptionFault>
</detail>

Response headers we received:

RequestId	00051e1ae5e0cfe00a0de108b6048a22
ResponseTime	11807

The call was made for account 182-109-9525.


Any idea what caused the error?
Thanks in advance!

[Josh Radcliff (AdWords API Team)]

Hi Zweitze,

Looking at our logs, this appears to be a transient error on our side. Did you retry the request some time after the initial set of retries, and if so, did it ever succeed?

Thanks,

Josh, AdWords API Team

[Zweitze]

Josh,

Indeed the software retried. After 30 seconds the call was tried again, which also failed, and then two more retries after waiting 30 seconds each time. So in total there were four attempts in 90 seconds. The response I sent was the response of the last attempt. (Unfortunately we don't record errors when a retry will come up.)

About that same time, say within five minutes, the same call was issued for other accounts - and succeeded.

This particular piece of software has no UI, but runs somewhere on a server. It cannot call help from an operator, it can only retry or abandon. Is our retry strategy the right one? Or would a slightly modfied retry-scheme give better protection against these glitches?

[Josh Radcliff (AdWords API Team)]

Hi Zweitze,

In your retry policy, you may want to implement an exponential back-off policy, so that each retry attempt waits longer than the previous one. I just tried your request again and it succeeded, which further suggests that this was a temporary issue.

Aside from the request ID you mentioned, have you seen this occurring on multiple occasions?

Thanks,

Josh, AdWords API Team

[Zweitze]

Hi Josh,

Thanks for your answer. The situation here is that we cannot afford to wait much longer. If you said that three minutes waiting may have produced better results we would consider that, but in this case we have not much to work on.
What I mean: we noticed that about 50% of this type of error disappear when retrying using our current retry-strategy (this figure is about all errors with retries, including HTTP 50x). If you would have suggested a retry-strategy with significant better results in a short time we would consider that.

BTW It would be great if this error came with a suggested waiting period, as some other errors do.

We experienced tons of UNEXPECTED_INTERNAL_API_ERROR errors, but this was the first time with ConversionTrackerService.get().

[Josh Radcliff (AdWords API Team)]

Hi Zweitze,

The underlying error here was basically a time out when trying to read the data, so unlike with rate limits, we (unfortunately) can't return a recommended wait period for retry in the exception.

You mentioned that your current retry approach fixes about 50% of these errors. What do you do with the remaining 50% for other services, and can you do something similar for the ConversionTracker.get case?

Thanks,

Josh, AdWords API Team

[Zweitze]

Josh,

What we do with the errors, where a retry doesn't help? We fail - we can't complete the task, so we log the problem. Using that log I ask what this error was about - because, sometimes it's about a new AdWords feature not correctly implemented in the API.

Zweitze.

[Josh Radcliff (AdWords API Team)]

Hi Zweitze,

I don't think this points to any issues specifically with ConversionTracker since the failed get requests succeeded later when I retried them, and according to our logs, the failures were due to timeouts trying to load the data, presumably because of high usage/load at that time.

In general, UNEXPECTED_INTERNAL_API_ERROR is usually transient, so if you find requests that are failing with this error even after retries, I'd recommend saving those in a queue for retry later (e.g., after waiting 5-10 minutes). I realize this still means a delay in executing these requests, but if a significant % of them succeed when retried later, that will at least save you the manual process of investigating failures that ultimately were due to temporary issues.

Thanks,

Josh, AdWords API Team