DeadlineExceed Error due to Authorization issue

[Vijay Kumar]

Hi,

We are observing the GRPC::DeadlineExceeded error while querying the google ads API via the ruby client. 

- Full error log:

```

W, [2022-06-13T15:38:53.292380 #1568]  WARN -- : CID: 9928077892, Host: googleads.googleapis.com:443, Method: /google.ads.googleads.v9.services.GoogleAdsService/Search, IsFault: yes, Request ID:
I, [2022-06-13T15:38:53.292674 #1568]  INFO -- : Outgoing request: Headers: {"developer-token":"REDACTED","login-customer-id":"3661020287","x-goog-api-client":"gl-ruby/2.7.5 gccl/15.0.0 gax/0.6.0 gapic/15.0.0 grpc/1.42.0","x-goog-request-params":"customer_id=9928077892"} Payload: {"customerId":"9928077892","query":"SELECT metrics.cost_micros FROM campaign WHERE segments.date BETWEEN '2022-06-01' AND '2022-06-13' AND metrics.impressions > 0"}
I, [2022-06-13T15:38:53.292834 #1568]  INFO -- : Incoming response (errors):
  GRPC::Unavailable(14:Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}>. debug_error_string:{"created":"@1655134733.291585418","description":"Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:\n{\n  "error": "invalid_grant",\n  "error_description": "Bad Request"\n}>","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":91,"grpc_status":14}):
  called from: /home/deploy/.rvm/gems/ruby-2.7.5/gems/activesupport-6.0.3.2/lib/active_support/logger_thread_safe_level.rb:53:in `add'
W, [2022-06-13T15:38:58.328831 #1568]  WARN -- : CID: 9928077892, Host: googleads.googleapis.com:443, Method: /google.ads.googleads.v9.services.GoogleAdsService/Search, IsFault: yes, Request ID:
I, [2022-06-13T15:38:58.329032 #1568]  INFO -- : Outgoing request: Headers: {"developer-token":"REDACTED","login-customer-id":"3661020287","x-goog-api-client":"gl-ruby/2.7.5 gccl/15.0.0 gax/0.6.0 gapic/15.0.0 grpc/1.42.0","x-goog-request-params":"customer_id=9928077892"} Payload: {"customerId":"9928077892","query":"SELECT metrics.cost_micros FROM campaign WHERE segments.date BETWEEN '2022-06-01' AND '2022-06-13' AND metrics.impressions > 0"}
I, [2022-06-13T15:38:58.329134 #1568]  INFO -- : Incoming response (errors):
  GRPC::Unavailable(14:Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:
{
  "error": "invalid_grant",
  "error_description": "Bad Request"
}>. debug_error_string:{"created":"@1655134738.328316778","description":"Getting metadata from plugin failed with error: #<Signet::AuthorizationError: Authorization failed.  Server message:\n{\n  "error": "invalid_grant",\n  "error_description": "Bad Request"\n}>","file":"src/core/lib/security/credentials/plugin/plugin_credentials.cc","file_line":91,"grpc_status":14}):
  called from: /home/deploy/.rvm/gems/ruby-2.7.5/gems/activesupport-6.0.3.2/lib/active_support/logger_thread_safe_level.rb:53:in `add'
W, [2022-06-13T15:39:04.836459 #1568]  WARN -- : CID: 9928077892, Host: googleads.googleapis.com:443, Method: /google.ads.googleads.v9.services.GoogleAdsService/Search, IsFault: yes, Request ID:
I, [2022-06-13T15:39:04.836762 #1568]  INFO -- : Outgoing request: Headers: {"developer-token":"REDACTED","login-customer-id":"3661020287","x-goog-api-client":"gl-ruby/2.7.5 gccl/15.0.0 gax/0.6.0 gapic/15.0.0 grpc/1.42.0","x-goog-request-params":"customer_id=9928077892"} Payload: {"customerId":"9928077892","query":"SELECT metrics.cost_micros FROM campaign WHERE segments.date BETWEEN '2022-06-01' AND '2022-06-13' AND metrics.impressions > 0"}
I, [2022-06-13T15:39:04.836906 #1568]  INFO -- : Incoming response (errors):
  GRPC::DeadlineExceeded(4:Deadline Exceeded. debug_error_string:{"created":"@1655134744.835871906","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4}):
  called from: /home/deploy/.rvm/gems/ruby-2.7.5/gems/activesupport-6.0.3.2/lib/active_support/logger_thread_safe_level.rb:53:in `add'
Traceback (most recent call last):
/home/deploy/.rvm/gems/ruby-2.7.5/gems/grpc-1.42.0-x86_64-linux/src/ruby/lib/grpc/generic/active_call.rb:29:in `check_status': 4:Deadline Exceeded. debug_error_string:{"created":"@1655134744.835871906","description":"Deadline Exceeded","file":"src/core/ext/filters/deadline/deadline_filter.cc","file_line":81,"grpc_status":4} (GRPC::DeadlineExceeded)

```

- Configuration

```

STREAM_TIMEOUT_SECONDS = 10

@client = Google::Ads::GoogleAds::GoogleAdsClient.new do |config|
config.refresh_token = refresh_token
config.client_id = ADWORDS_CONFIG['CLIENT_ID']
config.client_secret = ADWORDS_CONFIG['CLIENT_SECRET']
config.developer_token = ADWORDS_CONFIG['DEVELOPER_TOKEN']
config.logger = Rails.logger
config.login_customer_id = login_customer_id
end

@google_ads_service = @client.service.google_ads
@google_ads_service.configure do |config|
config.rpcs.search.timeout =
config.rpcs.search_stream.timeout =
STREAM_TIMEOUT_SECONDS
end

```

It seems, the client lib internally handles the OAuth token update: https://developers.google.com/google-ads/api/docs/first-call/refresh-token?authuser=4, which fails, and results in Signet::AuthorizationError. This somehow gets wrapped in a GRPC::Unavailable error. And, in case of a GRPC::Unavailable error the client lib keeps on retrying until the deadline exceeds: https://github.com/googleapis/gapic-generator-ruby/blob/69bad632b8275478d97dd8f7e53571a63676714a/gapic-common/lib/gapic/grpc/service_stub/rpc_call.rb#L125. That seems to be a bug to wrap this in the GRPC::Unavailable error, because by definition the GRPC::Unavailable error signifies a transient condition that may be corrected with a retry but that doesn't hold for the Authorization errors (at least when that fails post refresh token).

Can you confirm the situation?

Best Regards,

Vijay Kumar

[Google Ads API Forum Advisor]

Hi Vijay,

Thanks for reaching out. I see that you are also getting an invalid_grant error. Are you following the instructions given here?

Can you please try regenerating your OAuth credentials and refresh token to see if this resolves the issue? 

Thanks,

Matt Google Ads API Team

ref:_00D1U1174p._5004Q2bvrfZ:ref

[Vijay Kumar]

Can you explain why the Ruby client library is retrying in case of an invalid grant error? That's the problem. That leads to exceeding the deadline limit. Rather, the client library should raise the AuthorizationError.

[Google Ads API Forum Advisor]

Hi Vijay,
 

Thanks for your response. I am Darwin also, a member of the Google Ads API team and let me provide support to your concern.
 

An invalid_grant error usually indicates an issue with your refresh token. That said, I would recommend that you regenerate the said refresh token. Since you are using RUBY, you can refer to this guide on how to set up your authentication credentials.

Let me know how this goes, and should the issue persist, you may enable logging by referring to this RUBY guide, and provide to our team the complete request and response logs, with the request-id.
 

You may then send the requested information via the Reply privately to author option. If this option is not available, you may send the details directly to our googleadsa...@google.com alias instead.

Regards,

Darwin Google Ads API Team

ref:_00D1U1174p._5004Q2bvrfZ:ref

[Vijay Kumar]

Hi Darwin,

Thanks for your response. We already know that the account has an authorization issue, and needs to be reauthorized - that's not the problem. The problem is the behavior of the ruby client lib. The client library is infinitely retrying until the deadline exceeds in case of 'invalid_grant'. Theoretically 'invalid_grant' is not even a retriable error.

Regards,

Vijay Kumar

[Google Ads API Forum Advisor]

Hi Vijay,

This issue seems to be specific to the Ruby client library. As our team provides general Ads API support, I would recommend reaching out to the Ruby client library owners. 

Regards,

Matt Google Ads API Team

ref:_00D1U1174p._5004Q2bvrfZ:ref

[Vijay Kumar]

Thanks for the support.

Side note: This is a known bug in the ruby client library, posting the issue link here for the public reference: https://github.com/googleads/google-ads-ruby/issues/246 which is happening due to an the underlying issue in the google-auth lib https://github.com/googleapis/google-auth-library-ruby/issues/366.

[Google Ads API Forum Advisor]

Hi Vijay,

Thanks for posting the client library forum link for reference.