Migrated to V8 : Started Seeing Increased Rate of DEADLINE_EXCEEDED

82 views
Skip to first unread message

Pete Lavetsky (AdWords API Guru)

unread,
Jun 22, 2021, 8:23:06 PM6/22/21
to AdWords API and Google Ads API Forum
We put V8 into production earlier today. Everything was fine for about 6.5 hours and then we started seeing an increased rate of DEADLINE_EXCEEDED 

Our GrpcCallContext is configured for a 5m timeout and that's been fine for months

I reverted back to V7 and we haven't seen a DEADLINE_EXCEEDED for the last 20 minutes but I'm not sure if that proves anything

Is it perhaps a transient issue on Google's end for maintenance work around increased capacity for V8?

Here's the traceroute ... we've been asked to provide this before when we witnessed a much higher rate of DEADLINE_EXCEEDED

traceroute to googleads.googleapis.com (172.217.9.74), 50 hops max, 60 byte packets

 1  ip-172-31-67-206.us-east-2.compute.internal (172.31.67.206)  0.104 ms  0.096 ms  0.102 ms

 2  ec2-52-15-0-37.us-east-2.compute.amazonaws.com (52.15.0.37)  8.985 ms ec2-52-15-0-57.us-east-2.compute.amazonaws.com (52.15.0.57)  8.931 ms ec2-52-15-0-43.us-east-2.compute.amazonaws.com (52.15.0.43)  5.446 ms

 3  100.65.27.16 (100.65.27.16)  3.778 ms 100.65.31.16 (100.65.31.16)  6.846 ms 100.65.28.0 (100.65.28.0)  3.651 ms

 4  100.66.12.132 (100.66.12.132)  1.994 ms 100.66.12.144 (100.66.12.144)  3.632 ms 100.66.13.36 (100.66.13.36)  1.978 ms

 5  100.66.14.196 (100.66.14.196)  16.689 ms 100.66.14.202 (100.66.14.202)  22.094 ms 100.66.14.130 (100.66.14.130)  12.970 ms

 6  100.66.6.77 (100.66.6.77)  21.618 ms 100.66.6.131 (100.66.6.131)  50.483 ms 100.66.7.235 (100.66.7.235)  17.491 ms

 7  100.66.4.101 (100.66.4.101)  32.121 ms 100.66.4.117 (100.66.4.117)  31.610 ms 100.66.4.125 (100.66.4.125)  42.423 ms

 8  100.65.10.129 (100.65.10.129)  1.140 ms 100.65.13.161 (100.65.13.161)  0.937 ms 100.65.8.161 (100.65.8.161)  0.945 ms

 9  15.230.134.209 (15.230.134.209)  2.346 ms 15.230.134.161 (15.230.134.161)  1.961 ms 15.230.134.203 (15.230.134.203)  2.729 ms

10  15.230.39.184 (15.230.39.184)  2.850 ms 15.230.39.10 (15.230.39.10)  2.794 ms 52.95.2.198 (52.95.2.198)  3.692 ms

11  52.95.2.205 (52.95.2.205)  2.088 ms 52.95.1.235 (52.95.1.235)  1.820 ms 52.93.239.50 (52.93.239.50)  1.313 ms

12  100.92.53.124 (100.92.53.124)  12.442 ms 100.92.53.76 (100.92.53.76)  14.483 ms 100.92.53.72 (100.92.53.72)  12.509 ms

13  100.92.43.34 (100.92.43.34)  12.711 ms 100.92.48.100 (100.92.48.100)  16.661 ms 100.92.48.10 (100.92.48.10)  13.001 ms

14  100.92.43.85 (100.92.43.85)  12.143 ms 100.92.48.139 (100.92.48.139)  12.547 ms 100.92.48.135 (100.92.48.135)  12.539 ms

15  100.92.49.66 (100.92.49.66)  12.591 ms 100.92.44.90 (100.92.44.90)  18.647 ms  18.368 ms

16  100.92.49.7 (100.92.49.7)  12.986 ms 100.92.44.9 (100.92.44.9)  12.990 ms 100.92.44.123 (100.92.44.123)  18.647 ms

17  52.93.132.62 (52.93.132.62)  12.399 ms 52.93.130.134 (52.93.130.134)  12.463 ms 52.93.133.56 (52.93.133.56)  12.744 ms

18  100.91.163.4 (100.91.163.4)  13.681 ms 100.91.167.8 (100.91.167.8)  18.976 ms 100.91.163.96 (100.91.163.96)  13.129 ms

19  100.91.163.89 (100.91.163.89)  12.558 ms 100.91.168.43 (100.91.168.43)  13.010 ms 100.91.167.17 (100.91.167.17)  13.632 ms

20  100.91.164.108 (100.91.164.108)  12.366 ms 100.91.159.64 (100.91.159.64)  17.061 ms 100.91.164.60 (100.91.164.60)  12.693 ms

21  100.91.159.73 (100.91.159.73)  17.008 ms 100.91.164.105 (100.91.164.105)  13.024 ms 100.91.165.129 (100.91.165.129)  13.239 ms

22  100.91.177.41 (100.91.177.41)  11.699 ms 100.91.177.55 (100.91.177.55)  11.945 ms 100.91.176.241 (100.91.176.241)  11.779 ms

23  100.100.6.21 (100.100.6.21)  12.162 ms 100.100.8.11 (100.100.8.11)  14.884 ms 100.100.8.117 (100.100.8.117)  12.359 ms

24  100.100.93.8 (100.100.93.8)  12.228 ms 100.100.77.198 (100.100.77.198)  12.200 ms 100.100.76.134 (100.100.76.134)  12.463 ms

25  100.100.93.195 (100.100.93.195)  12.901 ms 100.100.73.221 (100.100.73.221)  13.835 ms 100.100.85.221 (100.100.85.221)  12.385 ms

26  100.100.28.124 (100.100.28.124)  12.296 ms 100.100.34.92 (100.100.34.92)  12.247 ms 100.100.2.36 (100.100.2.36)  12.940 ms

27  72.14.203.158 (72.14.203.158)  12.939 ms 99.82.181.23 (99.82.181.23)  12.316 ms 72.14.203.158 (72.14.203.158)  12.906 ms

28  108.170.246.66 (108.170.246.66)  12.874 ms *  12.818 ms

29  108.170.246.33 (108.170.246.33)  13.149 ms 72.14.203.158 (72.14.203.158)  17.185 ms 99.82.181.23 (99.82.181.23)  15.395 ms

30  108.170.246.66 (108.170.246.66)  12.841 ms * 108.170.240.98 (108.170.240.98)  12.831 ms

31  * * *

32  * * *

33  * * *

34  * * *

35  * * *

36  * * *

37  * * *

38  * * *

39  * * *

40  * * *

41  * * *

42  * * *

43  * * *

44  * * *

45  * * *

46  * * *

47  * * *

48  * * *

49  * * *

50  * * *

Looking closer it looks like almost all of the DEADLINE_EXCEEDED were on 

google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets calls

Request

-------

MethodName: google.ads.googleads.v8.services.CampaignBudgetService/MutateCampaignBudgets

Endpoint: googleads.googleapis.com:443

Headers: {developer-token=REDACTED, login-customer-id=5741853629, x-goog-api-client=gl-java/1.8.0_201 gapic/ gax/1.65.1 grpc/1.37.0}

Body: customer_id: "REDACTED"

operations {

  update {

    resource_name: "customers/REDACTED/campaignBudgets/7001096614"

    status: ENABLED

    delivery_method: STANDARD

    id: 7001096614

    amount_micros: 10000

    explicitly_shared: false

  }

  update_mask {

    paths: "resource_name"

    paths: "id"

    paths: "amount_micros"

    paths: "status"

    paths: "delivery_method"

    paths: "explicitly_shared"

  }

}

partial_failure: true



Response

--------

Headers: null

Body: null

Failure message: null

Status: Status{code=DEADLINE_EXCEEDED, description=deadline exceeded after 3599.999166595s. [buffered_nanos=65828859, buffered_nanos=107502844, remote_addr=googleads.googleapis.com/142.250.190.74:443], cause=null}.

2021-06-22 23:49:42,443 INFO  RequestLogger - FAILURE REQUEST DETAIL.

There's no request-id generated / given on a DEADLINE_EXCEEDED so I can't provide one.

Pete

Pete Lavetsky (AdWords API Guru)

unread,
Jun 22, 2021, 9:49:41 PM6/22/21
to AdWords API and Google Ads API Forum
We're now seeing this on other calls as well. These are across multiple customerIds.

2021-06-23 01:19:33,532 WARN  RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdService/MutateAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: DEADLINE_EXCEEDED, Fault: deadline exceeded after 3599.998632475s. [buffered_nanos=67392207, buffered_nanos=93613556, remote_addr=googleads.googleapis.com/142.250.191.138:443].

2021-06-23 01:22:48,579 WARN  RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: DEADLINE_EXCEEDED, Fault: deadline exceeded after 3599.985958315s. [buffered_nanos=65378596, buffered_nanos=67500777, remote_addr=googleads.googleapis.com/172.217.8.202:443].

20212021-06-23 01:30:08,166 WARN  RequestLogger - FAILURE REQUEST SUMMARY. Method: google.ads.googleads.v7.services.AdGroupAdService/MutateAdGroupAds, Endpoint: googleads.googleapis.com:443, CustomerID: REDACTED, RequestID: null, ResponseCode: DEADLINE_EXCEEDED, Fault: deadline exceeded after 3599.998821696s. [buffered_nanos=66057372, buffered_nanos=77619715, remote_addr=googleads.googleapis.com/142.250.191.202:443].

Pete Lavetsky (AdWords API Guru)

unread,
Jun 22, 2021, 10:22:52 PM6/22/21
to AdWords API and Google Ads API Forum
Even on mutates that don't make it to our DEADLINE threshold we're seeing a massive slowdown in responsiveness in mutate calls.  Below, hour 20 corresponds to 4pm EST on the 22nd of June.

day hour avg(unix_timestamp(date_finished)-unix_timestamp(date_started))
22 12 10.4448
22 13 10.3481
22 14 9.4694
22 15 13.9076
22 16 13.1181
22 17 20.0910
22 18 16.3803
22 19 19.1051
22 20 38.2070
22 21 480.0347
22 22 61.2185
22 23 227.2379

Mat

unread,
Jun 23, 2021, 1:20:09 AM6/23/21
to AdWords API and Google Ads API Forum
Hi there,

we seem to be in the same boat:

Bildschirmfoto vom 2021-06-23 07-16-40.png


Regards
Mat

Pete Lavetsky (AdWords API Guru)

unread,
Jun 23, 2021, 5:46:52 AM6/23/21
to AdWords API and Google Ads API Forum
For us, this cleared up overnight:

day hour avg_job_time
23 0 31.2911
23 1 101.0072
23 2 246.6130
23 3 1827.2152
23 4 296.0434
23 5 7.2689
23 6 19.2890
23 7 6.0186
23 8 11.5061
23 9 13.0516

Google Ads API Forum Advisor

unread,
Jun 24, 2021, 3:56:40 PM6/24/21
to pete.l...@gmail.com, adwor...@googlegroups.com
Hi Pete and Mat,

Thank you for the logs and dashboard pictures. Based on Pete's last response it seems that the issue is transient and it cleared up.  If DEADLINE_EXCEEDED occurs again please get back to us.

Regards,

Google Logo
Aryeh Baker
Google Ads API Team
 


ref:_00D1U1174p._5004Q2IxwR1:ref
Reply all
Reply to author
Forward
0 new messages