Batch Jobs stuck on RUNNING state on GoogleAds API

[Miquel Isern Roca]

Hello,

we have noticed that since yesterday, some of the Batch Jobs we pushed through the GoogleAds API are not finishing.

We are seeing an increased amount of BatchJobs in this status that have been on the RUNNING status for over 8 hours. And we don't know how to react and deal with them. Because they don't fail nor finish, we can't fetch the results and process it, so we don't know if operations have been executed or not.

We can't think of any changes on our side that could affect the processing of these operations. And this problem seems to have just started yesterday. Is there any known issue with BatchJob processing on Google's side?

I can provide data examples on a private e-mail if needed.

Thanks

[Skander Mansouri]

Hello,

We are having the same issue since yesterday but especially for a large chunk of operations. 
If we launch with less than 2000 operations in a batch, it works properly.

This is blocking our operations, can you confirm that it is a bug from your side?

Thanks,
Skander

[Ciel Eijssen]

Hi Google Team,

We currently have over 1300 batch jobs for multiple client customer ids with this issue.

They do not leave the RUNNING state so we do not know whether we can resubmit a new batch job, without it leading to duplicate objects in their Google Ads accounts.

If you can confirm that they will not be processed, that would be great so we can ignore the batch jobs on our end :-) 

Let me know if you would like to have the batch job ids so I can provide these to you.

Thanks in advance.

Best,

Ciel

[Skander Mansouri]

Hey again,

To add on this issue, currently even with 400 operation, the batch job service is having issues.

This is an example of a response I'm currently receiving:

Request ------- MethodName: google.longrunning.Operations/GetOperation Endpoint: googleads.googleapis.com:443 

Headers: {developer-token=REDACTED, login-customer-id=XXXXX, x-goog-api-client=gl-java/11.0.14.1 gccl/18.0.0 gapic/18.0.0 gax/2.6.1 grpc/1.42.1} 

Body: name: "customers/XXX/operations/XXX"

 Response -------- Headers: Metadata(content-type=application/grpc,request-id=XXXX,date=Thu, 05 May 2022 13:56:17 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43")

 Body: null 

Failure message: null

Status: Status{code=INTERNAL, description=Internal error encountered., cause=null}

[Google Ads API Forum Advisor]

Hello Everyone,

Thanks for reaching out to the Google Ads API Forum.

I can see that you are all having issues with BatchJobs requests. However, could you please provide us with the complete request and response logs with the request-id generated for the BatchJobs requests where we can see said observed behavior via the Reply privately to author option, so I can report this to my rest of the team? 

For the client library, logging can be enabled by navigating to the Client libraries > Your client library (ex Java) > Logging documentation, which you can access from this link. 

Regards,

Yasar Google Ads API Team

ref:_00D1U1174p._5004Q2aQjmG:ref

[Skander Mansouri]

Hi,

I sent the request/response privately, I'm not sure if it was well received?

do you have an update on this issue as it's blocking our business operations!

[Jaume Florez]

Hi,

We've replied with some logs as well.

Also, we are over 2000 BatchJobs for different customers that are in this state right now...

We have seen that one of the affected BatchJobs (with 7 CREATE AdGroupAd operations!) finished after 48 hours.

So it seems to be that it's just very slowly being processed, but waiting for 48hours doesn't seem like a good approach.

Hope to hear from you,

Thanks.

[Christian]

Hi,

We are facing the same issue since May 5. Today (May 8), we see batch jobs in state RUNNING since May 6:

SELECT batch_job.id, batch_job.status, batch_job.metadata.creation_date_time, batch_job.metadata.estimated_completion_ratio FROM batch_job WHERE batch_job.status != DONE ORDER BY batch_job.metadata.creation_date_time LIMIT 4

batch_job {
  resource_name: "customers/50.....307/batchJobs/9016872863"
  metadata {
    creation_date_time: "2022-05-06 06:23:50"
    estimated_completion_ratio: 0.0
  }
  status: RUNNING
  id: 9016872863
}

batch_job {
  resource_name: "customers/50.....307/batchJobs/9016708636"
  metadata {
    creation_date_time: "2022-05-06 06:23:51"
    estimated_completion_ratio: 0.0
  }
  status: RUNNING
  id: 9016708636
}

batch_job {
  resource_name: "customers/50.....307/batchJobs/9020761205"
  metadata {
    creation_date_time: "2022-05-06 12:33:30"
    estimated_completion_ratio: 0.0
  }
  status: RUNNING
  id: 9020761205
}

batch_job {
  resource_name: "customers/50.....307/batchJobs/9030551684"
  metadata {
    creation_date_time: "2022-05-07 12:14:41"
    estimated_completion_ratio: 0.0
  }
  status: RUNNING
  id: 9030551684
}

Is the job Id sufficient to investigate this on your side or should I send the customer ID in private as well?

Thanks for your support.

Best,

Christian

[Ciel Eijssen]

Hi,

Over the weekend the problem has only gotten worse and we now have over 2000 batch jobs stuck in the RUNNING state since at least yesterday morning, May 8th.

I already supplied you a list of the affected batch job ids and customer ids from before the weekend, but I can give you an updated list if needed.

Appreciate your time.

Best,

Ciel

[Jaume Florez]

Hi Google,

Expanding a bit more on this issue, we actually have some customers that are "stuck" with 100 running BatchJobs (affecting different campaigns).
This means that we can't do anything for them anymore unless we use the API directly, which, as mentioned by our google partner, is not a good option with our use cases.

Hope this can be resolved soon.

Jaume

[Skander Mansouri]

is there any updates on this?

[Matteo Medda]

Hi,
the situation is becoming more and more difficult to manage.

Do you have any news on this problem?

Regards,

Matteo

[Google Ads API Forum Advisor]

Hi Matteo,

Thanks for your patience on this. This issue should be resolved now. Can you please retry your batch job request and let us know if there are any reoccurring issues?

Thanks,
Matt
Google Ads API Team

Matt Google Ads API Team

ref:_00D1U1174p._5004Q2aQjmG:ref

[Silvia]

Hi,

the issue does not seem to have been solved for us.

Do you have any news?

Thanks

Silvia

[Ciel Eijssen]

Hi,

We also still have batch jobs which have been running since yesterday. Also newly submitted batch jobs are slower than usual.

Best,

[Matteo Medda]

Hi Team,

on our side, we don't see any improvements at the moment. 

Batch jobs are still very slow,

Regards,

Matteo

[Prototype.co]

Hi Team,

We are also experiencing the same issue, Batch Jobs seem to be running forever, and the operations do not seem to be running in any way.

[Christian]

Hi,

for us it looks like things got better today. Overall batch jobs got processed in the usual time.

Thanks for the support and resolution.

Best,

Christian

[Prototype.co]

Unfortunately, we experienced no difference on our part. Batch Jobs are still "running forever", and if we wait long enough, we end up with a "DeadlineExceeded" error.

Do note that in some cases, the Batch Jobs run fine (for example, updating and creating Asset-based extensions work just fine), however when trying to update keywords, we are still stuck in the RUNNING phase.

[Silvia]

Hi,

for us it looks like things got better today, but the overall time for each process is much higher than in the past (20-30 minutes compared to the maximum 1-2 minutes previously).
Do you think that batch processing will return to times closer to those of the past?

Best,
Silvia

[Google Ads API Forum Advisor]

Hello,

Thanks for your patience on this. Please allow me to share your experience with our team and I will provide an update shortly.

[Skander Mansouri]

Is the issue still persisting on your side everyone? 

[Silvia Oggiano]

Yes, we have the same issue on the long time of processing

[Prototype.co]

Yes, we are still experiencing this issue. Note that some Batch Jobs seem to run just fine (in our case, those that create a small number of ad extensions and link them to campaigns or ad groups - a batch job like this typically has 10-20 operations in total). However, Batch Jobs that update keywords seem to be stuck in 'RUNNING' state forever. In this case, operations count is hundreds to thousands of operations - the most recent, smallest example being 967 operations, that is 967 keyword updates.

[Google Ads API Forum Advisor]

Greetings All,

Thank you for chiming in on this. It looks like we have multiple reasons at this time as to why a batch job could be either stuck running or slower than expected.

First, there were general issues with Batch Jobs that were resolved on May 11th. We had a bit of a backlog, and we burned through that backlog. So, that did solve the problem for many of you.

For other developers on the 'stuck running' state continuing, I've checked in with the folks looking into this, and it appears there is a common theme amongst many of the jobs. It appears that at times the batch job will retry when an error will permanently result in failure. So, the job runs for much longer trying to retry failures that it should give up on right away. 

So, here's what I need from you folks to figure out what's happening in your individual cases. If you could reply privately to author on the forum (or send an email to googleadsa...@google.com if replying privately to me does not work) with less than 5 samples of customer IDs and batch jobs IDs (so I don't get too many at one time) where you see the constant running or significant slowdown with a note saying whether or not it's a slowdown or stuck running, that would be a huge help. I believe based on this sample of descriptions here that there may be slightly different issues happening, and I want to make sure to address all of them.

Thanks,

Nadine Wang Google Ads API Team

ref:_00D1U1174p._5004Q2aQjmG:ref

[Prototype.co]

I've sent an e-mail with some customer and batch job ids.

adsapi a következőt írta (2022. május 18., szerda, 19:46:25 UTC+2):

Greetings All,

Thank you for chiming in on this. It looks like we have multiple reasons at this time as to why a batch job could be either stuck running or slower than expected.

First, there were general issues with Batch Jobs that were resolved on May 11th. We had a bit of a backlog, and we burned through that backlog. So, that did solve the problem for many of you.

For other developers on the 'stuck running' state continuing, I've checked in with the folks looking into this, and it appears there is a common theme amongst many of the jobs. It appears that at times the batch job will retry when an error will permanently result in failure. So, the job runs for much longer trying to retry failures that it should give up on right away. 

So, here's what I need from you folks to figure out what's happening in your individual cases. If you could reply privately to author on the forum (or send an email to googleadsapi-support@google.com if replying privately to me does not work) with less than 5 samples of customer IDs and batch jobs IDs (so I don't get too many at one time) where you see the constant running or significant slowdown with a note saying whether or not it's a slowdown or stuck running, that would be a huge help. I believe based on this sample of descriptions here that there may be slightly different issues happening, and I want to make sure to address all of them.

[Skander Mansouri]

did the same! 

Status stuck on Pending and we are getting kicked out of the /GetOperation with a Task was cancelled error. not sure if someone is facing the same issue here?

Le jeudi 19 mai 2022 à 16:28:17 UTC+2, Prototype.co a écrit :

I've sent an e-mail with some customer and batch job ids.

adsapi a következőt írta (2022. május 18., szerda, 19:46:25 UTC+2):

Greetings All,

Thank you for chiming in on this. It looks like we have multiple reasons at this time as to why a batch job could be either stuck running or slower than expected.

First, there were general issues with Batch Jobs that were resolved on May 11th. We had a bit of a backlog, and we burned through that backlog. So, that did solve the problem for many of you.

For other developers on the 'stuck running' state continuing, I've checked in with the folks looking into this, and it appears there is a common theme amongst many of the jobs. It appears that at times the batch job will retry when an error will permanently result in failure. So, the job runs for much longer trying to retry failures that it should give up on right away. 

So, here's what I need from you folks to figure out what's happening in your individual cases. If you could reply privately to author on the forum (or send an email to googleadsa...@google.com if replying privately to me does not work) with less than 5 samples of customer IDs and batch jobs IDs (so I don't get too many at one time) where you see the constant running or significant slowdown with a note saying whether or not it's a slowdown or stuck running, that would be a huge help. I believe based on this sample of descriptions here that there may be slightly different issues happening, and I want to make sure to address all of them.

[Google Ads API Forum Advisor]

Hello All,

Thank you for providing all the additional details. I'm currently working through each of the messages sent to determine what specifically is causing the issue for your individual batch jobs. I'll reply privately to each of you.

Regards,

[bigapp...@gmail.com]

Just chipping in here - we're seeing the same issue and are working with the support team. A couple of notes on what we're seeing:

- It only appears to affect AdGroupCriterion operations. In our case, we're creating product groups in Shopping campaigns. Where we submit operations to create ad groups , set bid modifiers or create ads, the batches complete efficiently.

- We found that batches with more than 1k operations would all be slow to complete with all operations after the thousandth failing with "Duplicate Element" errors. Batches of 1k were working yesterday, but from late last night, small batches including batches of 1k, 500 and 50 do not seem to be completing and remain in RUNNING status

[Google Ads API Forum Advisor]

Hello Geoff,

Thanks for sharing that information here for those who are affected by this particular case. I can confirm there is an issue that we're working through with product groups. Thanks for mentioning that you're already working with one of my teammates on this so we don't duplicate the effort.

What I've learned by privately chatting with folks on this forum thread is that we have a few themes in play here:

• The issues with product groups that Geoff mentioned. I have a teammate who's taken point on sorting that one out.
• A few timeouts on our back-end that we're working through.
• If an operation fails, the batch jobs retry it; that's what a batch job should do. However, there are some types of errors we should not retry as the result could be the job running for a very long time. We're working on solving that one as well.

I'll continue to chat with all of you privately so we can get to a resolution on each of your specific issues.

Thanks,

[Silvia Oggiano]

I replied with some customerIDs and batchJobIDs with slow running.

Thanks!

[Google Ads API Forum Advisor]

Hello Silvia,

Thank you for the information. I sent you a private email that you should see that acknowledges I received it.

Regards,

[Google Ads API Forum Advisor]

Hi all,

I'm a colleague of Nadine's on the Google Ads API Team and I have some updates on the points she highlighted earlier.

1. The issues with product groups that Geoff mentioned. I have a teammate who's taken point on sorting that one out.
2. A few timeouts on our back-end that we're working through.
3. If an operation fails, the batch jobs retry it; that's what a batch job should do. However, there are some types of errors we should not retry as the result could be the job running for a very long time. We're working on solving that one as well.

We've rolled out various fixes for these on the afternoons of May 24 and 25, and have confirmed the batch job pipeline is looking much better, and we're investigating to see where we can make further improvements.

One significant point everyone should be aware of is that you will still see long running jobs if you are using any of the following operation types:

• AssetGroupListingGroupFilterOperation
• AssetGroupOperation
• CampaignConversionGoalOperation
• ConversionGoalCampaignConfigOperation
• CustomConversionGoalOperation
• CustomerConversionGoalOperation
• CustomerOperation

These operations do not support partial failure , and as such, are not supported in batch jobs and currently will lead to long running jobs. Please avoid including any of the above operation types in your jobs, and instead use GoogleAdsService.mutate with partial_failure set to false to execute those operations. I'm updating our documentation to reflect this limitation, and we are investigating validation checks to prevent adding these operations to a job.

In the meantime, if you are experiencing long running jobs that you started on or after May 25, please continue to share those with us by clicking Reply privately to author on this thread.

Many thanks,
Josh Radcliff, Google Ads API Team
 

ref:_00D1U1174p._5004Q2aQjmG:ref