Batch jobs stay in state processing for several hours and finally get cancelled

[Christian]

For one of our AdWords account we have around 800,000 CPC change operations per day. We usually submit batch jobs with 25.000 operations per job. Most of these jobs stay in state processing for several hours and then eventually are reported to be cancelled. In few cases at least some operations of the batch jobs have been executed. The overall rate of operations we get executed varies per day and is something between 10% and 50%.

Some questions about this:

1) Do we reach the limits of what the batch job service is capable to do?

2) Do we have to lower the number of operations?

3) (feature request) Would it be possible to get a more informative error (like some rate exceeding error) instead of a cancelled job with no error details?

Here is an example batch job ID: 563857775

Thanks in advance for some feedback on this issue.

Best regards,

Christian

[Shwetha Vastrad (AdWords API Team)]

Hi Christian, 

BatchJobService automatically retries operations that fail due to transient errors such as RateExceededError. If the submitted operations encounter an error during execution, the results in the downloadUrl should contain the list of processingErrors on the batch job. I'll get in touch with the team to check why the BatchJobs are getting cancelled. Could you retry the BatchJob request with fewer number of operations? 

Thanks,
Shwetha, AdWords API Team.

[Christian]

Hi Shwetha,

> If the submitted operations encounter an error during execution, the results in the downloadUrl should contain the list of processingErrors on the batch job

The problem here is that most of the cancelled jobs have no error information -- neither in the job status we get when polling the jobs nor in the result file pointed to by the download URL. If some of the operations executed, we see them in the result file but not executed operations are simply not referenced in the result file.

> Could you retry the BatchJob request with fewer number of operations? 

For now we tried to reduce the total number of operations. I'll check if this improves the situation. After that we will have a try with smaller batch jobs.

Best,

Christian

[Nadine Sundquist (AdWords API Team)]

Hello Christian,

I've taken a look into what's happening, and initially I don't see a root cause. My team will dig further into it, and I'll get back to you when I have the reason as to why these jobs are not successful.

Best,

Nadine, AdWords API Team

[Christian]

Hi Nadine,

thanks for looking into that.

Today we reduced the total number of set-max-cpc operations to be below 100 000 and all 4 jobs (each at most 25 000 operations) completed successfully within a few minutes. Is there already a fix live? Let me know if we should have a try again with a larger number of operations.

In case it is helpful for debugging, here are the IDs of the jobs successfully completed today: 565656811, 565639265, 565650926, 565663576.

By the way: we also submit more than 1 000 000 bid modifier operations each day. The corresponding jobs do not get cancelled but still require several hours to complete. Do we have to limit our daily number of operations here or is that something the batch job service should be able to handle faster in theory? An example job ID would be 565684901.

Best regards,

Christian

[Nadine Sundquist (AdWords API Team)]

Hello Christian,

Thanks for the update. I don't know of any code fixes that went in within the last day.

Batch job service can theoretically handle quite a few operations, but even 1 000 000 is a lot when you think of all the updates that have to happen on the back end. Some changes are more expensive than others and take a bit longer because even though it looks like you're updating maybe one field, there is logic that needs to check on things like if something can be overridden at specific levels or if other objects in the account will be affected by the change. That's why some things do take a bit longer.

Regards,

Nadine, AdWords API Team 

[Nadine Sundquist (AdWords API Team)]

Greetings Christian!

Here's an update on where we are on this. It turns out that the batch job service attempts to send all the requests, but is internally getting back a REQUEST_SIZE_LIMIT_EXCEEDED error. That happens when there are more than 2000 operations in a normal service. What batch job service is doing in the background is that it's breaking up the request when that happens and then sends fewer requests. This is happening so often that the batch job finally cancels itself. That does appear to be an issue on our side because of this. We're exploring solutions, but there isn't a solution, yet. I'll update you when we have more information. At least we now know the root cause of your issue.

Best,

Nadine, AdWords API Team

[Christian]

Hi Nadine,

thanks for the update and for looking into this. For now we'll stay with our limit of not more than 100 000 set-bid operations per day as this seems to be a working scale. Still, I really appreciate it when you report some news here once you have them. Thanks again!

Best,

Christian

[Christian]

Hi Nadine,

just want to give an update from our side. We incrementally increased our set-bid-operations-per-day limit again up to 400 000. At the same time we reduced our batch jobs to have at most 10 000 operations (25 000) before. So far everything looks good. Jobs usually complete in less than an hour for large accounts, which is good enough for us.

Best,

Christian

[Nadine Sundquist (AdWords API Team)]

Hello Christian,

I'm pleased to hear that. Thanks for providing an update!

Cheers,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hi Christian,

You've been so helpful in providing updates; now I get to provide the update. Yay! I just got notice from the engineer on my team working on this that the issue should be resolved now. Please get back to me if you have any further issues with this.

Thanks,

Nadine, AdWords API Team