batch-jobs taking too long or hanging indefinetely

[domenico....@kaufda.de]

Hello everyone,
it is since last night we are experiencing many batch-jobs running very slow (in some cases completing after 30 minutes) in other cases hanging indefinitely.
The problem is occurring also now.

Can this problem be fixed as soon as possible?

Thanks and regards

[Stefano Fontana]

Hello, 

I'm experiencing the same so after a while I cancel the job and retry which is working "fine" for now.

(Tip: just wait for the job's status to be "CANCELING" and not "CANCELED" because that also takes ages!).

Of course this is just a stopgap solution...waiting a couple of minutes for 2 operations to complete is very inefficient.

[domenico....@kaufda.de]

Hi Stefano, thanks for the tips.
At the moment we are doing the same by manually restarting the jobs.
In general the problem is that we have many jobs running at any time in the 24h (some of them are critical) so we cannot manually restart them all every time.

Hope the problem will be fixed soon on the AdWords platform side.

[Stefano Fontana]

You could implement some logic that cancels the jobs and resubmit it if 0 operations are executed after X seconds.

But yeah, hope this is solved asap by Google.

[Hunter]

We are also experiencing this issue as of this morning. Our BatchServiceJob jobs stay at 0% completed, and then time out.

Thanks,
Hunter.

[Michael Cloonan (AdWords API Team)]

Hello,

Could everyone experiencing issues with slowness in their batch jobs please provide me an example batch job that is having issues?

Please provide both the customer ID and the batch job ID for the job.

Regards,

Mike, AdWords API Team

[domenico....@kaufda.de]

Sent privately the required data

Regards

[Michael Cloonan (AdWords API Team)]

Hello,

A few people responded with information, which I have forwarded to our engineering team to investigate more thoroughly. I will let you know when I hear something back.

Regards,

Mike, AdWords API Team

[Stefano Fontana]

FWIW today I'm not getting any slowjobs.

[Michael Cloonan (AdWords API Team)]

Hello,

Engineering has found the root cause of this issue. Upon further investigation, it seems that in all these cases the job executes quite quickly once it begins, but sits in the queue for quite some time before it is able to start. We are working on improving the time it takes to start an individual job when there may be in-progress extremely large jobs already being executed from other users. Please be aware that we will not be able to get this time down to guarantee an immediate start for your job, but these reports have helped us identify a case that needs improvement.

Regards,

Mike, AdWords API Team

[domenico....@kaufda.de]

Thanks for taking care and feedbacks

Regards

[Stefano Fontana]

Hello Michael,

Is there an update about this?

Best,

   Stefano-

[Michael Cloonan (AdWords API Team)]

Hello,

We are continually working to improve the performance of the BatchJobService. Reports like these really help narrow down specific problems, but since our progress on improvement is incremental, we aren't able to update here every time something changes. Know that we take these reports seriously. Some improvements have already been made because of this thread, and some more are still in the works. I will not be reporting each individual one, but please keep reports coming if you are still running into this issue especially frequently or run into any other performance issues with this service.

Regards,

Mike, AdWords API Team

[Stefano Fontana]

Hello,

this morning some BatchJobs are extremely slow.

Usually cancelling and resubmitting the job did the trick but now even that doesn't work.

One example is the BatchJob with Id "326707007" it only contains 1 operation: 

{u'operator': u'ADD', u'operand': {u'status': u'ENABLED', u'name': u'12345', u'campaignId': 629535826, u'biddingStrategyConfiguration': {u'bids': [{u'bid': {u'microAmount': '2560000'}, u'xsi_type': u'CpcBid'}

Halp :(

[Dorian Kind]

Just here to confirm that—since this morning the BatchJob service doesn't show any progress for any submitted jobs. As an example, I currently have a job with ~40'000 operations that has been at 0% progress for the last hour. Would job IDs help to diagnose this issue?

[Michael Cloonan (AdWords API Team)]

Hello,

I will pass this information along to engineering. A batch job ID would help immensely.

I will post here when I have anything to report about this.

Regards,

Mike, AdWords API Team

[Brian Bachtel]

Having the same issue. Created a new job with only 2 operations and it is hanging in the ACTIVE state along with the rest of my jobs.

On Wednesday, July 20, 2016 at 8:06:19 AM UTC-4, Dorian Kind wrote:

[Brian Bachtel]

A batch job ID, 326746595

[Michael Cloonan (AdWords API Team)]

Hello,

Engineering is currently investigating the issue and I will update here as soon as I have more to share. Thanks for the reports, everyone.

Regards,

Mike, AdWords API Team

[an...@clicksco.com]

Having the same issue here too

[Dorian Kind]

Another batch job ID: 326763413. It contains only 8 operations and has been ACTIVE at 0% for around half an hour now.

[Michael Cloonan (AdWords API Team)]

Hello,

Engineering has found the root cause of the issue and jobs are being processed again now. Please note that there is a very large backlog of jobs from the duration of the issue, so it may take some time for your specific jobs to be handled. Engineering and I will both continue to monitor the situation and I will update this thread as necessary.

Regards,

Mike, AdWords API Team

[Stefano Fontana]

Hello Michael,

Thank you for solving the problem.

Would it be possible to have some technical insights about the problem? Would love to know what's happening behind the scenes :)

Best,

   S-

[dlisi...@senetic.com]

I have the same issue. My jobs has status ACTIVE with 0 percent from 30 minutes.
JobIDs : 327017976, 327090161

Yesterday, none of my job was done.

[gdoe...@intelliad.de]

We were also seeing lots of batch jobs running indefinitely. Today morning, the situation seems a little better, but we are still having problems with jobs running without progress.

Some of the jobs running longer than 20 minutes with 2,000 elements each:

327058106

327004624

327016504

327007494

327087581

327015171

327015105

... and many more. Looking forward to this sitation beeing solved. If I can help you with more information, please tell me.

Best regards,

Georg

[dlisi...@senetic.com]

I send some new jobs to queue.
All have status ACTIVE and percent 0.

IDs:

327104183,
327103697,
327103541,
327102302,
327102209,
327101831,
327101810,
327049528,
327049516,
327049510,
327049492,
327049489,
327049471,
327049159,
327048802,
327048790,
327048760,
327048298,
327048274,
327048136,
327048133,
327048127,
327043972,
327032010,
327031932,
327031902,
327031005,
327030690,
327030687,
327030684,
327029844,
327029832,
327029829,
327029817,
327028209

W dniu poniedziałek, 2 maja 2016 15:32:47 UTC+2 użytkownik domenico....@kaufda.de napisał:

[Dorian Kind]

Michael, seeing as we have jobs that have been at 0% for hours on end, would you recommend keeping them waiting or cancelling them and re-submitting?

[Biniam Asnake]

We are still unable to process jobs. This is very crucial for our business. Please help!!!

FYI: MutateJobService is still working although it is slow. Please do not discontinue MutateJobService unless BatchJobService is working properly!

On 27 Aug, MutateJobService will sunset which is a very bad news for our business.

Regards,

Biniam

[Michael Cloonan (AdWords API Team)]

Hello,

The same problem from yesterday has recurred this morning. Yesterday, we did eventually work through all the jobs in the queue, and the same will happen today while engineering works to determine the root cause.

If you have a job stuck at 0%, do not cancel it and retry it. Please instead just wait for it to complete, which could take multiple hours today.

The engineering team is very hard at work on this issue. They do know how critical this service is and getting it back up and running is their top priority until it's completely back to normal.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello everyone,

Thanks for your patience during this issue. All backed up batch jobs have been processed and we are back to operating normally.

I apologize for the inconvenience. If anyone else starts running into this issue again, please let me know.

Thanks,

-Mike, AdWords API Team