Stalling batch jobs may be back?

[Adrian Ghizaru]

Hello!

We are experiencing problems similar to the stalling batch jobs from May - July of this year [1]. On our test account at Busbud (760-779-2587) we have 4 jobs stalled for over a week and 2 more since last night:

387400054 - CANCELING - 23% - 80000 ops executed

387460843 - ACTIVE - 0% - 0 ops executed

387571183 - CANCELING - 0% - 0 ops executed

387643878 - ACTIVE - 0% - 0 ops executed

393500269 - ACTIVE - 19% - 65000 ops executed

393620888 - ACTIVE - 2% - 5000 ops executed

I cancelled the first job after about 4 days of being stuck at 80K ops, and the second one after about 7 days because we started getting DISK_QUOTA_EXCEEDED errors.

Thank you for your help.

[1] https://groups.google.com/forum/#!topic/adwords-api/J6fDOsLQkpc/discussion

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for the report. I'm going to look into it and let you know what I find.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

These jobs actually ran into a very rare issue that caused the stall. My suggestion is to retry these specific operations, and it will likely not recur. You may be able to reduce the likeliness even more by splitting the job into a few smaller batches.

Engineering is planning a fix for this issue as well, although I don't have a timeframe for when it might be available. For now, you should take the steps I mentioned above if you run into more cases like this.

Regards,

Mike, AdWords API Team

[Adrian Ghizaru]

Hi Mike!

Thank you so much for looking into it! Unfortunately, we're unable to start any new jobs due to DISK_QUOTA_EXCEEDED errors. It doesn't seem like we can do anything about it on our end, whenever I try to cancel one of those jobs it stays stuck in a CANCELING state. Would you be able to remove them from your end please?

Thanks again for your assistance,

- Adrian.

[Michael Cloonan (AdWords API Team)]

Hello,

I apologize for the slow response. I will see what we can do about the currently running jobs to clear things up and let you run new ones.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

We have cleared out the current queue of stalled jobs for your account, and have generated download URLs for those jobs so you can fetch results for each one.

Please let me know if this issue recurs while we're working on fixing the underlying problem.

Regards,

Mike, AdWords API Team

[Adrian Ghizaru]

Thank you so much, we've been able to upload all batches... except the last one (job id 397817899). It seems to be stuck at 100,000 ops executed.

Could you please give us a hint as to what edge case we're tripping? Given that it's happened on 3 separate occasions, it's probably worth it for us to invest some effort in avoiding this case in our scripts.

Cheers, and apologies for pestering you!

- Adrian.

[Michael Cloonan (AdWords API Team)]

Hello,

I've passed along the ID you gave, and the team will check whether that's the same issue as the previous one you encountered.

I believe that you may be able to reduce risk of this occurring by splitting jobs into multiple smaller batches, but I am confirming with the team about whether or not that will actually help, and if there are any other actions you can take to minimize the chances.

Regards,

Mike, AdWords API Team

[Adrian Ghizaru]

Hi Mike!

I just checked and it seems the job completed at some point over the weekend. It was however stuck at 100,000 ops for at least 40 minutes (which is the longest we'll poll for), and most healthy jobs of the same size don't seem to do this.

So we're good on our end for now, but it's not the last time we're doing this and we'd love guidance on avoiding this problem. Splitting the job is feasible but would require considerable refactoring, so we'd do it only if it's a very good chance it'll fix the problem.

Thanks you so much for your assistance!

- Adrian.

[Džiugas Petkevičius]

I think we experiencing same problem. We have 25 batch jobs with status canceling stalled for more than 20 hours. All new batch jobs are stalled too. But at least new ones can be canceled.

Stalled batch job ids: 504630851, 504630905, 504630932, 504630965, 504630983, 504631139, 504631166, 504631226, 504653183, 504654437, 504784504, 504784513, 504784528, 504961074, 504961098, 504961116, 504961122, 504961242, 504961272, 504961275, 504961299, 504961332, 504982419, 504982437, 504982470

It would be perfect if someone from AdWords could remove them.  

[Michael Cloonan (AdWords API Team)]

Hello,

I have contacted the team to have them investigate those IDs and see if they can get them canceled for you. I'll let you know when I hear back.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

We have finalized all those jobs for you. We're looking into making sure that this doesn't continue to happen going forward with new jobs as well.

Regards,

Mike, AdWords API Team