Issues with BatchJobService - jobs remain stuck

[AdWordsApiUser]

As mentioned by few other folks over the last few days, we're having issues with BatchJobService taking very long to process jobs. As requested, here are a few job IDs:

Job Ids:

441159032

441282849

These add AdGroupCriteria, but other jobs (e.g. that change bids) also get stuck. We would really appreciate some insight into this.

Regards,

MG

[Vishal Vinayak (Adwords API Team)]

Hi MG,

Issues related to batch job performance tend to be intermittent some of the times. In such cases, cancelling the job and sending a new request solves the issue. Could you please try cancelling all current jobs with the status AWAITING_FILE or ACTIVE and retry? If that doesn't help, please feel free to revert and I'll check with my team to see if we can dig into this issue further. 

Regards,

Vishal, AdWords API Team

[Brian Bachtel]

We are also seeing an issue with the BatchJobService as well. Jobs are stuck in an "ACTIVE" state and when we check the Progress Stats everything says 0 even though we definitely uploaded a file with operations. It appears to be intermittent.

Job IDs

441119824

441270816

441145043

[AdWordsApiUser]

Hi Vishal,

Thanks for your reply. However, we have had 100% of the jobs we've submitted in the last 6 hours take over 15 mintues to complete. 

Regards,

MG

[Nadine Sundquist (AdWords API Team)]

Greetings Fellow AdWordsians,

Thanks for the batch job IDs. They are a BIG help. I've taken a look at the history of those batch jobs in our internal systems, and I've found a pattern. I need to run it by a few teammates here to see if my observations result in finding a root cause. I'll update you when I've made a bit more progress.

Cheers,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hello Everyone,

I have good news! We found the root cause, and the issue has been fixed. Your batch job processing times should be back to normal now. Thanks again for reporting this!

Take care,

Nadine, AdWords API Team

[AdWordsApiUser]

Thanks Nadine! You and your team saved us a costly rewrite that we had started planning for (without using the bulk job service). We'll keep an eye out and report back if we see issues. Thanks team - you're the most responsive team at Google I've interacted with ;)

[AdWordsApiUser]

Hello team,

The issue seems to happen again, though not as bad as last time, so I thought I'd let you know before it becomes worse. Some jobs are taking a long time to process, while others process very fast. Last time it happened, over 2-3 days, the jobs became slower, and a higher proportion of jobs started getting stuck till you guys did some magic to make everything fast again:

stuck/took very long: 

445491084

445522515

445523004 

not stuck/ran fast: 

445491588

445352074

Cheers,

MG

[Nadine Sundquist (AdWords API Team)]

Hi MG,

Thanks for sending over the batch job IDs. I took a look at the time it took for the jobs to run for the slow running jobs. I didn't see any that exceeded 15 minutes and most were completing within 5 minutes. There is a bit of load balancing going on with a lot of job requests, so it's expected that while some jobs may only take 2 minutes, there will be a few that may exceed 10 minutes. Luckily, the jobs don't appear to be getting stuck indefinitely, and they're not hanging for hours on end. I'll keep an eye on things as I did see a spike in processing time this morning, but it looks like things are returning to normal again. Please reach out if you see the times increasing even more.

Thanks again,

Nadine, AdWords API Team

[AdWordsApiUser]

Thanks Nadine - appreciate your quick reply. We'll keep an eye out and let you know if things get out of hand.

[Stefan Kleine Stegemann]

Hi,

sorry for hijacking this thread but since yesterday morning, we also started to have significant troubles with batch job service. Jobs remained active for a very long time. The following jobs have been started yesterday morning, german time (see start time is noted for each job) and were still active this morning:

Job 445516765 | 2017-02-14T03:34

Job 445553000 | 2017-02-14T03:20

Job 445532407 | 2017-02-14T03:58

Job 445532515 | 2017-02-14T03:58

Job 445693959 | 2017-02-14T03:58

Job 445694061 | 2017-02-14T03:59

Job 445694595 | 2017-02-14T03:59

We've canceled the jobs this morning (after they are still reported to be active, how can that be?), but now they are in canceling state for about an hour (at the time of this writing). Furthermore, the situation got worse this morning, with 30 batch jobs being active for a couple of hours now. The ids of these jobs are attached to the end of this post. We will cancel these jobs as well because we do not expect them to finish, based on yesterday's experience.

As an additional information, the problems seem to affect accounts with campaigns with a large to huge number or product partitions / keywords. The problematic jobs were "spreaded" over three adwords accounts yesterday and three accounts today. Two of the accounts affected yesterday are also affected today. We do use these batch jobs to set cpc bids for keywords and product partitions and use a maximum of 40.000 operations per job. We experimented with this number in the past and found that to be a "suitable" balance between performance and reliability of batch jobs. All jobs contain only one type of operation. While multiple jobs might affect the same campaign / adgroups, they do not affect the same keywords / product partitions. I was told on the past API workshop that this is ok and will not result in concurrency issues. In addition to concrete information on the issue, we would be happy to know if it makes sense to "play" with the limit of 40.000 operations per job again.

As a general note, batch job continously tend to keep us busy since we switched to them last year. The runtime performance seems to be rather unpredictable. Especially on test accounts, batch jobs can run for a very long time (up to 30 minutes) which is painful if you want to run automated tests. Combined with the unclear advise on the number of operations to put in a batch job and best practices in general, that leaves us with a very uncertain feeling. Given the reliabilty issues we experienced in the past half year, we're currently thinking about ditching batch jobs entirely and implement batching on our site which would give us at least more control. We understand that the intend of batch jobs is like "fire and forget" (yeah, not really forget but check results etc.), we offload operations to AdWords and it takes care of it. Currently its more like "fire and pray". While we do not want to bring back the past, but we never had such troubles with mutate job service. I apologize for this bit of ranting, but it's constantly keeping us busy for a while now.

Finally, here are the job ids of the still-active (soon to be canceled) jobs from this morning.

445954525

445954540

445989185

446102085

446102088

445952002

445986527

445986596

445986662

445986857

445987025

445987106

445987562

445987577

445987715

446107815

446107821

446108097

446108442

446108472

446108553

446108598

446108796

446108910

446108958

45948708

445983389

445983824

446105442

446105460

Thanks for taking care of this and regards,

Stefan

[Nadine Sundquist (AdWords API Team)]

Hi Stefan,

Thanks for reporting this. I personally experienced some longer than usual running batch jobs last night as well. We're looking into what caused this last spike in execution time.

Cheers,

Nadine, AdWords API Team

[Stefan Kleine Stegemann]

Hi Nadine,

To update you on this issue, the situation has not improved but rather got slightly worse. Today we have 33 stuck batch jobs (in status ACTIVE) with eight adwords accounts being affected. Yesterday, it was 30 batch jobs with three accounts being affected and on tuesday it was seven jobs, also with three accounts. So we see a slightly "negative" tendency here. I also checked some of the batch jobs we canceled yesterday and they are still in status "CANCELING". Even the jobs which got stuck on tuesday and which we also have canceled yesterday remain in "CANCELING". It seems that these jobs are not doing anything anymore and that there is something broken fundamentally.

Do you have any update on this issue? Can we expect batch jobs to come back to "normal"? We'd like to know that because if not, we need to take action to deal with such scenarios.

Best,

Stefan

[Biniam Asnake]

We are also experiencing this issue since some time.

The following batches are stuck on the same percentage for morethan 24 hours! Please fix.

Here is some information from our log file.

The batch-job 446153939 returned the status ACTIVE.
Progress Stats:
          46% Executed
          1934 Operations Executed
          1932 Operations Succeeded
          0 Results Written


The batch-job 446111143 returned the status ACTIVE.
Progress Stats:
          53% Executed
          1733 Operations Executed
          1725 Operations Succeeded
          0 Results Written


The batch-job 446270445 returned the status ACTIVE.
Progress Stats:
          61% Executed
          2191 Operations Executed
          2181 Operations Succeeded
          0 Results Written


Batch job status check for ID: 446153939 (Status: In-Progress) is attempted 59 times while checking every 5 minutes
Batch job status check for ID: 446111143 (Status: In-Progress) is attempted 59 times while checking every 5 minutes
Batch job status check for ID: 446270445 (Status: In-Progress) is attempted 59 times while checking every 5 minutes

[Nadine Sundquist (AdWords API Team)]

Hi Biniam,

It looks like we found the culprit on our side, but we haven't found the solution, yet. The workaround for now is to cancel the job that is hanging and to retry the operations that have not executed. I'll keep you in the loop as we make progress.

Cheers,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hi Stefan,

The good news is that we found where the hanging is happening, but the bad news is that we still don't have a complete fix. If the cancel continues to hang, please get back to me. Right now, it looks like you're doing the right thing by cancelling and retrying. 

Best,

Nadine, AdWords API Team

[Stefan Kleine Stegemann]

Hi Nadine,

Thanks for the update, we canceled the jobs from this morning as well. While doing so, we observed that for some jobs, it was not possible to cancel them although they have been in state active. For these jobs, the progress showed 100% completed. Retrying is not really an option for us, if the jobs are not completed by early morning (around 07:00am german time), it's pointless or even harmful for us to retry. I suppose that, under normal conditions, we can expect a job to be finished in time if they have been submitted by 5:00am? At least that was not a problem in the past. Also, it feels like pushing more and more jobs to the "queue" might make things even worse.

I checked again some of the jobs we did cancel (including jobs from tuesday) and they all are still in status canceling, so no progress on that side.

Best,

Stefan

[Nadine Sundquist (AdWords API Team)]

Greetings Stefan,

This is useful information when we're trying to figure out how to find a solution for this one. Thanks for passing it along. If jobs have been finishing for you normally by a certain time, then I would expect it to continue to do so. If it doesn't, then it's probably fallen into the issue that we're working on. I'll continue to track this so we can find a solution.

Regards,

Nadine, AdWords API Team

[Stefan Kleine Stegemann]

Hi Nadine,

still no improvement, today we have 28 stuck batch jobs across seven adwords accounts. I also checked a sample of 10 batch jobs out of the ones we have canceled over the past days and 2 jobs are actually canceled but 8 are still in state canceling. As this issue is present for four days now, it becomes more critical for us because we are unable to deliver the full service to some of our customers. Thus, any advise for how the deal with the "broken" batch jobs is appreciated. Does it, for example, make sense to try smaller or larger jobs?

Best,

Stefan

Am Donnerstag, 16. Februar 2017 23:23:22 UTC+1

[Nadine Sundquist (AdWords API Team)]

Hi Stefan,

We're actively working on clearing the queue, which includes those jobs in the canceling state, as we're searching for a fix for this. Unfortunately, it won't make a difference whether or not it's a small job versus a larger job. One of my teammates put it quite well when he said that a larger job may have a higher chance of getting stuck, but on the other hand when it does go through, it gets through more operations. Just so you know, when a job is in a canceling state, it will not execute any more operations, so you can submit the rest of the operations that did not execute in that canceling job. It might take awhile for the job to reach a canceled state, but at least it won't execute any more operations. I realize this isn't optimal, but I do want you to know that is of higher priority to us. It's just really tricky trying to find a solution for what you're seeing. I'll get back to you as we make more progress. Thanks again for providing so much helpful information!

Cheers,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hello Stefan,

I have good news! We've been working through processing through the queue of hanging jobs. You should be seeing an improvement in that these hanging jobs should be reaching a final state e.g. canceling should now be canceled. You may still see a few hanging, but the majority should have reached a final state. We're still working on fixing the root cause.

Take care,

Nadine, AdWords API Team

[Stefan Kleine Stegemann]

Hi Nadine,

good news on our side as well. Today, we did not have stuck jobs any more and all jobs have been processed in time as they have before this issue. We did decrease the batch job size to 20k though, I don't know whether this relevant or not. Furthermore, I've checked a couple of jobs from the past days and they all are canceled now.

At the moment, it looks like the issue is solved. We'll keep an I on it and let you know if it happens again. Thanks for taking care of this and your constant updates on this issue.

Best,

Stefan

[Nadine Sundquist (AdWords API Team)]

Hi Stefan,

Thanks for the update! I'm keeping an eye on this as well.

Take care,

Nadine, AdWords API Team

[Mike]

Hi Nadine   

   Now in 24 Feb it's stuck again with job ID 449587116. Can you check what happen please? 

[Nadine Sundquist (AdWords API Team)]

Hi Mike,

It looks like that job was submitted over 10 hours ago, but it wasn't picked up to be processed. I'm taking a look into it. In the meantime, please try cancelling the job and resubmitting.

Thanks,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hi Mike,

It looks like the job is still waiting for a file to upload with operations. If you haven't cancelled yet, you can try uploading a file with operations to the upload URL.

Cheers,

Nadine, AdWords API Team

[Stefan Kleine Stegemann]

Hi Nadine,

over the past weeks, batch jobs have been stable and also much faster than before. However, starting yesterday, we have some

stuck batch jobs again. This is annoying, to be honest. It creates a lot of effort on our side. Do we have to expect that to

happen on a regular base?

Here are the job ids of jobs stuck today:

    - 456510200

    - 456526321

    - 456728007

    - 456485951

    - 456703203

    - 456703755

I did cancel the jobs around 8pm german time after they have remained active for around 14 hours. Could you check whether this

is the "old issue" appearing again?

Best,

Stefan

[Stefan Kleine Stegemann]

Hi,

the problem remains and "increases" as far as we can see. This morning (german time), we have 22 jobs stuck in ACTIVE for around two to three hours now. The "old" jobs from yesterday and tuesday are in state CANCELED meanwhile. However, as mentioned before in this thread, it makes no sense to re-submit a job if it does not get through in the early morning.

Regards,

Stefan

[Josh Radcliff (AdWords API Team)]

Hi Stefan,

Could you share the job IDs for some of the 22 jobs you mentioned?

Thanks,

Josh, AdWords API Team

[Stefan Kleine Stegemann]

Hi Josh,

i turned out that the jobs eventually went through, it just took very long (around 3 hours). Here are some job ids:

458099621

458108326

458358726

458161757

This morning, everything went back to normal and jobs go through as the weeks before. So no issue any more at the moment.

Thanks for taking care of this.

Best,

Stefan

[Josh Radcliff (AdWords API Team)]

Hi Stefan,

Glad to hear that things are back to normal! We had some issues earlier this week with long-running jobs + delayed job activation, but those should be resolved now. If you see this occurring again, please let me know (and send over the impacted job IDs) and I'll follow up with the batch job team.

Cheers,

Josh, AdWords API Team

[ro...@growth-rocket.com]

Hi Nadine,

Can you check this Batch Job Id:575510402

My first batch job, its been 4 hrs and status did not change. 

[Nadine Sundquist (AdWords API Team)]

Hello Roger,

Could you please open a new forum thread for this? This one is pretty old, and it really helps to have a new forum thread for a new issue.

Thanks,

Nadine, AdWords API Team

[Vlad Gukasov]

Hi Nadine!

I the same problem with my batch job. The ID is 743497970. It hangs a 3 hours. Thanks for the help in advance!

Regards,

Vlad, Independent Developer

[Thorsten Merz]

Hey there,

it seems that starting today we are experiencing performance issues with the batch job service (API v201710).

Usually our batchjobs take only seconds to finish up to 1-2 minutes. Since today our batch jobs run within 10, 20, 30 or even 60 minutes with similar amount of operations and operation types.

Are there any specific issues currently ongoing with the batch  job queue / service / API?

BatchID  example: 743564948

Some help on how to resolve these issues would be appreciated.

Regards

Thorsten

--

Seems i am not able to create a new discussion :/

[Nadine Sundquist (AdWords API Team)]

Hello Vlad,

I took a look at your job. It looks like it took 4 hours to complete, but it did finally complete. There was a bit of a load balancing glitch on our batch jobs that is currently clearing up, so your batch jobs should be getting faster again.

Best,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hi Thorsten,

You are correct. I do see an anomaly in our batch job processing where the time to process has spiked for a number of people. Thank you for giving us a heads-up! It started May 21st. I can see that it's getting better, so the processing should be speeding up. I'm working with a couple people here to make sure that the processing time stays down. Please get back to me if the processing time continues to be troublesome tomorrow.

Thanks,

Nadine, AdWords API Team