BatchJobService taking too long since yesterday?

[AdWordsApiUser]

Hello,

Some jobs on the BatchJobService seem to be taking a long time - the jobs seem to be in ACTIVE state for a long time before they get processed. The wait times seem to be increasing. Some of them do get processed fast though. Thought I'd bring this to notice before this becomes too problematic :) 

Cheers,

MG

[Shwetha Vastrad (AdWords API Team)]

Hi MG,

Thanks for letting us know. If you see batch jobs taking unusually longer to complete, please provide the BatchJob IDs so we can take a look. 

Regards,

Shwetha, AdWords API Team.

[Stefano Fontana]

Hello Swetha,

We are experiencing the same here.

Some jobs Ids are: 437684280 437574575 437670102 437670783

[Stefano Fontana]

Here you can appreciate the amount of the increase (chart shows execution time per build, there's one build a day). I started to notice a difference on Jan 11 2017.

We also have an increase of Slow Jobs that are hanging (which we cancel and then restart), some ids are in the previous post.

Cheers,

   Stefano-

[Shwetha Vastrad (AdWords API Team)]

Hi Stefano,

We found the root cause of the issue, and things should be back to normal now. We've put measures in place to make sure this root cause does not happen again. Please let me know if your batch jobs in the past day have encountered this problem. If you still encounter slowness with your batch jobs, please provide the BatchJobIds.

Thanks,

Shwetha, AdWords API Team.

[Stefano Fontana]

Hello Shwetha,

Thank you for looking into it.

I'm noticing other slow jobs though.

Example: 

 JobId 438029657 (add/remove 54 Ads)

 JobId 438145065 (add 2 Ads) 

The same happens for other operation types as well.

Both jobs were taking more than than 2 minutes to complete (even a small percentage of the task) at which point I canceled and resubmitted the job.

Interestingly enough after cancelling and resubmitting, the job *always* starts within 10 seconds, not matter what size of what kind of operations I am submitting.

The most annoying part of a slow job is having to re-submit hundreds of thousands of operations, which takes a repugnant amount of time in Python (see my issue here https://github.com/googleads/googleads-python-lib/issues/118)

Hope this helps :)

[Nadine Sundquist (AdWords API Team)]

Hi Stefano,

Thanks for providing the additional information. There was a recent spike in execution time, which is now back to normal. I can see why this would be frustrating, and thank you for logging that on GitHub. Those additional job IDs should help us pinpoint what's going on. We'll have to do a bit of digging on our side, so I'll get back to you when we have more information. In the meantime, things should have gotten a bit better for you.

Cheers,

Nadine, AdWords API Team

[Nadine Sundquist (AdWords API Team)]

Hello Stefano,

We've resolved any of the outstanding issues with performance in BatchJobService. If you see any further spikes in performance latency, please get back to me.

Thanks,

Nadine, AdWords API Team