BatchJob runtime performance issue

[di...@easyleads.com]

Hi,

Our software manages customers budgets and as result it performs a lot of BudgetOperations, thus the performance is critical.

I just had a job that updates budgets/delivery methods of 15 campaigns and it took several minutes.

Am I doing something wrong?

Thanks

Job ID is 455295008

Output:

Sleeping 2 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 4 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 8 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 16 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 32 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 64 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 128 seconds...

Batch job ID 455295008 has status 'ACTIVE'.

Sleeping 256 seconds...

Batch job ID 455295008 has status 'DONE'.

Code:

BatchJobServiceInterface batchJobService = adwordsServices.get(session, BatchJobServiceInterface.class);

// Create a BatchJob.

BatchJobOperation addOp = new BatchJobOperation();

addOp.setOperator(Operator.ADD);

addOp.setOperand(new BatchJob());

BatchJob batchJob = batchJobService.mutate(new BatchJobOperation[] {addOp}).getValue(0);

// Get the upload URL from the new job.

String uploadUrl = batchJob.getUploadUrl().getUrl();

// Use a BatchJobHelper to upload all operations.

BatchJobHelper batchJobHelper = new BatchJobHelper(session);

batchJobHelper.uploadBatchJobOperations(operations, uploadUrl);

List<BatchJobStatus> pendingStatuses = Arrays.asList(BatchJobStatus.ACTIVE, BatchJobStatus.AWAITING_FILE);

int pollAttempts = 0;

boolean isPending = true;

Selector selector = new SelectorBuilder().

fields(BatchJobField.Id, BatchJobField.Status, BatchJobField.DownloadUrl, BatchJobField.ProcessingErrors, BatchJobField.ProgressStats)

.equalsId(batchJob.getId())

.build();

int initialWait = 2;

if(operations.size() > 100){

if(operations.size() < 200){

initialWait = 4;

}

else if(operations.size() < 500){

initialWait = 6;

}

else{

initialWait = 10;

}

}

do {

long sleepSeconds = (long)(Math.scalb(initialWait, pollAttempts));

System.out.printf("Sleeping %d seconds...%n", sleepSeconds);

Thread.sleep(sleepSeconds * 1000);

batchJob = batchJobService.get(selector).getEntries(0);

System.out.printf("Batch job ID %d has status '%s'.%n", batchJob.getId(), batchJob.getStatus());

pollAttempts++;

isPending = pendingStatuses.contains(batchJob.getStatus());

} while (isPending && pollAttempts < 10);

[di...@easyleads.com]

Another job, running right now:

Sleeping 2 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 4 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 8 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 16 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 32 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 64 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 128 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 256 seconds...

Batch job ID 455533540 has status 'ACTIVE'.

Sleeping 512 seconds...

[Vishal Vinayak (Adwords API Team)]

Hi,

The code that you shared looks correct. There might be an intermittent issue with performance of the batch jobs which might have caused delays on your side. Could you please confirm if you saw any batch job failures? If you are still facing issues, please let me know and I would dig deeper into this issue. Meanwhile, you can check out this Best Practices Guide to optimize your batch job.

Regards,

Vishal, AdWords API Team

[di...@easyleads.com]

Hi,

Sorry for the delay.

All jobs succeed, but some of them take a lot of time. These jobs are from last two hours (size is number of operations in job). I understand that due to sleep actual time may be less (for example 300 seconds instead of 500), but it's still seems for me too long.

job_id	size	duration (sec)
463701054	1	1088
463504394	1	1081
463506035	7	1078
463504838	3	1074
463499033	1	1063
463526266	3	1063
463501124	14	1054
463695663	3	1047
463695669	3	1047
463515697	5	1044
463689093	2	1026
463516076	3	562
463709946	2	556
463492631	3	552
463696620	1	546
463526995	2	543
463493558	1	528
463518640	3	526
463695759	4	523
463529416	4	513

Thanks

[Vishal Vinayak (Adwords API Team)]

Hi,

Batch jobs usually do not take a long time to complete when there are only a few operations in the request. However, fewer larger jobs are preferred over many smaller jobs and you might have faced intermittent delays when sending too many smaller jobs in a short interval of time. Could you please check if some of these operations can be grouped in a single job? If there are dependent operations, you can use temporary IDs to reference to the result of an ADD operation that precedes a given operation. 

[di...@easyleads.com]

Hi,

I group operations of the same customer. The problem is that we manage hundreds of customers and as far as I know I can't group their operations in one job (because I can't specify multiple customers/credentials in the session).

But the real problem is that there is a severe degradation during past weeks. This logic runs for months and the problem began 2-3 weeks ago.

[Vishal Vinayak (Adwords API Team)]

Hi,

In case you need only a few operations per customer, you can send the operations directly via the service (rather than using BatchJobService). Meanwhile, I'll check with my team on why the performance has been slow for the last few weeks. Myself or someone from my team would get back to you on this thread. 

[di...@easyleads.com]

Great.

Is there "rule of thumb" on number of operations that is better to execute directly via the service?

Thank you!

[Michael Cloonan (AdWords API Team)]

Hello,

There are no particular performance guidelines or guarantees I can share with you. The team is currently looking into why the BatchJobService may be performing slower than expected, and I will let you know when I have any more information.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

The team has determined that the issues at this time were transient and that everything should be going smoothly again now. I apologize for the inconvenience.

Please make a new thread if the slowdowns occur again so that we can investigate!

Regards,

Mike, AdWords API Team

[di...@easyleads.com]

Thank you!