Batch job service is not efficient, stucked

[mi...@adgorithms.com]

Hello,

Im trying to use the BatchJob service in python.

Im creating adgroups with in total ~100000 keywords and it takes a lot of time(sometimes also stucked).

I have read the code in the api documentation.

My question is:

1. There a efficient way to create this batch jobs?\

2. There is a limit of operations?

3. How can we improve it(best practices)?

This is my code(took from internet):

def run_operations_as_batch_job(account_id, wait_for_response=False, *operations):
    set_adwords_client(account_id)
    job_helper = adwords_client.GetBatchJobHelper()
    job = create_batch_job()
    upload_url = job['uploadUrl']['url']
    batch_job_id = job['id']

    job_helper.UploadOperations(upload_url, *operations)
    while wait_for_response:
        try:
            sleep(5)
            url = GetBatchJobDownloadUrlWhenReady(batch_job_id)
            response = urllib2.urlopen(url).read()
            return job_helper.ParseResponse(response)
        except:
            pass
    return batch_job_id


def GetBatchJobDownloadUrlWhenReady(batch_job_id, max_poll_attempts=10):
    batch_job = get_batch_job(batch_job_id)
    poll_attempt = 0
    while poll_attempt < max_poll_attempts and batch_job['status'] in ('DONE', 'ACTIVE', 'AWAITING_FILE', 'CANCELING'):
        if 'downloadUrl' in batch_job:
            url = batch_job['downloadUrl']['url']
            print ('Batch Job with Id "%s", Status "%s", and DownloadUrl "%s" ready.'
                   % (batch_job['id'], batch_job['status'], url))
            return url
        sleep_interval = (30 * (2 ** poll_attempt) +
                          (random.randint(0, 10000) / 1000))
        print 'Batch Job not ready, sleeping for %s seconds.' % sleep_interval
        sleep(sleep_interval)
        batch_job = get_batch_job(batch_job_id)
        poll_attempt += 1

    raise Exception('Batch Job not finished downloading. Try checking later.')


def get_batch_job(batch_job_id):
    batch_job_service = get_service(Services.BATCH_JOB_SERVICE)
    selector = {
      'fields': ['Id', 'Status', 'DownloadUrl'],
      'predicates': [{'field': 'Id', 'operator': 'EQUALS', 'values': [batch_job_id]}]
    }
    return batch_job_service.get(selector)['entries'][0]

[Vishal Vinayak (Adwords API Team)]

Hi Mike,

Looks like you are referring to our python example to add keywords using a batch job, which is definitely recommended. Please find the rest of my answers inline:

Im creating adgroups with in total ~100000 keywords and it takes a lot of time(sometimes also stucked).

Could you please provide SOAP XML request and response logs for the job that you feel got stuck? If not, if you have the batch job ID, that would help me in debugging the issue.

1. There a efficient way to create this batch jobs?

The recommended way to create a batch job is explained in our Batch Processing Guide. Although the process is explained in Java, the same guidelines are referred to when creating the Python library examples. 

2. There is a limit of operations?

You may see a RateExceededError if you perform more than 10,000 operations in a day. Please refer to our Rate Sheet to understand what counts as an operation. In this case, the only way to increase the operational limit for your account would be to apply for a Standard Access. You can, however, also see this error if you send too many requests to the API in a short amount of time (few minutes for example). This usually happens if you do not group your operations into a batch job and instead prefer to send one operation with each API request. This rate limit is in place to stop malicious software from crashing our servers. Rate limits (not to be confused with System Limits) are transient and expire after a period of time.The retryAfterSeconds property of the RateExceededError would tell you the appropriate time interval after which it is safe to retry. Also, there are no penalties associated with hitting the rate limit.

3. How can we improve it(best practices)?

You can find a list of best practices for batch jobs here.

Regards,

Vishal, AdWords API Team

[mi...@adgorithms.com]

Thanks for the quick response!

The problem we see is not the job in google. Its when building the SOAP xml to upload. The function that do this is job_helper.UploadOperations() from the adwprds library in python.
We checked and it takes a very long time before it sends to google (creating ~100000 keywords).
Do you a better way to do this? Maybe a workaround or another protocol.

Thanks!

[Vishal Vinayak (Adwords API Team)]

Hi Mike,

The advantage of batch jobs is in the fact that you can group many operations of various kinds into a single request and the API would automatically handle retrying and rate limiting logic for you. It may, however, take some time to complete the batch job if there are hundreds of thousands of operations in a single request. If you follow the best practices guide (which I shared in my previous response), you should be able to achieve optimum performance. Since AdWords API uses SOAP protocol in the back-end, unfortunately I cannot think of a workaround to the way the jobs are sent via a SOAP XML. If you have more questions related to the Python client library in general, I would recommend opening an issue here. 

[Stefano Fontana]

See this issue https://github.com/googleads/googleads-python-lib/issues/118