MutateJobService - slow response

[GDZ]

When creating a mutate call submitting ~1000 keywords via MutateJobService, response time is always around 50 sec.

Wasn't the idea of this service to just submit operations in bulk quickly, and only after that periodically check for job status and result?

50 sec for just submitting (not actual processing) of 1000 operations seems kinda too much. And yes I know that we can do that concurrently but still...

Thank you,

GDZ 

[Josh Radcliff (AdWords API Team)]

Hi GDZ,

Do you have the requestId from the response of one of your MutateJobService requests that took 50 seconds? I'd like to take a look at it from our side.

One thing to keep in mind is that the benefits of MutateJobService go beyond just quickly submitting large requests. Although it does allow you to "fire and forget" large changes, it also attempts to handle transient errors such as concurrent modification errors and rate limit errors.

Thanks,

Josh, AdWords API Team

[GDZ]

Here's the requestId: 00051557bc394c380aecb348e200683b

Thank you.

[Josh Radcliff (AdWords API Team)]

Hi GDZ,

Thanks for sending over the request ID. According to our records, the response was returned in 844 milliseconds. I'm wondering if there's either a bottleneck in the library you're using or your code that's actually causing the delay.

Could you let me know which client library you're using to send your requests? Also, have you confirmed that the 50 second delay is actually from the time the request is sent to the time the response is received?

Thanks,

Josh, AdWords API Team

[GDZ]

Using this example here:

https://github.com/googleads/googleads-python-lib/blob/master/examples/adwords/v201502/campaign_management/add_keywords_in_bulk.py

And yes, I measure time before and after "response = mutate_job_service.mutate(operations, policy)"

[Josh Radcliff (AdWords API Team)]

Hi,

I retrieved the XML response for your request, and that also claims that the request only took 844 milliseconds, so I suspect the delay is in the preparation of the request, most likely within the client library itself or (less likely) in your code that constructs the objects.

Could you enable debug logging when submitting one of your large requests by adding the following to your code?

import logging

logging.basicConfig(level=logging.DEBUG)

logging.getLogger('suds.transport').setLevel(logging.DEBUG)

In the output generated, you should see something like this:

DEBUG:suds.metrics:method 'mutate' invoked: 183 (ms)

I'd like to know what that shows in terms of milliseconds, as that reflects the actual request/response time over the wire.

Thanks,

Josh, AdWords API Team

[GDZ]

Submitting job of 237 keywords, took around 20 sec.

Output:  DEBUG:suds.metrics:method 'mutate' invoked: 1.167 (seconds)

[Josh Radcliff (AdWords API Team)]

Hi,

Thanks for getting the log message. That clearly shows that out of the 20 seconds it took to:

1. Prepare the request (including converting it to SOAP XML)
2. Send the request
3. Process the response

only 1.167 seconds was actually spent on step 2. This means that something else besides the sending of the request is the bottleneck. I highly doubt that step 3 is an issue because the response is rather small, so that points to step 1 as the culprit.

Do the logs give any more insights as to which portion of step 1 is taking the remaining 18.833 seconds?

Thanks,

Josh, AdWords API Team

[GDZ]

Hope not a double post:  another message I found in the log:  DEBUG:suds.metrics:message for 'mutate' created: 30.551 (seconds)

Emailed you full log also.

Thank you!

[Josh Radcliff (AdWords API Team)]

Hi,

I spoke with our Python library guru, and he informed me that this is a known performance issue with the library. Specifically, suds (the SOAP library used by our Python client library) can take a significant amount of time to prepare large requests and parse large responses. This is consistent with the timing you sent over, as it shows that the actual processing time (once the request is prepared) is just a small fraction of the overall time elapsed.

Unfortunately, there's not much you can do to resolve this. From tests on our side, if you have a large # of operations to send, you'll see slightly better performance if you send fewer larger requests (more operations per request) than if you send a large number of small requests (fewer operations per request).

Thanks,

Josh, AdWords API Team

[GDZ]

Yes I realize that there's not much WE can do to resolve this. But will this be fixed on your side in the next version? Such slow processing kinda defeats the whole idea of quickly submitting bulk jobs...

Thank you.

[Josh Radcliff (AdWords API Team)]

Hi,

You will encounter this issue whether you submit a large number of operations via MutateJobService or via the type-specific services such as AdGroupCriterionService, since the bottleneck is in how suds handles marshalling and unmarshalling large requests and responses.

I'm not aware of any immediate plans to try to switch to another SOAP library, which is what would be required to improve performance of extremely large requests using the Python client library (unless, of course, the owners of the suds library address this issue). If you feel that this is a significant problem for your app, please open an issue on the googleads-python-lib github issues page. The library owners may have additional info, and this will allow other users to chime in on the issue if it's also a problem for them.

Thanks,

Josh, AdWords API Team

[GDZ]

Josh,

I ended up forming and submitting SOAP requests directly. Super fast and works like a charm. 

Thank you,

GDZ