Issue with uploading large amounts of data using MutateJobService

[Alan Coleman]

Hello everyone,

I'm creating some functionality in our app to build a fresh copy of a campaign based on a current  campaign's ALL_TIME sqpr report. The report has the potential to produces tens of thousands of search terms - Which will in turn be created into AdGroups, TextAds, Keywords and Negatives.

I'm using MutateJobService to bulk upload data in batches of 1000 records. So for example if I need to create 16,000 AdGroups I'm chunking the array of AdGroups into 16 x 1000 and making 16 MutateJobService requests in a loop.

This works, up to a point.

I can created around 20,000 TextAds before  the script stops and I receive an error (See below), although confusingly it just stops and I don't see an error at all, or even an entry into my various logs.

Can anyone think of something I could be doing wrong?

Am I asking for too much by pushing this much data through the API, even using MutateJobService?

To give you an idea of what I'm trying to achieve, this is what the page of my app looks like:

Fatal error: Uncaught SoapFault exception: [soap:Server] [InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.<init>(InternalApiErro] in /home1/adwordsl/public_html/app/adwords_api_php_5.3.2/src/Google/Api/Ads/Common/Lib/AdsSoapClient.php:216 Stack trace: #0 /home1/adwordsl/public_html/app/adwords_api_php_5.3.2/src/Google/Api/Ads/Common/Lib/AdsSoapClient.php(216): SoapClient->__soapCall('mutate', Array, NULL, Array, Array) #1 /home1/adwordsl/public_html/app/adwords_api_php_5.3.2/src/Google/Api/Ads/AdWords/v201402/MutateJobService.php(19169): AdsSoapClient->__soapCall('mutate', Array) #2 /home1/adwordsl/public_html/app/php_inc/app_addtextadsinbulk.php(73): MutateJobService->mutate(Array, Object(BulkMutateJobPolicy)) #3 /home1/adwordsl/public_html/app/appcampaignreset.php(484): AddTextAdsInBulk(Object(AdWordsUser), Array) #4 {main} thrown in /home1/adwordsl/public_html/app/adwords_api_php_5.3.2/src/Google/Api/Ads/Common/Lib/AdsSoapClient.php on line 216

[Anash P. Oommen (AdWords API Team)]

Hi Alan,

InternalApiError.UNEXPECTED_INTERNAL_API_ERROR means that something unexpected happened at our end. MutateJobService should be able to process the kind of batch size you provide without any issues.

Can you please provide the SOAP request id (or the sanitized SOAP logs)? You can turn on SOAP logging in the client library's settings.ini.

Cheers,
Anash P. Oommen,
AdWords API Advisor.

[Alan Coleman]

Thanks Anash,

I caught another such error in my SOAP Log this morning, although this time the error returned before any results.

I'd really appreciate it if you could take a quick look and let me know what you think.

Many thanks.

HTTP/1.1 500 Internal Server Error

Content-Type: text/xml; charset=UTF-8

Date: Thu, 28 Aug 2014 09:28:48 GMT

Expires: Thu, 28 Aug 2014 09:28:48 GMT

Cache-Control: private, max-age=0

X-Content-Type-Options: nosniff

X-Frame-Options: SAMEORIGIN

X-XSS-Protection: 1; mode=block

Server: GSE

Transfer-Encoding: chunked

<?xml version="1.0"?>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">

  <soap:Header>

    <ResponseHeader xmlns="https://adwords.google.com/api/adwords/cm/v201402">

      <requestId>000501ad297264b00a4254a6960046b8</requestId>

      <serviceName>MutateJobService</serviceName>

      <methodName>mutate</methodName>

      <operations>0</operations>

      <responseTime>8909</responseTime>

    </ResponseHeader>

  </soap:Header>

  <soap:Body>

    <soap:Fault>

      <faultcode>soap:Server</faultcode>

      <faultstring>[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.&lt;init&gt;(InternalApiErro]</faultstring>

      <detail>

        <ApiExceptionFault xmlns="https://adwords.google.com/api/adwords/cm/v201402">

          <message>[InternalApiError.UNEXPECTED_INTERNAL_API_ERROR @ com.google.ads.api.services.common.error.InternalApiError.&lt;init&gt;(InternalApiErro]</message>

          <ApplicationException.Type>ApiException</ApplicationException.Type>

          <errors xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="InternalApiError">

            <fieldPath/>

            <trigger/>

            <errorString>InternalApiError.UNEXPECTED_INTERNAL_API_ERROR</errorString>

            <ApiError.Type>InternalApiError</ApiError.Type>

            <reason>UNEXPECTED_INTERNAL_API_ERROR</reason>

          </errors>

        </ApiExceptionFault>

      </detail>

    </soap:Fault>

  </soap:Body>

</soap:Envelope>

[Oliver]

I'm about to do something similar to what Alan is doing and, hence, why I'm also interested in this.

I see on this page:

https://developers.google.com/adwords/api/docs/guides/batch-processing

there is a paragraph that says:

• When submitting a lot of concurrent jobs for the same customerId, try to reduce the likelihood of jobs operating on the same AdGroups at the same time, while maintaining large job sizes. Many unfinished jobs (with status of PENDING or PROCESSING) that try to mutate the same set of AdGroups may lead to deadlock-like conditions resulting in severe slow-down and even job failures.

I don't know exactly what is being referred to by this term:

>> jobs operating on the same AdGroups

Is the creation of multiple Ads within the same AdGroup considered as operating on the adgroup itself?

If so, is this implying that we shouldn't attempt to create multiple Ads within the same adgroup using MJS?

Oliver

[Alan Coleman]

Hi Oliver,

Thanks for getting involved, that's an interesting find in the documentation that I haven't seen before. I started this post about the UNEXPECTED_INTERNAL_API_ERROR which I'm sometimes receiving when trying bulk uploads using MJS - Hopefully this is being looked into.

However there has been issues with MJS nearly finishing my jobs but then just stopping in a deadlock-like condition. The SOAP Logs report back as normal and indicate that the job is pending.

There's no way round using the same customerId as that would defeat the object of what we're trying to achieve. 

If I use adding TextAds as an example, the array that I currently pass to the MJS is constructed in AdGroup order, which is how it comes out of my db, and it's not unusual to be trying to add several TextAds in each AdGroup.

I'm chunking my ordered array into arrays of 1000 and then looping round through MJS to create jobs of that size, so it's possible that a concurrent job may contain the same AdGroup if the TextAds for that AdGroup have been chunked half way through. I think that is what the following means:

>> jobs operating on the same AdGroups

It may be a long shot but a good place for me to start would be to ensure that a particular Adgroup can only appear in one job.

With regards to:

Is the creation of multiple Ads within the same AdGroup considered as operating on the adgroup itself?
If so, is this implying that we shouldn't attempt to create multiple Ads within the same adgroup using MJS?

I may be wrong, but I've never experienced any problems with writing multiple TextAds or Keywords to a single AdGroup until I started using MJS, and even then it's only when maintaining large job sizes.

As I said earlier, probably unrelated to the UNEXPECTED_INTERNAL_API_ERROR, but I'm going to make the subtle changes to constructing my MJS anyway.

Thanks again,

[Oliver]

Thanks Alan for the update. Let us know if the changes you make will have an effect.

Hopefully also the API team will shed some light on this.

Oliver

[Anash P. Oommen (AdWords API Team)]

Hi Alan, Oliver,

We have identified the issue at our end, it's a flaky connection to our datastore when creating and reading jobs. We are working on a fix, but meanwhile you can put a retry loop around your code to resubmit the job. MJS can handle the batch size you are providing, so this error needn't be a limiting factor when deciding whether or not to use MJS.

Cheers,
Anash P. Oommen,

AdWords API Advisor.

[Oliver]

Anash,

Can you please clarify what the documentation (I pointed out earlier) is referring to when it says:

>> try to reduce the likelihood of jobs operating on the same AdGroups at the same time

Is creating multiple Ads within the same adgroup using MJS something that the above is saying shouldn't be done?

[360gam...@gmail.com]

Anash,

What is the status of this datastore connection issue?  Is it still unresolved?  We are having problems with jobs working for a little while and then all jobs just start returning Pending status.  We have already added the recommended retry code and they never return Complete status after retries.  The jobs also never seem to actually write their info to the account..

We posted regarding our issue here: https://developers.google.com/adwords/api/community/?place=topic%2Fadwords-api%2FPlFEA-V4tpI%2Fdiscussion

The job ids were sent to you as a follow up but we have not received any reply.  I am not sure that of our issue is the same as this or a different one, but our development is blocked and we really need some feedback.

Thanks,

-Brian

[Anash P. Oommen (AdWords API Team)]

Hi Brian,

A fix is being worked upon, it should hopefully be in place during the next server update.

Cheers,
Anash P. Oommen,
AdWords API Advisor.

[Alan Coleman]

Hi Anash,

Is there any update on this fix? Or a timescale in place?

We'd really like to be able to make use of this aspect of the API.

Many thanks.

[Anash P. Oommen (AdWords API Team)]

Hi Alan,

This fix should be live now, let me know if you are still facing issues.

Cheers,
Anash P. Oommen,
AdWords API Advisor.