BatchJobService not returning downloadUrl

[wes...@channable.com]

We are heavy users of the Adwords API (on behalf of our customers) and are experiencing some issues with batch jobs.

When we query the BatchJobService for a specific batch job id, we don't get a downloadUrl in the response. We were under the assumption that we should always get a downloadUrl from the response.

We've had these issues before, but they have been exacerbated in the last two days.

Concretely, we are using the BatchJobService to send BatchJob's to e.g. create new campaigns, adgroups, ads etc.

Here are the API docs for it: https://developers.google.com/adwords/api/docs/reference/v201802/BatchJobService.BatchJob
The docs indicate that the field 'downloadUrl' is *always* present in a response.

We are using the `googleads` python library (version 10.1.0), with adwords version `v201802`
 
Here is an example of the selector we use

    selector = {
        'fields': ['Id', 'Status', 'DownloadUrl'],
        'predicates': [
            {
                'field': 'Id',
                'operator': 'EQUALS',
                'values': [batch_job_id]
            }
        ]
    }

and an example `batch_job_id` we use is `745158738` (the client's account id is available upon request)

The return we get from using this selector is:

(BatchJob){
   id = 745158738
   status = "DONE"
   progressStats = 
      (ProgressStats){
         numOperationsExecuted = 2
         numOperationsSucceeded = 2
         estimatedPercentExecuted = 2147483647
         numResultsWritten = 2
      }
 }


If we instead use the AWQL:

`SELECT Id, Status, DownloadUrl WHERE Id = 745158738`

then we get

{
    'id': 745158738,
    'status': 'DONE',
    'progressStats': {
        'numOperationsExecuted': 2,
        'numOperationsSucceeded': 2,
        'estimatedPercentExecuted': 2147483647,
        'numResultsWritten': 2
    },
    'uploadUrl': None,
    'downloadUrl': None,
    'processingErrors': [],
    'diskUsageQuotaBalance': None
}

which more accurately shows that we are not receiving anything for the `downloadUrl`
(We also get a strange number for `estimatedPercentExecuted`, but we assume this is because the number is 2^31-1, the largest signed 32-bit integer. 
Not sure if this matters, but figured we should point it out)

Any idea what is going on? Because we send so many batch jobs per day for varying customers, this is giving us a lot of errors.

I've seen in the forum that the BatchJobService is experiencing difficulties, but like I said, we also have gotten this before.


Another user that had a similar issue: https://groups.google.com/forum/#!searchin/adwords-api/downloadUrl|sort:relevance/adwords-api/ntmtnFLPhWo/SSO4Qr-KDQAJ

[Michael Cloonan (AdWords API Team)]

Hello,

That is strange; I will get in contact with the team in charge of that service and see what we can do to get you the downloadUrl, and also investigate why this happened.

Regards,

Mike, AdWords API Team

[GregT]

I've also been having the same type of issue. I emailed API support directly yesterday. We've been having around a thousand or so jobs per day the past few days with this type of issue (the status showing as DONE, but there is no download URL to be able to get the results).  Here are a handful of job ids from the past couple hours with this issue (I also just emailed these in a reply to the support response I got, FYI):

745536273

745442031

745255265

Thanks,

Greg

[wes...@channable.com]

It's not just happening for this one batch job. We have a lot that have been having this issue.

[wes...@channable.com]

Also, the batch job id listed above was for only a few sent operations. We have some that we have no dowloadUrl for that contain a lot more operations.

batch job id 745031759 gave a response of

'numOperationsExecuted': 2104, 'numOperationsSucceeded': 2104, 'estimatedPercentExecuted': 2147483647, 'numResultsWritten': 0

but we still have no downloadUrl

On Friday, May 25, 2018 at 9:03:17 PM UTC+2, Michael Cloonan (AdWords API Team) wrote:

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for the additional details. Since it seems to be more widespread than just a one-off error, I've increased the priority internally so that we can try to get to the bottom of this sooner.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

We have regenerated download URLs for all jobs that had completed without one, and are working to improve our processes so that we can reduce the chance of this happening in the future.

Please let me know if you still have any trouble retrieving the download URLs for your batch jobs, and please include any relevant batch job IDs for ones that aren't fixed so we can investigate.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hi Michael,

We actually noticed that you regenerated, because now we are getting a different error.

For batch job id: 744483600, we now get a `downloadUrl`, but it returns a 404. We have a few other batch jobs with the same problem.

Another with the same issue:

batch_job_id = 745104354

If you need more, I can provide them.

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for letting me know. I've passed this information along to the batch job team and will let you know as soon as I hear anything back. I apologize for the inconvenience; obviously having a URL that results in a 404 is no better than having no URL at all, so I'm making sure that this is still at a high priority for us.

Regards,

Mike, AdWords API Team

[GregT]

Thanks for the update, Mike. As I posted earlier, we were having around 1,000 jobs a day have this issue. Looks like for the past day or two, we haven't had any, so it's looking good. 

Thanks,

Greg

[Michael Cloonan (AdWords API Team)]

Hello,

We made another pass regenerating download URLs, and they should no longer result in 404s. Please let me know if you are having any more difficulties.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hi Michael,

I think most of the issues we had are solved. I am still seeing a few batch jobs that when we download them.

batch_job_id: 744483651

This returns a "ParseResponse" (using the python adwors library) of:

{'mutateResponse': {'@xmlns': 'https://adwords.google.com/api/adwords/cm/v201802'}}

We expect there to be an `rval` key within the `mutateResponse`. This is usually where we get the data from the URL from.

Any advice there?

...

[Michael Cloonan (AdWords API Team)]

Hello,

Can you provide the full SOAP request/response for that particular batch job? You can reply privately to me using the arrow at the top-right of my post.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

I am having this issue (with the same result as what I sent in the private message) with at least the following 3 batch_job_ids:

744483651

744483600

745226632

That is all I can find in the error logs, so it should just be these 3.

Also, should we always expect to have a `downloadUrl` from now on? If we don't have one, should I just raise the issue here again? 

...

[Michael Cloonan (AdWords API Team)]

Hello,

Yes, when the job is done there should always be a downloadUrl available so you can fetch the results. I'll pass these IDs along to the team.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hi Michael,

Any news? I have customers waiting to here from me about this.

Thanks,

Wesley

...

[Michael Cloonan (AdWords API Team)]

Hello,

We're still investigating these last few. I have followed up with the team to see if they have any information they can share, which I will pass along here when possible.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hello,

I have just ran into the issue of having no `dowloadUrl` again for batch job id 759856332

...

[Michael Cloonan (AdWords API Team)]

Hello,

Thanks for the report, we are working hard to identify the underlying issue here, and I will let you know if we regenerate the download URL for that batch job ID, or if I have any other information to share regarding the previous ones you mentioned.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hi all,

Was just checking back in to see if there was any new news, since the thread hasn't been updated in over a week.

Cheers,

Wesley

...

[wes...@channable.com]

Hi all,

I also found another batch job today that has no `downloadUrl`

batch job id: 759856332

...

[Michael Cloonan (AdWords API Team)]

Hello,

I apologize for the delayed turnaround here. I am following up with the team to try to find a long-term solution. I think that's the same ID that you had shared previously, so I will try to get that download URL regenerated for you.

Regards,

Mike, AdWords API Team

[Michael Cloonan (AdWords API Team)]

Hello,

We have regenerated the download URL for that batch job.

Regards,

Mike, AdWords API Team

[wes...@channable.com]

Hello Michael,

I just had the same problem where I get a `downloadUrl`, but when we try to call that URL, we get an invalid status code.

This happened for the following batch job ids:

| batch_job_id |
| ----- |
| 787630151 |
| 787630490 |
| 797647414 |
| 802682768 |