403 Access denied when calling Ad Manager Api with report query

[James Tweedie]

Hello 

we are experiencing an issue this morning with some code that has been running steady for 3 years  

we connect to and report from several Ad Manager and Ad Manager 360

this morning our data collection process is unable to consistently download reports 

the issue is intermittent but frequent and not always the same Ad Manager instance

if i run a single task, it sometimes succeeds and sometimes fails 

the error is always 403 access denied, which is incorrect, we can see the service account in each of the GAM instances. 

So far the issue seems to be isolated to Ad Manager, we have no reported issues with Ad Manager 360 

Regards

James 

[James Tweedie]

update on this: we have observed the same behavior on GAM360 

Is anyone else having intermittent 403 errors on DownloadReportToFile method? 

@moderator - at least one other person also submitted a similar question - with no response 

-james   

[Wouter]

Hi James,

I'd like to confirm that we are suffering the same issue as you have described: a permission error when downloading the file from the URL provided by the Google Ad Manager API ReportService.getReportDownloadURL. The error is as follows (masked the service account):

<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>XXXX...@developer.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object.</Details></Error>

Our temporary solution for this is to do a retry: request a new URL using ReportService.getReportDownloadURL and try to see if that download succeeds. This usually is the case upon the second or third try. 

I'd like to suggest to the Google Ad Manager API team to investigate this though, as it seems to be causing a lot of people their flows to be broken.

Best,

Wouter

[Manish Saraf]

Hello everyone,

I am also having same exception from last two days,

Response:
REDACTED REPORT DATA
--------
Error:
Client error: `GET https://storage.googleapis.com/dfp-report-export/960bfebb-53d1-498d-8cdc-48479d67bdfa?GoogleAccessId=376921769015-q0sqoemp6...@developer.gserviceaccount.com&Expires=1596256535&Signature=xd6jYfB8ZM%2BTjjtk8fpmhLIZZQqHH9rPY9abmsAPCEY50CL0qojHCwMkikhiN4qLIc%2BbwLTz%2BBd%2BP3i01iHZYD2WJABrI9oYYiXM5Aez9VYu2b0G2gfHTa3tDeFkuSrIoJZRSUGsvrShXaTQwHUNaxtZarafG4P1DxxwoknYDq5YJ92iNGZC1vjUoS1hQomKue9V9rMsdV89bnY%2FGibaPjNhSUXhX4zXtNb%2BqgZGPGb5CbR8Sdpqe7U1rp49%2BQ4%2F814Foaa%2FyeO2ffR1wbU09C1zkmViB%2Fn2GwYa7TiTtO74Mru0xKYiJtHtbc1XuJb0Wbya5p4DiWMcZyqU9vPgJg%3D%3D` resulted in a `403 Forbidden` response:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>37692176 (truncated...)


PHP Fatal error:  Uncaught GuzzleHttp\Exception\ClientException: Client error: `GET https://storage.googleapis.com/dfp-report-export/960bfebb-53d1-498d-8cdc-48479d67bdfa?GoogleAccessId=376921769015-q0sqoemp6...@developer.gserviceaccount.com&Expires=1596256535&Signature=xd6jYfB8ZM%2BTjjtk8fpmhLIZZQqHH9rPY9abmsAPCEY50CL0qojHCwMkikhiN4qLIc%2BbwLTz%2BBd%2BP3i01iHZYD2WJABrI9oYYiXM5Aez9VYu2b0G2gfHTa3tDeFkuSrIoJZRSUGsvrShXaTQwHUNaxtZarafG4P1DxxwoknYDq5YJ92iNGZC1vjUoS1hQomKue9V9rMsdV89bnY%2FGibaPjNhSUXhX4zXtNb%2BqgZGPGb5CbR8Sdpqe7U1rp49%2BQ4%2F814Foaa%2FyeO2ffR1wbU09C1zkmViB%2Fn2GwYa7TiTtO74Mru0xKYiJtHtbc1XuJb0Wbya5p4DiWMcZyqU9vPgJg%3D%3D` resulted in a `403 Forbidden` response:
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>37692176 (truncated...)
 in /home/centos/googleAPI/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113
Stack trace:
#0 /home/centos/googleAPI/vendor/guzzlehttp/guzzle/src/Middleware.php(65): GuzzleHttp\Exception\Reques in /home/centos/googleAPI/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php on line 113

--
You received this message because you are subscribed to the Google Groups "Google Ad Manager API Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-doubleclick-for-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-doubleclick-for-publishers-api/54c1c813-87ac-4a40-a1b7-0ea5a605c89dn%40googlegroups.com.

[James Tweedie]

Update:

kudos Wouter !, Manesh have a look at what Wouter posted as a temporary solution. 

As Wouter suggested implementing a retry mechanism alleviated some of the issues we were having. It was a simple matter of activating a flag we had for this purpose [some of our other API connectors have low ceiling quotas]. It did not solve all the issues until we put a wait mechanism between API calls. The combination of the retry and wait got us back up and running. 

This leads me to believe that there is a latency issue with Google Cloud Storage and how it applies ACL on the bucket. We had a similar issue  in the past when using a GCS bucket for a front end cache.  When we updated the cache during heavy traffic we were getting lots of 403 errors within the first 30 seconds or so. We since scrapped that architecture. 

The thing that bothers me the most about this is the error message itself. If the object is not ready, the method should continue to poll until it becomes ready. We had nagios and zabbix alerts going off like wildfire chasing a security problem that wasn't.

@Google Ad Manager API team : This issue requires more investigation and more importantly - do not throw a 403 unless it is really a 403.  

James 

[ib...@cafemedia.com]

Same problem here. Sporadic 403 errors from very similar report queries issued by the same client.

I totally agree with James Tweedle above. This is a bogus 403 error.

[Oded Shafran]

We are having 403 as well when downloading the report from Google Storage

[Ad Manager API Forum Advisor Prod]

Hi everyone,

 

We're currently investigating your concern. Rest assured that we'll reach out to you once we have more information.

 

Thanks,

Kevin Soriano, Ad Manager API Team

ref:_00D1U1174p._5004Q22ag6r:ref

[Nepomuk Seiler]

Hi,

Just want to confirm that we see this issue as well. 

• We are using the Java SDK
• The failing queries only differ in the ad unit selection
• Queries don't fail consistently - Sometimes rerunning it multiple times works, sometimes not
• There are no specific ad units that cause this issue. Rather it seems completely random which one fails

Why is there no status update here: https://ads.google.com/status ?

This is issues seems rather big as reporting is broken for over 4 days now. We are paying a lot of money for GAM 360
and an outage like this isn't even communicated, which is very unfortunated.

hopefully you find the issue soon,
Muki

[Ad Manager API Forum Advisor Prod]

Hi Muki,

 

I'm sorry for the inconvenience that this has caused. We are already working to fix this as soon as possible. I'll also discuss with our team if we can update the ads status dashboard so everyone can follow this for updates instead.

[Rustam]

Hi,

We get the same errors with our integrations too.

[Kueez Network]

We also suffer from this issue in the past 3 hours.

[Nepomuk Seiler]

Hi,

So retrying works with sufficient backoff (500 - 1000ms) and regenerating the download url.

This seems to be a wider problem with Google Storage as it seems. Our Google Data Studio reports are also

very laggy which are backed by Google Sheets.

cheers,

Muki

[Ad Manager API Forum Advisor Prod]

Hi everyone,

I can confirm that we're aware of this issue and the reporting engineering team is working on a fix. This is an intermittent error as some of you have noticed, and retrying with a backoff will help mitigate this in the meantime. I'll post again once I have more information.

Chris Seeley Ad Manager API Team Share your feedback

ref:_00D1U1174p._5004Q22ag6r:ref

[hu...@tapclicks.com]

Hi,

We are experiencing the same problem since July 31st and while the re-try was working for a couple of times this morning. Now it's not working any more. Trying 5 times with a 10 second wait between re-tries

Hope this is properly fixed soon

Thanks,

[James Tweedie]

@hugo we were able to work around this issue with 4 tries and an incremental backoff with an initial 30 seconds and then 10 second increments. 

Others that are having this issue. It is important to understand the underlying issue is how the report data is handed off to Cloud Storage prior to the Download action. Each of our workloads will be different based on our use cases. For some this might be a longer wait time while others can tweak to reduce the overall workload.  

@Chris Sealy and other Googlers - to Muki's point earlier, we, and our customers pay some good money for GAM360. Why was there no alert sent on this? Why does it still show Green on the Ad Manager status[https://ads.google.com/status]?  Normally an outage of this magnitude requires a Severity 1 response model, which includes time to resolution and periodic updates. None of this happened and I think is right for the community to know why, regardless of whether the issue was surfaced through the community forum or through Google support channels.  We started off this conversation thinking there was something we did wrong or something our clients did wrong with their GAM configuration. 

Regards

James 

[Ad Manager API Forum Advisor Prod]

Hi everyone,

The fix has been rolled out now, please let me know if you're still running into this issue.

As James pointed out, the issue was related to the Cloud Storage Signed URLs that Ad Manager uses for downloading completed reports. Some of the URLs created by the getReportDownloadUrl methods were returning 403 errors. If you retried reading the URL, the error was consistent. Clients had to retry both the API call to getReportDownloadUrl and the actual download. 

The Ads Status Dashboard has been updated now, and I'll be following up with the reporting engineering team to determine what improvements can be made in preventing and detecting these issues going forward.

[James Tweedie]

@hugo the problem has been solved, as Chris noted, so you should have no issue. Chris raised an important distinction that I may have glossed over, and may be helpful for yourself and others. We wrap the entire request within a python method, meaning we recreate the entire call chain, except for the credentials,  and pass in the same request payload. This method is then wrapped and handled by the retry and incremental backoff code. This is also useful when dealing with usage quotas. Depending on the quota infraction you would want to adjust your incremental backoff accordingly or even adjust the retry ceiling. We have used this methodology with other ad platforms that, lets just say, are not as friendly with their quotas as GAM. A very handy tool in your toolkit for marketing api wrangling. 

Thanks Chris, Kevin, and others in the GAM API team,  in helping bring this to resolution. 

cheers 

James

[Nepomuk Seiler]

Thanks for fixing this and for your patience. I appreciate that you are doing some sort of post mortem on how to prevent this,

but it would also be helpful to update the status dashboard with the correct data:

https://ads.google.com/status#hl=en&v=issue&sid=24&iid=ffb7a789b2100d0d677b456eaaec58df

The outage started on 31.07.2020 and lasted until yesterday 03.08.2020. This doesn't give a lot of trust in the transparency how
things will be handled in the future. It would save a lot of head aches, time and frustration if things are communicated faster and
more transparent.

stay safe,

Muki

[Ad Manager API Forum Advisor Prod]

Those are definitely valid concerns Muki and ones that we'll be discussing on our side. Thank you for your patience here, I know this issue was difficult to work around.

[hu...@tapclicks.com]

thanks, works now

[Rustam]

I think that issue started over again. We are experiencing that issue in all of our GAM networks.

[erika.w...@rdbrck.com]

We are also seeing issues intermittently since 14:30 UTC. Generally, our retry is working, but the code was stable before this.

[ib...@cafemedia.com]

Confirming. I see 403 errors again.

[Ad Manager API Forum Advisor Prod]

Hi everyone,

We're aware of this and a fix is in on the way. I'll post again once it's been completely rolled out.

Chris Seeley Ad Manager API Team

Take our developer survey

ref:_00D1U1174p._5004Q22ag6r:ref

[Ad Manager API Forum Advisor Prod]

The fix is now live. Please let me know if you're still getting this error.

Chris Seeley Ad Manager API Team

Share your feedback

ref:_00D1U1174p._5004Q22ag6r:ref