Recent Increase of 403 Errors

153 views
Skip to first unread message

John White

unread,
Feb 8, 2016, 9:29:53 PM2/8/16
to Google App Engine
The last two weeks I've seen an uptick of 403 responses come back from my endpoint.  It's been random.  At first it was only every once every hour or so.  Then over the last two weeks the 403 responses increased more and more until tonight things are almost unusable.  I can't get through to my endpoint and the data within datastore at all.

Two different apps access the same endpoint.  One is a JavaFX app (JDK 8) developed in NetBeans.  The other is an Android app (JDK 7) being developed in Android Studio.  Here's the specifics of the GAE and endpoints version I'm using, taken from my build.gradle file of the endpoint backend within Android Studio:
com.google.appengine:appengine-java-sdk:1.9.14
com.google.appengine:appengine-endpoints:1.9.14
com.google.appengine:appengine-endpoints-deps:1.9.14

Like I said, the 403 has been happening intermittently and increasing in volume, and it's happening during both read and write operations.  The response I get is very generic.  I read somewhere in the help docs here that the returned errors from the endpoint can sometimes return more detail.  Would any one know how to capture that?  This is all I'm getting in my console:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
<HTML>
<HEAD>
<TITLE>Forbidden</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Forbidden</H1>
<H2>Error 403</H2>
</BODY>
</HTML>

	at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:145)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
	at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:312)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1056)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
	at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
	at com.white.chordkendev.services.PlayFieldViewService$1.call(PlayFieldViewService.java:46)

----or, something like----

02-08 20:45:51.877    6296-6596/com.nessylum.sudochord E/PlayFieldService﹕ findPlayFieldViews
    com.google.api.client.googleapis.json.GoogleJsonResponseException: 403 Forbidden
    <HTML>
    <HEAD>
    <TITLE>Forbidden</TITLE>
    </HEAD>
    <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
    <H1>Forbidden</H1>
    <H2>Error 403</H2>
    </BODY>
    </HTML>
            at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
            at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
            at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:312)
            at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1049)
            at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
            at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
            at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)
            at com.nessylum.sudochord.service.PlayFieldServiceImpl.findPlayFieldViews(PlayFieldServiceImpl.java:123)


I had absolutely no problems of this kind before a couple weeks ago.  I haven't changed anything in the endpoint code within that time, nor anything in the JavaFX app's code.  Things are working fine when I debug against the endpoint deployed locally.  Sorry if I didn't provide enough detail, this is all I've been able to find so far.  Is anyone else having this problem?

Saket Kumar

unread,
Feb 9, 2016, 5:23:35 AM2/9/16
to Google App Engine
Hi,

We are aware of an increase in 403 errors for Cloud Endpoints starting 1300 (US/Pacific), 02/08. We are currently rolling out a fix that should be available in next few hours.


Thanks!
Saket Kumar
Google Cloud Platform Support.

Ryan Fagan

unread,
Feb 9, 2016, 10:07:54 AM2/9/16
to Google App Engine
We are seeing the 403 on all sites on every request to cloud endpoints at this time.

Saket Kumar

unread,
Feb 9, 2016, 10:11:38 AM2/9/16
to google-a...@googlegroups.com
Hi Ryan,

Can you please send me the App Id privately? This would let us verify if these 403s are related.

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/3ZIbuw_ppNc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/13361456-e974-4a19-80b2-6ee8051ed491%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Saket
Message has been deleted

Saket Kumar

unread,
Feb 9, 2016, 12:20:10 PM2/9/16
to google-a...@googlegroups.com
Thanks. This problem should have been resolved ~30 mins ago.

On 9 February 2016 at 16:50, Ryan Fagan <ry...@gannettpeaktech.com> wrote:
This appears resolved for us at this time.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/78dcb728-4c20-416f-a538-b065f1ceae98%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Saket

Ryan Fagan

unread,
Feb 9, 2016, 12:21:46 PM2/9/16
to Google App Engine
Agreed, thank you.

Ryan
Message has been deleted

John White

unread,
Feb 9, 2016, 9:04:44 PM2/9/16
to Google App Engine
Thank you for your help, gentlemen.  It appears resolved for me, as well. Both apps are communicating with the backend fine.

On Tuesday, February 9, 2016 at 12:21:46 PM UTC-5, Ryan Fagan wrote:
Agreed, thank you.

Ryan
Reply all
Reply to author
Forward
0 new messages