Issue 879 in google-api-java-client: Invalid_request from credential.refreshToken()

[google-api-...@googlecode.com]

Status: New
Owner: wonder...@google.com
Labels: Type-Defect Priority-Medium

New issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Version of google-api-java-client (e.g. 1.15.0-rc)?
1.18.0

Java environment (e.g. Java 6, Android 2.3, App Engine)?
Java 7 with AppEngine

Describe the problem.
We're seeing a lot of invalid_request errors happening ( stack trace below
). We opened a ticket with the Google API team and they said it was an
issue with the java client library and to post here. Below is there
response for this and the stack trace for the error.

--- GOOGLE SUPPORT ---
Hello Nathan,

Thank you for your response. I do apologize due to the amount of time that
has taken for us to get back to you. We have isolated the issue and it
seems that it is being caused by the API Client Library that is being used.
Unfortunately, we do not offer support for client library issues through
this support portal.

Simply post your questions on the issue tracker maintained by the Client
library team
(http://code.google.com/p/google-api-java-client/downloads/list), a member
of our global developer community will respond shortly.

NOTE: make sure you list the correct client libraries issue tracker for
your user (Python, Java, Etc.)
--- END GOOGLE SUPPORT ---
--- STACK TRACE ---
com.google.api.client.auth.oauth2.TokenResponseException: 403 OK
{
"error" : "invalid_request"
}
at
com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at
com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at
com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at
com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:268)
at
com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)
--- END STACK TRACE ---

How would you expect it to be fixed?
NA



--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings

[google-api-...@googlecode.com]

Comment #1 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for posting the issue. We are investigating and will get back to you
shortly. By the way, were you running your application on a large number of
AppEngine instances simultaneously?

[google-api-...@googlecode.com]

Comment #2 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for the quick response. No, we do run some things simultaneously
but we get this on just one off user actions ( user clicking a button in
our application ). I can send you over a request / response if that will
assist in narrowing down the issue.

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #3 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Here is an example of a failure and a successful call below. I commented
out some of the codes that are sent back and forth for security reasons.


ERRORED

CONFIG: -------------- REQUEST --------------
POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 568

May 19, 2014 3:45:14 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -X POST -H 'Accept-Encoding: gzip'
-H 'User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)' -H 'Content-Type:
application/x-www-form-urlencoded; charset=UTF-8' -d '@-'
-- 'https://accounts.google.com/o/oauth2/token' << $$$
May 19, 2014 3:45:14 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 568 bytes
May 19, 2014 3:45:14 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG:
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
<< commented out for security
May 19, 2014 3:45:14 PM
com.google.apphosting.utils.jetty.AppEngineAuthentication$AppEngineAuthenticator
authenticate
FINE: Got /_ah/channel/dev, returning NOBODY to imply authentication is in
progress.
May 19, 2014 3:45:14 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
403 OK
content-type: application/json; charset=utf-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Fri, 01 Jan 1990 00:00:00 GMT
date: Mon, 19 May 2014 15:45:00 GMT
content-disposition: attachment; filename="json.txt";
filename*=UTF-8''json.txt
content-encoding: gzip
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: GSE
alternate-protocol: 443:quic
transfer-encoding: chunked

May 19, 2014 3:45:14 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 33 bytes
May 19, 2014 3:45:14 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: {
"error" : "invalid_request"
}





GOOD ONE
CONFIG: -------------- REQUEST --------------
POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip
User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 568

May 19, 2014 3:59:05 PM com.google.api.client.http.HttpRequest execute
CONFIG: curl -v --compressed -X POST -H 'Accept-Encoding: gzip'
-H 'User-Agent: Google-HTTP-Java-Client/1.15.0-rc (gzip)' -H 'Content-Type:
application/x-www-form-urlencoded; charset=UTF-8' -d '@-'
-- 'https://accounts.google.com/o/oauth2/token' << $$$
May 19, 2014 3:59:05 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 568 bytes
May 19, 2014 3:59:05 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG:
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
<< commented out for security
May 19, 2014 3:59:05 PM
com.google.apphosting.utils.jetty.AppEngineAuthentication$AppEngineAuthenticator
authenticate
FINE: Got /_ah/channel/dev, returning NOBODY to imply authentication is in
progress.
May 19, 2014 3:59:05 PM com.google.api.client.http.HttpResponse <init>
CONFIG: -------------- RESPONSE --------------
200 OK
content-type: application/json; charset=utf-8
cache-control: no-cache, no-store, max-age=0, must-revalidate
pragma: no-cache
expires: Fri, 01 Jan 1990 00:00:00 GMT
date: Mon, 19 May 2014 15:58:51 GMT
content-disposition: attachment; filename="json.txt";
filename*=UTF-8''json.txt
content-encoding: gzip
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
server: GSE
alternate-protocol: 443:quic
transfer-encoding: chunked

May 19, 2014 3:59:05 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: Total: 120 bytes
May 19, 2014 3:59:05 PM
com.google.api.client.util.LoggingByteArrayOutputStream close
CONFIG: {
"access_token" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX", <--
replaced for security.
"token_type" : "Bearer",
"expires_in" : 3600

[google-api-...@googlecode.com]

Comment #4 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Great. Thanks.

[google-api-...@googlecode.com]

Comment #5 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Looks like you were using google-http-java-client 1.15.0-rc. Did you try
with the latest version, ie., 1.18.0-rc?

[google-api-...@googlecode.com]

Comment #6 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Sorry about that. That was from other testing. Here are some from 1.18.

com.google.api.client.http.HttpRequest execute: curl -v --compressed -X

POST -H 'Accept-Encoding: gzip' -H 'User-Agent:

Google-HTTP-Java-Client/1.18.0-rc (gzip)' -H 'Content-Type:

application/x-www-form-urlencoded; charset=UTF-8' -d '@-'
-- 'https://accounts.google.com/o/oauth2/token' << $$$

D2014-05-19 11:49:45.370
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826
bytes
D2014-05-19 11:49:45.370
com.google.api.client.util.LoggingByteArrayOutputStream close:
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:45.463
com.google.api.client.http.HttpResponse <init>: -------------- RESPONSE

-------------- 403 OK content-type: application/json; charset=utf-8
cache-control: no-cache, no-store, max-age=0, must-revalidate pragma:
no-cache expires: Fri, 01 Jan 1990 00:00:00 GMT date: Mon, 19 May 2014

15:49:45 GMT content-disposition: attachment; filename="json.txt";

filename*=UTF-8''json.txt content-encoding: gzip x-content-type-options:
nosniff x-frame-options: SAMEORIGIN x-xss-protection: 1; mode=block server:

GSE alternate-protocol: 443:quic x-google-cache-control: remote-fetch via:
HTTP/1.1 GWA
D2014-05-19 11:49:45.463

2014-05-19 11:49:45.476
com.google.api.client.http.HttpRequest execute: -------------- REQUEST
-------------- POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip User-Agent: Google-HTTP-Java-Client/1.18.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 826
D2014-05-19 11:49:45.477
com.google.api.client.http.HttpRequest execute: curl -v --compressed -X

POST -H 'Accept-Encoding: gzip' -H 'User-Agent:

Google-HTTP-Java-Client/1.18.0-rc (gzip)' -H 'Content-Type:

application/x-www-form-urlencoded; charset=UTF-8' -d '@-'
-- 'https://accounts.google.com/o/oauth2/token' << $$$

D2014-05-19 11:49:45.477
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826
bytes
D2014-05-19 11:49:45.477
com.google.api.client.util.LoggingByteArrayOutputStream close:
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:45.559

D2014-05-19 11:49:46.574
com.google.api.client.http.HttpRequest execute: -------------- REQUEST
-------------- POST https://accounts.google.com/o/oauth2/token
Accept-Encoding: gzip User-Agent: Google-HTTP-Java-Client/1.18.0-rc (gzip)
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Content-Length: 826
D2014-05-19 11:49:46.574
com.google.api.client.http.HttpRequest execute: curl -v --compressed -X

POST -H 'Accept-Encoding: gzip' -H 'User-Agent:

Google-HTTP-Java-Client/1.18.0-rc (gzip)' -H 'Content-Type:

application/x-www-form-urlencoded; charset=UTF-8' -d '@-'
-- 'https://accounts.google.com/o/oauth2/token' << $$$

D2014-05-19 11:49:46.574
com.google.api.client.util.LoggingByteArrayOutputStream close: Total: 826
bytes
D2014-05-19 11:49:46.575
com.google.api.client.util.LoggingByteArrayOutputStream close:
grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer&assertion=
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX<<---REMOVED FOR SECURITY
D2014-05-19 11:49:46.651

[google-api-...@googlecode.com]

Comment #7 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Also, here is the stack trace from 1.18

com.google.api.client.auth.oauth2.Credential handleResponse: unable to
refresh token

com.google.api.client.auth.oauth2.TokenResponseException: 403 OK
{
"error" : "invalid_request"
}
at
com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
at
com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
at
com.google.api.client.auth.oauth2.TokenRequest.execute(TokenRequest.java:307)
at
com.google.api.client.googleapis.auth.oauth2.GoogleCredential.executeRefreshToken(GoogleCredential.java:268)
at
com.google.api.client.auth.oauth2.Credential.refreshToken(Credential.java:489)

at
com.google.api.client.auth.oauth2.Credential.handleResponse(Credential.java:272)
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:999)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:410)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:343)
at
com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:460)

[google-api-...@googlecode.com]

Comment #8 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

If you need anything else just let me know. We're seeing these errors
consistently throughout the day so I can get you more debug info if you
need.

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #9 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

FYI, not sure if you can see logs or anything, but were getting this error
now with the domain gpanel.info.

[google-api-...@googlecode.com]

Comment #10 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Mike, What's the email address of the service account used to create the
assertion? That would be helpful for our debugging.

[google-api-...@googlecode.com]

Comment #11 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Can you please shoot me an email and I'll respond to you with the service
account, I'd rather not post it so everyone can see it.

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #12 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Just responded to your email. If you need anything definitely let us
know. We're seeing this error a lot lately.

[google-api-...@googlecode.com]

Comment #13 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Also, not sure if you can get into the Google support portal, but the case
that we had opened up and referred us here is 04248873.

[google-api-...@googlecode.com]

Comment #14 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Just another FYI. If you can see the backend logs ( not sure if you can or
not ), we're seeing a lot of invalid_requests at the moment.

[google-api-...@googlecode.com]

Comment #15 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Just wondering if you've had any luck in replicating the issue?

[google-api-...@googlecode.com]

Comment #16 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

We've got some results but are still investigating. Stay tuned.

[google-api-...@googlecode.com]

Comment #17 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Awesome. Is it something that we are doing wrong or is it something on
Google's side of things? If there is anything you would like us to try
definitely let me know.

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #18 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Hi Mike, we took a dump of your recent requests history and it turns out
that you were sending 100+ requests in 5 minutes, which apparently exceeds
our quota limit for number of active access tokens. Are you having a
service account impersonating a large number of users?

[google-api-...@googlecode.com]

Comment #19 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Hi, well we use the same service account ( the one that I emailed you ) for
accessing resources on different domains ( this is an apps marketplace
application ). I thought the limit for tokens was on a per user basis (
not a service account basis )?

[google-api-...@googlecode.com]

Comment #20 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

That explains. When using service accounts, the requests are sent on behalf
of that particular service account, instead of end users, and thus the
limit is per service account basis. You can read more on
https://developers.google.com/accounts/docs/OAuth2ServiceAccount.

We are making changes to our client library so that it will return "Rate
limit exceeded." when this happens and do an exponential backoff
accordingly. At the server side, we are also working on a project that will
remove this rate limit completely and you can expect this to be done pretty
soon.

[google-api-...@googlecode.com]

Comment #21 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for the update. We've got a big problem then. We have hundreds of
domains that actively use our product ( you can find our product in the
chrome web store and in the apps marketplace ). So are you saying that if
100 of those domains login and take an action in 5 minutes, we'll see the
invalid_requests? Or is it just 100 requests by a service account within 5
minutes ( no matter the number of different domains )?

Do you know of a work around? Can we request a quota increase so that we
won't hit this error until the server project you mentioned is put in place
so we don't have to worry about the rate limit?

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #22 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Hi Mike, from the logs, we suspect that you were using service account with
impersonation. With impersonation, token requests are sent on behalf of a
particular domain, specified by an email address, instead of your service
account. The rate limit is per domain basis in that case. The domain that's
sending most of the requests is admini...@gpanel.info. You probably
want to balance your requests among all domains. You may want to read
the "Additional claims" section here:
https://developers.google.com/accounts/docs/OAuth2ServiceAccount#formingclaimset

[google-api-...@googlecode.com]

Comment #23 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for the info. So does this refer to "api requests" or "token
requests". for example, if I make an api call to google, does that count
towards the rate limit or does the rate limit just get impacted when we
call credential.refresh, or get a new token?

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #24 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

This rate limit is for "token requests", i.e., when you call
credential.refresh(). You would have another rate limit for api request but
that's something you can config/raise through Google Cloud Console.

[google-api-...@googlecode.com]

Comment #25 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for the quick response. So just to confirm, if we request on behalf
of different users within the same domain, that still counts towards
the "domain rate limit" correct?

Thanks again,
Mike

[google-api-...@googlecode.com]

Comment #26 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

I should probably have used the accurate term. Impersonation works for a
particular "user" specified by an email address, through the "sub" field of
a JWT.

[google-api-...@googlecode.com]

Comment #27 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Thanks for the clarification. So basically if a domain has 100 users on
our application, and they all choose to view the documents ( using the
drive service ) for their own accounts using our application at the same
time, we're going to hit the limit? because that would be 100 ( or however
many ) requests to get a token and that would count to the domain that they
are all on. Am I correct in saying this?

( sorry, I'm just trying to understand exactly what will cause the limit so
we can figure out a way to fix this and not get this error )

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #28 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

And FYI, I used drive specifically because the only way to get the drive
files for a specific user is to specify the setServiceAccountUser as the
user you want to retrieve the files for ( this is similar for other
services as well ).

[google-api-...@googlecode.com]

Comment #29 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

No, this limit is per user basis. So if you have 100 users within one
domain, and you impersonate them differently(passing different user emails
to setServiceAccountUser()), you won't hit that limit as long as you don't
send requests for one user too quickly.

I see you sent a lot of token requests for user admini...@gpanel.info,
you probably want to slow down a little bit.

[google-api-...@googlecode.com]

Comment #30 on issue 879 by mike.sch...@promevo.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

Hello,

We've made some changes last night and this morning. Can you compare the
requests that you were seeing yesterday with what we're requesting now and
see if there is a big difference?

Thanks,
Mike

[google-api-...@googlecode.com]

Comment #31 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
http://code.google.com/p/google-api-java-client/issues/detail?id=879

We still see a lot of token requests for admini...@gpanel.info. Do you
have a log at your side that can help us debugging? What was your use case
and why were you sending so many requests on behalf of this user?

[google-api-...@googlecode.com]

Comment #32 on issue 879 by wonder...@google.com: Invalid_request from
credential.refreshToken()
https://code.google.com/p/google-api-java-client/issues/detail?id=879

Is this still an issue?