twitter API suddenly returning error 32 “Could not authenticate you.” only in prod GAE

[Ryan Barrett]

hi all! a number of us have started seeing twitter API errors in prod GAE in the last week or so. write calls and some search calls are returning HTTP 401 {"errors":[{"code":32,"message":"Could not authenticate you."}]} for all users with no code changes on our end. the particularly odd part is that the same calls with the same code, twitter app key/secret, and even the exact same user access token key/secret work fine in dev_appserver.

we're on python and mostly using tweepy. our current theory is that twitter has blacklisted or graylisted app engine's external facing IPs. can you think of any other ideas?

more details in this twitter dev forum post. thanks in advance!

[David Fischer]

I'm seeing the same problem on Appengine production (and not with dev_appserver) and I'm using the same Twitter library. However, I was seeing some tweets succeeding and some failures. After another user in the Twitter Dev forum posted, I looked at the content of the tweets. The failing tweets all contained the "!" character. After removing that character from tweets, I'm seeing a 100% success rate.

My working theory is that urlfetch is somehow changing how parameters are escaped and that's causing the HMAC in Twitter's Oauth to be rejected.

[Dave Loomer]

I am reaching similar conclusions as David (I am @kidneybingos on that Twitter dev thread). I have tried lots of punctuation characters, and most succeed, but the following always cause errors:

' (apostrophe)

!

*

(

)

My conclusion is similarly that urlfetch is encoding something different as of a week ago. I, like others in the thread, had been running the exact same code with 100% success prior to then, and had been issuing these API calls hundreds of times per day for a few years.

[Dave Loomer]

And just to reiterate - the characters seem to encode correctly (e.g. apostrophe encodes to %27), and at any rate, when I log the URLs generated by Tweepy the URLs generated on my dev instance are identical to prod. Yet it works in dev, not in prod. My guess is something is happening in the urlfetch code which we can't debug in prod.

[Dave Loomer]

All: I won't have time to do this in the next several hours (my app engine app is not my day job), but I think tonight I'll try changing the api.twitter.com endpoint to point to a URL I own, and then examine the URLs / query strings and headers received from my prod vs. dev code to see if the encoding is any different.

I figure that could help point to whether it's a urlfetch issue. Only other thing I can think of, if it's not urlfetch, is that the oauth signatures are being generated differently in prod vs. dev, due to a difference in some built-in python library on the prod server, and that will be much more difficult to debug since the timestamp and nonce prevent you from comparing dev vs prod requests character-by-character.

[Dave Loomer]

(what I was getting it with the previous post is: if any of you need this for your day jobs, you are welcome to try the same yourselves in the meantime ;) )

[Ryan Barrett]

thanks for the comprehensive debugging, Dave and David! it definitely sounds like new header munging on app engine's part may be the problem. Dave, your idea of sending the API calls to a different endpoint and examining the headers sounds like a great lead. i'm looking forward to the results!

as another data point, there may be more to it than just punctuation characters. i see the error with pretty much any tweet content and username. i reproduced it just now by attempting to tweet "foo" (just those three characters) as @schnarfed. sigh.

[Dave Loomer]

Well bad news, the URL and headers received from my hosted app from a Tweepy GET request were identical in prod vs. dev. We might need to go back to attacking from the Twitter side to see if they can explain why they're rejecting our API requests.

I realized when requesting from a dummy URL I didn't need to worry about changing nonce and timestamp; I just hard-coded them (to realistic values) to see if the oauth signatures were the same. And they were. So I have failed to identify what these punctuation characters have to do with the authentication error.

Caveat: the signatures are the same, but the 'Authorization' request header is slightly different; its elements are ordered differently in prod vs dev for whatever reason. But I'm not really concerned because the ordering is the same regardless of whether I use the punctuation; i.e. dev and prod Authorization headers differ in the same way even for requests that would return a successful response from the Twitter API.

To see what I'm talking about, here is the Authorization request header in production:

'OAuth realm="", oauth_nonce="12345678", oauth_timestamp="1430361562", oauth_consumer_key="xxxxxxxx", oauth_signature_method="HMAC-SHA1", oauth_version="1.0", oauth_token="yyyyyyyyyyyyy", oauth_signature="AiqwN51QaZnwsBkikcVh9rENc84%3D"'

And here it is in dev:

'OAuth realm="", oauth_token="yyyyyyyyyyyyy", oauth_nonce="12345678", oauth_timestamp="1430361562", oauth_signature="AiqwN51QaZnwsBkikcVh9rENc84%3D", oauth_version="1.0", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="xxxxxxxx"'

These are for the query "dogs & cat)s" which I confirmed returns the authentication error from prod (but not dev). In each case the URL requested of my hosted app is:

/1.1/users/search.json?q=dogs%20%26%20cat)s

Note that the ) is not encoded. However, that remains the case when calling from dev, so I don't see why this could be the cause of the problem.

[Waleed]

Thanks, Dave. I did a similar test but I got different results. I used an ssh tunnel and tcpdump to monitor the request coming from App Engine production and dev server. In both cases, my tweet text was: "Test!'*". This is what I got:

Request coming from App Engine Production:

POST /home/wizards?status=Test!'*&in_reply_to_status_id= HTTP/1.1

Host: 173.203.209.168

Content-Type: multipart/form-data; boundary=----------bundary------

Authorization: OAuth oauth_version="1.0", oauth_token="41002916-JMWvFP2KG920Gq0pkybnPhhQwTOph7Craz4BUp", oauth_nonce="79779727", oauth_timestamp="1430365200", oauth_signature="8jTYjCYZig8pwfnc6rd1RUdc%3D", oauth_consumer_key="jTkKoDhBrAPGCuOvkBw", oauth_signature_method="HMAC-SHA1"

X-Cloud-Trace-Context: ChIJlNDTg-mu4-0RHx_ztO5m71wRzwsjCEJb3XMYAA

Content-Length: 110478

Connection: Keep-alive

Accept-Encoding: gzip,deflate

User-Agent: AppEngine-Google; (+http://code.google.com/appengine; appid: s~crowdc)

Request coming from App Engine Dev Server:

POST /home/wizards?in_reply_to_status_id=&status=Test%21%27%2A HTTP/1.1

Content-Length: 1024214

Accept-Encoding: gzip

User-Agent: AppEngine-Google; (+http://code.google.com/appengine)

Host: 173.203.209.168

Content-Type: multipart/form-data; boundary=----------bundary------

Authorization: OAuth oauth_token="41002916-JMWvFP2KG920Gq0pkybnPhhQwTOph7Craz4BUp", oauth_timestamp="1430365345", oauth_consumer_key="jTkKoDhBrAPGCuOvkBw", oauth_nonce="01368662", oauth_signature_method="HMAC-SHA1", oauth_version="1.0", oauth_signature="%2B0xFtIJZBBtCsLduLfYA19wY%3D"

Looks like requests coming from production are not being encoded with %. Not sure why or how to get around it.

[Waleed]

I simplified the test to one line:

urlfetch.fetch("http://173.203.209.168/home/wizards?status=Test%27", method="POST")

On production, I receive:  

POST /home/wizards?status=Test' HTTP/1.1

And in local dev server, I receive:

POST /home/wizards?status=Test%27 HTTP/1.1

[Dave Loomer]

Interesting. Perhaps the difference is that I tested only GET and not POST (still, I get behavioral differences between dev and test with GET, even if the request looks the same in my tests).

Or, maybe your method uncovers something that mine does not. Could you try a GET (maybe using the search API) to see if there's a difference?

--
You received this message because you are subscribed to a topic in the Google Groups "Google App Engine" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/google-appengine/6ILDt39anbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/69cc83ab-30ce-48d3-a27b-fff113f44623%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Barrett]

On Wednesday, April 29, 2015 at 11:29:34 AM UTC-7, Ryan Barrett wrote:

as another data point, there may be more to it than just punctuation characters. i see the error with pretty much any tweet content and username. i reproduced it just now by attempting to tweet "foo" (just those three characters) as @schnarfed. sigh.

sorry guys, disregard this, i was mistaken. i was accidentally including parentheses in the tweet text after all. tweeting just 'foo' does work for me, so it probably is indeed the difference in URL-encoding those characters.

[Waleed]

Another update. It seems that the issue is happening in App Engine release 1.9.20. Some of the instances in my app are on this release, and others are on the previous 1.9.19. We didn't a full scan, but based on manual inspection it seems that all the errors are happening on requests that are on the 1.9.20 release. 

[Ryan Barrett]

On Thursday, April 30, 2015 at 1:10:23 PM UTC-7, Waleed wrote:

Another update. It seems that the issue is happening in App Engine release 1.9.20. Some of the instances in my app are on this release, and others are on the previous 1.9.19. We didn't a full scan, but based on manual inspection it seems that all the errors are happening on requests that are on the 1.9.20 release. 

nice sleuthing! i can confirm that app engine 1.9.19 doesn't exhibit this behavior. my app was rolled back to 1.9.19 earlier today, and i'm now able to post a tweet with the problematic characters again.

[Ryan Barrett]

...on an unrelated note, 1.9.20 was *way* better with utilization and scheduling instances efficiently for my app. app engine team, i hope that part survives!

[Jesse Scherer (Google Cloud Support)]

Hey guys, great sleuthing indeed!

I think this is related to another issue which was reported yesterday, https://code.google.com/p/googleappengine/issues/detail?id=11917

I'm going to add a summary of this thread to that issue, and I suggest you all "star" the issue, both to be copied on updates and questions from the engineers, and to underscore the impact of it.

[Waleed]

My app was also rolled back to 1.9.19 yesterday. Then later it was back on 1.9.20, but the error seems to have been fixed. I also got email confirmations from customers that they're not seeing this problem any more. Great team work. 

[Ryan Barrett]

On Friday, May 1, 2015 at 4:13:03 PM UTC-7, Waleed wrote:

My app was also rolled back to 1.9.19 yesterday. Then later it was back on 1.9.20, but the error seems to have been fixed. I also got email confirmations from customers that they're not seeing this problem any more. Great team work. 

confirmed! same with my app. thanks for the heads up!