Just started getting HTTP_REFERER problem

[todd]

Howdy,

In the last few days my Twitter API calls started failing with this
error:

403 Forbidden: The server understood the request, but is refusing to
fulfill it. If you are posting from an API tool please ensure that the
HTTP_REFERER header is not set.

To my knowledge I'm not setting the referer. In my search of this
group it seemed all the posts related to referer were in 2007. My code
has been working since then so something changed. Anything changed on
the Twitter side? This code runs on Google AppEngine so I suppose it
could be something on their end. My guess is it is probable in GAE
because the code seems to work in their local test fixture, but I just
want to check.

Any thoughts on what might be happening or what I can do about it?

thanx

class SendChime(webapp.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/plain'
username = self.request.get("username")

logging.debug("Sending chime for user=" + username)
login = username;
password = password"
chime = self.get_chime()
payload= {'status' : chime,
'source' : "innertwitter"}
payload= urllib.urlencode(payload)

base64string = base64.encodestring('%s:%s' % (login, password))
[:-1]
headers = {'Authorization': "Basic %s" % base64string}

url = "http://twitter.com/statuses/update.xml"
result = urlfetch.fetch(url, payload=payload,
method=urlfetch.POST, headers=headers)

self.response.out.write(result.content)

[Ed Finkler]

Try using a proxy tool like Charles and capturing *exactly* what
you're sending to the server, and what it's returning. That's going to
be a lot more useful than your source code in this case.

--
Ed Finkler
http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[todd]

That is good advice Ed, thanks. I was just hoping there was something
obvious in the code.

Using Sir Charles here is what I found:

POST /statuses/update.xml HTTP/1.1
Authorization: Basic xxxxxxxx
User-Agent: AppEngine-Google; (+http://code.google.com/appengine)
Referer: http://xxx.appspot.com/
Host: x.x.x.x
Content-Length: 88
Accept-Encoding: gzip


status=........%2Achime%2A..............Thu+Jun+
+5+22%3A05%3A50+2008&source=innertwitter


This seems to be a recent change in GAE and GAE doesn't allow referer
to be set. Is the canonical solution still to write another command to
execute the twitter command?

thanx

On Jun 3, 4:00 pm, "Ed Finkler" <funkat...@gmail.com> wrote:
> Try using a proxy tool like Charles and capturing *exactly* what
> you're sending to the server, and what it's returning. That's going to
> be a lot more useful than your source code in this case.
>
> --

> Ed Finklerhttp://funkatron.com

> AIM: funka7ron
> ICQ: 3922133
> Skype: funka7ron
>

[James Strachan]

I've just hit this too (thanks for figuring this out todd!)

I just wondered why twitter returns a 403 if the Referer is set? As
this kinda means we can't post to twitter from Google's AppEngine.
Couldn't twitter just ignore the Referer header? :)

[James Strachan]

I just spotted this thread...

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/ebf637d1ea971247/bb8c1c9c23901ecb?lnk=gst&q=referer#bb8c1c9c23901ecb

I wonder if certain domains could be allowed as referrers? e.g. could
we allow http://*.appspot.com/ as values for the Referer header - then
folks could post to twitter from Google's AppEngine?

[Ryan W]

I just discovered this too, and am not too thrilled to have wasted the
evening trying to post to twitter from AppEngine. It all worked fine
in dev, then deploy and bam, shot down. The issue with allowing
*.appspot.com is that a lot of GAE are running on real domains using
Google Apps. Perhaps allowing the AppEngine user-agent would be
another possibility? What's the latest response from twitter?

On Jun 10, 4:04 am, James Strachan <james.strac...@gmail.com> wrote:
> I just spotted this thread...
>

> http://groups.google.com/group/twitter-development-talk/browse_thread...

[Evan Weaver]

No one has given me any reason why referrer is disallowed, so I will
remove the constraint.

Evan

--
Evan Weaver

[Ed Finkler]

If you do, I wonder if it would be best to ensure that you *cannot*
post via GET. Otherwise I think a CSRF attack would be really simple.

(One could still trick someone into doing a POST CSRF via a form, but
it would hook a lot less people).

--
Ed Finkler

http://funkatron.com
AIM: funka7ron
ICQ: 3922133
Skype: funka7ron

[Ryan W]

Thanks Evan! Glad to hear it.

[Evan Weaver]

Okay, I think it's there for cross site request forgery prevention.
I'm not sure that it helps, because I think you can remove the
referrer with a double meta refresh.

Nevertheless, would it be appropriate to limit the method to only
basic auth requests?

Evan

--
Evan Weaver

[Ryan W]

Not sure I understand your question, nor whether I'm in a position to
lend much weight to security implications. But, if Twitter doesn't
need the referer for anything, I'm still not sure why it cares about
it. Which method are you referring to? I was under the impression
all API interaction was done using basic auth, and in my case, it's
what I'm using to post from app engine.

But, anyway, I'm still hoping there's some way we can post to twitter
from app engine.

[Ryan W]

Any updates on this?

[todd]

It seems to work now.

[todd]

Apparently I was wrong about this. I'm getting the problem again:

add_post_to_profile: 403 Forbidden: The server understood the request,

but is refusing to fulfill it. If you are posting from an API tool
please ensure that the HTTP_REFERER header is not set.

Please pretty please would it be possible to remove this restriction?

thanx

On Jul 26, 12:31 pm, todd <t...@possibility.com> wrote:
> It seems to work now.
>
> On Jul 25, 4:43 pm, Ryan W <rwilli...@gmail.com> wrote:
>
> > Any updates on this?
>
> > On Jul 15, 3:09 pm, Ryan W <rwilli...@gmail.com> wrote:
>
> > > Not sure I understand your question, nor whether I'm in a position to
> > > lend much weight to security implications.  But, if Twitter doesn't

> > > need therefererfor anything, I'm still not sure why it cares about

> > > > >> >> we allow http://*.appspot.com/ as values for theRefererheader - then

> > > > >> >> folks could post to twitter from Google's AppEngine?
>
> > > > >> >> On Jun 7, 6:37 am, James Strachan <james.strac...@gmail.com> wrote:
>
> > > > >> >> > I've just hit this too (thanks for figuring this out todd!)
>

> > > > >> >> > I just wondered why twitter returns a 403 if theRefereris set? As

> > > > >> >> > this kinda means we can't post to twitter from Google's AppEngine.

> > > > >> >> > Couldn't twitter just ignore theRefererheader? :)

> > > > >> >> > > > > To my knowledge I'm not setting thereferer. In my search of this
> > > > >> >> > > > > group it seemed all the posts related torefererwere in 2007. My code

[Alex Payne]

Absolutely. We'll lift this restriction starting with a deploy this week.

--
Alex Payne
http://twitter.com/al3x

[todd]

That will be sweet. Thanks.