Search API to require HTTP Referrer and/or User Agent

[Doug Williams]

Hi all,
The Search API will begin to require a valid HTTP Referrer, or at the very least, a meaningful and unique user agent with each request. Any request not including this information will be returned a 403 Forbidden response code by our web server.

This change will be effective within the next few days, so please check your applications using the Search API and make any necessary code changes.

Thanks,
Doug

[Matt Sanford]

Hi all,

Let me clarify a bit. For server-side processing please set the
User-Agent header. I recommend using your domain name, or if you don't
have one (which is odd) your appname. Something like "myapp.com" or
"myapp". By using domain name we'll be able to check out the site and
reach out to contact people if we suspect them of abuse. Spammers
often don't respond to questions from the services they abuse, and if
someone is using your user-agent falsely you'll have the possibility
of saying "That's not me, I'm not on app engine". For client-side
processing like TweetGrid the browser will send a User-Agent and
referrer unless you're doing something exceedingly odd, so you should
be fine.

This change is mostly to combat an increasing amount of spam coming
from "cloud" services like ecs and appengine. At first we'll only be
applying this restriction to those IP addresses but it may need to be
broadened as time goes on. If you're writing client software please
add a user-agent in case we end up having to widen this in the future.
This seems like a better plan than the Media Temple fiasco we went
though last time we blocked a shared service for hosting spammers [1].

Thanks;
– Matt Sanford / @mzsanford
Twitter Dev

[1] - https://twitter.com/mzsanford/status/1924718435