Hi Jarrett,
Depending on how you go about shortening a URL, Bitly will crawl your URL to find various data about the page to display in our web UI. You can disallow our crawler by using an industry-standard robots.txt file (
http://www.robotstxt.org) at the root of your site. Our crawler identifies as "bitlybot", so if you wanted to block our crawler specifically you could set your robots.txt file to have the following:
User-Agent: bitlybot
Disallow: /
If you wanted to block all crawlers that respect robots.txt, you could instead have the following:
User-Agent: *
Disallow: /
Our crawler caches a site's robots.txt file for 24 hours, so it may take a day or so for our crawler to look for your new or edited robots.txt and take it into account. That said, there are crawlers out there that do NOT respect robots.txt, and some folks who do crawl Bitlinks, which are not hard to discover by random walk from a known starting point. A better strategy is to architect your web application so that HTTP GETs don't trigger one-time effects. Instead, you should ideally be using POST method for those kind of tasks. But that's more ideal than real. Start with robots.txt and see if that takes care of the issue. I suspect you'll find that various crawlers will hit those URLs at some point anyway, though likely not before they're triggered by the intended user.
Regards,
---Peter
Peter Herndon
Sr. Application Engineer
@Bitly