link wrapping on the API

[Raffi Krikorian]

hi all.

twitter has been wrapping links in e-mailed DMs for a couple months now. with that feature, we're trying to protect users against phishing and other malicious attacks. the way that we're doing this is that any URL that comes through in a DM gets currently wrapped with a twt.tl URL -- if the URL turns out to be malicious, Twitter can simply shut it down, and whoever follows that link will be presented with a page that warns them of potentially malicious content. in a few weeks, we're going to start slowly enabling this throughout the API for all statuses as well, but instead of twt.tl, we will be using t.co.

practically, any tweet that is sent through statuses/update that has a link on it will have the link automatically converted to a t.co link on its way through the Twitter platform. if you fetch any tweet created after this change goes live, then its text field will have all its links automatically wrapped with t.co links. when a user clicks on that link, Twitter will redirect them to the original URL after first confirming with our database that that URL is not malicious.  on top of the end-user benefit, we hope to eventually provide all developers with aggregate usage data around your applications such as the number of clicks people make on URLs you display (it will, of course, be in aggregate and not identifiable manner). additionally, we want to be able to build services and APIs that can make algorithmic recommendations to users based on the content they are consuming. gathering the data from t.co will help make these possible.

our current plan is that no user will see a t.co URL on twitter.com but we still have some details to work through. the links will still be displayed as they were sent in, but the target of the link will be the t.co link instead. and, we want to provide the same ability to display original links to developers. we're going to use the entities attribute to make this possible.

let's say i send out the following tweet: "you have to check out http://dev.twitter.com!"

a returned (and truncated) status object may look like:

{

  "text" : "you have to check out http://t.co/s9gfk2d4!",

  ...

  "user" : {

    "screen_name" : "raffi",

    ...

  },

  ...

  "entities" : {

    "urls" : [

      {

        "url" : "http://t.co/s9gfk2d4",

        "display_url" : "http://dev.twitter.com",

        "indices" : [23, 43]

      }

    ],

    ...

  },

  ...

}

two things to note: the text of the returned status object doesn't have the original URL and instead it has a t.co URL, and the entities block now has a display_url attribute associated with it. what we're hoping is that with this data, it should be relatively easy to create a UI where you replace the http://t.co/s9gfk2d4 in the text with the equivalent of

<a href="http://t.co/s9gfk2d4">http://dev.twitter.com</a>

this means the user would not see the t.co link, but we all can still provide the protection and gather data from the wrapped link. for the applications that don't choose to update, the t.co link will be shown (and the goal to protect users will be met). i just want to emphasize -- we really do hope that you all render the original URL, but please send the user through the t.co link.   if you do choose to prefetch all the URLs on a timeline, then, when a user actually clicks on one of the links, please still send him or her through t.co. We will be updating the TOS to require you to check t.co and register the click.

related to this: the way the Twitter API counts characters is going to change ever so slightly. our 140 characters is now going to be defined as 140 characters after link wrapping. t.co links are of a predictable length -- they will always be 20 characters. after we make this live, it will be feasible to send in the text for a status that is greater than 140 characters. the rule is after the link wrapping, the text transforms to 140 characters or fewer. we'll be using the same logic that is in twitter-text-rb to figure out what is a URL.

look for an update to dev.twitter.com where we'll have a best practices document on how to use these t.co links.

what's the timeline?  "soon" we'll enable this on @twitterapi, @rsarver, @raffi, and a few other test accounts so you all have live data to play with.  on the timescale of weeks (to potentially a month or two), we'll roll this out to everybody.

of course, if there are any questions, just feel free to direct them to @twitterapi!

--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

[Dewald Pretorius]

Raffi,

I'm fine with everything up to the new 140 character count.

If you count the characters *after* link wrapping, you are seriously
going to mess up my system. My short URLs are currently 18 characters
long, and they will be 18 long for quite some time to come. After that
they will be 19 for a very long time to come.

If you implement this change, a ton, and I mean a *huge* number of my
system's updates are going to be rejected for being over 140
characters.

On Jun 8, 7:57 pm, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months

> now<http://bit.ly/twttldmemail>.

> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"

>
> a returned (and truncated) status object may look like:
>
> {

>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",

>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with

> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

[DeWitt Clinton]

Hi Raffi,

Interesting...  A couple of quick questions:

1) Will the redirect from t.co -> domain.com be a 301 Moved Permanently or a 302 Found response?

2) Will the t.co URL redirect point to the URL in the original tweet, or will it point to the ultimate resolved URL?  

I.e., if I post "Check out my site at http://bit.ly/abcd" where bit.ly/abcd redirects to domain.com, and the resultant tweet becomes "Check out my site at http://t.co/abcd", will the t.co URL redirect like this:

  a) t.co/abcd -> domain.com

Or like this:

  b) t.co/abcd -> bit.ly/abcd -> domain.com

3) In the above scenario, will the 'display_url' contain 'http://bit.ly/abcd' or 'http://domain.com'?

4) Why redirect all URLs, btw?  Why not just redirect the malicious ones?

Thanks!

-DeWitt

--
Twitter API documentation and resources: http://apiwiki.twitter.com
API updates via Twitter: http://twitter.com/twitterapi
Change your membership to this group: http://groups.google.com/group/twitter-api-announce?hl=en

[Twitlonger]

How will this affect links for third party services that clients
handle natively, such as Twitpic (and obviously TwitLonger, which
already has shorter dedicated short urls for its posts)?

I'll also be interested to see how this goes down with the privacy
types who will now be paranoid that Twitter is tracking the sites
they're going to, even if they are going through a third party client.
If I'm clicking a link on a desktop client, should Twitter really be
getting that information?

What about links through bit.ly etc? Will I still be able to see the
analytics that they provide for my links? If so, does that mean there
will be at least two levels of redirection from the ultimate
destination?

On Jun 8, 11:57 pm, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months

> now<http://bit.ly/twttldmemail>.

> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"

>
> a returned (and truncated) status object may look like:
>
> {

>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",

>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with

> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

[John Barratt]

Hi Raffi,

On 9/06/10 8:57 AM, Raffi Krikorian wrote:
> "url" : "http://t.co/s9gfk2d4",
> "display_url" : "http://dev.twitter.com",
> "indices" : [23, 43]

Any chance of getting the title of the resolved URL added in here too if
available?

Then we could display a link like :

<a title="Twitter Dev" href="http://t.co/s9gfk2d4">
http://dev.twitter.com
</a>

or :

<a title="http://dev.twitter.com" href="http://t.co/s9gfk2d4">
Twitter Dev
</a>

This would give even more context to users, without having to follow the
redirect path, & load and parse the page to extract it as well.

Thanks,

JB.

[Dewald Pretorius]

This is not unique to me. This will be problematic for anyone who uses
a shortening service that shortens URLs to less than 20 characters.

In these cases, you are basically adding characters to the submitted
text, and then rejecting the submitted text as being too long.

> > this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4inthe text with the equivalent of

[Raffi Krikorian]

1) Will the redirect from t.co -> domain.com be a 301 Moved Permanently or a 302 Found response?

301!

 

2) Will the t.co URL redirect point to the URL in the original tweet, or will it point to the ultimate resolved URL?  

I.e., if I post "Check out my site at http://bit.ly/abcd" where bit.ly/abcd redirects to domain.com, and the resultant tweet becomes "Check out my site at http://t.co/abcd", will the t.co URL redirect like this:

  a) t.co/abcd -> domain.com

Or like this:

  b) t.co/abcd -> bit.ly/abcd -> domain.com

we're not modifying or tampering with URLs - if you send us a bit.ly link, we will wrap that bit.ly link.  analytics will still work, etc.

 

3) In the above scenario, will the 'display_url' contain 'http://bit.ly/abcd' or 'http://domain.com'?

bit.ly!

 

4) Why redirect all URLs, btw?  Why not just redirect the malicious ones?

in the case of malicious URLs, you sometimes don't know it at the time of tweet creation.  or the URL may eventually become malicious.  this allows us to do shutdown after tweet creation.

 

Thanks!

that's what i'm here for :P

[Raffi Krikorian]

How will this affect links for third party services that clients
handle natively, such as Twitpic (and obviously TwitLonger, which
already has shorter dedicated short urls for its posts)?

that's why we are providing all the data back out in the API.  while the tweet itself may have t.co, we do include, in the entities, the original URL.  our hope, honestly, is that final users never have to see t.co -- we want to provide enough data back to developers so they can create URLs that look like

<a href="http://t.co/blahblah">http://mycoolsite.com</a>

all those URLs should still show through.

 

What about links through bit.ly etc? Will I still be able to see the
analytics that they provide for my links? If so, does that mean there
will be at least two levels of redirection from the ultimate
destination?

yes - we won't be touching the original URL.  all analytics that users want to see on bit.ly will still be there.  this is what we do on URLs in DMs right now.

[Raffi Krikorian]

that would be an awesome service!

[Raffi Krikorian]

its true, and we understand that.

just to correct my previous post, however -- t.co links are 19 characters.

[Dewald Pretorius]

So what are you saying? Suck it up? That's what I am hearing.

I have a work-around for the problem, in that I can simply adjust my
in-house shortening service to start generating 19-character URLs. But
other developers don't have that option.

> > > > our current plan is that no user will see a t.co URL on twitter.combut we

> > replace thehttp://t.co/s9gfk2d4inthetext with the equivalent of

[Jim Gilliam]

Will we be able to get matches on the original URL through the streaming API?

For example, I'm tracking "act" so I can match tweets that link to 'http://act.ly'.  Will I still be able to do that?

Jim Gilliam

http://act.ly/

http://twitter.com/jgilliam

[DeWitt Clinton]

Awesome, thanks for the quick response!  

Those are the right answers, too.  : )

Though there's an inconsistency with returning 301's and also requiring every click to go through the t.co link (as required by the ToS).  A 301 means that the redirect is cacheable by any intermediary (because it is permanent and will never change).

The 301 also implies that you actually can replace only the malicious links, (not every link), because clients will already have resolved and cached the 301 redirects (which again, can never change), so you won't be able to change the redirect down the road anyway.

So, I think you might actually have meant to use 302's, not 301's, if redirecting every click is the goal.

But then again, 301's really are the (philosophically? morally?) right answer, so maybe I don't want you to fix that. : )

Or better still, resolving all URLs upfront and returning the full URL inline, making tweets longer than 140 characters, and stopping this whole URL shortening nonsense to begin with.  (But you knew I'd say that...!)

-DeWitt

[Dewald Pretorius]

Raffi: Never mind. I just saw the Twitter blog post. The motivation
for this is to get metrics for Promoted Tweets and Resonance. Hence,
the answer is: Suck it up.

DeWitt: Yikes, discarding all shortens between t.co and the final link
will seriously mess with the click stats of a few million people.

[John Barratt]

Hi Raffi,

On 9/06/10 9:57 AM, Raffi Krikorian wrote:
> that would be an awesome service!

Currently we use one our own services (http://metauri.com/) to do this
for http://trendsmap.com/. In addition to the title, it also gives the
content type, which can be useful in determining how, or if to use the
link in your display.

I was just wondering if Twitter were going to possibly supply this extra
data, as it would remove a time & resource intensive step in the tweet
analysis process :).

Another question, will you wrap links that have no protocol in tweets, eg :

"Hey check out www.mysite.com/evil_page"

and :

"Hey check out mysite.com/evil_page"

I imagine many clients will currently link these out as urls and link
them up automatically?

Thanks again,

JB.

[Brian Smith]

If you do this, you will literally be forcing app developers to waste users time and money, especially over metered GRPS/3G connections.

 

If the user can see the full URL, then why do they need to be “protected” any more than they are when they use any other service? If anything, you should be cutting through any and all redirects (shorteners) so that the application can show the final URL to the user and avoid multiple useless, latency-inducing redirects that reduce reliability and increase costs for end-users and network operators. Cutting through all the redirects would improve security AND improve on the users’ privacy, instead of hurting it.

 

And, what about the user’s right to privacy? You’re basically forcing every Twitter app to become spyware. Who wants to create spyware? No developers with a conscience—and I’m sure that includes you guys at Twitter. Please ask whoever’s making these horrible decisions lately to reconsider at least this one.

 

Sincerely,

Brian Smith

@BRIAN_____

[Damon C]

+1 on this, I'd like to know the answer as well.

Damon/@dacort

On Jun 8, 4:43 pm, Jim Gilliam <j...@gilliam.com> wrote:
> Will we be able to get matches on the original URL through the streaming
> API?
>
> For example, I'm tracking "act" so I can match tweets that link to 'http://act.ly'.  Will I still be able to do that?
>

> Jim Gilliamhttp://act.ly/http://twitter.com/jgilliam

> > thehttp://t.co/s9gfk2d4inthe text with the equivalent of

[Sami]

I don't see how this feature could impact user privacy more than what
it is right now. Today Twitter stores all links for all users and they
can spy on them and the t.co shortner is not changing that :)

My question is, will developers have access to analytics from t.co
through API?

Thanks

[Alex B]

What's the algorithm for the display url? Ideally it will be a
predictable length, to aid predictability in tweet display code.

If the motive is really to protect us from malicious URLs, what about
giving a service we can call to route links through your protective
redirect servers? Then we can give users the option to be protected by
your malicious detection algorithms if they want.

If you want to click track every URL for whatever reason, ask client
developers to hit a ping URL if the user clicks? I'm not sure
otherwise how you will tell in a software client if it's the user
requesting the t.co URL or the software.

On Jun 9, 6:57 am, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months

> now<http://bit.ly/twttldmemail>.

> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"

>
> a returned (and truncated) status object may look like:
>
> {

>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",

>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with

> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

[Brian Smith]

Sami wrote:
> I don't see how this feature could impact user privacy more than what it
is right
> now. Today Twitter stores all links for all users and they can spy on them
and the
> t.co shortner is not changing that :)

Right now, Twitter can see all the links that users *post*, but they don't
see which links users *click*.

In order to implement this feature, Twitter has already built the framework
that does all the hard work that applications need to protect users' privacy
against (link-shortener) click-tracking. Twitter will be withholding that
final URL from applications, preventing us (through the ToS) from
implementing our own anti-click-tracking privacy measures. If, instead, they
gave the final URL to the application and let the application use that URL,
then applications could implement anti-click-tracking privacy measures with
an even greater degree of privacy than they could by using a third-party
service.

In other words, instead of Twitter using their existing link-unshortening
technology to let applications tell *fewer* companies what links your users
are clicking on, they are using it to force applications to tell *more*
companies what your users are clicking on.

Only advertisers could build a privacy-improving technology and using it for
the exact opposite purpose. :(

http://mashable.com/2010/06/03/alex-payne-twitter-interview/

Regards,
Brian Smith
@BRIAN_____

[Raffi Krikorian]

our hope is to eventually provide this analytics.

[Andy Matsubara]

Raffi wrote:
> related to this: the way the Twitter API counts characters is going to change ever so slightly. our 140
> characters is now going to be defined as 140 characters after link wrapping. t.co links are of a
> predictable length -- they will always be 20 characters. after we make this live, it will be feasible to
> send in the text for a status that is greater than 140 characters. the rule is after the link wrapping,
> the text transforms to 140 characters or fewer. we'll be using the same logic that is in twitter-text-rb
> to figure out what is a URL.

I guess this change will make frontend text handling more difficult.
Counting characters in a text box must figure out what is a URL. I
hope Twitter will publish JavaScript library for realtime character
counts. I also want APIs to make shortened URL.

Andy Matsubara

[Raffi Krikorian]

What's the algorithm for the display url? Ideally it will be a
predictable length, to aid predictability in tweet display code.

i'm not sure why the display_url would be of predictable length?  the display_url is -exactly- the URL that the user has sent into the system.  so, that may be of varying length.

 

If the motive is really to protect us from malicious URLs, what about
giving a service we can call to route links through your protective
redirect servers? Then we can give users the option to be protected by
your malicious detection algorithms if they want.

If you want to click track every URL for whatever reason, ask client
developers to hit a ping URL if the user clicks? I'm not sure
otherwise how you will tell in a software client if it's the user
requesting the t.co URL or the software.

i guess i'm confused on this as well?  isn't that what t.co does?

 

[Raffi Krikorian]

Right now, Twitter can see all the links that users *post*, but they don't
see which links users *click*.

In order to implement this feature, Twitter has already built the framework
that does all the hard work that applications need to protect users' privacy
against (link-shortener) click-tracking. Twitter will be withholding that
final URL from applications, preventing us (through the ToS) from
implementing our own anti-click-tracking privacy measures. If, instead, they
gave the final URL to the application and let the application use that URL,
then applications could implement anti-click-tracking privacy measures with
an even greater degree of privacy than they could by using a third-party
service.

hey brian - just wanted to point out - the "Twitter will be withholding that final URL from applications" is NOT true at all.  we are providing, as part of the "entities" the original URL back to the developers.  stated another way - we are giving all the data back and we are not withholding the data.

 

[Raffi Krikorian]

yeah - its definitely case that counting characters will become a bit more subtle.  i hope that we can provide a really good and easy way to help you all out.  at the very least we are going to update documentation, but i know we can do better than that.

[Brian Smith]

I was basing my statement on the blog post, which indicated that at least some “display URLs” will be truncated:

 

http://blog.twitter.com/2010/06/links-and-twitter-length-shouldnt.html

 

“A really long link such as http://www.amazon.com/Delivering-Happiness-Profits-Passion-Purpose/dp/0446563048 might be wrapped as http://t.co/DRo0trj for display on SMS, but it could be displayed to web or application users as amazon.com/Delivering-.”

 

Will the application be doing the truncation from the full URL to the truncated one (“amazon.com/Delivering-“), or will the API?

 

And, if the application really will get the full destination URL, then it is even more ridiculous to prevent them (through the ToS) from using it to improve the user’s privacy, don’t you think?

 

Regards,

Brian

[Hwee-Boon Yar]

Are all links going to be wrapped or only long links? If it's the
latter, what's the definition?

1. This affects how we count characters before sending and has quite a
potential to go wrong, since we'll now need to know exactly which
links are going to be wrapped in a tweet.

2. It's also going to be tricky for apps that currently show a live
character count (like SimplyTweet and many other iPhone Twitter apps
and possibly web sites) as users type.

--
Hwee-Boon

> related to this: the way the Twitter API counts characters is going to
> change ever so slightly. our 140 characters is now going to be defined as
> 140 characters after link wrapping. t.co links are of a predictable length
> -- they will always be 20 characters. after we make this live, it will be
> feasible to send in the text for a status that is greater than 140
> characters. the rule is after the link wrapping, the text transforms to 140
> characters or fewer. we'll be using the same logic that is in
> twitter-text-rb to figure out what is a URL.
>

> look for an update to dev.twitter.com where we'll have a best practices
> document on how to use these t.co links.
>
> what's the timeline?  "soon" we'll enable this on @twitterapi, @rsarver,
> @raffi, and a few other test accounts so you all have live data to play
> with.  on the timescale of weeks (to potentially a month or two), we'll roll
> this out to everybody.
>
> of course, if there are any questions, just feel free to direct them to
> @twitterapi!
>

[Alex B]

OK, it's a little confusing naming for display URL, as that implies
that is what clients should show directly to the users, as most of the
time I would imagine that field should be cut for brevity.

The difference between having a ping service that can help twitter
track clicks and a redirect service that can help twitter protect
users, and having twitter simply force-edit everyone's tweet texts, is
that instead of providing a new service that developers and users can
opt to use, you are providing a service that everyone _must_ use or
add code to work around.

You could have simply provided an extension to posting new tweets that
identified the real urls of shortened urls, that would have protected
short url providers who invested in your platform and allowed
developers to add the improvement on their own time frames.

I like the general idea of embedding real links in the twitter
metadata even if it adds to an already bloated tweet data payload, but
Twitter isn't respecting its ecosystem here by forcing complexity on
all developers and giving a time frame of weeks to change established
code developed over years.

> > thehttp://t.co/s9gfk2d4inthe text with the equivalent of

[John Kalucki]

All links will be wrapped. It's not about length.

[John Kalucki]

Existing url shortners will continue to work just fine. We're not
going to resolve them to their final link and remove them from the
chain.

By redirecting all links, we can protect all users and the entire
ecosystem much faster. The adoption via opt-in would be slower, and
might never reach critical mass.

Apps that don't update will continue to work, they will just display
something different than they do now.

On Tue, Jun 8, 2010 at 9:06 PM, Alex B <alex.b...@gmail.com> wrote: