link wrapping on the API

377 views
Skip to first unread message

Raffi Krikorian

unread,
Jun 8, 2010, 6:57:21 PM6/8/10
to twitter-deve...@googlegroups.com, twitter-ap...@googlegroups.com
hi all.

twitter has been wrapping links in e-mailed DMs for a couple months now. with that feature, we're trying to protect users against phishing and other malicious attacks. the way that we're doing this is that any URL that comes through in a DM gets currently wrapped with a twt.tl URL -- if the URL turns out to be malicious, Twitter can simply shut it down, and whoever follows that link will be presented with a page that warns them of potentially malicious content. in a few weeks, we're going to start slowly enabling this throughout the API for all statuses as well, but instead of twt.tl, we will be using t.co.

practically, any tweet that is sent through statuses/update that has a link on it will have the link automatically converted to a t.co link on its way through the Twitter platform. if you fetch any tweet created after this change goes live, then its text field will have all its links automatically wrapped with t.co links. when a user clicks on that link, Twitter will redirect them to the original URL after first confirming with our database that that URL is not malicious.  on top of the end-user benefit, we hope to eventually provide all developers with aggregate usage data around your applications such as the number of clicks people make on URLs you display (it will, of course, be in aggregate and not identifiable manner). additionally, we want to be able to build services and APIs that can make algorithmic recommendations to users based on the content they are consuming. gathering the data from t.co will help make these possible.

our current plan is that no user will see a t.co URL on twitter.com but we still have some details to work through. the links will still be displayed as they were sent in, but the target of the link will be the t.co link instead. and, we want to provide the same ability to display original links to developers. we're going to use the entities attribute to make this possible.

let's say i send out the following tweet: "you have to check out http://dev.twitter.com!"

a returned (and truncated) status object may look like:

{
  "text" : "you have to check out http://t.co/s9gfk2d4!",
  ...
  "user" : {
    "screen_name" : "raffi",
    ...
  },
  ...
  "entities" : {
    "urls" : [
      {
        "url" : "http://t.co/s9gfk2d4",
        "display_url" : "http://dev.twitter.com",
        "indices" : [23, 43]
      }
    ],
    ...
  },
  ...
}

two things to note: the text of the returned status object doesn't have the original URL and instead it has a t.co URL, and the entities block now has a display_url attribute associated with it. what we're hoping is that with this data, it should be relatively easy to create a UI where you replace the http://t.co/s9gfk2d4 in the text with the equivalent of


this means the user would not see the t.co link, but we all can still provide the protection and gather data from the wrapped link. for the applications that don't choose to update, the t.co link will be shown (and the goal to protect users will be met). i just want to emphasize -- we really do hope that you all render the original URL, but please send the user through the t.co link.   if you do choose to prefetch all the URLs on a timeline, then, when a user actually clicks on one of the links, please still send him or her through t.co. We will be updating the TOS to require you to check t.co and register the click.

related to this: the way the Twitter API counts characters is going to change ever so slightly. our 140 characters is now going to be defined as 140 characters after link wrapping. t.co links are of a predictable length -- they will always be 20 characters. after we make this live, it will be feasible to send in the text for a status that is greater than 140 characters. the rule is after the link wrapping, the text transforms to 140 characters or fewer. we'll be using the same logic that is in twitter-text-rb to figure out what is a URL.

look for an update to dev.twitter.com where we'll have a best practices document on how to use these t.co links.

what's the timeline?  "soon" we'll enable this on @twitterapi, @rsarver, @raffi, and a few other test accounts so you all have live data to play with.  on the timescale of weeks (to potentially a month or two), we'll roll this out to everybody.

of course, if there are any questions, just feel free to direct them to @twitterapi!


--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi

Dewald Pretorius

unread,
Jun 8, 2010, 7:33:02 PM6/8/10
to Twitter Development Talk
Raffi,

I'm fine with everything up to the new 140 character count.

If you count the characters *after* link wrapping, you are seriously
going to mess up my system. My short URLs are currently 18 characters
long, and they will be 18 long for quite some time to come. After that
they will be 19 for a very long time to come.

If you implement this change, a ton, and I mean a *huge* number of my
system's updates are going to be rejected for being over 140
characters.

On Jun 8, 7:57 pm, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months
> now<http://bit.ly/twttldmemail>.
> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"
>
> a returned (and truncated) status object may look like:
>
> {
>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",
>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with
> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

DeWitt Clinton

unread,
Jun 8, 2010, 7:45:16 PM6/8/10
to twitter-deve...@googlegroups.com
Hi Raffi,

Interesting...  A couple of quick questions:

1) Will the redirect from t.co -> domain.com be a 301 Moved Permanently or a 302 Found response?

2) Will the t.co URL redirect point to the URL in the original tweet, or will it point to the ultimate resolved URL?  

I.e., if I post "Check out my site at http://bit.ly/abcd" where bit.ly/abcd redirects to domain.com, and the resultant tweet becomes "Check out my site at http://t.co/abcd", will the t.co URL redirect like this:


Or like this:


3) In the above scenario, will the 'display_url' contain 'http://bit.ly/abcd' or 'http://domain.com'?

4) Why redirect all URLs, btw?  Why not just redirect the malicious ones?

Thanks!

-DeWitt



--
Twitter API documentation and resources: http://apiwiki.twitter.com
API updates via Twitter: http://twitter.com/twitterapi
Change your membership to this group: http://groups.google.com/group/twitter-api-announce?hl=en

Twitlonger

unread,
Jun 8, 2010, 7:49:22 PM6/8/10
to Twitter Development Talk
How will this affect links for third party services that clients
handle natively, such as Twitpic (and obviously TwitLonger, which
already has shorter dedicated short urls for its posts)?

I'll also be interested to see how this goes down with the privacy
types who will now be paranoid that Twitter is tracking the sites
they're going to, even if they are going through a third party client.
If I'm clicking a link on a desktop client, should Twitter really be
getting that information?

What about links through bit.ly etc? Will I still be able to see the
analytics that they provide for my links? If so, does that mean there
will be at least two levels of redirection from the ultimate
destination?

On Jun 8, 11:57 pm, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months
> now<http://bit.ly/twttldmemail>.
> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"
>
> a returned (and truncated) status object may look like:
>
> {
>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",
>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with
> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

John Barratt

unread,
Jun 8, 2010, 7:50:44 PM6/8/10
to twitter-deve...@googlegroups.com
Hi Raffi,

On 9/06/10 8:57 AM, Raffi Krikorian wrote:
> "url" : "http://t.co/s9gfk2d4",
> "display_url" : "http://dev.twitter.com",
> "indices" : [23, 43]

Any chance of getting the title of the resolved URL added in here too if
available?

Then we could display a link like :

<a title="Twitter Dev" href="http://t.co/s9gfk2d4">
http://dev.twitter.com
</a>

or :

<a title="http://dev.twitter.com" href="http://t.co/s9gfk2d4">
Twitter Dev
</a>

This would give even more context to users, without having to follow the
redirect path, & load and parse the page to extract it as well.

Thanks,

JB.

Dewald Pretorius

unread,
Jun 8, 2010, 7:53:47 PM6/8/10
to Twitter Development Talk
This is not unique to me. This will be problematic for anyone who uses
a shortening service that shortens URLs to less than 20 characters.

In these cases, you are basically adding characters to the submitted
text, and then rejecting the submitted text as being too long.
> > this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4inthe text with the equivalent of

Raffi Krikorian

unread,
Jun 8, 2010, 7:54:10 PM6/8/10
to twitter-deve...@googlegroups.com
1) Will the redirect from t.co -> domain.com be a 301 Moved Permanently or a 302 Found response?

301!
 
2) Will the t.co URL redirect point to the URL in the original tweet, or will it point to the ultimate resolved URL?  

I.e., if I post "Check out my site at http://bit.ly/abcd" where bit.ly/abcd redirects to domain.com, and the resultant tweet becomes "Check out my site at http://t.co/abcd", will the t.co URL redirect like this:


Or like this:


we're not modifying or tampering with URLs - if you send us a bit.ly link, we will wrap that bit.ly link.  analytics will still work, etc.
 
3) In the above scenario, will the 'display_url' contain 'http://bit.ly/abcd' or 'http://domain.com'?

 
4) Why redirect all URLs, btw?  Why not just redirect the malicious ones?

in the case of malicious URLs, you sometimes don't know it at the time of tweet creation.  or the URL may eventually become malicious.  this allows us to do shutdown after tweet creation.
 
Thanks!

that's what i'm here for :P

Raffi Krikorian

unread,
Jun 8, 2010, 7:56:09 PM6/8/10
to twitter-deve...@googlegroups.com
How will this affect links for third party services that clients
handle natively, such as Twitpic (and obviously TwitLonger, which
already has shorter dedicated short urls for its posts)?

that's why we are providing all the data back out in the API.  while the tweet itself may have t.co, we do include, in the entities, the original URL.  our hope, honestly, is that final users never have to see t.co -- we want to provide enough data back to developers so they can create URLs that look like


all those URLs should still show through.
 
What about links through bit.ly etc? Will I still be able to see the
analytics that they provide for my links? If so, does that mean there
will be at least two levels of redirection from the ultimate
destination?

yes - we won't be touching the original URL.  all analytics that users want to see on bit.ly will still be there.  this is what we do on URLs in DMs right now.

Raffi Krikorian

unread,
Jun 8, 2010, 7:57:48 PM6/8/10
to twitter-deve...@googlegroups.com
that would be an awesome service!

Raffi Krikorian

unread,
Jun 8, 2010, 7:58:32 PM6/8/10
to twitter-deve...@googlegroups.com
its true, and we understand that.

just to correct my previous post, however -- t.co links are 19 characters.

Dewald Pretorius

unread,
Jun 8, 2010, 8:02:53 PM6/8/10
to Twitter Development Talk
So what are you saying? Suck it up? That's what I am hearing.

I have a work-around for the problem, in that I can simply adjust my
in-house shortening service to start generating 19-character URLs. But
other developers don't have that option.
> > > > our current plan is that no user will see a t.co URL on twitter.combut we
> > replace thehttp://t.co/s9gfk2d4inthetext with the equivalent of

Jim Gilliam

unread,
Jun 8, 2010, 7:43:04 PM6/8/10
to twitter-deve...@googlegroups.com
Will we be able to get matches on the original URL through the streaming API?

For example, I'm tracking "act" so I can match tweets that link to 'http://act.ly'.  Will I still be able to do that?

Jim Gilliam

DeWitt Clinton

unread,
Jun 8, 2010, 8:08:22 PM6/8/10
to twitter-deve...@googlegroups.com
Awesome, thanks for the quick response!  

Those are the right answers, too.  : )

Though there's an inconsistency with returning 301's and also requiring every click to go through the t.co link (as required by the ToS).  A 301 means that the redirect is cacheable by any intermediary (because it is permanent and will never change).

The 301 also implies that you actually can replace only the malicious links, (not every link), because clients will already have resolved and cached the 301 redirects (which again, can never change), so you won't be able to change the redirect down the road anyway.

So, I think you might actually have meant to use 302's, not 301's, if redirecting every click is the goal.

But then again, 301's really are the (philosophically? morally?) right answer, so maybe I don't want you to fix that. : )

Or better still, resolving all URLs upfront and returning the full URL inline, making tweets longer than 140 characters, and stopping this whole URL shortening nonsense to begin with.  (But you knew I'd say that...!)

-DeWitt

Dewald Pretorius

unread,
Jun 8, 2010, 8:15:23 PM6/8/10
to Twitter Development Talk
Raffi: Never mind. I just saw the Twitter blog post. The motivation
for this is to get metrics for Promoted Tweets and Resonance. Hence,
the answer is: Suck it up.

DeWitt: Yikes, discarding all shortens between t.co and the final link
will seriously mess with the click stats of a few million people.

John Barratt

unread,
Jun 8, 2010, 8:20:23 PM6/8/10
to twitter-deve...@googlegroups.com, Raffi Krikorian
Hi Raffi,

On 9/06/10 9:57 AM, Raffi Krikorian wrote:
> that would be an awesome service!

Currently we use one our own services (http://metauri.com/) to do this
for http://trendsmap.com/. In addition to the title, it also gives the
content type, which can be useful in determining how, or if to use the
link in your display.

I was just wondering if Twitter were going to possibly supply this extra
data, as it would remove a time & resource intensive step in the tweet
analysis process :).

Another question, will you wrap links that have no protocol in tweets, eg :

"Hey check out www.mysite.com/evil_page"

and :

"Hey check out mysite.com/evil_page"

I imagine many clients will currently link these out as urls and link
them up automatically?

Thanks again,

JB.

Brian Smith

unread,
Jun 8, 2010, 8:26:59 PM6/8/10
to twitter-deve...@googlegroups.com

If you do this, you will literally be forcing app developers to waste users time and money, especially over metered GRPS/3G connections.

 

If the user can see the full URL, then why do they need to be “protected” any more than they are when they use any other service? If anything, you should be cutting through any and all redirects (shorteners) so that the application can show the final URL to the user and avoid multiple useless, latency-inducing redirects that reduce reliability and increase costs for end-users and network operators. Cutting through all the redirects would improve security AND improve on the users’ privacy, instead of hurting it.

 

And, what about the user’s right to privacy? You’re basically forcing every Twitter app to become spyware. Who wants to create spyware? No developers with a conscience—and I’m sure that includes you guys at Twitter. Please ask whoever’s making these horrible decisions lately to reconsider at least this one.

 

Sincerely,

Brian Smith

@BRIAN_____

Damon C

unread,
Jun 8, 2010, 9:46:34 PM6/8/10
to Twitter Development Talk
+1 on this, I'd like to know the answer as well.

Damon/@dacort

On Jun 8, 4:43 pm, Jim Gilliam <j...@gilliam.com> wrote:
> Will we be able to get matches on the original URL through the streaming
> API?
>
> For example, I'm tracking "act" so I can match tweets that link to 'http://act.ly'.  Will I still be able to do that?
>
> Jim Gilliamhttp://act.ly/http://twitter.com/jgilliam
> > thehttp://t.co/s9gfk2d4inthe text with the equivalent of

Sami

unread,
Jun 8, 2010, 10:14:18 PM6/8/10
to Twitter Development Talk
I don't see how this feature could impact user privacy more than what
it is right now. Today Twitter stores all links for all users and they
can spy on them and the t.co shortner is not changing that :)

My question is, will developers have access to analytics from t.co
through API?

Thanks

Alex B

unread,
Jun 8, 2010, 9:59:07 PM6/8/10
to Twitter Development Talk
What's the algorithm for the display url? Ideally it will be a
predictable length, to aid predictability in tweet display code.

If the motive is really to protect us from malicious URLs, what about
giving a service we can call to route links through your protective
redirect servers? Then we can give users the option to be protected by
your malicious detection algorithms if they want.

If you want to click track every URL for whatever reason, ask client
developers to hit a ping URL if the user clicks? I'm not sure
otherwise how you will tell in a software client if it's the user
requesting the t.co URL or the software.

On Jun 9, 6:57 am, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months
> now<http://bit.ly/twttldmemail>.
> let's say i send out the following tweet: "you have to check outhttp://dev.twitter.com!"
>
> a returned (and truncated) status object may look like:
>
> {
>   "text" : "you have to check outhttp://t.co/s9gfk2d4!",
>   ...
>   "user" : {
>     "screen_name" : "raffi",
>     ...
>   },
>   ...
>   "entities" : {
>     "urls" : [
>       {
>         "url" : "http://t.co/s9gfk2d4",
>         "display_url" : "http://dev.twitter.com",
>         "indices" : [23, 43]
>       }
>     ],
>     ...
>   },
>   ...
>
> }
>
> two things to note: the text of the returned status object doesn't have the
> original URL and instead it has a t.co URL, and the entities block now has a
> display_url attribute associated with it. what we're hoping is that with
> this data, it should be relatively easy to create a UI where you replace thehttp://t.co/s9gfk2d4in the text with the equivalent of

Brian Smith

unread,
Jun 8, 2010, 10:47:56 PM6/8/10
to twitter-deve...@googlegroups.com
Sami wrote:
> I don't see how this feature could impact user privacy more than what it
is right
> now. Today Twitter stores all links for all users and they can spy on them
and the
> t.co shortner is not changing that :)

Right now, Twitter can see all the links that users *post*, but they don't
see which links users *click*.

In order to implement this feature, Twitter has already built the framework
that does all the hard work that applications need to protect users' privacy
against (link-shortener) click-tracking. Twitter will be withholding that
final URL from applications, preventing us (through the ToS) from
implementing our own anti-click-tracking privacy measures. If, instead, they
gave the final URL to the application and let the application use that URL,
then applications could implement anti-click-tracking privacy measures with
an even greater degree of privacy than they could by using a third-party
service.

In other words, instead of Twitter using their existing link-unshortening
technology to let applications tell *fewer* companies what links your users
are clicking on, they are using it to force applications to tell *more*
companies what your users are clicking on.

Only advertisers could build a privacy-improving technology and using it for
the exact opposite purpose. :(

http://mashable.com/2010/06/03/alex-payne-twitter-interview/

Regards,
Brian Smith
@BRIAN_____

Raffi Krikorian

unread,
Jun 8, 2010, 11:16:50 PM6/8/10
to twitter-deve...@googlegroups.com
our hope is to eventually provide this analytics.

Andy Matsubara

unread,
Jun 8, 2010, 11:17:22 PM6/8/10
to twitter-deve...@googlegroups.com
Raffi wrote:
> related to this: the way the Twitter API counts characters is going to change ever so slightly. our 140
> characters is now going to be defined as 140 characters after link wrapping. t.co links are of a
> predictable length -- they will always be 20 characters. after we make this live, it will be feasible to
> send in the text for a status that is greater than 140 characters. the rule is after the link wrapping,
> the text transforms to 140 characters or fewer. we'll be using the same logic that is in twitter-text-rb
> to figure out what is a URL.
I guess this change will make frontend text handling more difficult.
Counting characters in a text box must figure out what is a URL. I
hope Twitter will publish JavaScript library for realtime character
counts. I also want APIs to make shortened URL.

Andy Matsubara

Raffi Krikorian

unread,
Jun 8, 2010, 11:18:24 PM6/8/10
to twitter-deve...@googlegroups.com
What's the algorithm for the display url? Ideally it will be a
predictable length, to aid predictability in tweet display code.

i'm not sure why the display_url would be of predictable length?  the display_url is -exactly- the URL that the user has sent into the system.  so, that may be of varying length.
 
If the motive is really to protect us from malicious URLs, what about
giving a service we can call to route links through your protective
redirect servers? Then we can give users the option to be protected by
your malicious detection algorithms if they want.

If you want to click track every URL for whatever reason, ask client
developers to hit a ping URL if the user clicks? I'm not sure
otherwise how you will tell in a software client if it's the user
requesting the t.co URL or the software.

i guess i'm confused on this as well?  isn't that what t.co does?
 

Raffi Krikorian

unread,
Jun 8, 2010, 11:24:14 PM6/8/10
to twitter-deve...@googlegroups.com
Right now, Twitter can see all the links that users *post*, but they don't
see which links users *click*.

In order to implement this feature, Twitter has already built the framework
that does all the hard work that applications need to protect users' privacy
against (link-shortener) click-tracking. Twitter will be withholding that
final URL from applications, preventing us (through the ToS) from
implementing our own anti-click-tracking privacy measures. If, instead, they
gave the final URL to the application and let the application use that URL,
then applications could implement anti-click-tracking privacy measures with
an even greater degree of privacy than they could by using a third-party
service.

hey brian - just wanted to point out - the "Twitter will be withholding that final URL from applications" is NOT true at all.  we are providing, as part of the "entities" the original URL back to the developers.  stated another way - we are giving all the data back and we are not withholding the data.
 

Raffi Krikorian

unread,
Jun 8, 2010, 11:27:45 PM6/8/10
to twitter-deve...@googlegroups.com
yeah - its definitely case that counting characters will become a bit more subtle.  i hope that we can provide a really good and easy way to help you all out.  at the very least we are going to update documentation, but i know we can do better than that.

Brian Smith

unread,
Jun 8, 2010, 11:33:30 PM6/8/10
to twitter-deve...@googlegroups.com

I was basing my statement on the blog post, which indicated that at least some “display URLs” will be truncated:

 

http://blog.twitter.com/2010/06/links-and-twitter-length-shouldnt.html

 

“A really long link such as http://www.amazon.com/Delivering-Happiness-Profits-Passion-Purpose/dp/0446563048 might be wrapped as http://t.co/DRo0trj for display on SMS, but it could be displayed to web or application users as amazon.com/Delivering-.

 

Will the application be doing the truncation from the full URL to the truncated one (“amazon.com/Delivering-“), or will the API?

 

And, if the application really will get the full destination URL, then it is even more ridiculous to prevent them (through the ToS) from using it to improve the user’s privacy, don’t you think?

 

Regards,

Brian

Hwee-Boon Yar

unread,
Jun 9, 2010, 12:05:49 AM6/9/10
to Twitter Development Talk
Are all links going to be wrapped or only long links? If it's the
latter, what's the definition?

1. This affects how we count characters before sending and has quite a
potential to go wrong, since we'll now need to know exactly which
links are going to be wrapped in a tweet.

2. It's also going to be tricky for apps that currently show a live
character count (like SimplyTweet and many other iPhone Twitter apps
and possibly web sites) as users type.

--
Hwee-Boon
> related to this: the way the Twitter API counts characters is going to
> change ever so slightly. our 140 characters is now going to be defined as
> 140 characters after link wrapping. t.co links are of a predictable length
> -- they will always be 20 characters. after we make this live, it will be
> feasible to send in the text for a status that is greater than 140
> characters. the rule is after the link wrapping, the text transforms to 140
> characters or fewer. we'll be using the same logic that is in
> twitter-text-rb to figure out what is a URL.
>
> look for an update to dev.twitter.com where we'll have a best practices
> document on how to use these t.co links.
>
> what's the timeline?  "soon" we'll enable this on @twitterapi, @rsarver,
> @raffi, and a few other test accounts so you all have live data to play
> with.  on the timescale of weeks (to potentially a month or two), we'll roll
> this out to everybody.
>
> of course, if there are any questions, just feel free to direct them to
> @twitterapi!
>

Alex B

unread,
Jun 9, 2010, 12:06:29 AM6/9/10
to Twitter Development Talk
OK, it's a little confusing naming for display URL, as that implies
that is what clients should show directly to the users, as most of the
time I would imagine that field should be cut for brevity.

The difference between having a ping service that can help twitter
track clicks and a redirect service that can help twitter protect
users, and having twitter simply force-edit everyone's tweet texts, is
that instead of providing a new service that developers and users can
opt to use, you are providing a service that everyone _must_ use or
add code to work around.

You could have simply provided an extension to posting new tweets that
identified the real urls of shortened urls, that would have protected
short url providers who invested in your platform and allowed
developers to add the improvement on their own time frames.

I like the general idea of embedding real links in the twitter
metadata even if it adds to an already bloated tweet data payload, but
Twitter isn't respecting its ecosystem here by forcing complexity on
all developers and giving a time frame of weeks to change established
code developed over years.
> > thehttp://t.co/s9gfk2d4inthe text with the equivalent of

John Kalucki

unread,
Jun 9, 2010, 12:13:24 AM6/9/10
to twitter-deve...@googlegroups.com
All links will be wrapped. It's not about length.

John Kalucki

unread,
Jun 9, 2010, 12:18:17 AM6/9/10
to twitter-deve...@googlegroups.com
Existing url shortners will continue to work just fine. We're not
going to resolve them to their final link and remove them from the
chain.

By redirecting all links, we can protect all users and the entire
ecosystem much faster. The adoption via opt-in would be slower, and
might never reach critical mass.

Apps that don't update will continue to work, they will just display
something different than they do now.


On Tue, Jun 8, 2010 at 9:06 PM, Alex B <alex.b...@gmail.com> wrote:

Hwee-Boon Yar

unread,
Jun 9, 2010, 1:19:04 AM6/9/10
to Twitter Development Talk
But if apps don't update and user sends a tweet which is just below
140 characters say, 139, and which contain a link(s) shorter than 19
(or is it 20) characters will mysteriously fail. The user will wonder
why the app doesn't let them send the tweet when their app clearly
says it's still within 140 characters, because Twitter is now counting
the longer 19/20 character t.co link.

Is this considered a rare scenario?

--
Hwee-Boon

Bernd Stramm

unread,
Jun 9, 2010, 1:37:43 AM6/9/10
to twitter-deve...@googlegroups.com, hwee...@gmail.com
On Tue, 8 Jun 2010 22:19:04 -0700 (PDT)
Hwee-Boon Yar <hwee...@gmail.com> wrote:

> But if apps don't update and user sends a tweet which is just below
> 140 characters say, 139, and which contain a link(s) shorter than 19
> (or is it 20) characters will mysteriously fail. The user will wonder
> why the app doesn't let them send the tweet when their app clearly
> says it's still within 140 characters, because Twitter is now counting
> the longer 19/20 character t.co link.
>
> Is this considered a rare scenario?

The right way to do this for twitter would be to count the characters
submitted to the API, before twitter changes the content.

That way, the API acceptance of a well formed post is predictable.
Otherwise it's not, since an application really doesn't know what
twitter will do with the content.

>
> --
> Hwee-Boon
>
> On Jun 9, 12:18 pm, John Kalucki <j...@twitter.com> wrote:
> > Apps that don't update will continue to work, they will just display
> > something different than they do now.

--
Bernd Stramm
<bernd....@gmail.com>

Harshad RJ

unread,
Jun 9, 2010, 1:45:21 AM6/9/10
to twitter-deve...@googlegroups.com
  • Privacy of click history. Your changes are such a big compromise of privacy that I might stop using Twitter altogether and not just stop making applications for it. (To get a feel of how seriously I take privacy of click-history, see an old blog post of mine).
  • By changing the TOS your are acting like FaceBook. In fact, it is even worse, because you are not just changing the default settings, you are saying that this is the only way to go. 
  • The concept of a URL shortener that expands links that are too short is so wrong I can't begin to describe it. It complicates the application logic at best, and leaves the user utterly bewildered at worst.

On Wed, Jun 9, 2010 at 4:27 AM, Raffi Krikorian <ra...@twitter.com> wrote:
hi all.

twitter has been wrapping links in e-mailed DMs for a couple months now. with that feature, we're trying to protect users against phishing and other malicious attacks. the way that we're doing this is that any URL that comes through in a DM gets currently wrapped with a twt.tl URL -- if the URL turns out to be malicious, Twitter can simply shut it down, and whoever follows that link will be presented with a page that warns them of potentially malicious content. in a few weeks, we're going to start slowly enabling this throughout the API for all statuses as well, but instead of twt.tl, we will be using t.co.

practically, any tweet that is sent through statuses/update that has a link on it will have the link automatically converted to a t.co link on its way through the Twitter platform. if you fetch any tweet created after this change goes live, then its text field will have all its links automatically wrapped with t.co links. when a user clicks on that link, Twitter will redirect them to the original URL after first confirming with our database that that URL is not malicious.  on top of the end-user benefit, we hope to eventually provide all developers with aggregate usage data around your applications such as the number of clicks people make on URLs you display (it will, of course, be in aggregate and not identifiable manner). additionally, we want to be able to build services and APIs that can make algorithmic recommendations to users based on the content they are consuming. gathering the data from t.co will help make these possible.

our current plan is that no user will see a t.co URL on twitter.com but we still have some details to work through. the links will still be displayed as they were sent in, but the target of the link will be the t.co link instead. and, we want to provide the same ability to display original links to developers. we're going to use the entities attribute to make this possible.

let's say i send out the following tweet: "you have to check out http://dev.twitter.com!"

a returned (and truncated) status object may look like:

{
  "text" : "you have to check out http://t.co/s9gfk2d4!",
  ...
  "user" : {
    "screen_name" : "raffi",
    ...
  },
  ...
  "entities" : {
    "urls" : [
      {
        "url" : "http://t.co/s9gfk2d4",
        "display_url" : "http://dev.twitter.com",
        "indices" : [23, 43]
      }
    ],
    ...
  },
  ...
}

two things to note: the text of the returned status object doesn't have the original URL and instead it has a t.co URL, and the entities block now has a display_url attribute associated with it. what we're hoping is that with this data, it should be relatively easy to create a UI where you replace the http://t.co/s9gfk2d4 in the text with the equivalent of


this means the user would not see the t.co link, but we all can still provide the protection and gather data from the wrapped link. for the applications that don't choose to update, the t.co link will be shown (and the goal to protect users will be met). i just want to emphasize -- we really do hope that you all render the original URL, but please send the user through the t.co link.   if you do choose to prefetch all the URLs on a timeline, then, when a user actually clicks on one of the links, please still send him or her through t.co. We will be updating the TOS to require you to check t.co and register the click.

related to this: the way the Twitter API counts characters is going to change ever so slightly. our 140 characters is now going to be defined as 140 characters after link wrapping. t.co links are of a predictable length -- they will always be 20 characters. after we make this live, it will be feasible to send in the text for a status that is greater than 140 characters. the rule is after the link wrapping, the text transforms to 140 characters or fewer. we'll be using the same logic that is in twitter-text-rb to figure out what is a URL.

look for an update to dev.twitter.com where we'll have a best practices document on how to use these t.co links.

what's the timeline?  "soon" we'll enable this on @twitterapi, @rsarver, @raffi, and a few other test accounts so you all have live data to play with.  on the timescale of weeks (to potentially a month or two), we'll roll this out to everybody.

of course, if there are any questions, just feel free to direct them to @twitterapi!


--
Raffi Krikorian
Twitter Platform Team
http://twitter.com/raffi



--
Harshad RJ
http://hrj.wikidot.com

M. Edward (Ed) Borasky

unread,
Jun 9, 2010, 2:28:39 AM6/9/10
to twitter-deve...@googlegroups.com
Quoting DeWitt Clinton <dcli...@gmail.com>:

> Awesome, thanks for the quick response!
>
> Those are the right answers, too. : )
>
> Though there's an inconsistency with returning 301's and also requiring
> every click to go through the t.co link (as required by the ToS). A 301
> means that the redirect is cacheable by any intermediary (because it is
> permanent and will never change).
>

> The 301 also implies that you actually *can* replace only the malicious


> links, (not every link), because clients will already have resolved and
> cached the 301 redirects (which again, can never change), so you won't be
> able to change the redirect down the road anyway.
>
> So, I think you might actually have meant to use 302's, not 301's, if
> redirecting every click is the goal.
>
> But then again, 301's really are the (philosophically? morally?) right

> answer, so maybe I *don't* want you to fix that. : )
>
> Or better still, resolving *all* URLs upfront and returning the full URL


> inline, making tweets longer than 140 characters, and stopping this whole
> URL shortening nonsense to begin with. (But you knew I'd say that...!)
>
> -DeWitt

Well ... while we're on the subject ... since we're talking *tweets*
and not arbitrary text from just anywhere, why do we have to waste
seven characters with the "http://"? Can't it just be "t.co/abcdef"?
After all, the receiver got the message from Twitter, not from thin
air, and can easily supply the "http://". You're never going to have
an "ftp://t.co", "file://t.co", etc., right?


Ken

unread,
Jun 9, 2010, 4:41:21 AM6/9/10
to Twitter Development Talk
Sorry if this is pedantic, but can you point to Twitter's definition
of "malicious" ?

Obviously, viruses, phishing etc. Presumably, "fraudulent" or
"illegal" would be included, but this might vary depending on the
jurisdiction.

Also, if a site is banned in country x, can the government of x
request that Twitter disable redirection for users in its
jurisdiction? Twitter might prefer this to a complete ban on Twitter
itself...

Rich

unread,
Jun 9, 2010, 4:45:33 AM6/9/10
to Twitter Development Talk
Please make sure the timeline for this is LONGER than 2 weeks please,
some of us have to code this and then wait at least a week to get
through Apple's approval system. This is especially important when it
comes to detecting images, etc.

Richard

On Jun 9, 4:27 am, Raffi Krikorian <ra...@twitter.com> wrote:
> yeah - its definitely case that counting characters will become a bit more
> subtle.  i hope that we can provide a really good and easy way to help you
> all out.  at the very least we are going to update documentation, but i know
> we can do better than that.
>

artesea

unread,
Jun 9, 2010, 5:04:09 AM6/9/10
to Twitter Development Talk
At present my code reads a tweet in plain text format. It then adds
<a> tags to any links, the @username and the #tags. It then looks for
twitpic/flickr/youtube etc urls it will add a clickable thumbnail to
the top of the tweet.
As I can see it, my code will need to put the t.co in the <a> tag, do
a lookup on the full url and replace the t.co in the text of the
tweet, check for twitpic links, add the thumbnail, BUT ensure that the
link on the thumbnail goes via t.co else it will be a breach of the
TOS.
Will not parsing the image url and displaying the thumbnail in the
first place be a breach of the TOS as I haven't sent the users browser
via t.co?

To repeat another request will you be providing search results with
words inside the URLs redirects like you already do for tinyurl
bit.ly?

Also two weeks, with just a couple of feeds to test against and no
sample code???

Cameron Kaiser

unread,
Jun 9, 2010, 8:49:34 AM6/9/10
to twitter-deve...@googlegroups.com
> But if apps don't update and user sends a tweet which is just below
> 140 characters say, 139, and which contain a link(s) shorter than 19
> (or is it 20) characters will mysteriously fail. The user will wonder
> why the app doesn't let them send the tweet when their app clearly
> says it's still within 140 characters, because Twitter is now counting
> the longer 19/20 character t.co link.
>
> Is this considered a rare scenario?

I wouldn't consider this a rare scenario; in fact, I think this might be
common with some clients. The proper way to solve this is to offer a
shortener API that a front end can talk to. *cough*

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- We aren't your science project! -- Akane, "Ranma 1/2" (OAV 9) --------------

Terence Eden

unread,
Jun 9, 2010, 6:58:13 AM6/9/10
to Twitter Development Talk
Would it be possible to use a 3rd party URL lengthener (like
http://longurlplease.com/) then use that?

Alice posts bit.ly/foo
Twitter changes it to t.co/bar
LongUrl returns bit.ly/foo resolves to example.com
App shows <a href="http://t.co/bar">example.com</a>
When Bob clicks on example.com, he hits t.co, which then redirects to
bit.ly, which then redirects to example.

Aside from the annoyance / speed on a mobile of seeing the page reload
several times (assuming example.com doesn't redirect) does that seems
to be ok?

Terence

Dewald Pretorius

unread,
Jun 9, 2010, 9:18:34 AM6/9/10
to Twitter Development Talk
This thing really is a can of worms, in terms of:

1) The impact on the ecosystem if characters are counted after link
wrapping.
2) The impact on Twitter's core system and SMS communication if
characters are counted before link wrapping.
3) The perception that there is no guaranteed stability in the TOS,
which makes developing and investing in the ecosystem increasingly
unattractive.

I don't buy the click tracking privacy argument. Twitter will have no
more insight into clicks than what bit.ly or any other shortening
service has, and they won't have any insight into clicks on links that
are not in tweets. For example, if I tweet something with a bit.ly
link and I use that same bit.ly link in other places, Twitter will
track only the clicks on the link in the tweet. They are not able to
track the clicks on that bit.ly link in the other places I've used it
(Facebook, Buzz, StatusNet, blog, etc.). If the privacy argument
carried any water, then bit.ly would be a far greater privacy threat
than Twitter.

Brian Smith

unread,
Jun 9, 2010, 9:30:13 AM6/9/10
to twitter-deve...@googlegroups.com
Dewald Pretorius wrote:
> StatusNet, blog, etc.). If the privacy argument carried any water, then
bit.ly
> would be a far greater privacy threat than Twitter.

I agree, and that's why I (and others) were working on solutions to get
around bit.ly and other link shorteners. And, Twitter has already developed
the ultimate solution to that privacy issue, but they just are refusing to
let API developers use it. That is what is so incredibly frustrating.

Regards,
Brian

Rich

unread,
Jun 9, 2010, 10:46:53 AM6/9/10
to Twitter Development Talk
I'm a +1 one for the huge can of worms this is going to cause with
realtime character counting. Here are my concerns

1) An app realtime character counts what is entered in a text box and
that could end up being wrong when posted (and in the mobile space
making ANOTHER http call just get yet another shortener is not
realistic)

2) If the user has a bit.ly link, for example, and it's only say 15
characters long and they post at 136 characters, it'll get rejected
from Twitter whilst the client app said it was ok.

3) After the system has launch we could dynamically look at urls,
ignore their character count and substitute it for 20 characters,
however for iPhone developers we have AT LEAST a week's wait to get
these things through Apple's approval process.

But it's not just iPhone developers, all developers can't just push
out code that quickly without a decent amount of testing

We can't realistically do that until the system is live... and yet the
apps will be 'broken' in the time between 'go live' and end of testing/
approval.

Do you see the chicken and the egg scenario here?

My proposal is you need to keep the 140 limit as it is now, if your
own link shortener makes it over 140 then so be it, that's your
decision, not ours... but the original posted text was 140.

Thomas Woolway

unread,
Jun 9, 2010, 10:55:38 AM6/9/10
to twitter-deve...@googlegroups.com
As well as everything that others have asked, I'm also concerned about the following:

1) Does this mean that tweets can effectively be infinite length - say I write a tweet with 120 chars of text and then http://www.a-really-really-really-long-url-maybe-with-a-message-encoded.com/in-the-middle-of-it - that will then get counted as 140 chars after the link has been wrapped. When we come to display that, we have to be able to correctly format with realistically, anything up to 200 char statuses.

2) What will be coming out of the search API? Will those tweets have t.co links in them? If so, will we get the additional entity data?

Tom

Harshad RJ

unread,
Jun 9, 2010, 11:17:37 AM6/9/10
to twitter-deve...@googlegroups.com
On Wed, Jun 9, 2010 at 6:48 PM, Dewald Pretorius <dpr...@gmail.com> wrote:

I don't buy the click tracking privacy argument. Twitter will have no
more insight into clicks than what bit.ly or any other shortening
service has,

The difference being that the user who clicks the links in Twitter will have most probably logged into Twitter. Thus, Twitter can directly associate a click with a user.

When clicking on bit.ly shortened URLs it is very very unlikely that the user is logged into bit.ly. That is because only people who shorten URLs need a bit.ly account (which is a very small percentage).

Dewald Pretorius

unread,
Jun 9, 2010, 11:18:39 AM6/9/10
to Twitter Development Talk
Rich,

That of course would be the right thing for Twitter to do. In other
words, if they want this change, then they must take the bulk of the
pain, instead of creating a problem and dumping it on the developers
to solve.

Raffi Krikorian

unread,
Jun 9, 2010, 11:24:31 AM6/9/10
to twitter-deve...@googlegroups.com
-totally-

the timeline on this is, i think (and i'll check to be sure), sometime like 6-8 weeks.

Scott Mills

unread,
Jun 9, 2010, 11:28:44 AM6/9/10
to Twitter Development Talk

I, like others, have problems with this new "service". My main
concern is this. In the past, we had an expectation that what we post
to twitter will be shown verbatim. Some people will want to post a
url and know that it will be shown that way across all sites/apps.
Like it or not, urls have meaning.

A person will no longer be able to post something like "This is/was
the best site ever: http://twitter.com" and have the same intent/
meaning.

This could lead to other things like twitter replacing bad words in
the interest of "protecting users" or s/coke/pepsi/ for the right
price.

I know this is probably paranoid talk. I just hope this is not a
shark jump moment for Twitter.

Also...

On Jun 8, 11:18 pm, John Kalucki <j...@twitter.com> wrote:
> Apps that don't update will continue to work, they will just display
> something different than they do now.

This is completely false with one of my apps. It will completely
break it and it will show nothing. The app (peephole for iphone)
provides an alternate view of Twitter by mostly showing the photos
people are posting.

As someone else asked, will the search api still return results for
searches like "twitpic"?


Most of the twitter photo services have a documented way to get
thumbnail or full size images from the photo url.
ex. with this url http://twitpic.com/1e10q you know that:
- http://twitpic.com/show/thumb/1e10q will give you the thumbnail

Most twitter apps don't use the posted photo url. Does inferring a
link to show a photo to a user violate the TOS?


A time scale of weeks is a very short time frame for apps that go
through Apple's app store. It seems like you would have a period of
at least a few months where all tweets have the url in the entities
before you start replacing the urls in the text.


Thanks for considering my comments and questions.

Scott Mills
IFWORLD Inc

janole

unread,
Jun 9, 2010, 11:52:17 AM6/9/10
to Twitter Development Talk
I think it pretty much depends on what Twitter will require us to do
when following t.co links. On their own website and in their own
Twitter clients I could imagine that they will track link usage from
people clicking on the link (sooner or later.)

If they don't require third party clients to send any user related
info when accessing t.co links, they only have the IP and possibly the
user agent header.

Cheers,
ole @ mobileways.de / @janole on Twitter

Ron B

unread,
Jun 9, 2010, 12:05:36 PM6/9/10
to Twitter Development Talk
Users are accustomed to the fact that use of *free* services is
entirely *as is* and at their own risk, so none of us should feel we
have to protect their privacy or their security beyond this original
expectation. If they don't like the performance, security, or privacy
implications of this change, they'll leave Twitter, just like they
left Facebook. But that is ultimately Twitter's responsibility to
control and react to. Our responsibility is to keep rowing
collectively in the same direction, or get off the boat. :)

M. Edward (Ed) Borasky

unread,
Jun 9, 2010, 12:47:14 PM6/9/10
to twitter-deve...@googlegroups.com, Ron B, Twitter Development Talk
Quoting Ron B <rbthe...@gmail.com>:

> Users are accustomed to the fact that use of *free* services is
> entirely *as is* and at their own risk, so none of us should feel we
> have to protect their privacy or their security beyond this original
> expectation. If they don't like the performance, security, or privacy
> implications of this change, they'll leave Twitter, just like they
> left Facebook.

Ah, but they *aren't* leaving Facebook! They're continuing to join and
share. Facebook and its hundreds of millions of users are continuing
to co-evolve. And *when* - not *if* - the FUD brigade realizes it
can't take Facebook out, it will turn its energies on Twitter. I hope
as a community / ecosystem / whatever, we're ready for that.

> But that is ultimately Twitter's responsibility to
> control and react to. Our responsibility is to keep rowing
> collectively in the same direction, or get off the boat. :)

Well, to the extent that it's possible, sure. But "groupthink" can
kill a product / service too. There are clearly some *technical* rough
edges on link wrapping that need to be discussed / debated.


Chris Barr

unread,
Jun 9, 2010, 1:03:37 PM6/9/10
to Twitter Development Talk
My 2 pence:

The difference with bit.ly is that I choose to use it. If I don't want
to use it I'm not forced to.

Additionally, what happens if the t.co service goes down? All links
will be temporarily broken until the service goes back up.

Ron B

unread,
Jun 9, 2010, 1:21:09 PM6/9/10
to Twitter Development Talk
I just had one of those *rough edges* brought to my attention that I
think may also have already been alluded to on this thread. Some
users use long URLs as a portion or even their entire tweet. This
tweet technique is being used significantly in tweets about the Gulf
oil spill crisis, for example. (i.e.
http://www.grist.org/aritcle/2010-06-08-does-obama-need-more-not-fewer-oilmen-in-his-administration).
The author of such a tweet fully expects that this message will be
conveyed in the tweet itself, by virtue of the message conveyed by the
long URL - not just the URL destination's content. It doesn't sound
like this interesting and useful melding of tweet text and tweet
attachments will be possible any longer after the t.co wrapper
initiative is implemented. That would be a real shame...

Good point Ed, about staying in the game. Fortunately, we can row and
talk at the same time. :)

On Jun 9, 11:47 am, "M. Edward (Ed) Borasky" <zn...@borasky-
research.net> wrote:

Ken

unread,
Jun 9, 2010, 1:28:58 PM6/9/10
to Twitter Development Talk
Chris, I am not worried about that or any of this, but agree that it's
unfortunate to lose the choice. And it feels wrong to be obfuscating
links to my own website...

For apps that display tweets, I understand that the t.co link must be
used and not the display link. But what does it mean, "require you to
check t.co and register the click" ? How do we "check" and
"register" ?

M. Edward (Ed) Borasky

unread,
Jun 9, 2010, 3:22:03 PM6/9/10
to twitter-deve...@googlegroups.com, Ron B, Twitter Development Talk
Quoting Ron B <rbthe...@gmail.com>:

> I just had one of those *rough edges* brought to my attention that I
> think may also have already been alluded to on this thread. Some
> users use long URLs as a portion or even their entire tweet. This
> tweet technique is being used significantly in tweets about the Gulf
> oil spill crisis, for example. (i.e.
> http://www.grist.org/aritcle/2010-06-08-does-obama-need-more-not-fewer-oilmen-in-his-administration).
> The author of such a tweet fully expects that this message will be
> conveyed in the tweet itself, by virtue of the message conveyed by the
> long URL - not just the URL destination's content. It doesn't sound
> like this interesting and useful melding of tweet text and tweet
> attachments will be possible any longer after the t.co wrapper
> initiative is implemented. That would be a real shame...

That particular example looks like a blatant "search engine
optimization" ploy and not the "human-to-human communication" that is,
IIRC, the "Spirit of Twitter". ;-)

Bonus points for correcting the spelling of "article" in the link? ;-)

Bernd Stramm

unread,
Jun 9, 2010, 3:45:42 PM6/9/10
to twitter-deve...@googlegroups.com, zn...@borasky-research.net, Ron B

Really now, what is wrong with a person expressing themselves by making
human readable links?

If an application wants to provide the original intent of the user, it
is forced (by ToS), to present a link that doesn't go to where it says
it does. That is problematic, the application acts as spyware.

--
Bernd Stramm
<bernd....@gmail.com>

janole

unread,
Jun 9, 2010, 3:54:32 PM6/9/10
to Twitter Development Talk
Hi Raffi,

my Twitter client has an option to post links to del.ico.us, Read-it
Later and Instapaper. Do I have to post the t.co links there or will I
be allowed to use the expanded links?

ole @ mobileways.de / @janole on Twitter ( Gravity Twitter Client for
Symbian )

On 9 Jun., 00:57, Raffi Krikorian <ra...@twitter.com> wrote:
> hi all.
>
> twitter has been wrapping links in e-mailed DMs for a couple months
> now<http://bit.ly/twttldmemail>.
> let's say i send out the following tweet: "you have to check outhttp://