Updates to the retweet API payload

[Marcel Molina]

We've updated the retweet payload to look a lot more like a regular
tweet's payload. We find this change makes the retweet API
conceptually simpler, easier and more convenient to work with and
better overall. I've linked to examples of the new payload below, and
will be updating documentation shortly, but first I'd like to share
some background on what we've learned working on the retweet API over
the past few weeks.

One of the big objectives of the retweet feature is discovering new
users to follow. We wanted the original status and its author
highlighted front and center so they get due credit for the tweet
someone you follow found worthy of retweeting. To that end the API was
designed to emphasize the original tweet by having it be the top level
object. Embedded within it we provided details about who retweeted it
and when.

Though this makes a statement to how we'd like the feature to be
thought about, logistically it doesn't degrade gracefully for clients
who don't implement any retweet logic in their code. Tweets appear in
timelines from people who you do not follow and unless clients
differentiate those tweets visually letting you know that someone you
follow has shared this tweet, confusion ensues.

To mitigate this, we opted to not include retweets into the existing
friends timeline. Instead we created a new parallel home timeline that
would be just like the friends timeline but included retweets. That
way developers could opt in to the retweet feature at their leisure.
Even with this opt in policy, though, there is still a critical
responsibility imposed on developers to clearly communicate to their
users that a given tweet in their timeline is a retweet and who the
retweet is from. The likelihood that confusion would result turned out
to be too high.

Additionally, with the original status as the top level object in the
payload, that means consumers of the API would receive what appears to
be the same tweet multiple times. In a lot of cases that's fine, in
others it's problematic.

After experimenting with this approach we've decided that it's a
better bet to craft a payload that will degrade far more gracefully.
So we've redesigned the retweet payload. Rather than having the
original tweet as the top level status with embedded details about who
retweeted it, the retweet is the top level object and within it we
include the original tweet and its author. You'll have full access to
the entire retweeted status and user as well as the original tweet and
its user. So you don't have to make any additional API calls and
should have everything you need to display a retweet distinctively
with attribute to both the original author and the retweeter. In other
words, these retweeted statuses look a whole lot like regular
statuses. This new design was our instinct to begin with and we
probably should have gone with it. We think it's better and hope you
do too. All the same documented resources will exist. Only the
payloads change.

The following timeline should contain examples of the updated retweet
payload. You can use this to test consuming retweets.
curl http://twitter.com/statuses/user_timeline/testiverse.xml
curl http://twitter.com/statuses/user_timeline/testiverse.json

In the event that new tweets go into the above timeline that push the
retweets out, you can access a few directly at the following urls:
curl http://twitter.com/statuses/show/4452134416.xml
curl http://twitter.com/statuses/show/4452466408.xml
curl http://twitter.com/statuses/show/4349744308.xml

The above payloads don't contain a "retweet_count" element yet and
they probably will. Other than that we don't suspect any more major
changes as we approach a full public launch. As always, though, we're
open and solicitous of everyone's feedback.

Thanks.

--
Marcel Molina
Twitter Platform Team
http://twitter.com/noradio

[Amitab]

Great, I think this is much better and provides a better way of
identify which tweet is a retweet and which is not.

/Amitab

Follow Twalle @mytwaller

> curlhttp://twitter.com/statuses/user_timeline/testiverse.json

>
> In the event that new tweets go into the above timeline that push the
> retweets out, you can access a few directly at the following urls:
> curlhttp://twitter.com/statuses/show/4452134416.xml
> curlhttp://twitter.com/statuses/show/4452466408.xml

> curlhttp://twitter.com/statuses/show/4349744308.xml

[dean.j.robinson]

Looking at the examples I think this works much nicer, and should be
'friendlier' to implement. Looking forward to playing with it further.

> curlhttp://twitter.com/statuses/user_timeline/testiverse.json

>
> In the event that new tweets go into the above timeline that push the
> retweets out, you can access a few directly at the following urls:

> curlhttp://twitter.com/statuses/show/4452134416.xml
> curlhttp://twitter.com/statuses/show/4452466408.xml
> curlhttp://twitter.com/statuses/show/4349744308.xml

[Brian Smith]

Marcel Molina wrote:
> The above payloads don't contain a "retweet_count" element yet and
> they probably will. Other than that we don't suspect any more major
> changes as we approach a full public launch. As always, though, we're
> open and solicitous of everyone's feedback.

This is a great improvement over the previous way.

Is Twitter going to still merge retweets of the same tweet together in the
website UI? I think merging is a good idea when a tweet is re-tweeted by
multiple people in a short amount of time, but it causes problems with the
temporal presentation of timelines. I would like my app to be consistent
with the website, so more information about your plans for the website UI
would be very helpful.

How will this be encoded in in the home timeline XML/JSON?:

A: hello, world

B: RT @A hello, world

C: RT @B RT @A hello, world

D: RT @C RT @B RT @A hello, world

Also, replies and re-tweet merging do not interact well with each other:
what do you set in_reply_to_status_id to? In which timelines does a reply to
a re-tweeted tweet show up in?

Thanks,
Brian

[Rich]

I don't like this approach purely based on the fact it's no longer
backwards compatible, it's going to break a lot of clients who aren't
looking for a retweet_status tag.

[Rich]

Sorry ignore me, lack of sleep and I missed the <text> tag outside the
<retweet_status> one!

On Oct 1, 2:57 am, "Brian Smith" <br...@briansmith.org> wrote:

[Rich]

One question though I notice you're adding the RT @user at the
beginning, is this intentional or can we add it ourselves (as some
people prefer the RT <message> (via @user) format)

[Rich]

A final question, will home_timeline also follow this same format, as
the docs still have the old format.

[Marco Kaiser]

Hi,

first of all, let me say that I think this change to the relation between the retweet and the retweeted status makes much sense - it feels much more natural to see it as "user A retweets user B" instead of "user B retweeted by user A", especially if you don't follow B.

A couple of questions inline below:

2009/10/1 Marcel Molina <mar...@twitter.com>

After experimenting with this approach we've decided that it's a
better bet to craft a payload that will degrade far more gracefully.
So we've redesigned the retweet payload. Rather than having the
original tweet as the top level status with embedded details about who
retweeted it, the retweet is the top level object and within it we
include the original tweet and its author.

In your original design, the retweeted message was about to appear in a user's stream, by the original author. I understand that one of the main objectives of this change was to avoid the confusion of people appearing in home timelines that users aren't following. Does that also mean that you'll now show the retweet as sent by the retweeter, not the original poster, and give credit to the original poster in the meta information for a tweet on your web interface? (kind of the opposite of what the first version did)

 

You'll have full access to
the entire retweeted status and user as well as the original tweet and
its user. So you don't have to make any additional API calls and
should have everything you need to display a retweet distinctively
with attribute to both the original author and the retweeter.

Are you considering a retweet as a single entity, or as part of a number of retweets of the original message? In other words, is the "an by 5 others" (if multiple people retweeted a message) still something you want to show somehow? If more than one of my friends retweets a message, should this be displayed as multiple messages now, coming from different people, or are you still collapsing retweets? (the latter doesn't make much sense anymore IMO, as with the inverted relation, the retweet is now coming from the retweeter, not the original poster anymore)
 

In other
words, these retweeted statuses look a whole lot like regular
statuses. This new design was our instinct to begin with and we
probably should have gone with it. We think it's better and hope you
do too. All the same documented resources will exist. Only the
payloads change.

The /statuses/retweets/id.format endpoint returned a list of retweet_details before - as I understand it, this data structure no longer exists. What will it return instead?
 

The following timeline should contain examples of the updated retweet
payload. You can use this to test consuming retweets.
curl http://twitter.com/statuses/user_timeline/testiverse.xml
curl http://twitter.com/statuses/user_timeline/testiverse.json

In the event that new tweets go into the above timeline that push the
retweets out, you can access a few directly at the following urls:
curl http://twitter.com/statuses/show/4452134416.xml
curl http://twitter.com/statuses/show/4452466408.xml
curl http://twitter.com/statuses/show/4349744308.xml

The above payloads don't contain a "retweet_count" element yet and
they probably will. Other than that we don't suspect any more major
changes as we approach a full public launch. As always, though, we're
open and solicitous of everyone's feedback.

If there is no retweet_count, how can apps know if they need to display something like "and by 5 others"? How can they know if it is worth requesting a list of retweets (doesn't make sense if there is just one retweet, as it would waste an API request)? Or (see question above) is showing the list of retweets considered a deprecated feature?
 

Thanks.

And finally: when will the retweet API be available for beta testers again?

Thanks,
Marco
 

[Coderanger]

I am a bit confused as to how we can use these new APIs.

If I use the new retreet API in my app then no-one will see it if they
are using any app that isnt using the new home_timeline (perhaps no-
one) ... so how can it ever be used unless everyone (I mean client
apps and sites) uses the new APIs all at the same time.

Maybe someone can clarify this for me as I would like to add support
but I cannot see how.

[Rich]

All apps will eventually have to use home_timeline as the platform
guys have said they will remove the old timeline in a matter of months
after the retweet api launches

[Coderanger]

So you have to wait until then before you can even support the retweet
posting?

[Rich]

Not at all, just that retweets won't show in any app using the new re-
tweet API.

However I expect many apps that care about their users will switch
because retweets from the biggest source i.e. web will be using it!

[Zaudio]

This sounds a lot more sensible...

Importantly, where can we expect this new payload to be returned...
any of the following methods as well?

REST API
statuses/mentions
statuses/user

Streaming API/Shadow

I need to know in advance of such an addition to any of these, as
otherwise the parser on one of our apps gets broken until it's recoded
to handle the retweet payload...

or is the ONLY was to get these vie the retweet methods and the new
home_timeline ? If so, what about apps that mainly make use of the
Streaming API for tweets?

Thanks

Simon (Zaudio)

[John Kalucki]

There are plans to filter retweets from various resource, see the
documentation. However, it would be most prudent to assume that
retweets will eventually show up everywhere, and handle them
appropriately, or at least tolerate them wherever they are found.

They should start appearing in low volumes in all /1/statuses/*
resources on the Streaming API very soon.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.

[Zaudio]

Hey John,

Thanks for that... can you put an earliest date on 'very soon' please
- just so I know how long we've got?

Thanks

Simon (Zaudio)

On Oct 3, 8:15 pm, John Kalucki <jkalu...@gmail.com> wrote:
> There are plans to filter retweets from various resource, see the
> documentation. However, it would be most prudent to assume that
> retweets will eventually show up everywhere, and handle them
> appropriately, or at least tolerate them wherever they are found.
>
> They should start appearing in low volumes in all /1/statuses/*
> resources on the Streaming API very soon.
>

> -John Kaluckihttp://twitter.com/jkalucki

> Services, Twitter Inc.
>
> On Oct 3, 10:33 am, Zaudio <si...@z-audio.co.uk> wrote:
>
> > This sounds a lot more sensible...
>

> > Importantly, where can we expect this newpayloadto be returned...

> > any of the following methods as well?
>
> > REST API
> > statuses/mentions
> > statuses/user
>
> > Streaming API/Shadow
>
> > I need to know in advance of such an addition to any of these, as
> > otherwise the parser on one of our apps gets broken until it's recoded
> > to handle theretweetpayload...
>

> > or is the ONLY was to get these vie theretweetmethods and the new

[John Kalucki]

Retweet is an invasive feature with many deep dependency paths. Firm
dates would be useful, but they aren't possible in this particular
situation. This makes planning for downstream folks difficult.

I'd be ready for the slight possibility of low-volume retweets mid-to-
late week, with a high chance the following week, and perhaps a near-
unity chance of low-volume retweets the week after that. So, for
critical code, any time now. As for full-roll-out, that would be even
more of a guessing game.

-John Kalucki

http://twitter.com/jkalucki
Services, Twitter Inc.

[Zaudio]

Another significant thought... could you 'please' consider changing
the name of the <user> node INSIDE the retweeted_status node to say
<retweeted_user> ?

Thius will make JSON parsing way simpler... especially if the goal is
to extract the retweeted_status when present; or do things like
quickly find the date of the tweet... I alreayd have to contend with a
created_at field in the user and status nodes... now that could double
up... so owuld appreciate an easier to find retweeted user node for
JSON parsability

Thanks

Simon (Zaudio)

On Oct 4, 8:16 am, John Kalucki <jkalu...@gmail.com> wrote:
> Retweetis an invasive feature with many deep dependency paths. Firm

> dates would be useful, but they aren't possible in this particular
> situation. This makes planning for downstream folks difficult.
>
> I'd be ready for the slight possibility of low-volume retweets mid-to-
> late week, with a high chance the following week, and perhaps a near-
> unity chance of low-volume retweets the week after that. So, for
> critical code, any time now. As for full-roll-out, that would be even
> more of a guessing game.
>

> > > > Simon (Zaudio)- Hide quoted text -
>
> - Show quoted text -

[Marco Kaiser]

No, please don't change that to retweeted_user ... the data structure included as the retweeted status is a status, and that data structure has a user property. That's a very clear object model, and should map very well to JSON, as it's nested, not at the same level as the main user the retweet is received from. So by doing that change, you'd break the data model for a status, in that there are two version that need to be taken care of.

Or can you explain in more depth why this would cause problems with reasonable JSON parsers that map strings to objects?

2009/10/6 Zaudio <si...@z-audio.co.uk>

[Zaudio]

Sure, I'll justify this mroe...

One of our apps receives updates via the Streaming API and the various
REST api methods (mentions, user timelines, friends/home timelines).
We collect data as JSON as it's to date been faster and mor compact
than alternatives... we can also then go on to use this directly
client side in Jscript if we wish...
We are caching all updates of interest in a db... thus parse them for
the required data fields before storing them there. Currently the
parser has to pull out the user node, and is then left with the root
status node.... it is then simple to parse the separated nodes for all
fields pertinant to the current operation.
We make quick checks initially to determine the relevance of the
message to the app's cached stream.... we want to check things like
JUST created_at for the status (and not user) and then check the text
property for certain markers.
It is easy to find these efficiently in a JSON string without parsing
the entire thing to objects as things stand... so we save a lot of
server cpu cycles. It's fastest to this from the inner node
outwards... this is where the 'wasted' cpu cycles are coming in here
with the change for retweeted_status
Status's of interest only have the further fields parsed that we want
for out db copy...

for example.. say you want to quickly check the id of the status to
confirm if you have it in you db already or not... currently we just
excluded the user node, and thent he id is in the remainder without
conflict.

Now add retweeted_status with it's user subnode

To now get the id of the root status without parsing the entire string
to objects... we pull it apart again from the inner node outwards...
we now need to exclude the retweeted_status user subnode... this no
longer has a unique start tag/definition... as there are TWO identical
start tags in the string.... so we do a lot more work to ensure we get
the retweeted_status and it's user node that we would if it had an
alternative start tag.
If it were instead retweeted_user, then we could extract that directly
and easily, exclude it, then exclude the surrounding retweeted_status
tag... and we've got the retweeted_status node separated... then we
can procede as we do now with the rest... and if necessary use the
retweeted_status as well.

Hope that makes sense....
I agree that keeping it as user also makes good object sense... but
then the retweeted_status is not 'status' anymore... and it is a
status.... I'm suggesting soemthing similar for it's inner node..

Simon (Zaudio)

[Marco Kaiser]

Well, you're not making use of JSON as JSON, and instead use "brute-force" methods to extract parts of it... I think it's a bit unfair to request that change to be made, as it would complicate things for everyone doing "real" JSON-to-OO mapping. Just my 2c.

And no, retweet_status is perfectly valid - it's the property name in a status model, and it is assigned a status data model. Just a nested object. You don't have to name a member "status" just because it is a Status data type...

Marco

2009/10/7 Zaudio <si...@z-audio.co.uk>

[Zaudio]

We use JSON thoughout... as JSON and also parsing to db fields... I
love that JSON converts directly to OO.... but then again what works
for one app, may hinder others... no ideal solution.

I suspect that things will stay as they are anyway... so your our app
lose on this one and will just have to do more during our to db
parsing operations ...

Simon

> > > > > - Show quoted text -- Hide quoted text -

[Mark Hawker]

Hi, John. Are you able to update the API documentation with the latest
payloads?

- statuses/retweet and statuses/retweets have an embedded
retweet_details element. Is retweet_id the identifier for the status
of the retweeting user? Shouldn't statuses/retweet return the
retweeted_status element?
- statuses/home_timeline has an embedded retweeted_status element (I'm
with this one)
- statuses/retweeted_by_me, statuses/retweeted_of_me and statuses/
retweeted_to_me are random. Which form will they take?

It's fine that there are two distinct retweet elements, but
documentation doesn't seem to make this very clear.

Best,

On Oct 4, 3:16 pm, John Kalucki <jkalu...@gmail.com> wrote:
> Retweetis an invasive feature with many deep dependency paths. Firm

> dates would be useful, but they aren't possible in this particular
> situation. This makes planning for downstream folks difficult.
>
> I'd be ready for the slight possibility of low-volume retweets mid-to-
> late week, with a high chance the following week, and perhaps a near-
> unity chance of low-volume retweets the week after that. So, for
> critical code, any time now. As for full-roll-out, that would be even
> more of a guessing game.
>