Download entire twitter archive of a user

[jmhol...@gmail.com]

Hi Guys
I want to be able to download my entire twitter archive in text
format, not just the last 20. Is there a way to do this?
Thanks
Jenna

[Cameron Kaiser]

> I want to be able to download my entire twitter archive in text
> format, not just the last 20. Is there a way to do this?

I think with the database performance issues lately, that would probably
earn you execution by firing squad from the developers.

I don't think paging support is back either.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- Funk is its own reward. -- George Clinton ----------------------------------

[Alex Payne]

The thing is, we have code to do this, but actually getting that many
records back from the database without it locking up on some other
query is... unlikely. I'll try to make a reliable export feature
available, as it's come up before. Thanks for your patience.

--
Alex Payne
http://twitter.com/al3x

[jmhol...@gmail.com]

Thanks! Do let me know when this feature is available!

On Nov 2, 1:30 pm, "Alex Payne" <a...@al3x.net> wrote:
> The thing is, we have code to do this, but actually getting that many
> records back from the database without it locking up on some other
> query is... unlikely. I'll try to make a reliable export feature
> available, as it's come up before. Thanks for your patience.
>

> On 11/2/07, Cameron Kaiser <spec...@floodgap.com> wrote:
>
>
>
> > > I want to be able to download my entire twitter archive in text
> > > format, not just the last 20. Is there a way to do this?
>
> > I think with the database performance issues lately, that would probably
> > earn you execution by firing squad from the developers.
>
> > I don't think paging support is back either.
>
> > --
> > ------------------------------------ personal:http://www.cameronkaiser.com/--

> > Cameron Kaiser * Floodgap Systems *www.floodgap.com* ckai...@floodgap.com

[Uldis Bojars]

On Nov 2, 5:30 pm, "Alex Payne" <a...@al3x.net> wrote:
> The thing is, we have code to do this, but actually getting that many
> records back from the database without it locking up on some other
> query is... unlikely. I'll try to make a reliable export feature
> available, as it's come up before. Thanks for your patience.

That would be a great feature to have. Good ideas++

Alex, how do external Twitter search engines usually get the data? How
can they get a large amount of tweets from everyone (ideally - all
tweets that are there) w/o hitting the 70 api calls per hour limit and
w/o dropping too many tweets (dropping tweets is a problem with some
desktop Twitter clients)?

Are they using an API? Requesting RSS feed often enough? Instant
messaging (can you ask to IM you the public Tweet feed at all?)?
Something else?

Thanks,
Uldis

[ http://captsolo.net/info/ ]

[Alex Payne]

Some people just scrape the public_timeline feed. Our preference for
high-volume API clients is to send them a Jabber stream of all public
updates. We've been doing this on an ad-hoc basis, but we'll be
releasing a PubSub API to enable this automatically.

[Max Engel]

i'd love to see this export functionality as well. i am trying to
assemble an art piece based on aggregating 1-years worth of my twitter
updates, and would definitely love a simple way to do this.

On Nov 5, 2:36 pm, "Alex Payne" <a...@al3x.net> wrote:
> Some people just scrape the public_timeline feed. Our preference for
> high-volume API clients is to send them a Jabber stream of all public
> updates. We've been doing this on an ad-hoc basis, but we'll be
> releasing a PubSub API to enable this automatically.
>

[Max Engel]

i'd love to see this export functionality as well. i am trying to
assemble an art piece based on aggregating 1-years worth of my twitter
updates, and would definitely love a simple way to do this.

On Nov 5, 2:36 pm, "Alex Payne" <a...@al3x.net> wrote:
> Some people just scrape the public_timeline feed. Our preference for
> high-volume API clients is to send them a Jabber stream of all public
> updates. We've been doing this on an ad-hoc basis, but we'll be
> releasing a PubSub API to enable this automatically.
>

[jmhol...@gmail.com]

any word on this yet?

On Nov 2, 12:10 pm, "jmhollo...@gmail.com" <jmhollo...@gmail.com>
wrote:

[Alex Payne]

Still on the list, should be working on it today along with a couple
of other API improvements. I've been trying to wrap up another
feature for the last couple weeks so API work has gotten pushed aside.
Thanks for your patience.

[Alex Payne]

Okay, got this deployed just now. Still need to add it to the
official documentation, but you should be able to hit:

http://twitter.com/account/archive.[xml|json]

...and you'll get back 80 statuses at a time. You can paginate back
(append "?page=x" to the request) as far as you need to. This request
operates against a read-only slave database, so it should be
reasonably fast. Please use it responsibly.

[jmhol...@gmail.com]

Thanks!~ Now just to figure out how to use it! =D

On Nov 16, 6:57 pm, "Alex Payne" <a...@al3x.net> wrote:
> Okay, got this deployed just now. Still need to add it to the
> official documentation, but you should be able to hit:
>
> http://twitter.com/account/archive.[xml|json]
>
> ...and you'll get back 80 statuses at a time. You can paginate back
> (append "?page=x" to the request) as far as you need to. This request
> operates against a read-only slave database, so it should be
> reasonably fast. Please use it responsibly.
>

> On 11/16/07, Alex Payne <a...@al3x.net> wrote:
>
>
>
> > Still on the list, should be working on it today along with a couple
> > of other API improvements. I've been trying to wrap up another
> > feature for the last couple weeks so API work has gotten pushed aside.
> > Thanks for your patience.
>

[gareth rushgrove]

Hi Alex

On 11/16/07, Alex Payne <al...@al3x.net> wrote:
>
> Okay, got this deployed just now. Still need to add it to the
> official documentation, but you should be able to hit:
>
> http://twitter.com/account/archive.[xml|json]

Great, thanks. A (very) quick hack for grabbing everything up on my
blog: http://morethanseven.net/posts/archiving-twitter-data-with-python/

I want to see how easy it is to feed the JSON into something like
http://simile.mit.edu/timeline/ but that will have to wait for later.

G

>
> ...and you'll get back 80 statuses at a time. You can paginate back
> (append "?page=x" to the request) as far as you need to. This request
> operates against a read-only slave database, so it should be
> reasonably fast. Please use it responsibly.
>
> On 11/16/07, Alex Payne <al...@al3x.net> wrote:
> > Still on the list, should be working on it today along with a couple
> > of other API improvements. I've been trying to wrap up another
> > feature for the last couple weeks so API work has gotten pushed aside.
> > Thanks for your patience.
> >
> > On 11/16/07, jmhol...@gmail.com <jmhol...@gmail.com> wrote:
> > >
> > > any word on this yet?
> > >
> > > On Nov 2, 12:10 pm, "jmhollo...@gmail.com" <jmhollo...@gmail.com>
> > > wrote:
> > > > Hi Guys
> > > > I want to be able to download my entire twitter archive in text
> > > > format, not just the last 20. Is there a way to do this?
> > > > Thanks
> > > > Jenna
> > >
> >
> >
> > --
> > Alex Payne
> > http://twitter.com/al3x
> >
>
>
> --
> Alex Payne
> http://twitter.com/al3x
>

--

Gareth Rushgrove
garethrushgrove.com
morethanseven.net
refreshnewcastle.org