Using max_id to navigate to pages after 15

[Karthik]

Please confirm, if the following is allowed?

1. Set rpp=100 and retrieve 15 pages search results by incrementing
the param 'page'
2. Get the id of the last status on page 15 and set that as the max_id
for the next query
3. If we have more results, go to step 1

[Matt Sanford]

Hi there,

From the parameter names I assume you mean in the search API.
That will certainly work but there is a limit on how far back in time
we allow a max_id to be used. When you reach the limit you'll see a
warning message (in the atom feed it's a twitter:warning tag) telling
you so.

Thanks;
— Matt

[nish]

Hi,

Were you able to get more than 1500 tweets using the max_id method?

I somehow cannot get it. I am using the search JSON API and whenever
I set the max_id looks like the server resets it to something else.

Thanks.

[Doug Williams]

Nish,
The way I read your question, it sounds like you are trying to use the max_id parameter incorrectly. max_id allows you to specify the maximum status id to return. It is the way to limit the results to statuses before a given status id. It is complementary to the since_id parameter, capping the results of a query at the top of the result list where since_id closes the bottom.

Doug Williams
Twitter API Support
http://twitter.com/dougw

[ronin]

Did anyone ever manage to get more than 1500 or past 15 pages worth of
tweets by:

1. Set rpp=100 and retrieve 15 pages search results by incrementing
the param 'page'
2. Get the id of the last status on page 15 and set that as the max_id

for the next query,

3. If we have more results, go to step 1

as mentioned by Karthik?



For me, once the first 15 pages are retrieved and the max_id is set to
the id of the last tweet of page 15, the results returned are not
below that max_id.

E.g. after first 15 pages, the last id is 1000

The first query (for the next 15 pages) would be something like
http://....&max_id=1000&since_id=100&page=1
The ids returned would still be greater than 1000 and it's as though
the max_id is ignored

Any idea why? Is it because the limit has been reached for max_id ?



** The id quoted are just sample values, the realistic number is way
higher.

On Mar 10, 4:45 pm, Doug Williams <d...@twitter.com> wrote:
> Nish,
> The way I read your question, it sounds like you are trying to use the
> max_id parameter incorrectly. max_id allows you to specify the maximum
> status id to return. It is the way to limit the results to statuses before a
> given status id. It is complementary to the since_id parameter, capping the
> results of a query at the top of the result list where since_id closes the
> bottom.
>
> Doug Williams

> Twitter API Supporthttp://twitter.com/dougw

[ronin]

Hi Doug,

I tried the approach that Karthik mentioned but it didn't quite work.

So after the first 15 pages worth of tweets were retrieved, I set the
max_id to the id of the last tweet and page =1

For example,

id of last tweet = 1000
max_id = 1000
page = 1

then the next query (for the next 15 pages) http://search...?max_id=1000&since_id=100&page=1...

The results returned basically ignored the max_id and the ids were all
greater than the stipulated 1000.

Any idea why the max_id was ignored ? Is there a way to retrieve
tweets beyond 15 pages?

Thanks for the help in advance,
Ronnie

On Mar 10, 4:45 pm, Doug Williams <d...@twitter.com> wrote:

> Nish,
> The way I read your question, it sounds like you are trying to use the
> max_id parameter incorrectly. max_id allows you to specify the maximum
> status id to return. It is the way to limit the results to statuses before a
> given status id. It is complementary to the since_id parameter, capping the
> results of a query at the top of the result list where since_id closes the
> bottom.
>
> Doug Williams

> Twitter API Supporthttp://twitter.com/dougw

[Matt Sanford]

Hi Ronnie,

    Did the output contain any sort of warning message? When the max_id is too old we have to adjust it and the output will contain a warning saying as much.

Thanks;

  — Matt Sanford

[nish]

HI,

I am facing the same problem as Ronin. The output does not contain
any warning message. Just that the max_id passed in is ignored and a
new max_id much higher than the one passed in is assigned.


Thanks,
Nish.

PS: If since_id is not specified, then I do see a warning message
that it was adjusted, but thats a different case.

[Ronnie]

Hey Matt,

Nish is right, I don't see a warning message that the max_id that
max_id is adjusted unless since_id is not specified

Here's a sample json request:
http://search.twitter.com/search.json?ors=sad=all&rpp=100&since_id=1309564503&max_id=1311104069&page=1

The id of the first result is higher than the requested max_id but
there was no warning message

Thanks,
Ronnie

[Matt Sanford]

Hi,

    That query of 'ors=sad=all' looks a little goofy, I would suggest 'q=sad+OR+all' if you're looking for both. I just tried a few different things and it seems like this is an issue with combining max_id and since_id. The max_id parameter was added to make sure the page parameter works correctly. I've oddly never tested using it to circumvent the pagination limit. I'll take a look at it but please open a Google Code issue (http://code.google.com/p/twitter-api/issues/entry) so I don't forget.

   A quick aside about the pagination limit: It's not there to make it hard on people or somehow hide our data, it's there to make searches faster. When you go back in time we have to read data from disk and replace recent data in memory with that older data. The pagination limit is there to prevent too much of our memory space being taken up by old data that a very small percentage of requests need.

Thanks;

  — Matt

[Ronnie]

Hi Matt,

The bug has been logged over at:
http://code.google.com/p/twitter-api/issues/detail?id=342&colspec=ID%20Stars%20Type%20Status%20Priority%20Owner%20Summary%20Opened%20Milestone

Thanks for replying and looking into it.

On Mar 11, 9:29 am, Matt Sanford <m...@twitter.com> wrote:
> Hi,
>
>      That query of 'ors=sad=all' looks a little goofy, I would suggest  
> 'q=sad+OR+all' if you're looking for both. I just tried a few  
> different things and it seems like this is an issue with combining  
> max_id and since_id. The max_id parameter was added to make sure the  
> page parameter works correctly. I've oddly never tested using it to  
> circumvent the pagination limit. I'll take a look at it but please  
> open a Google Code issue (http://code.google.com/p/twitter-api/issues/entry
> ) so I don't forget.
>
>     A quick aside about the pagination limit: It's not there to make  
> it hard on people or somehow hide our data, it's there to make  
> searches faster. When you go back in time we have to read data from  
> disk and replace recent data in memory with that older data. The  
> pagination limit is there to prevent too much of our memory space  
> being taken up by old data that a very small percentage of requests  
> need.
>
> Thanks;
>    — Matt
>
> On Mar 11, 2009, at 09:11 AM, Ronnie wrote:
>
>
>
> > Hey Matt,
>
> > Nish is right, I don't see a warning message that the max_id that
> > max_id is adjusted unless since_id is not specified
>
> > Here's a sample json request:

> >http://search.twitter.com/search.json?ors=sad=all&rpp=100&since_id=13...

[basha]

Hi I am Alos facing the same problem. If any one knows the solution
please tell me. i too tried max_id with since_id. when i try with
max_id only i got tweets correctly as karthik said. but when i tried
max_id with since_id , the response seems that the api is ignoring
max_id when given with since_id.

On Mar 11, 7:10 pm, Ronnie <ronniel...@gmail.com> wrote:
> Hi Matt,
>

> The bug has been logged over at:http://code.google.com/p/twitter-api/issues/detail?id=342&colspec=ID%...

[Beier]

Maybe I'm not getting it, but from Search API documentation page, I
don't see max_id as an available parameter. was it added recently?

On Mar 11, 2:34 am, ronin <ronniel...@gmail.com> wrote:
> Hi Doug,
>
> I tried the approach that Karthik mentioned but it didn't quite work.
>
> So after the first 15 pages worth of tweets were retrieved, I set the
> max_id to the id of the last tweet and page =1
>
> For example,
>
> id of last tweet = 1000
> max_id = 1000
> page = 1
>

> then the next query (for the next 15 pages)http://search...?max_id=1000&since_id=100&page=1...