since_id basic usage confusion

[Joel Hughes]

Hi all,
I have looked at the API and searched this group but can't find the
answer.

I've created some code which is doing a search and doing some analysis
on the results and then poking the results into a database. Simple
enough.

This code then runs a few minutes later.

I obviously don't want to process the same tweets again so I'm
assuming that I need to use the "since_id" to help me out here and to
filter out previously seen tweets.

With simplexml in php I get this kinda XML back
<id>tag:search.twitter.com,2005:1730670929</id>

..so at the end of my process run I'm parsing out that number on the
end as that seems like the status_id to me

I then store that in the database and to feed it back into the process
when it NEXT runs

.............the issue is though

If I run the process a couple of times on a row I can still see the
same tweets coming back - this foxes me as I would have thought that
by using status_id in this way I was effectively creating some sort of
"sliding window" where I could NOT see tweets older than this previous
id.

Is this assumption right (and, therefore, I'm looking at a code issue
here) or have I misunderstood the nature of status_id?

Thanks for any help/thoughts you can provide

Joel

[Chad Etzel]

Your understanding of since_id is correct. If you specify since_id=x
then you should only get back tweet ids strictly greater than x. Are
you using PHP 5? If so, I strongly recommend using the .json feed
instead of .atom and using json_decode() to parse the data.

-Chad

[Matt Sanford]

Hi there,

Just as a general software comment if you're sending since_id
it's probably a good idea to also double check your results. That
said: since_id should work like you expect. One case where it does not
work is if you follow the next_page URL for pagination. When you do
that there will be warning message that the since_id has been
removed. Are you going beyond page one be chance?

Thanks;
– Matt Sanford / @mzsanford
Twitter Dev

[Joel Hughes]

...ok, I THINK I may have answered my own question

I believe I made an assumption (warning sounds going off anyone?) that
the XML returned would be in status_id order - nope!

Now that I've adjusted for that I think I'm back on course.

...next issue for me us trying to look at paging

Joel

[Sherif]

Joel - with paging. The basic principle is this
1) As you start, keep the last since_id persisted somewhere
2) When you reach the end of the page, progress to the second page,
with the same since_id etc.. and you keep going till you cant get any
more.

Next time you want to refresh/resume data polling/loads you just pick
it up form the last since_id you had loaded.
Sherif