Since_ID and Search Problem

[Spode]

Hi,

I've been noticing some strange behaviour since last night - but
that's not to say it's not been going on longer.

http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=5402169538

Performing this search turns up 0 results when it should show 4.

http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=

Run it with no since_id and it works. Then straight away run the
original query...

http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=5402169538

And you get the expected 4 results.

What's going on?

Spode (journotwit.com)

[Marcel Molina]

I've talked with the search team and there's a known condition where
10 of our servers are behind the others that can cause varying results
across requests. We're aware of the issue, it is ticket and we are
working on it.

--
Marcel Molina
Twitter Platform Team
http://twitter.com/noradio

[twitter-development]

I think a few people including myself have seen this. Best guess is
that it's a caching issue on the twitter server side. Here are some
other threads with similar issues:

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/e368ccc88f393885/a4f217e36b0d0657?#a4f217e36b0d0657

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/e6289b6439c1d26d/01bcafab31516f7a?lnk=gst&q=since_id#01bcafab31516f7a

http://groups.google.com/group/twitter-development-talk/browse_thread/thread/ed72429eef055cb3/23cf597ef030ca62?lnk=gst&q=since_id#23cf597ef030ca62

On Nov 4, 3:45 am, Spode <sp...@justfdi.com> wrote:
> Hi,
>
> I've been noticing some strange behaviour since last night - but
> that's not to say it's not been going on longer.
>

> http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=5...

>
> Performing this search turns up 0 results when it should show 4.
>
> http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=
>
> Run it with no since_id and it works. Then straight away run the
> original query...
>

> http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=5...

[Brian Morearty]

I've been seeing this problem for the last couple of weeks too. I
thought I was going crazy.

Most of the time when I do a search with since_id, it returns zero
results. My searches always include "to:screenname" in addition to
since_id. For example:

With since_id: curl "http://api.twitter.com/1/search.json?
since_id=5627135528&q=to%3APogue"

result:
{
"results": [],
"max_id": 5637056671,
"since_id": 5627135528,
"refresh_url": "?since_id=5637056671&q=to%3APogue",
"results_per_page": 15,
"page": 1,
"completed_in": 0.012596,
"query": "to%3APogue"
}

Without since_id: curl "http://api.twitter.com/1/search.json?q=to
%3APogue"
(notice that the ids are higher than the since_id I passed in):

result:
{
"results": [{
"profile_image_url": "http://a3.twimg.com/profile_images/
351653493/IMG_7001_normal.JPG",
"created_at": "Thu, 12 Nov 2009 03:52:08 +0000",
"from_user": "timstratton",
"to_user_id": 2053065,
"text": "@Pogue You have seen this right ? http://twitter.com/big_ben_clock,
I have been laughing all day. Odd though, wife does not agree at
all.",
"id": 5639849021,
"from_user_id": 5772395,
"to_user": "pogue",
"geo": null,
"iso_language_code": "en",
"source": "&lt;a href=&quot;http://twitter.com/
&quot;&gt;web&lt;/a&gt;"
},
{
"profile_image_url": "http://a3.twimg.com/profile_images/
242892621/Jombi_Ambigram_Square_normal.jpg",
"created_at": "Thu, 12 Nov 2009 03:32:15 +0000",
"from_user": "JombiDotNet",
"to_user_id": 2053065,
"text": "@Pogue Google Suggest and Sociology http://www.slate.com/id/2234019",
"id": 5639382121,
"from_user_id": 30310966,
"to_user": "pogue",
"geo": null,
"iso_language_code": "fr",
"source": "&lt;a href=&quot;http://twitter.com/
&quot;&gt;web&lt;/a&gt;"
},
{
"profile_image_url": "http://a1.twimg.com/profile_images/
335299002/IMG_0030_normal.JPG",
"created_at": "Thu, 12 Nov 2009 01:07:06 +0000",
"from_user": "AaronIsaacs",
"to_user_id": 2053065,
"text": "@Pogue google &quot;where can I find Chuck
Norris&quot; and click &quot;I'm feeling lucky&quot;",
"id": 5635640329,
"from_user_id": 21018646,
"to_user": "pogue",
"geo": null,
"iso_language_code": "en",
"source": "&lt;a href=&quot;http://www.atebits.com/&quot;
rel=&quot;nofollow&quot;&gt;Tweetie&lt;/a&gt;"
},
etc. etc. etc.
[ ... snip ... ]

My only workaround is to ignore the results I've already seen before
but no one wins--Twitter has to send me more results and I have to
process more results.

A fix soon would be very much appreciated. Thanks!

Brian Morearty



On Nov 4, 10:50 am, twitter-development <emmanuel....@gmail.com>
wrote:

> I think a few people including myself have seen this. Best guess is
> that it's a caching issue on the twitter server side. Here are some
> other threads with similar issues:
>

> http://groups.google.com/group/twitter-development-talk/browse_thread...
>
> http://groups.google.com/group/twitter-development-talk/browse_thread...
>
> http://groups.google.com/group/twitter-development-talk/browse_thread...

>
> On Nov 4, 3:45 am, Spode <sp...@justfdi.com> wrote:
>
> > Hi,
>
> > I've been noticing some strange behaviour since last night - but
> > that's not to say it's not been going on longer.
>
> >http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=5...
>
> > Performing this search turns up 0 results when it should show 4.
>
> >http://search.twitter.com/search.json?q=JournoTwit&rpp=100&since_id=
>

> > Run it with nosince_idand it works. Then straight away run the

[Spode]

This is STILL a problem. I have a possible theory on what is happening
at a server level.

The search phrases I am using are quite specific - i.e, I may well be
the only person in the world searching for that particular string.


First Call - Because it has a since_id, it assumes that it checks
against the server cache ONLY - and doesn't update the search results.

Second Call - Because there is no since_id, it updates the search
results.

Third Call - Once again, with a since_id, it checks against the cache
only - but because of the previous call, now there are tweets there.


I think Twitter is assuming that if there is a since_id, that it
doesn't need to update the original search. The only way around this
problem is to make TWO API calls, or to process the since_id myself.
That's just not acceptable.

[Spode]

I've created a screen-cam video to illustrate this problem and how
much of a problem it is...

http://www.youtube.com/watch?v=3a-88MSdqes

[Spode]

The last video I posted was garbled - but I've re-done it.

http://www.youtube.com/watch?v=qedssv3cBCA