from:user and since_id breaking Search API

[Christopher Warren]

We have an app that runs searches regularly, and recently stopped
receiving new tweets. After investigating we found a search
combination that seems to break the search API. Instead of getting a
response with no tweets, an .atom request errors and a .json request
404s.

http://search.twitter.com/search.json?q=from:silent_tester02&since_id=4979161317
http://search.twitter.com/search.atom?q=from:silent_tester02&since_id=4979161317

Changing the query to not use from:username works as expect, but I've
put several usernames in and they all respond the same way. I haven't
managed to narrow down the cause of the problem much further than
that, but we're handling it in our code by rescuing any failed
searches and appending since: with the date of the most recent tweet
to the q.

Any thoughts on what might be causing this would be appreciated.

[Marc W]

We've started seeing this too. If we specify a since_id, then
periodic refetches will frequently return "no new tweets" (but
sometimes they will return new ones). If we simply drop the
since_id, then all new tweets are fetched.

Some new server optimization thing gone wrong?

Thanks!


On Oct 20, 10:06 pm, Christopher Warren <christopher.war...@gmail.com>
wrote:

> We have an app that runssearchesregularly, and recently stopped

> receiving newtweets. After investigating we found a search

> combination that seems to break the search API. Instead of getting a

> response withnotweets, an .atom request errors and a .json request
> 404s.
>
> http://search.twitter.com/search.json?q=from:silent_tester02&since_id...http://search.twitter.com/search.atom?q=from:silent_tester02&since_id...

>
> Changing the query to not use from:username works as expect, but I've
> put several usernames in and they all respond the same way. I haven't
> managed to narrow down the cause of the problem much further than

> that, but we're handling it in our code by rescuing any failedsearchesand appending since: with the date of the most recent tweet

[harshavs]

We are experiencing the same thing as Marc. We are not using from:
though.
But intermittently it gets tweets in significant number. If I try to
get the same set again using the since_id for this particular crawl, I
again get no tweets.

Any Idea on this issue?

Thanks.

On Oct 23, 10:48 am, Marc W <marcwanchipm...@gmail.com> wrote:
> We've started seeing this too.  If we specify a since_id, then
> periodic refetches will frequently return "no new tweets" (but
> sometimes they will return new ones).    If we simply drop the
> since_id, then all new tweets are fetched.
>
> Some new server optimization thing gone wrong?
>
> Thanks!
>
> On Oct 20, 10:06 pm, Christopher Warren <christopher.war...@gmail.com>
> wrote:
>
> > We have an app that runssearchesregularly, and recently stopped
> > receiving newtweets. After investigating we found a search
> > combination that seems to break the search API. Instead of getting a
> > response withnotweets, an .atom request errors and a .json request
> > 404s.
>

> >http://search.twitter.com/search.json?q=from:silent_tester02&since_id......

[Damon Clinkscales]

Christopher

To my recollection, for search with since_id to work properly, the
tweet id must be in the search index. In this case:

http://search.twitter.com/search?q=+from%3Asilent_tester02

does not yield the "Dinner, movie, drinks." tweet in the index.

As an aside, I did an "exact match search" on that phrase above and it
returned many results that are not exact matches. But that's a
separate issue.

You could file an issue about the fact that the results coming back
are not always consistent, but the first thing I would do is make sure
that I am using a since_id that actually exists in the search index.
Granted this can be a bit of a pain to verify this 100% of the time
because sometimes tweets do not end up in the search index (which
appears to be the case here). But in my experience, most of the time,
they do. So as a test, pick a tweet you know is in the index and make
some calls with it over a period of time. See if the results are
consistent.

Best,
-damon
--
http://twitter.com/damon

[Dave B]

I'm getting the same issue as Marc and harshavs too.

I'm using since_id to start a search from the last set of resuts but
generally get 0 results most of the time, even though the same search
without since_id yields lots of results.

I could start logging tweet IDs against user IDs, and then work
backwards down the list of past results until I get a since_id that
yields results, but it does seem like a bit of a pain.

This is my first time using the search API so I can't even tell you
when this started happening. It just happens!

For info: search is using negation operators, OR and from:

Dave

[Marc W]

Hello Damon!

I'm not 100% sure I buy this explanation:

1. This problem wasn't happening a day or two ago.

2. I tried executing the query on the command line, and incremented
the since_id by 1 maybe 8-10x ... it just doesn't return any results.

Even weirder is that if I wait 20 minutes, and execute the same query
with the same original since_id, then I might get some of the results,
but not all of them.

Currently, the only solution I can see is to simply never use since_id
and just filter out - on the client - those tweetids I've seen
already. Seems like a horrible waste of bandwidth and computing
power, and especially strange given that this largely worked a few
days ago, right before some other changes were rolled out that also
seemed to cause a sudden increase in 500-series errors being returned
from the servers and other weirdnesses.


Thanks,
Marc.

On Oct 23, 9:31 pm, Damon Clinkscales <sca...@pobox.com> wrote:
> Christopher
>
> To my recollection, for search with since_id to work properly, the
> tweet id must be in the search index.  In this case:
>
> http://search.twitter.com/search?q=+from%3Asilent_tester02
>
> does not yield the "Dinner, movie, drinks." tweet in the index.
>
> As an aside, I did an "exact match search" on that phrase above and it
> returned many results that are not exact matches. But that's a
> separate issue.
>
> You could file an issue about the fact that the results coming back
> are not always consistent, but the first thing I would do is make sure
> that I am using a since_id that actually exists in the search index.
> Granted this can be a bit of a pain to verify this 100% of the time
> because sometimes tweets do not end up in the search index (which
> appears to be the case here). But in my experience, most of the time,
> they do.  So as a test, pick a tweet you know is in the index and make
> some calls with it over a period of time.  See if the results are
> consistent.
>
> Best,
> -damon

> --http://twitter.com/damon

>
> On Tue, Oct 20, 2009 at 9:06 AM, Christopher Warren
>
>
>
> <christopher.war...@gmail.com> wrote:
>
> > We have an app that runs searches regularly, and recently stopped
> > receiving new tweets. After investigating we found a search
> > combination that seems to break the search API. Instead of getting a
> > response with no tweets, an .atom request errors and a .json request
> > 404s.
>

> >http://search.twitter.com/search.json?q=from:silent_tester02&since_id...
> >http://search.twitter.com/search.atom?q=from:silent_tester02&since_id...

[arjun]

I am seeing issues with twitter search using since_id.

The search results the query returns is not correct for atleast few
hours. I think, some synchronization happens every certain number of
hours. Also, if since_id is old, it expires.

I don't find it to be that useful.

Thanks,
Arjun.

The synchronization is very

[George V. Reilly]

I'm *not* using since_id, but I'm sporadically getting warning
messages about it. Most of the time, my queries work.

d:\>curl "http://search.twitter.com/search.json?q=
%23jquery&lang=en&page=1&show_user=true&rpp=15"

{
"results": [{
"profile_image_url": "http://a1.twimg.com/profile_images/353780044/
avatar_normal.png",
"created_at": "Thu, 29 Oct 2009 00:05:02 +0000",
"from_user": "2twittdotcom",
"to_user_id": null,
"text": "create a simple infinite carousel with #jquery
http://2twitt.com/3u7",
"id": 5244850622,
"from_user_id": 33556519,
"geo": null,
"iso_language_code": "en",
"source": "&lt;a href=&quot;http://apiwiki.twitter.com/&quot;
rel=&quot;nofollow&quot;&gt;API&lt;/a&gt;"
}],
"max_id": 5244850622,
"since_id
": 5056020580,
"refresh_url": "?since_id=5244850622&q=%23jquery",
"results_per_page": 15,
"page": 1,
"completed_in": 0.02854,
"warning": "adjusted since_id to 5056020580 (2009-10-22 00:00:00
UTC), requested since_id was older than allowed",
"query": "%23jquery"
}

It's very annoying. Any ideas?