Quality Filter in Streaming API with follow predicates

[Jonathan Strauss]

From http://apiwiki.twitter.com/Streaming-API-Documentation#ResultQuality:
"Removing the quality filter from streams with follow predicates is,
quite reasonably, an often requested feature."

Does anyone have any ideas on the status of this feature or any
suggested workarounds?

We are working on a new version of TweetPo.st powered by the Streaming
API. However, some of our beta testers are apparently being quality
filtered and we are trying to figure out a way to get their tweets.
They are OAuthed, so we can in theory poll their accounts directly for
updates. However, we don't have a systematic way of figuring out
whether they are being quality filtered in the first place or if that
status changes subsequently.

So, our specific questions are:
A) Is there any update on the status of removing the quality filter
from streams with follow predicates?
B) Is there any way to programmatically check if a given user is being
quality filtered?
C) Is there any way to be notified when a given user's quality filter
status changes?

Any help here would be greatly appreciated as we would ideally like to
use the Streaming API as much as possible and it's hard to explain to
users when their tweets don't show up.

Thanks!
-jonathan

=====
Jonathan Strauss, Co-Founder
http://snowballfactory.com

Campaign tracking for social media - http://awe.sm
A smarter way to update Facebook from Twitter - http://tweetpo.st
Sharecount button for Facebook - http://www.fbshare.me

[John Kalucki]

This has been the "next generally scheduled" user-facing feature for quite some time on the Streaming API. We've been pushing scale, operational and efficiency branches recently, and haven't exposed anything user-facing in quite some time. If we can't keep well ahead of the organic tweet growth, we're in a tough spot. I hope that we can transition back to features soon enough, and this feature should be near, if not at the top of, the list. There's a git branch that has this feature largely working, but I'm afraid that the merge back to master is going to be a very painful one. The bits, they rot.

As far as programmatic detection, there are significant policy issues in play around filtered users. Getting this feature shipped is the real solution.

-John Kalucki
http://twitter.com/jkalucki
Infrastructure, Twitter Inc.

[M. Edward (Ed) Borasky]

Quoting John Kalucki <jo...@twitter.com>:

> We've been pushing scale, operational and
> efficiency branches recently, and haven't exposed anything user-facing in
> quite some time. If we can't keep well ahead of the organic tweet growth,
> we're in a tough spot.

At 50 million tweets a day, and an average tweet size (JSON, measured
from "sample") of about 1400 bytes, you're gaining about 70 GB of
"stuff" a day without *any* "geometric" growth by my calculations. And
if I read Kevin Weil's blog post correctly, that 50 million tweets a
day is *after* spam removal!

BTW, "sample" is delivering about 90,000 tweets per hour at the peak
part of the day now. This mathematician hasn't had his coffee yet for
24 February 2010, so there are no theorems - just a simple "wow!" or
perhaps "OMG!" ;-)

--
M. Edward (Ed) Borasky
borasky-research.net/m-edward-ed-borasky/

"A mathematician is a device for turning coffee into theorems." ~ Paul Erdos

[Jonathan Strauss]

On Feb 23, 11:45 pm, John Kalucki <j...@twitter.com> wrote:

> As far as programmatic detection, there are significant policy issues in
> play around filtered users. Getting this feature shipped is the real
> solution.

Thanks for the quick response John! We suspected that shipping this
feature was the right solution, and it's good to know it's at the top
of the list. However, we know first-hand how big a time sink
scalability can be, so we won't hold our breath on seeing it too
soon ;-)

In the meantime, we may be able to do a workaround using user input
(i.e. "Click here if your tweets aren't showing up on Facebook").
We're already doing direct polling for protected updates, so it would
just be a matter of detecting the quality filtered public updates and
flipping the direct polling bit on our side.

Thanks again, and we look forward to seeing this feature when you're
able to get to it.

-jonathan

=====
Jonathan Strauss, Co-Founder
http://snowballfactory.com

Campaign tracking for social media -http://awe.sm
A smarter way to update Facebook from Twitter -http://tweetpo.st

[John Kalucki]

I don't know if you could detect this via Facebook updates. You could, perhaps, start following them on the stream and poll their timelines in parallel until you determine that their tweets are flowing -- then turn off the polling.

-John

[Jonathan Strauss]

Heh :-) The app we're building, TweetPo.st, is designed to post the
user's tweets to Facebook. So, the call-to-action I proposed would be
specific to our app: i.e. if TweetPo.st users were not seeing the
expected behavior from the app.

But, we have decided to do basically what you suggest to start.
However, this will not help us if a user starts out in the clear and
is then subsequently quality filtered for whatever reason. But,
hopefully that's a real edge-case.

Thanks again!
-jonathan

[John Kalucki]

FYI: There's already an app that posts Tweets to Facebook.

-John

[Jonathan Strauss]

Indeed several, but people love ours because we do it smarter:
* change @mentions to Twitter real names
* post links to FB wall so your friends can see previews, watch
videos, and/or play audio inline
* give you the option of an inclusive (#fb) or exclusive (!fb) filter
on what tweets to post

See #3 on http://mashable.com/2009/05/25/twitter-to-facebook/ :-)