OT - where's the proper place to talk about search.twitter.com?

[TjL]

Specifically

1) There are WAY to many "trending topic" bots which fill search
results with useless clutter

2) I'd love to see a "trending topics" list that does NOT include hash
tags, you know, to find out what ordinary people are talking about :-)

I know this is the wrong place for it (sorry) but I'm not sure where else to go.

TjL

[Chad Etzel]

1) I agree, so I wrote a greasemonkey script to zap them from
http://search.twitter.com/ results:
http://jazzychad.com/stuff/?p=37
(disclosure: i also run one of the trending bots... i discuss this in the post)

2) http://twitscoop.com to the rescue! I have been told that they are
hoping to create an API for their site (right now they have special
mojo with tweetdeck integration).

-Chad

[Burhan TANWEER]

I agree with him. Search trends are not available in xml format. I will appreciate, if twitter can provide search trends in xml and so that i can update my social search engine ExploreWWW.com with search trends in real time.

 

Thanks

Burhan

 

On Thu, Mar 5, 2009 at 11:09 PM, TjL <luo...@gmail.com> wrote:

--
Sincerely,

Burhan Tanweer
www.explorewww.com
exp...@explorewww.com

[Doug Williams]

Chad,
In your experience, do trending bots have a disproportionate
participation in the search results for trending topics? Have you done
any analysis like that?

Doug Williams
@dougw

--
Doug Williams

do...@igudo.com
http://www.igudo.com

[TjL]

On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams <do...@igudo.com> wrote:
> In your experience, do trending bots have a disproportionate
> participation in the search results for trending topics? Have you done
> any analysis like that?

I'm not Chad :-) but if you click on any of the Trending Topics and
watch for any length of time you'll see scads of trending topic bots
popping up.

I think the most I counted at one point was like 12 out of the top 20 results.

It's insane.

TjL

[Chad Etzel]

Well, it's kind of a weird feedback loop.

Say you are following a trending bot (many many people do, a
surprising number to me). As soon as you see a tweet from your
favorite trending bot, you click the link and head over to see the
results.... Well, all the other bots are tweeting at about the same
time, so as soon as a new trend appears you get a dozen or so
trend-bot tweets appearing in the results you just loaded up. I will
admit this can be semi-annoying. Disproportionate? I guess it depends
on how many results your browser loads by default. Mine is always set
to 100, so I can scroll by the bots pretty quickly, but if people are
only seeing 25 at a time, they'd have to click "Next" or "Older" to
get past the bots.

Like I said in my blog post, once you are actually searching for a
trend, you don't need a dozen things telling you it's a trend again..
you're already there! Some bots are "worse offenders" than others and
just spew all the trends every 5 minutes or retweet people (randomly
it seems) that match the trend (not naming names, I'm sure you can
figure them out).

As a means of driving traffic they are very effective (at least the
one I run seems to be). A little over 50% of the traffic to
tweetgrid.com/search comes from links posted by my bot. I am not sure
if the effectiveness can be attributed to the mere fact that the bot
exists, or because it has some useful information attached (e.g.
#trend has risen to the #3 trend! <link>). Very few of the bots seem
to talk about the rank of the trend, but mine does, so it has some
added value. I think this has helped my bot, and it also means that
it gets retweeted quite a bit (another big surprise to me).

In all honesty, I started my bot because one of my competitors
convinced one of the existing trend bots to link to their site instead
of search.twitter.com. I launched my bot in defense.

A long, meandering answer to a short question. I am somewhat
conflicted on the issue since I run one of these bots, but I will
admit I find the greasemonkey script to blow them away quite nice.
How's that for a definite maybe?

-Chad

On Fri, Mar 6, 2009 at 11:11 AM, Doug Williams <do...@igudo.com> wrote:
>

> Chad,
> In your experience, do trending bots have a disproportionate
> participation in the search results for trending topics? Have you done
> any analysis like that?
>
> Doug Williams
> @dougw

> - Show quoted text -

[Peter Denton]

what I think is also important is that the bots are then contributing to the trending itself, and eliminating other legitimate topics from trends.

[Matt Sanford]

Hi there,

    We've talked about this among the search folks a few times. We exclude a bunch of bots and things from influencing trends but then they still get displayed. I just opened a ticket for someone to fix that so we can exclude the trend bots using a parameter or search operator.

    As far as if this is the correct place for search or not, I think it is. If other Twitter API developers disagree please let me know and I'll start a second group. From my perspective keeping up with one is easier for me to manage … and we're planning to merge the APIs in the next version of the API.

Thanks;

  — Matt Sanford

[Nick Arnett]

On Fri, Mar 6, 2009 at 8:34 AM, Matt Sanford <ma...@twitter.com> wrote:

Hi there,

    We've talked about this among the search folks a few times. We exclude a bunch of bots and things from influencing trends but then they still get displayed. I just opened a ticket for someone to fix that so we can exclude the trend bots using a parameter or search operator.

    As far as if this is the correct place for search or not, I think it is. If other Twitter API developers disagree please let me know and I'll start a second group. From my perspective keeping up with one is easier for me to manage … and we're planning to merge the APIs in the next version of the API.

It would be terrific if users could self-identify as bots and that data became part of the user profile.  Although I'm sure that many people would not bother, we'd at least know that some of them definitively are bots. My bots self-identify in their description, which people seem to appreciate.   

Hmm.  Maybe it would be far easier to simply encourage a hashtag in the description - how about #bot?  That's something we could do now, without Twitter having to make any code changes.  Thoughts?

Nick

[Peter Denton]

I am skeptical that bot devs, (outside of the integrious Jazzy Chad), will do anything to encourage segregation, as it would probably lead to a nuking list at some point. I would say this has to be done programatically, with a "secret sauce" that is known to twitter only.  As search is more and more the golden goose apparent, gaming will be enemy number 1.

[Chad Etzel]

I agree, most ppl probably won't abide by any guidelines that they
have to 'voluntarily' follow in order to identify themselves at bots.
It's pretty darn easy to tell if something is a trend bot or not...
especially with the username :) Matt even said they've identified
them (uh oh, i'm on some kind of twitter watchlist.... but who watches
the watchlist?)

If twitter themselves ever incorporate auto-updating search results
like the special election pages, my bot and its links would pretty
much be rendered useless D:

-Chad

On Fri, Mar 6, 2009 at 11:53 AM, Peter Denton <peterm...@gmail.com> wrote:
> I am skeptical that bot devs, (outside of the integrious Jazzy Chad), will
> do anything to encourage segregation, as it would probably lead to a nuking
> list at some point. I would say this has to be done programatically, with a
> "secret sauce" that is known to twitter only.  As search is more and more
> the golden goose apparent, gaming will be enemy number 1.

> - Show quoted text -
>

[Nick Arnett]

Wrong place?  I don't think so...

I've got code that's doing a fairly job of identifying robots -- essentially any user who posts an unusually high number of tweets in a short time and doesn't follow many people.  (My goal is to analyze tweets by real people, not automated systems.)  The code is relatively simple -- it looks at how many updates the user has made over time, short-term and long-term.  My long-term period is 12 days, but that misses the new ones that seem to pop up every day, so I also look at the last 12 hours.  The long-term one also looks at the follower count.  If the user is following less than 100 people, that seems to be a strong clue that it's a bot.  I'm sure that I can refine the rules further, but it hasn't been a priority, since this seems to be working reasonably well.

There are a bit more than 1,800 in the list, which isn't just for trending bots, but all sorts of automated users.

I've added it as a blog page here: http://www.twurlednews.com/twitter-bots/

Grab everything between the pre tags and you'll have a CSV list of screen names and ids.

I assume it would be more useful as a plain CSV file.  If there's interest in that, I'll make it available and keep it updated.  The current update schedule for TwURLed News is 15 minutes, so it should stay fairly fresh.  Other than an API, which I don't really have time to create right now, is there another form this data could take that would be useful?  I could create a Twitter user and send a tweet every time I identify a new possible bot.  But if this is sufficient, I have other things I'd like to work on... ;-)

I've been thinking about adding a page that lists them, which would give a filter list that you could use to remove them from results.  One of these days, I might even put that in an API. This is something that's been done for years for user-agents on the web, so that advertisers and others can isolate robots from web analytics.

[Cameron Kaiser]

> I agree, most ppl probably won't abide by any guidelines that they
> have to 'voluntarily' follow in order to identify themselves at bots.
> It's pretty darn easy to tell if something is a trend bot or not...
> especially with the username :) Matt even said they've identified
> them (uh oh, i'm on some kind of twitter watchlist.... but who watches
> the watchlist?)

Dibs on @drmanhattanbot.

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com
-- All things are possible, except skiing through a revolving door. -----------

[Nick Arnett]

On Fri, Mar 6, 2009 at 11:49 AM, Cameron Kaiser <spe...@floodgap.com> wrote:

> I agree, most ppl probably won't abide by any guidelines that they
> have to 'voluntarily' follow in order to identify themselves at bots.
> It's pretty darn easy to tell if something is a trend bot or not...
> especially with the username :)  Matt even said they've identified
> them (uh oh, i'm on some kind of twitter watchlist.... but who watches
> the watchlist?)

Dibs on @drmanhattanbot.

I created @RoboTweeters this morning.  I'll probably start feeding it screen names and ids of the ones I find, since that's quite simple.

Nick 

[Chad Etzel]

On Fri, Mar 6, 2009 at 2:56 PM, Nick Arnett <nick....@gmail.com> wrote:
> I created @RoboTweeters this morning.  I'll probably start feeding it screen
> names and ids of the ones I find, since that's quite simple.
> Nick

Just make sure not to feed @RobotTweeters to itself... you may rip a
blackhole in the tweet/space continuum!

[Nick Arnett]

On Fri, Mar 6, 2009 at 12:05 PM, Chad Etzel <jazz...@gmail.com> wrote:

Just make sure not to feed @RobotTweeters to itself... you may rip a
blackhole in the tweet/space continuum!

I endeavor to never cross the streams.

Nick 

[TjL]

Here's the latest example of bots drowning out actual posts:

http://tntluoma.com/temp/TrendBots.png (screenshot)

Of 11 visible results:

4 are actual content (although 2 are ReTweets, which is another issue
altogether, but at least they are humans)

7 are bots.

Almost 2x as many bots as actual results.

IMO, "trend bots" should have to be registered with Twitter (they say
what they are going to use their API access for, right?) and should
excluded from Twitter search.

If people want to FOLLOW one of the Trend bots, fine, but they are
choking the usefulness of the trending topics list and with nothing
but echoing noise.

It's getting as bad as trying to go to Google to find hotel information.

TjL

[Cameron Kaiser]

> IMO, "trend bots" should have to be registered with Twitter (they say
> what they are going to use their API access for, right?) and should
> excluded from Twitter search.

How do you enforce bots registering as bots, however?

--
------------------------------------ personal: http://www.cameronkaiser.com/ --
Cameron Kaiser * Floodgap Systems * www.floodgap.com * cka...@floodgap.com

-- In defeat, unbeatable; in victory, unbearable. -- Churchill, on Montgomery -

[TjL]

On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser <spe...@floodgap.com> wrote:
>
>> IMO, "trend bots" should have to be registered with Twitter (they say
>> what they are going to use their API access for, right?) and should
>> excluded from Twitter search.
>
> How do you enforce bots registering as bots, however?

Well, revoking API whitelisting for any that don't register properly
would be a good first step.

Just a checkbox/radio button on the API whitelisting form should do.

That will deal with any new ones.

As for existing ones, well, just a matter of watching the Trending
Topics and ID'ing trending bots.

Add a banner on search.twitter.com which links to a blog post on the
Twitter blog for more information.

TjL

[Chad Etzel]

On Sun, Mar 8, 2009 at 9:04 PM, TjL <luo...@gmail.com> wrote:
>
> On Sun, Mar 8, 2009 at 7:37 PM, Cameron Kaiser <spe...@floodgap.com> wrote:
>>
>>> IMO, "trend bots" should have to be registered with Twitter (they say
>>> what they are going to use their API access for, right?) and should
>>> excluded from Twitter search.
>>
>> How do you enforce bots registering as bots, however?
>
> Well, revoking API whitelisting for any that don't register properly
> would be a good first step.

Huh? Bots don't need any sort of whitelisting to exist or function.
It's trivial to create and run one. It won't be so trivial once OAuth
hits, but I'm sure it won't be much of a barrier.

-chad

[TjL]

Ah. Well. My mistake.

Thanks

TjL

[Paul Kinlan]

On the topic of bots, http://www.itsabot.com works pretty well most of the time.

Paul

2009/3/9 TjL <luo...@gmail.com>

[Nick Arnett]

On Mon, Mar 9, 2009 at 6:26 AM, Paul Kinlan <paul....@gmail.com> wrote:

On the topic of bots, http://www.itsabot.com works pretty well most of the time.

My list is now updating live:

http://www.twurlednews.com/twitter-bots/

Nick

[TjL]

The more I think about this, the more I realize that there really
ought to be a "logged in" version of Twitter Search.

Not that you would HAVE to login, but IF you were logged in:

People you have BLOCKED would not appear.

People who have private accounts you follow WOULD appear.

That way you could just block bots and have them excluded from results.

Personal choice, FTW.

Now it just needs to be implemented :-)

TjL

[Chad Etzel]

oh noes... now even al3x is a trending bot:
http://twitter.com/al3x/status/1302097888

[Chad Etzel]

So it appears that most (if not all) of the trending bots have been
removed from search results as of 2 days ago, nice. I have also
noticed that the referral traffic from my bot links have dropped about
85% in the last two days.

Not complaining; I'm all for this change. Just noticing out loud.

-Chad