Twitter Harvesting Blocked?

[Stever]

Hello everyone,

I've been harvesting on Twitter for research etc. using pattern.web, and from a couple of days ago it stopped working - as in, Twitter is not responding to harvest requests.  I wonder what's going on, all other modules (Google, Newsfeed, etc.) work fine.  I wonder if my IP's gone on the blacklist somehow (pinging too much?)  If so is there a way to unblock this?

Steve

[Stever]

By the way, the code:

from pattern.web import Twitter
print Twitter().trends(cached=False)

produces this error:

Traceback (most recent call last):
  File "C:directory path\Twitter.py", line 5, in <module>
    print Twitter().trends(cached=False)
  File "C:\Python27\lib\site-packages\pattern\web\__init__.py", line 1234, in trends
    data = url.download(**kwargs)
  File "C:\Python27\lib\site-packages\pattern\web\__init__.py", line 394, in download
    data = self.open(timeout, proxy, user_agent, referrer, authentication).read()
  File "C:\Python27\lib\site-packages\pattern\web\__init__.py", line 363, in open
    raise HTTPError
HTTPError

[Stever]

Also I wonder if this (the new version of Twitter API) has to do with anything:

https://dev.twitter.com/blog/api-v1-is-retired

[Michele Orrù]

nope,

>>> from pattern import web
>>> web.TWITTER
'http://api.twitter.com/1.1/

--
ù

[Stever]

I got:

>>> web.TWITTER
'http://search.twitter.com/'

[Tom De Smedt]

Twitter updated their API a few days ago, so Pattern had to be updated too. You can download the latest revision from GitHub:

https://github.com/clips/pattern

There's a "ZIP" download button on the page. Twitter works again in the latest revision. Ideally, you should now pass a tweet id to the "start" parameter in Twitter().search().

In short:

from pattern.web import Twitter

twitter = Twitter()

last_id = None

for i in range(10):

for tweet in twitter.search("cats", start=last_id, count=100, cached=False):

print tweet.text

last_id = tweet.id

... which gives you a 1,000 tweets about cats.

Also check the example in:

pattern/examples/01-web/04-twitter.py

Best,

Tom

--
 
---
You received this message because you are subscribed to the Google Groups "Pattern" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pattern-for-pyt...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[modciv]

Tom: I've been getting a pattern.web.HTTP403Forbidden error w/ this lately. Has something changed again?

To unsubscribe from this group and stop receiving emails from it, send an email to pattern-for-python+unsub...@googlegroups.com.

[Tom De Smedt]

You should be fine if you use the latest GitHub revision (http://www.github.com/clips/pattern). Haven't had time yet to update the official download on http://www.clips.ua.ac.be/pattern