help with accessing the streaming api

[omri]

def run (self):
status_url = "http://stream.twitter.com/1/statuses/filter.json"
request = urllib2.Request(status_url)
print request
auth = base64.encodestring('Consumer key:Consumer secret')[:-1]
request.add_header('Authorization', 'basic %s' % auth)
firehose = urllib2.urlopen(request)
for tweet in firehose:
print tweet

firehose.close

when printing the request i have 500 error, it is not reachable.

do you have an idea why?

[Lenin]

Dont you know that the BasicAuth has been disabled for good?

[Tom van der Woerdt]

Not for Streams.

Tom

On 9/7/10 1:52 PM, Lenin wrote:
> Dont you know that the BasicAuth has been disabled for good?
>

> --
> Twitter developer documentation and resources: http://dev.twitter.com/doc
> API updates via Twitter: http://twitter.com/twitterapi
> Issues/Enhancements Tracker:
> http://code.google.com/p/twitter-api/issues/list
> Change your membership to this group:
> http://groups.google.com/group/twitter-development-talk?hl=en

[John Kalucki]

The Streaming API will print a short text message along with errors that is often a very explicit indication of the problem. I don't think it throws 500 errors (see: http://dev.twitter.com/pages/streaming_api_response_codes) so something odd is going on. If anything, it is throwing a 406, as you haven't provided parameters to track.

-John Kalucki

http://twitter.com/jkalucki

Twitter, Inc.

[omri]

hi,
now it returns the 401 error.
It seems like I don't know what is the username and password i should
insert.

is this ths oauth token?
what is the title of this fields?

API key, Consumer key, Consumer secret?

thanks

On 7 ספטמבר, 17:16, John Kalucki <j...@twitter.com> wrote:
> The Streaming API will print a short text message along with errors that is
> often a very explicit indication of the problem. I don't think it throws 500
> errors (see:http://dev.twitter.com/pages/streaming_api_response_codes) so
> something odd is going on. If anything, it is throwing a 406, as you haven't
> provided parameters to track.
>

> -John Kaluckihttp://twitter.com/jkalucki

[John Kalucki]

What text message does it return with the 401 error?

You can still use basic auth with streaming. Does that work for you? You should use your screenname and password for basic auth.

-John Kalucki

http://twitter.com/jkalucki

Twitter, Inc.

[John Kalucki]

Screenname and password. Your @username, not your email address.

-John

On Thu, Sep 9, 2010 at 6:56 AM, omri <omrid...@gmail.com> wrote:

this is the message i get :

 File &quot;C:\Python26\lib\urllib2.py&quot;, line 516, in
http_error_default
   raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Unauthorized

i think i dont know which username and password to type in the header
- 'Authorization' 'basic (username:password)
the twitter's username?
the twitter api oauth?

thanks

On 9 ספטמבר, 15:42, John Kalucki <j...@twitter.com> wrote:
> What text message does it return with the 401 error?
>
> You can still use basic auth with streaming. Does that work for you? You
> should use your screenname and password for basic auth.
>

[omri]

this is the message i get :

File &quot;C:\Python26\lib\urllib2.py&quot;, line 516, in
http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 401: Unauthorized

i think i dont know which username and password to type in the header
- 'Authorization' 'basic (username:password)
the twitter's username?
the twitter api oauth?

thanks


On 9 ספטמבר, 15:42, John Kalucki <j...@twitter.com> wrote:

> What text message does it return with the 401 error?
>
> You can still use basic auth with streaming. Does that work for you? You
> should use your screenname and password for basic auth.
>

[omri]

a

On 9 ספטמבר, 15:42, John Kalucki <j...@twitter.com> wrote:

> What text message does it return with the 401 error?
>
> You can still use basic auth with streaming. Does that work for you? You
> should use your screenname and password for basic auth.
>

[Taylor Singletary]

Hi Omri,

With basic auth, you don't use your consumer key or secret at all. Consumer key and secret are used when authorizing through OAuth, and the implementation is a bit more detailed than putting them in a header.

In your case, you want to use basic authorization, which should be easier for your initial implementation.

Here are some pointers in the right direction for using basic auth with Python's urllib2: http://docs.python.org/library/urllib2.html#examples

Taylor

On Thu, Sep 9, 2010 at 6:56 AM, omri <omrid...@gmail.com> wrote:

[omri]

so..
succeded with the username and password but now i have :
HTTPError: HTTP Error 403: Forbidden

I need something special to have access to the streaming resources?

On 9 ספטמבר, 17:03, Taylor Singletary <taylorsinglet...@twitter.com>
wrote:

[Taylor Singletary]

Hi Omri,

The Streaming API has various levels of access. The most basic doesn't require any kind of permission, and that's utilizing the sampling resource, documentation for which can be found here: http://dev.twitter.com/pages/streaming_api_methods#statuses-sample and http://dev.twitter.com/pages/streaming_api_concepts#sampling

For additional permissions, you'll want to fill out this form: http://twitter.com/help/request_streaming as well as send a note to a...@twitter.com (from the same email address associated with the account you used the form) specifically asking for the role that makes sense for your implementation. 

It's best to get started with the sampling stream, as it lets you begin coding against the streaming API from a stream that isn't too aggressive in the amount of data it will be throwing at you. 

What are you interested in building?

Taylor

[John Kalucki]

Which URL are you requesting? What is the text message returned? Does this doc help? http://dev.twitter.com/pages/streaming_api_response_codes

-John

On Thu, Sep 9, 2010 at 7:45 AM, omri <omrid...@gmail.com> wrote:

[omri]

I request this URL :
http://stream.twitter.com/1/statuses/firehose.json

i get this message on my compiler :
WARNING 2010-09-11 07:31:30,447 urlfetch_stub.py:284] Stripped
prohibited headers from URLFetch request: ['Host']

i think something is wrong with my permission.

I read this doc.

This is the code i wrote in prder to use the streaming :

status_url = "http://stream.twitter.com/1/statuses/firehose.json"
request = urllib2.Request(status_url)
auth = base64.encodestring('(username):(password)')[:-1]

request.add_header('Authorization', "basic %s" % auth)
firehose = urllib2.urlopen(request)
for tweet in firehose:
print tweet

firehose.close

thanks again

On 9 ספטמבר, 18:02, John Kalucki <j...@twitter.com> wrote:
> Which URL are you requesting? What is the text message returned? Does this

> doc help?http://dev.twitter.com/pages/streaming_api_response_codes

[John Kalucki]

I think the best debugging process is to reproduce your problem in curl, then show the curl -v output (passwords elided, naturally). If curl works and your home-rolled client does not, then you can use tcpdump(1) or some other packet sniffer and work out the deltas between the working solution and the non-working solution.

In this case, do you have access to the firehose? Can you use an existing client library? There are dozens of clients for the Streaming API out there.

-John Kalucki

http://twitter.com/jkalucki

Twitter, Inc.

[mynameisgabe]

I've been googling like crazy and searching the streaming api docs for
the answer to this question: Where would I find the latest definition
for json objects returned by the stream? Specifically I'm looking for
field names, data types and max lengths (if available) that will be
returned. I'm using the filter tracking several keywords.

Thanks!

Gabe

On Sep 11, 6:11 am, John Kalucki <j...@twitter.com> wrote:
> I think the best debugging process is to reproduce your problem in curl,
> then show the curl -v output (passwords elided, naturally). If curl works
> and your home-rolled client does not, then you can use tcpdump(1) or some
> other packet sniffer and work out the deltas between the working solution
> and the non-working solution.
>
> In this case, do you have access to the firehose? Can you use an existing
> client library? There are dozens of clients for the Streaming API out there.
>

[M. Edward (Ed) Borasky]

Quoting mynameisgabe <myname...@gmail.com>:

> I've been googling like crazy and searching the streaming api docs for
> the answer to this question: Where would I find the latest definition
> for json objects returned by the stream? Specifically I'm looking for
> field names, data types and max lengths (if available) that will be
> returned. I'm using the filter tracking several keywords.
>
> Thanks!
>
> Gabe

I don't think I've seen anything like that - Twitter tends to announce
format changes on this list, but I don't know that there's a
"reference document" anywhere. When I write Streaming tests, I usually
just grab whatever comes down the pipe and stash it away as text, or
parse the JSON to a Perl or Ruby object and "flatten" the resulting
hash. Of course, I'm just generating CSV - more advanced users might
simply be using a leading-edge NoSQL data store. ;-)

--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos

[Gabriel Harriman]

Oh wow, I like the NoSQL data store idea. =)

[Abraham Williams]

This is about as close as you will get and it is probably outdated already. http://mehack.com/map-of-a-twitter-status-object

Abraham
-------------
Abraham Williams | Hacker Advocate | http://abrah.am
@abraham | http://projects.abrah.am | http://blog.abrah.am
This email is: [ ] shareable [x] ask first [ ] private.

[M. Edward (Ed) Borasky]

Yeah - lots of them to choose from, although Twitter has invested a
fair amount of time in Cassandra and Hadoop/Pig.

--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos

[Abraham Williams]

They ditched the effort to switch the status store to Cassandra though. http://engineering.twitter.com/2010/07/cassandra-at-twitter-today.html

Abraham
-------------
Abraham Williams | Hacker Advocate | http://abrah.am
@abraham | http://projects.abrah.am | http://blog.abrah.am
This email is: [ ] shareable [x] ask first [ ] private.

[M. Edward (Ed) Borasky]

Yeah - at my current scale, I have absolutely no desire to use
anything but PostgreSQL. It has a bunch of neat stuff like full text
search and some key-value store capabilities, plus JSON storage is
coming (in about a year). Solid as a rock and no licensing problems
(except from Oracle pushing them off of previously "donated" SPARC
testing hardware.) ;-)

The only downside to PostgreSQL is that they really don't like Ruby or
Rails. But they're *very* Python and Perl friendly and the preferred
database for Django.

--
M. Edward (Ed) Borasky
http://borasky-research.net http://twitter.com/znmeb

"A mathematician is a device for turning coffee into theorems." - Paul Erdos