User Stream shows isn't catching any tweets
Tutorial
I have an issue with User Streams when using phirehose , so I was
hoping that someone here might have a good clue.
The problem is this: I make a successful connection using oAuth (at
least it appears so), but after the connection has been established,
there isn't any data flowing into the stream; I don't even get list of
friends. Output that phirehose creates looks like this:
Phirehose: Connecting to twitter stream: https://userstream.twitter.com/2/user.json
with params: array ( 'delimited' => 'length',)
Phirehose: Resolved host userstream.twitter.com to 199.59.148.139
Phirehose: Connecting to 199.59.148.139
Phirehose: Connection established to 199.59.148.139
Phirehose: DEBUG: 'GET&https%3A%2F%2Fuserstream.twitter.com
%2F2%2Fuser.json&delimited%3Dlength%26oauth_consumer_key
%3Dxxxxxxxxxxxxxxxxxxxx%26oauth_nonce%3Dxxxxxxxxxxxxxxxxxxxx
%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp
%3D1326993652%26oauth_token%3Dxxxxxxxxxxxxxxxxxxxx%26oauth_version
%3D1.0'
Phirehose: GET /2/user.json HTTP/1.0
Phirehose: Host: userstream.twitter.com:443
Phirehose: Content-type: application/x-www-form-urlencoded
Phirehose: Content-length: 16
Phirehose: User-Agent: phirehose
Phirehose: Authorization: OAuth realm="https://userstream.twitter.com/
2/user.json", oauth_consumer_key="xxxxxxxxxxxxxxxxxxxx",
oauth_token="xxxxxxxxxxxxxxxxxxxx",
oauth_nonce="xxxxxxxxxxxxxxxxxxxx", oauth_timestamp="1326993652",
oauth_signature_method="HMAC-SHA1", oauth_version="1.0",
oauth_signature="xxxxxxxxxxxxxxxxxxxx"
Phirehose:
Phirehose: delimited=length
Phirehose:
Phirehose: Consume rate: 0 status/sec (0 total), avg enqueueStatus():
0ms, avg checkFilterPredicates(): 0ms (0 total) over 69 seconds, max
stream idle period: 30 seconds.
This last line keeps repeating until I terminate phirehose's process.
I should probably also note that when I use curl with signature
provided by Twitter, the stream opens and data starts to flow. Also,
everything works just fine when I use Streaming API.
This has been really bugging me for the past few days and I feel like
there is a silly error somewhere, but can't really figure out what it
is. The line with "DEBUG" looks suspicious, but I can't be sure
because I obviously don't know enough about this library.
I've read about similar problems and the only thing I came up with was
to replace all post requests with get requests, but this made no
improvements whatsoever.
The code I used to try this out was userstream-example.php (https://
github.com/fennb/phirehose/blob/master/example/userstream-simple.php).
The only thing I changed were tokens.
Thank you in advance for any help.
Fenn Bailey
Tutorial
1) I'm using the same tokens that I'm using for curl, which works, so
I guess that those tokens should be correct. At first they really were
wrong, but then I was notified that authentication error has occurred.
I don't get this message any more, so I believe it's safe to assume
tokens are valid.
As far as I understand, I am watching accounts of people who have
granted access to my application. I've been testing this with my
application created only for testing purposes and I'm the only one who
granted access to this app, so I guess I should be getting tweets
written by people I follow and no-one else (unless someone else grants
access to the app).
2) I'm aware that this is a live stream, but the problem (for me) lies
in a fact that method enqueueStatus never gets invoked. By default,
this method prints out some received data, but I never get any output.
I've also tried storing data into a txt file, but this doesn't work
either (I didn't want to create a file in advance, but left it to the
function to open it itself, which it never does).
When using curl signature generated by Twitter, everything works as
expected. I get a list of friends and then I see new tweets after they
get written by someone I follow.
Thanks!
Hrvoje
Fenn Bailey
As far as I understand, I am watching accounts of people who have
granted access to my application.
Darren Cook
> in a fact that method enqueueStatus never gets invoked.
I just tried userstream-simple.php and noticed two things:
1. A minor bug:
Notice: Undefined variable: sz in ... line 38
2. I get the list of friends, but then I get:
Array
(
[errors] => Array
(
[0] => Array
(
[code] => WAR400
[message] => Http request illformed, some data ignored
)
)
)
I've not seen that before. At first glance it looks like something has
changed on the twitter side (I may not have run that test script since
Oct '11); perhaps phirehose relies on something that was undocumented
that they've now stopped doing. I'll do a bit more investigation now.
But, if the same problem you have, why don't you see that same error
message? Do you php.ini settings send everything to the log files and to
stdout/stderr? I suggest setting up a php.ini for the cli version, and
have startup and normal error messages go to stderr instead of the log
file (*).
Darren
*: Definitely for development machines; it is also my preference for
production machines. However, never do this for production machines for
a web-site.
--
Darren Cook, Software Researcher/Developer
http://dcook.org/work/ (About me and my work)
http://dcook.org/blogs.html (My blogs and articles)
Tutorial
answers and to try out everything you've suggested. I'll let you know
how it all went as soon as I try it all out.
Darren Cook
> I just tried userstream-simple.php and noticed two things:
>
> 1. A minor bug:
> Notice: Undefined variable: sz in ... line 38
Fixed in a Dec 11th update. (And, no, that wasn't what $sz was doing; it
was supposed to be the running total of all bytes received in that
session. But the fixed version is fine too.)
> [code] => WAR400
> [message] => Http request illformed, some data ignored
I don't get this any more. Either it was coincidence or fixed in the one
of the three changes [1] to UserstreamPhirehose.php that I just updated to.
Darren
[1]:
Tue Nov 29 23:14:27 2011 -0800:
FIX: get UserstreamPhirehose class working again (replace use of
TCP_BACKOFF and HTTP_BACKOFF with the class members)
Thu Dec 1 11:29:39 2011 +1100
Fixed inconsistent code, indentation and other cleanups.
Fixed inconsistent braces, mixed indentation, invalid comment chars (#is
not valid for PHP);
Switched raw var_dump to proper log output;
Other minor cleanups;
Sat Dec 10 17:46:26 2011 -0800:
Missing USER_AGENT class constant in UserstreamPhirehose
Darren Cook
> least it appears so), but after the connection has been established,
> there isn't any data flowing into the stream; I don't even get list of
> friends.
[1] shows the initial output for me.
It was doing nothing, so I logged in to twitter.com, and followed a more
active user, and then I did a test tweet, and then the new user gave a
tweet [2]. Later, I deleted my test tweet and saw [3].
I hope that helps. It shows you can still log in to twitter.com while
connected to a user stream, and can do follow/tweet/delete commands that
you should see reflected in your userstream immediately.
Darren
[1]
Phirehose: Connecting to twitter stream:
https://userstream.twitter.com/2/user.json with params: array (
'delimited' => 'length',)
Phirehose: Resolved host userstream.twitter.com to 199.59.148.139
Phirehose: Connecting to 199.59.148.139
Phirehose: Connection established to 199.59.148.139
Phirehose: DEBUG:
'POST&https%3A%2F%2Fuserstream.twitter.com%2F2%2Fuser.json&delimited%3Dlength%26oauth_consumer_key%3D########%26oauth_nonce%3D########%26oauth_signature_method%3DHMAC-SHA1%26oauth_timestamp%3D1327023222%26oauth_token%3D########%26oauth_version%3D1.0'
Phirehose: POST /2/user.json HTTP/1.0
Phirehose: Host: userstream.twitter.com:443
Phirehose: Content-type: application/x-www-form-urlencoded
Phirehose: Content-length: 16
Phirehose: User-Agent: Phirehose (https://github.com/fennb/phirehose)
Phirehose: Authorization: OAuth
realm="https://userstream.twitter.com/2/user.json",
oauth_consumer_key="#########", oauth_token="######",
oauth_nonce="#######", oauth_timestamp="1327023222",
oauth_signature_method="HMAC-SHA1", oauth_version="1.0",
oauth_signature="#####"
Phirehose:
Phirehose: delimited=length
Phirehose:
2012-01-20 10:33:42 (149):Array
(
[friends] => Array
(
[0] => 1234
....
[13] => 1234
)
)
Phirehose: Consume rate: 0 status/sec (1 total), avg enqueueStatus():
3.31ms, avg checkFilterPredicates(): 0ms (0 total) over 60 seconds, max
stream idle period: 30 seconds.
Phirehose: Consume rate: 0 status/sec (0 total), avg enqueueStatus():
0ms, avg checkFilterPredicates(): 0ms (0 total) over 60 seconds, max
stream idle period: 30 seconds.
...
[2]:
2012-01-20 10:43:55 (2764):Array
(
[target] => Array
(
[lang] => en
[geo_enabled] =>
...
)
[event] => follow
[created_at] => Fri Jan 20 01:43:48 +0000 2012
[source] => Array
(
[lang] => en
[geo_enabled] =>
)
)
2012-01-20 10:44:34 (1844):Array
(
[retweet_count] => 0
[favorited] =>
[text] => This is my test tweet, please ignore
[in_reply_to_status_id_str] =>
...
[in_reply_to_status_id] =>
)
Phirehose: Consume rate: 0 status/sec (2 total), avg enqueueStatus():
0.27ms, avg checkFilterPredicates(): 0ms (0 total) over 82 seconds, max
stream idle period: 25 seconds.
2012-01-20 10:45:10 (2226):Array
(
[retweet_count] => 0
[favorited] =>
[text] => Australia 4Q Import Price Index (QoQ) 2.5%
http://t.co/IDiqfZ0y #forex #trading
[in_reply_to_status_id_str] =>
...
[3]
2012-01-20 10:59:15 (125):Array
(
[delete] => Array
(
[status] => Array
(
[user_id_str] => 239273333
[id_str] => 160175827634503680
[id] => 160175827634503680
[user_id] => 239273333
)
)
)
Tutorial
I redid all the tests and now it works just fine, though I made no
changes to the code whatsoever. I don't know which is worse: that it's
working without making no changes to the code or the fact that I don't
know if it might happen again and how to fix it. I really hope that
this won't happen to anyone else.
Thank you both for your help, it is much appreciated. I apologize for
spending your time on my app that decide to be stochastic.
HG
On Jan 20, 3:02 am, Darren Cook <dar...@dcook.org> wrote:
> > The problem is this: I make a successful connection using oAuth (at
> > least it appears so), but after the connection has been established,
> > there isn't any data flowing into the stream; I don't even get list of
> > friends.
>
> [1] shows the initial output for me.
> It was doing nothing, so I logged in to twitter.com, and followed a more
> active user, and then I did a test tweet, and then the new user gave a
> tweet [2]. Later, I deleted my test tweet and saw [3].
>
> I hope that helps. It shows you can still log in to twitter.com while
> connected to a user stream, and can do follow/tweet/delete commands that
> you should see reflected in your userstream immediately.
>
> Darren
>
> [1]
> Phirehose: Resolved host userstream.twitter.com to 199.59.148.139
> Phirehose: Connecting to 199.59.148.139
> Phirehose: Connection established to 199.59.148.139
> Phirehose: DEBUG: