track syntax

[Joseph]

I am trying to use the track streaming API, but I'm having trouble
with the syntax. The example shows how to use a tracking list stored
in a file in the format: track = word1, word2, etc..

I tried the following (following a successful fsockopen call to
stream.twitter.api:

POST /track.json track = Palin, #fubar HTTP/1.1 Host:
stream.twitter.com User-Agent: UserAgent Authorization: Basic
cGXybmFzc3TzZGV2OlBhcm5hMzT1Mzl4MDMz Connection: Close

and I am getting the following error code: HTTP/1.1 400 Bad Request

The actual relevant test PHP code is:

$fp = fsockopen($url, 80, $errno, $errstr, 30);
if (!$fp) {
echo "$errstr ($errno)\n";
} else {
echo "file handle: ".$fp."<br/>";
$header = "POST /track.json track = $trackTerms HTTP/1.1\r\n";
$header .= "Host: stream.twitter.com\r\n";
$header .= "User-Agent: UserAgent\r\n";
$header .= "Authorization: Basic " . base64_encode($twitUsername .
':' . $twitPwd)."\r\n";
$header .= "Connection: Close\r\n\r\n";
echo $header."<br/>";
fwrite($fp, $header);
$line = fgets($fp, 4096);
echo $line;
fclose($fp);
}

[JDG]

a) the POST body (i.e. track=Palin) must come after ALL headers.
b) change your password, you just sent it to the whole dev community.

--
Internets. Serious business.

[Joseph]

Thanks. I should have mentioned that the password is not real (changed
the data, but kept the format to illustrate what exactly is being
used).

[John Kalucki]

At first glance, your question is really about HTTP, not Track, as
your request appears to be malformed. I'd suggest using an existing
HTTP client if at all possible. You could also contrast your
implementation with: http://tools.ietf.org/html/rfc2616#section-4.1.
Or, get your query working with curl(1) and capture the interaction
with a protocol decoder, perhaps Wireshark. Then analyze your client,
and fix the deltas.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.

[Joel Strellner]

Here is a working example of how to do /track:

$count = 1;
$startparsing = false;

$keyword_needles[] = 'keyword1';
$keyword_needles[] = 'keyword2';
$keyword_needles[] = 'keyword3';
$keyword_needles[] = 'keyword4';

// if your keywords have spaces, they must be urlencoded (twitter does not support phrases, only the first keyword will be used, the space character and after will be ignored)
foreach ($keyword_needles AS $i=>$needle) {
    $keyword_needles[$i] = urlencode($needle);
}

$poststr = 'track=' . implode(',', $keyword_needles);
$fp = fsockopen("stream.twitter.com", 80, $errno, $errstr, 30);

if (!$fp) {
    echo "$errstr ($errno)\n";
} else {

    $out = "POST /track.json HTTP/1.1\r\n";
    $out .= "Host: stream.twitter.com\r\n";
    $out .= "User-Agent: YourUserAgent\r\n";
    $out .= "Referer: http://yourdomain.com\r\n";
    $out .= "Content-Type: application/x-www-form-urlencoded\r\n";
    $out .= "Authorization: Basic " . base64_encode("username:password")."\r\n";
    $out .= "Content-length: " . strlen($poststr) . "\r\n";
    $out .= "Connection: Close\r\n\r\n";
    $out .= $poststr . "\r\n\r\n";
       
    fwrite($fp, $out);
    while (!feof($fp)) {
        $line = fgets($fp, 4096);
        if ($startparsing) {
            if (trim($line) != '') {
                //echo trim($line) . "\n";
                $tweet_obj = json_decode(trim($line));
                // do your stuff here
            }
        }
        else {
            // view the header lines: uncomment the below line
            //echo trim($line) . "\n";
              
            $header_arr[] = $line;
            $headercount = count($header_arr)-1;

            if (trim($header_arr[$headercount]) == '') {
                $startparsing = true;
                $count = 1;
                unset($header_arr, $headercount);
            }
        }
        if (trim($line) != '') $count++;
    }
    fclose($fp);

[Tom Fitzgerald]

Joel,

For some reason when I try your code I get a timeout error. Any
suggestions? What you have is exactly what I'm looking for. It could
really help me out a jam, thanks!

[Joel Strellner]

Other than my username and password, this is an example that I know is
working:

<?php

$count = 1;
$startparsing = false;

$keyword_needles[] = 'twitter';

$keyword_needles[] = 'keyword2';
$keyword_needles[] = 'keyword3';
$keyword_needles[] = 'keyword4';

// if your keywords have spaces, they must be urlencoded (twitter does
not support phrases, only the first keyword will be used, the space
character and after will be ignored)
foreach ($keyword_needles AS $i=>$needle) {
$keyword_needles[$i] = urlencode($needle);
}

$poststr = 'track=' . implode(',', $keyword_needles);

$fp = fsockopen("stream.twitter.com", 80, $errno, $errstr, 10);

if (!$fp) {
echo "$errstr ($errno)\n";
} else {

$out = "POST /track.json HTTP/1.1\r\n";
$out .= "Host: stream.twitter.com\r\n";
$out .= "User-Agent: YourUserAgent\r\n";
$out .= "Referer: http://yourdomain.com\r\n";
$out .= "Content-Type: application/x-www-form-urlencoded\r\n";
$out .= "Authorization: Basic " . base64_encode
("username:password")."\r\n";
$out .= "Content-length: " . strlen($poststr) . "\r\n";
$out .= "Connection: Close\r\n\r\n";
$out .= $poststr . "\r\n\r\n";

fwrite($fp, $out);
while (!feof($fp)) {
$line = fgets($fp, 4096);
if ($startparsing) {
if (trim($line) != '') {

echo trim($line) . "\n";
$tweet_obj = json_decode(trim($line));
// do your stuff here
}
}
else {
// view the header lines: uncomment the below line

echo trim($line) . "\n";

$header_arr[] = $line;
$headercount = count($header_arr)-1;

if (trim($header_arr[$headercount]) == '') {
$startparsing = true;
$count = 1;
unset($header_arr, $headercount);
}
}
if (trim($line) != '') $count++;
}
fclose($fp);
}

?>

The only changes I made was to echo things out and I added a keyword
that I was sure would have volume - twitter. It did take a little bit
of time to connect. I am assuming that that is because of their
current load though, and not the script.

[Tom Fitzgerald]

I appreciate the reply Joel. I'll give it a try. I also tried just
downloading from the stream api with the curl command line. However I
kept getting 'malformed xml' errors. It was weird, each tweet would
have the <?xml ... tag before it. That ring any bells with you? Same
thing with JSON format but it was a different error, still malformed.
All I'm doing is

curl http://stream.twitter.com/spritzer.xml -uuser:pass

<?xml version="1.0" encoding="UTF-8"?> is the exact line I get before
every status. If I manually clean up the XML (or JSON) it works great.

[Joel Strellner]

Hi Tom,

I am not sure about XML, since I use JSON - it has a much lower over-the-wire data size, and its easier to parse.

Let me know if the code works for you.

-Joel

[Tom Fitzgerald]

I'm sorry Joel I keep getting a PHP timeout on the code you sent. I'll
troubleshoot more and see if I can give you any more details (increase
the maximum time, etc). Who knows, maybe its Twitter. Any other
thoughts? I'll get back to you more with some detailed info. Are you
able to get that exact code working on your server?

On Aug 3, 2:12 pm, Joel Strellner <j...@twitturly.com> wrote:
> Hi Tom,
>
> I am not sure about XML, since I use JSON - it has a much lower
> over-the-wire data size, and its easier to parse.
>
> Let me know if the code works for you.
>
> -Joel
>
>
>
> On Mon, Aug 3, 2009 at 10:55 AM, Tom Fitzgerald <ccexpe...@gmail.com> wrote:
>
> > I appreciate the reply Joel. I'll give it a try. I also tried just
> > downloading from the stream api with the curl command line. However I
> > kept getting 'malformed xml' errors. It was weird, each tweet would
> > have the <?xml ... tag before it. That ring any bells with you? Same
> > thing with JSON format but it was a different error, still malformed.
> > All I'm doing is
>

> > curlhttp://stream.twitter.com/spritzer.xml-uuser:pass

[Joel Strellner]

Tom,

Yes, that code works perfectly for me exactly as is.  You might want to change the connect timeout from 10 to 30 seconds.

How long are you waiting before calling it quits?  It does take a few seconds for /track to start sending your results.

-Joel

[Polymatheus]

I've used the code above to start streaming and then dumping the
output to a text file every hour to process later. There are a few
things I want to clarify,

1) How can the above script be amended to show both follow and track?
Is this possible?
2) If I opened a stream to follow 10 users and then a further 5 users
joined my site, would I have to close the first stream then open a new
stream for the 15 follows? Or just a second stream with the additional
5 users? The second approach reduce the chance of tweets being lost
between closing and opening a new stream.
3) I can't seem to close a stream that I opened using the sample code
above, I have tried changing the file to simply say:

$fp = fsockopen("stream.twitter.com", 80, $errno, $errstr, 10);

fclose($fp);

But it doesn't appear to work :/

Thanks in advance

[John Kalucki]

You can set both the track and follow parameters when using the /1/
statuses/filter URL.

Best practices around changing your predicate:
http://apiwiki.twitter.com/Streaming-API-Documentation#UpdatingFilterPredicates

I can't answer PHP questions, sorry.

-John Kalucki
http://twitter.com/jkalucki
Services, Twitter Inc.

> ...
>
> read more »

[Abraham Williams]

On Mon, Aug 31, 2009 at 14:32, Polymatheus <world...@gmail.com> wrote:

3) I can't seem to close a stream that I opened using the sample code
above, I have tried changing the file to simply say:

$fp = fsockopen("stream.twitter.com", 80, $errno, $errstr, 10);

fclose($fp);

Maybe this will fix your problem: http://us.php.net/manual/en/function.fclose.php#65654

--
Abraham Williams | Community Evangelist | http://web608.org
Hacker | http://abrah.am | http://twitter.com/abraham
Project | http://fireeagle.labs.poseurtech.com
This email is: [ ] blogable [x] ask first [ ] private.
Sent from Madison, WI, United States

[Joseph Cheek]

are you really just opening stream.twitter.org? Normally you would want
to open http://stream.twitter.org/path/to/url.xml...

Joseph Cheek
jos...@cheek.com, www.cheek.com
twitter: http://twitter.com/cheekdotcom

[Joel Strellner]

Hi Polymatheus,

1) How can the above script be amended to show both follow and track?
Is this possible?

John's suggestion is the only way, you must use the new paths.

2) If I opened a stream to follow 10 users and then a further 5 users
joined my site, would I have to close the first stream then open a new
stream for the 15 follows? Or just a second stream with the additional
5 users? The second approach reduce the chance of tweets being lost
between closing and opening a new stream.

You must restart the stream, unfortunately, there is no way around this at this point.

3) I can't seem to close a stream that I opened using the sample code

Abraham gives a potential solution to this, and I believe its along the right path.  I normally only see this if we are lagging behind for whatever reason. 

Let me know if you have any further questions, since I am the person that wrote the code you're using.

-Joel

[Joel Strellner]

If he really is using only those two lines, then yes, it explains some of his error.  The header lines are what tells the server what page you're looking for on stream.twitter.com.

-Joel

[John OBrien (TwapperKeeper)]

Joel - thx for this bit of code... very helpful.

John
http://twapperkeeper.com

@jobrieniii
http://www.linkedin.com/in/jobrieniii