is tweetpayment notify broken?

[divesnob]

Sometimes I go to notify tipjoy about a tweet that has a payment and I
get the "Sorry, something went wrong! We're looking into it." page.
Which is odd - because I'd expect an error from the API - not a bunch
of HTML.

Here's the tweet id i'm trying to process. This happens 100% of the
time for some accounts and 0% for other accounts.

-matt

[Ivan Kirigin]

Can you give me a case where it happens 100% of the time? :-)

You're POSTing, right?

I should be returning a JSON error, not a bunch of HTML. That junk is
just the built in 500 error.

I should make a decorator for API endpoints that ensures the response
is easily readable from software.

Ivan

[divesnob]

Perhaps it's more related to the fact that my twitter account was set
to private? I have authorized tipjoy as a follower - so that account
should be able to view my tweets. Right?

When I make my status updates public, the api call works. However, it
works as long as I don't tip myself. If I tip myself and notify
tipjoy, I get an error:

{"reason": "Payment exceeds $500, so the user must pre-fill their
Tipjoy account", "runTime": "0:00:00.144357", "request": "/api/
tweetpayment/notify/", "result": "failure"}

Again though, the main problem appears that I shouldn't have to have a
public account for the tipjoy tweets to get verified via the notify
api call.

-matt

On Jun 4, 2:23 pm, Ivan Kirigin <ivan.kiri...@gmail.com> wrote:
> Can you give me a case where it happens 100% of the time? :-)
>
> You're POSTing, right?
>
> I should be returning a JSON error, not a bunch of HTML. That junk is
> just the built in 500 error.
>
> I should make a decorator for API endpoints that ensures the response
> is easily readable from software.
>
> Ivan
>

[divesnob]

To be clear, the account that I was using is "tobea_dev".

-matt

[Ivan Kirigin]

I need to check if I make the tweet retrieval authenticated. I'd run
into whiteist issues with that approach though, so might require some
kind of authentication.

Ivan
via iphone
http://tipjoy.com 617-777-4317

[divesnob]

Twitter's got the whitelisting request for those that are making many
requests/hour. Are you not whitelisted already? 20k requests/hour is
quite a lot.

As an alternative, couldn't tipjoy use the oauth headers that we pass
in? That way the onus of the rate limiter is on the user who is
requesting the tip.

-matt

On Jun 4, 3:45 pm, Ivan Kirigin <ivan.kiri...@gmail.com> wrote:
> I need to check if I make the tweet retrieval authenticated. I'd run  
> into whiteist issues with that approach though, so might require some  
> kind of authentication.
>
> Ivan
> via iphonehttp://tipjoy.com617-777-4317
>

[Ivan Kirigin]

I thought the limit was 20K daily. Hourly is so much better. I'll need
to update my time.sleep() calls in the processes that poll twitter.
This also could help that other bug with the double payments, as
that's probably a race condition. To that end, I'm getting on the
"firehose" - a realtime streaming API endpoint to get all the tweets
from those with Tipjoy accounts.

Note that even whitelisted, the vulnerability is that an attacker
would use up our requests. We not only need to add rate limiting to
our own endpoints (nothing significant, just to stop attackers), but
should also add authentication to the call.

Ivan
http://tipjoy.com

[dinosaur robot]

Ivan, congrats on getting the firehose! Good work :-)

[Ivan Kirigin]

Just to clarify, it isn't the "full firehose'. It's initially going to
be birddog.

For those curious, birddog requires approval from Twitter and a
licensing agreement. It's a "closed alpha"
http://apiwiki.twitter.com/Streaming-API-Documentation?SearchFor=streaming+api&sp=1#birddog

If we decide to automatically make Tipjoy accounts for anyone who
tweets "p $X @user", then we'll switch to the firehose. Switching to a
streaming client is the hard part. Dialing up from birddog to the
firehose should be easy. There isn't actually that much data.

I'm told you can process it on "a single box"

Ivan
http://tipjoy.com

[dinosaur robot]

Ivan,

>"Switching to a streaming client is the hard part."

I bet. I just learned about streaming data to a client, so I'd like to
try it out with the public twitter stream (spritzer, which I hear gets
about 5 tweets/sec).

Any recommendations or open source code on handling the connection?
Twitter recommends separate stream handlers and data processors. I'm
thinking my client will write rotating flat files that a separate
process ingests into the database. What do you do, and what should we
watch for? My current crawl server is cpu bound; I'm still tracking
down whether I'm doing something stupid or need to upgrade the RAM.

Thanks,
Lucy.

[Ivan Kirigin]

> Any recommendations or open source code on handling the connection?

I'm told using pycurl is easy. You can specify a callback for new data
as it comes in. You can then make a reader which pumps out complete
tweets to a queue. I'd make it multi-process, and have another process
troll through the queue looking for messages.

In cases like Tipjoy, where the payment tweets are a small percentage
of regular tweets, keeping it all in memory should work.

Ivan
http://tipjoy.com

[dinosaur robot]

Word. I just found http://gist.github.com/126173 Open a python
request and read.
I'll check out pycurl. Thanks!!

[dinosaur robot]

(Note there was some discussion about using foo.encode("utf-8") on the
json results. The gist is definitely just a gist to get you started)