avoiding RATE_LIMIT_EXCEEDED

[alexh]

I have a bulk operation I'm attempting to run and am running into
exceeding bit.ly's rate limit. I'm using bitlyj's Provider as a static
class. Does this help prevent creating new connecting to the API? I
still continue to get the error however. Are there other steps I can
take to preventing this? I need to convert about 250 URLs on each run.

[ch...@rosaloves.com]

Interesting question. I would firstly refer you to the bitly best-
practices:

http://code.google.com/p/bitly-api/wiki/ApiBestPractices

Additionally, I suggest posting to the bitly API group to see if
anyone from bitly can state the actual call limit. I suspect you might
be hitting that and not the connection limit, in which case any
connection tricker in bitlyj would be moot.

However, to entertain the possibility of gaining more throughput,
there is at least one thing to try. Bitly, at least according to the
headers from the API server, supports HTTP 1.1 persistent connections.
According to:

http://download.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive.html

the jdk >= 5 should cache and reuse the underlying TCP connections for
dealing with HTTP streams, if the streams are fully consumed. I need
to make sure the underlying XML parser is both fully consuming and
closing the stream, though I suspect it is. The only other thing I can
think of is to modify bitlyj to fully consume the error response,
which it doesn't currently do. According to the tiger docs, "Doing so
may results in idle TCP connections."

This would be a simple change, but I'm doubtful that it's at the root
of your problem. Again, see if the bitly API group will divulge the
actual call limit.

The only other thing you may be doing is using bitlyj in a
multithreaded fashion. Bitlyj maintains no mutable state, so it should
be thread-safe by default. However, for N threads you will have at
least N open connections, and if some are idling (which is possible
but I'd think unlikely), you would have N + idle connections.

I follow the bilty API group, so start a thread and I'll watch.

-chris

[alexh]

I got in touch with with support at bit.ly and it was a timing issue:
too many requests within a given time slice. They were nice enough to
remove this for me.

I am now bumping into an issue with the bitlyj library throwing an
exception when running on Google App Engine.

Caused by: java.lang.RuntimeException:
com.google.apphosting.api.ApiProxy$CancelledException: The API call
urlfetch.Fetch() was explicitly cancelled.
at com.rosaloves.bitlyj.SimpleProvider.fetchUrl(SimpleProvider.java:
106)
at com.rosaloves.bitlyj.SimpleProvider.call(SimpleProvider.java:45)
at com.nph.server.web.Canvas2.shorten(Canvas2.java:105)
at com.nph.server.web.Canvas2.createHtml(Canvas2.java:60)
at com.nph.server.web.Canvas2.get(Canvas2.java:46)

I'd like to research this a bit more and perhaps if I have time I can
send you a patch?

Alex

On Sep 20, 12:20 pm, "ch...@rosaloves.com" <ch...@rosaloves.com>
wrote:

> Interesting question. I would firstly refer you to the bitly best-
> practices:
>
> http://code.google.com/p/bitly-api/wiki/ApiBestPractices
>
> Additionally, I suggest posting to the bitly API group to see if
> anyone from bitly can state the actual call limit. I suspect you might
> be hitting that and not the connection limit, in which case any
> connection tricker in bitlyj would be moot.
>
> However, to entertain the possibility of gaining more throughput,
> there is at least one thing to try. Bitly, at least according to the
> headers from the API server, supports HTTP 1.1 persistent connections.
> According to:
>

> http://download.oracle.com/javase/1.5.0/docs/guide/net/http-keepalive...

[ch...@rosaloves.com]

Cool.

Always open to patches, but I expect you'll find something else at the
root of this issue. I have an app deployed to GAE that uses bitlyj
with no issues. For some reason GAE is killing your outgoing call, so
seek that out instead.

[alexh]

Ok thanks for pointing out you have successfuly used the API on App
Engine. I must be picking up some other dependency different than your
implementation.

[ch...@rosaloves.com]

Let me know how this plays out. Like I said, you should have no
problems on GAE, but I'm a bit curious about that exception.