Re: slurp on over slow HTTP crashes clojure?

[Colin Jones]

Is this using REPLy / a lein2-previewX repl, by chance? I fixed a memory leak in REPLy yesterday that happened in long-running command scenarios, and could have caused this behavior (see https://github.com/technomancy/leiningen/issues/691 for details). 

- Colin

On Wednesday, July 18, 2012 10:14:24 AM UTC-5, the80srobot wrote:

Hello,

I apologize if this isn't the right place to post this.

I've seen slurp exhibit some very strange behavior when querying a particular URL at certain times of day - it seems that a call to slurp as such:

(slurp "http://support.clean-mx.de/clean-mx/viruses.php?limit=0,150")

will crash clojure if that site is particularly slow to load. Downloading the same URL using curl succeeds eventually and the download is just over 2 MB, which shouldn't cause Java to run out of memory, but nevertheless that is what seems to be happening, as the following is sometimes thrown before the REPL dies:

Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap space
Bye for now!

I haven't been able to recreate this issue with other URLs. I realize this isn't much to go on, but has anyone else encountered this problem before?

-Adam

[Devin Walters]

I've had this issue when the network I was on was way overloaded. It seems like it might be nice to have it at least fail gracefully. IIRC I switched to using the http lib. It seems like at the very least a graceful failure from slurp would be an improvement.

Cheers,
'(Devin Walters)

On Jul 18, 2012, at 11:14 AM, the80srobot <ad...@ingenious.cz> wrote:

Hello,

I apologize if this isn't the right place to post this.

I've seen slurp exhibit some very strange behavior when querying a particular URL at certain times of day - it seems that a call to slurp as such:

(slurp "http://support.clean-mx.de/clean-mx/viruses.php?limit=0,150")

will crash clojure if that site is particularly slow to load. Downloading the same URL using curl succeeds eventually and the download is just over 2 MB, which shouldn't cause Java to run out of memory, but nevertheless that is what seems to be happening, as the following is sometimes thrown before the REPL dies:

Exception in thread "Thread-12" java.lang.OutOfMemoryError: Java heap space
Bye for now!

I haven't been able to recreate this issue with other URLs. I realize this isn't much to go on, but has anyone else encountered this problem before?

-Adam

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Stuart Sierra]

The 'slurp' function on a URL dispatches to java.net.HttpURLConnection, which is pretty primitive as HTTP clients go. If you need to handle slow sites or large responses, you'll probably be better off with a real HTTP client library.

-S