Re: Parallel connections are rejected by server at some point?

[Hans Hasselberg]

Hi Jose,


I suspect there is some kind of limit on the server you're requesting the websites from. Typhoeus is very fast and you're perhaps throttled. You could put the failed requests back in the queue and try them later.

You described that you're putting chunks of 40 requests in the hydra - thats not necessary. The hydra can handle that itsself with the option max_concurrency. If you say 'Typhoeus::Hydra.new(:max_concurrency => 40)' and queue _all_ your requests the hydra will never do more than 40 requests at a time.

--
Hans


On Wednesday, October 24, 2012 at 9:01 PM, Jose Luis Estrella Campaña wrote:

> Hi there !
>
> I've been using the typhoeus gem to increase the speed of scrapping of a huge website(amount of items wise), without typhoeus the scrapping finished correctly, but it was very slow. So I heard this gem could help me.
> Now, my problem is that Scrapping is stopping at the same point every single time I run it.
>
> I'm using the classic block you guys have to handle errors:
> if response.success?
> # hell yeah
> elsif response.timed_out?
> # aw hell no
> log("got a time out")
> elsif response.code == 0
> # Could not get an http response, something's wrong.
> log(response.curl_error_message)
> else
> # Received a non-successful http response.
> log("HTTP request failed: " + response.code.to_s)
> end
>
>
> and it always goes to the response.code == 0 of the handler, I reiterate it stops at the same point always(a specific URL), and I'm trying 40 simultaneous connections at a time (I finish those 40, and I try another 40, etc)
>
> Is this a common pitfall? or is there a configuration I should be aware of?
>
> Thanks in advance!
>
>
> --
> You received this message because you are subscribed to the Google Groups "Typhoeus" group.
> To view this discussion on the web visit https://groups.google.com/d/msg/typhoeus/-/5FpGLqpIl7QJ.
> To post to this group, send email to typh...@googlegroups.com (mailto:typh...@googlegroups.com).
> To unsubscribe from this group, send email to typhoeus+u...@googlegroups.com (mailto:typhoeus+u...@googlegroups.com).
> For more options, visit this group at http://groups.google.com/group/typhoeus?hl=en.

[Hans Hasselberg]

> Is it mandatory in my case to set the max_concurrency?

No, but it would take care of doing the chunking for you. You can do that of course on your own.
> On the other hand, yes, I'm doing chunks of 40 at a time.. and I noticed that when it's about to start chunk: 240-280, it always crashes with a Response Code of 0. Even If I set a different start point. What's your diagnosis?

Can you provide a gist to reproduce your issue? I still think the reason is a server which throttles you.
> PS. Why I don't see the max_concurrency option documented here: http://rubydoc.info/github/pauldix/typhoeus/master/Typhoeus/Hydra ? -just wondering.

Because this is not the latest version of the documentation. The github repo is here: https://github.com/typhoeus/typhoeus and the rdoc here: http://rubydoc.info/github/typhoeus/typhoeus/frames/Typhoeus. I suggest updating to version 0.5.0.rc (gem install typhoeus --pre).
> PS2. Could help asking.. Why is the following happenning:
>
> 1.9.3-p194 :004 > Typhoeus::Request.new("www.google.com (http://www.google.com)").run
> NoMethodError: undefined method `run' for :method => :get,
> :url => www.google.com (http://www.google.com):Typhoeus::Request
> from (irb):4
> from /Users/jose/.rvm/rubies/ruby-1.9.3-p194/bin/irb:16:in `<main>'

It works in 0.5.0.rc, please try to update.

--
Hans

[Hans Hasselberg]

> I've set the concurrency to 5, and it has started to work properly, yay! (I mistakenly thought # of connections = # of URLs I could scrap in parallel), Could you explain more as to why this? I mean, a single change of config, like this, and it started to work, I want to know why. Was the server preventing TOO MANY concurrent calls?

Yes, exactly. The server is obviously configured to accept only a few requests from the same ip at a time.
The max_concurrency option tells the hydra to only do X requests at the same time. When not specified you'll make up to 200 requests at the same time.
> How can I know beforehand How many concurrent calls does a given server accepts?

You cannot unless they tell you. You have to try it out.
> My complete program is a little big, but here is (https://gist.github.com/3948548) the snippet that might interest you, let me know what you think. (Refactoring Tips are welcome Too!), and btw, sending a chunks of urls at a time is a matter of my algorithm.

Ok, the code seems fine!
> You said that there's a way to retry failed requests, right? Could you please show how can I do this? -> How can I put a request back in the queue for processing later?

You could put failed requests into a seperate array and retry it afterwards. There's nothing like that buildin.

--
Hans