Is It Threadsafe™?

[Ilya Vassilevsky]

Hello :)

First of all, congratulations with the 1.0 release! Huge milestone!

I should also mention that this gem scores the highest CodeClimate GPA of all Ruby HTTP clients. The code is sure pleasant to read!

But is it threadsafe™?

Can multiple threads issue requests through the same connection?

Will it work fine in Sidekiq 2 based on Celluloid?

Thanks!

[Tony Arcieri]

On Tue, Dec 29, 2015 at 1:39 AM, Ilya Vassilevsky <vassi...@gmail.com> wrote:

But is it threadsafe™?

Can multiple threads issue requests through the same connection?

You'll need a thread-safe connection pool for this to work. You can use this one:

https://github.com/mperham/connection_pool 

Use it in combination with HTTP.persistent and you should be good. We use it in a similar capacity at Square.

--

Tony Arcieri

[Jonathan Swartz]

Hi Tony --

Let's say I don't need threads to issue requests through the same connection, and I'm ok with each thread creating a brand new connection. Is it thread-safe in that case? e.g. can I just do

  urls.each do |url|

    Thread.new do

         responses[url] = HTTP.get(url)

    end

  end

so that each of the urls are fetched in parallel?

(This is just an example - in reality I want to make asynchronous http requests in a more ad hoc way, rather than all at the same time).

I'm surprised the documentation doesn't mention anything about thread-safety, as it's quite difficult to find any thread-safe HTTP client in Ruby.

Also, I noticed that the timeout feature uses Ruby's Timeout, which I've heard is not thread-safe (or safe at all).

Thanks!

Jon

[Tony Arcieri]

On Mon, Jul 11, 2016 at 4:00 PM, Jonathan Swartz <swa...@pobox.com> wrote:

Hi Tony --

Let's say I don't need threads to issue requests through the same connection, and I'm ok with each thread creating a brand new connection. Is it thread-safe in that case? e.g. can I just do

  urls.each do |url|

    Thread.new do

         responses[url] = HTTP.get(url)

    end

  end

so that each of the urls are fetched in parallel?

Yes, this is fine.

 

(This is just an example - in reality I want to make asynchronous http requests in a more ad hoc way, rather than all at the same time).

I'm surprised the documentation doesn't mention anything about thread-safety, as it's quite difficult to find any thread-safe HTTP client in Ruby.

The library itself provides no specific thread safety features, but all state is localized to objects you instantiate, so as long as you don't share them across threads you get thread safety by default.

I can add a small thread safety notes section describing this.

Also, I noticed that the timeout feature uses Ruby's Timeout, which I've heard is not thread-safe (or safe at all).

For the most part it does NOT use Timeout: it specifically implements timeouts using asynchronous I/O specifically to address the very problem you're describing (and many others).

There's one exception: DNS. We use the system DNS resolver, which has a synchronous API, and is presently still wrapped in Timeout.

Fixing that is a very very deep rabbit hole. I've attempted it in Celluloid::IO and can't say I'm particularly happy with the results. Using anything other than the system resolver cuts you out of a lot of functionality.

--

Tony Arcieri

[Jonathan Swartz]

Thanks for your quick reply. I had trouble getting httpclient to work so I look forward to trying this!

[Tony Arcieri]

I did a quick writeup on thread safety:

https://github.com/httprb/http/wiki/Thread-Safety

It could probably use some examples of thread safe and unsafe code.

--
You received this message because you are subscribed to the Google Groups "http.rb: a fast, easy-to-use Ruby HTTP client with a chainable API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to httprb+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Tony Arcieri

[Swartz, Jonathan]

Thanks again. Gives me confidence. :)