Anyone on here an EventMachine expert?

[Travis Reeder]

Want to EventMachine'ize the aws and simple_record gems (optional) so it can take advantage of non-blocking IO and would like to chat with someone about the best way to go about it.

Cheers,

Travis

[pete]

--
You received this message because you are subscribed to the Google Groups "ruby-aws" group.
To post to this group, send email to ruby...@googlegroups.com.
To unsubscribe from this group, send email to ruby-aws+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/ruby-aws?hl=en.

I'm not confident enough to call myself an expert, but the best way to use aws with EM would probably be to refactor the right_http_connection gem to loosen its dependency on net/http. From there, you could use something like em-synchrony or faraday with its em-synchrony adapter to get synchronous http calls, or go whole hog and use em-http-request to go full async.

pete

[Travis Reeder]

Wonder if we should just skip the right_http_connection gem altogether if user specifies they want to use EM?

[pete]

I haven't used faraday for any projects, but I've looked through the code and I think most of the http request/response stuff done by the aws gem could be reimplemented using faraday middlewares. Abstracting the http stuff out is going to be necessary if you want to offer the option of using EM anyways.

pete

[Travis Reeder]

Hmm, that's interesting, hadn't seen Faraday before. 

So we could just do it all in Faraday, then swap out the adapter: 

builder.use Faraday::Adapter::NetHttp

with an EM one if the user specifies it. We could add an :adapter option when creating a new 'aws' service object, eg: Aws::Sdb.new(....., :adapter=>:em). Would have to fork faraday and add the EM one, then ditch right_http_connection.

[pete]

Faraday has support for EM through its em-synchrony adapter:

https://github.com/technoweenie/faraday/blob/master/lib/faraday/adapter/em_synchrony.rb

pete

[Travis Reeder]

ahh, nice. Trying to swap out the guts with Faraday right now, I'll see how easy (hard?) it is shortly.

On Tue, Mar 29, 2011 at 6:32 PM, pete <pe...@peterhiggins.org> wrote:

[Travis Reeder]

Ok, got faraday working now and I also have a first attempt at a fully asynchronous Faraday adapter. A performance test is here and results look pretty promising. With 50 items it was 2 seconds vs 13 seconds (ish). https://github.com/appoxy/aws/blob/faraday/test/sdb/test_performance.rb

[pete]

Excellent!

pete

[Ankur]

Will this non-blocking Faraday library work with non Eventmachine
based servers like Phusion Passenger?

I think I have observed Phusion blocking on S3 transfers with this
gem. I am trying to get that out of my controller ASAP.

Ankur

On Apr 14, 1:37 pm, pete <p...@peterhiggins.org> wrote:

> On Wed, Apr 13, 2011 at 10:42 PM, Travis Reeder <tree...@gmail.com> wrote:
> > Ok, got faraday working now and I also have a first attempt at a fully
> > asynchronous Faraday adapter. A performance test is here and results look
> > pretty promising. With 50 items it was 2 seconds vs 13 seconds (ish).
> >https://github.com/appoxy/aws/blob/faraday/test/sdb/test_performance.rb
>

> > On Tue, Mar 29, 2011 at 7:12 PM, Travis Reeder <tree...@gmail.com> wrote:
>
> >> ahh, nice. Trying to swap out the guts with Faraday right now, I'll see
> >> how easy (hard?) it is shortly.
>

> >> On Tue, Mar 29, 2011 at 6:32 PM, pete <p...@peterhiggins.org> wrote:

>
> >>> On Tue, Mar 29, 2011 at 6:20 PM, Travis Reeder <tree...@gmail.com>wrote:
>
> >>>> Hmm, that's interesting, hadn't seen Faraday before.
>
> >>>> So we could just do it all in Faraday, then swap out the adapter:
>
> >>>> builder.use Faraday::Adapter::NetHttp
>
> >>>> with an EM one if the user specifies it. We could add an :adapter option
> >>>> when creating a new 'aws' service object, eg: Aws::Sdb.new(.....,
> >>>> :adapter=>:em). Would have to fork faraday and add the EM one, then ditch
> >>>> right_http_connection.
>

> >>>> On Tue, Mar 29, 2011 at 5:59 PM, pete <p...@peterhiggins.org> wrote:

>
> >>>>> On Tue, Mar 29, 2011 at 4:18 PM, Travis Reeder <tree...@gmail.com>wrote:
>
> >>>>>> Wonder if we should just skip the right_http_connection gem altogether
> >>>>>> if user specifies they want to use EM?
>

> >>>>>> On Tue, Mar 29, 2011 at 2:29 PM, pete <p...@peterhiggins.org> wrote:

> >>>https://github.com/technoweenie/faraday/blob/master/lib/faraday/adapt...

[Travis Reeder]

In theory yes, but the EventMachine reactor will have to run in it's own thread so the idea is that an EventMachine thread/loop will always be running to handle EM async tasks and feel similar to typical multi-threading. In the meantime you could async your stuff already using normal threads,  check out the concur library https://github.com/appoxy/concur and try using a thread pool executor or multi threaded executor. The EventMachine stuff we're adding to aws will probably use concur anyways so when it's ready you should (in theory) be able to just swap out the executor type with the eventmachine_executor like you can see in this example: https://github.com/appoxy/aws/blob/faraday/test/sdb/test_performance.rb