Anyone on here an EventMachine expert?

[Travis Reeder]

Want to EventMachine'ize the aws and simple_record gems (optional) so it can take advantage of non-blocking IO and would like to chat with someone about the best way to go about it.

Cheers,

Travis

[joshmckin]

I'm not an expert but all my recent rails 3 aps are running async
(see: https://github.com/igrigorik/async-rails) and created a simple
em-http drive api for our zendesk needs using https://github.com/igrigorik/em-synchrony.

Mperham started working on simpldb in qanat, although that gem is
geared more towards AWS caching it does have a basic simpledb api:
https://github.com/mperham/qanat/tree/

Since I've been running rails as async I've wanted to get aws to be em
aware but could not find to time really get something working, the
older code was a bit funky, and I was not sure how to handle (or if it
would even matter) the threading that already exists in aws in
conjunction with fibers. In the end I felt as though it might be
easier to create a completely new aws gem that used fibers instead of
threading. Additionally, my needs are specific to rails and all the
apps I run are deployed on heroku which uses Thin, so I did not have
worry about an EM instance running. However, aws is pretty popular gem
and if it goes EM it probably needs to check if an EM aware server is
available and if not spin something up, I would check out Goliath for
handling non Rails-Thin (or any other em aware web server) situations:
http://postrank-labs.github.com/goliath/

Em-Aws would be really awesome. Let me know if there is anything I can
do.

Thanks,

Josh

[joshmckin]

Sorry that should read "... qanat is geared more towards AWS message
queue..."

Just for clarification.

On Mar 29, 9:11 am, joshmckin <joshmc...@gmail.com> wrote:
> I'm not an expert but all my recent rails 3 aps are running async
> (see:https://github.com/igrigorik/async-rails)  and created a simple

> em-http drive api for our zendesk needs usinghttps://github.com/igrigorik/em-synchrony.

[Travis Reeder]

That's interesting, thin was the reason I was thinking of this in the first place since it's single threaded, if one request is slow, nobody else can access the site. And a lot of the slowness was making calls to aws (SimpleDB for instance) so EventMachine enabling it should fix that. 

The old aws code is pretty funky, agreed. I think there is really only a couple spots that need changing though and that would probably be in awsbase.rb, request_info_impl method where it makes the actual HTTP request. Perhaps a setting can be set globally (or per service) that lets the gem know that you want it to use EM. Would it just be changing from using Net::HTTP to EventMachine::Protocols::HttpClient ?

http://eventmachine.rubyforge.org/EventMachine/Protocols/HttpClient.html

--
You received this message because you are subscribed to the Google Groups "SimpleRecord" group.
To post to this group, send email to simple...@googlegroups.com.
To unsubscribe from this group, send email to simple-recor...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/simple-record?hl=en.

[joshmckin]

Correct. If EM is running somewhere then you should be able to replace
the Net::HTTP with an em-http request, but i would probably use em-
synchrony as the em-http-request library, to make sure you yield to
the root fiber. The http error handling in AWS is what I found to be a
bigger issue, since it is build closely around Net::HTTP.

I had some problems getting started in my apps with testing and
running the em-http request from console since EM is normally started
with thin. Since my request code were generally small I would check to
make sure Em is running and if not wrap the request in an
EM.synchrony do block; see example below:

def post
EM::HttpRequest.new("some.url.com/data.json").post :body => {:foo
=> "bar"}
end

def fetch_response
if EM::reactor_running?
self.response = self.post.response
else
EM.synchrony do
http = self.post
http.errback { self.errors.add(:request, "failed.") }
http.callback {
self.response = http.response
EM.stop
}
end
end

[joshmckin]

Also, I has some fun one day trying to figure out why thin was dying.
Turned out I had EM.stop in a bit of request code that was running
when EM::reactor_running? returned true. EM.stop stops them all so
watch out. Seems obvious now, but at the time it was loads of fun ;)

[Travis Reeder]

Hah, ya, that does sound fun. ;)

Someone on the aws list suggested the faraday gem: https://github.com/technoweenie/faraday

If we swapped out all the http guts in aws with that, we could just replace what is used with adapters. 

https://groups.google.com/forum/#!topic/ruby-aws/deFei8Sok04

[joshmckin]

Faraday is new to me as well, but that migration plan sounds solid.
Building in the ability to use different adapters, em or otherwise,
would nice. EM is the hotness for ruby async request today but who
knows what options there may be in a time.

[Travis Reeder]

Ok, got faraday working now and I also have a first attempt at a fully asynchronous Faraday adapter. A performance test is here and results look pretty promising. With 50 items it was 2 seconds vs 13 seconds (ish). https://github.com/appoxy/aws/blob/faraday/test/sdb/test_performance.rb

[joshmckin]

Thats great. Looking at the commits, the faraday appeared pretty
conversion with only a couple instances of completely new lines of
code. Can't wait to give it a shot.

On Apr 14, 12:42 am, Travis Reeder <tree...@gmail.com> wrote:
> Ok, got faraday working now and I also have a first attempt at a fully
> asynchronous Faraday adapter. A performance test is here and results look
> pretty promising. With 50 items it was 2 seconds vs 13 seconds (ish).https://github.com/appoxy/aws/blob/faraday/test/sdb/test_performance.rb

>  <https://github.com/appoxy/aws/tree/faraday>