Support for Event Machine - Non Blocking ?

[SimonS]

Hi

Is RGeo able to work with EventMachine. i.e. are the underlying
database adapters or adapter that help integrate with the geo-location
services implemented in a non blocking way ?

Thanks
-Simon

[Daniel Azuma]

Hi Simon,

The short answer is yes and no. Yes if all you're using is RGeo-- the RGeo core doesn't do any I/O itself, so the question in that case is meaningless. It works fine in an evented architecture as long as the code surrounding it is evented. There are, I think, a few parts that aren't thread-safe; it's on my to-do list to go through the entire library and clean those up. But in a single-threaded evented environment, there shouldn't be an issue.

If, however, you're talking about the ActiveRecord adapters such as activerecord-postgis-adapter, then no. But that's primarily because ActiveRecord itself (as far as I know) doesn't support an evented architecture out of the box either. And in any case, activerecord-postgis-adapter is essentially a subclass of the stock postgres adapter provided by ActiveRecord, and inasmuch as those stock adapters are not evented, neither is activerecord-postgis-adapter. So the longer answer is, I think this is a larger discussion that involves the direction of Rails and whatever other frameworks you're using.

Daniel

[SimonS]

Hi Daniel

Event Machine does have an implementation of an evented Postgres 3
client: http://eventmachine.rubyforge.org/EventMachine/Protocols/Postgres3.html.

One wonders how difficult it would be to get the authors of
activerecord-postgis-adapter to follow that model ? Don't suppose you
have any influence there ?

Just a thought...

-Simon

[Daniel Azuma]

Hi Simon,

Oh... I'm the author of activerecord-postgis-adapter. :-) But the point is, that adapter is limited by the design of ActiveRecord as a whole, and of ActiveRecord's stock postgresql adapter (of which activerecord-postgis-adapter is a subclass). So my hands are somewhat tied unless there's movement in the framework itself.

That said, I think Mike Perham was working on EM-aware patches to ActiveRecord and its core adapters at some point, but I'm not sure what became of that project. It might be worth reaching out to him and finding out where all that went. Maybe it's already in Rails core and I'm just not aware of it. I'll send him an email.

Daniel

[Kyle Drake]

As an extension of this, Ruby does not block on IO when using threaded mode. A short description of this is in the sinatra synchrony site: http://kyledrake.net/sinatra-synchrony/#caveats

The "ruby blocks on IO" thing is a myth. It's simply not true. What it doesn't do is implement a full reactor pattern for web serving, which is fine actually, reactor patterns are really more designed for "lots of sleepy connections" type of work.

One thing you could look at for reactor pattern implementation is Celluloid, which gives you a hybrid of threads and reactor pattern via Celluloid::IO. But it's not really designed to plug into Rails.

My recommendation for people these days is to not use EventMachine unless you can demonstrate you have a clear need for it.. "expecting" a problem is not sufficient IMHO. It's a power toy that really shouldn't be used as often as it has been.

[Kyle Drake]

I've talked about this with Mike Perham. He has joined the "we should be using EM a lot less" side, and has been implementing things with threads. His latest project, Sidekiq, uses threading via Celluloid, and it performs significantly better than Resque did.. and does not block on IO!

I'm currently working on a book that discusses ruby concurrency issues, so that the shared knowledge in this area improves.

-Kyle

[SimonS]

All

Thanks for the comments!

Firstly, Daniel: I understand, but there is a postgres implementation
out there that could be reviewed with the option to leverage that to
adapt a version for your postgis adapter for use in em environments
http://eventmachine.rubyforge.org/EventMachine/Protocols/Postgres3.html.
I'm still learning many aspects of the project and delving into this
myself is beyond my skill and bandwidth.

Secondly, Kyle: I think I have a valid use of EM in that I'm pulling
real time event data of a TCP Socket, and processing, manipulating and
storing the data, and streaming it out again in a better format, a
pipeline of sorts. As an aside there be a web app with a restfull API
that can deliver snapshot or historical data in either an admin UI or
as XML/JSON/ODATA as clients need it.

The source data has a significant amount of geo data hence the desire
to use rGeo.

The "processing, manipulating and storing the data, and streaming it
out again" is where the need for a concurrent model is required. I
also want to leverage some rails functionality for the admin interface
(active scaffold), and that then becomes a constraint as well. It
would not surprise me if other applications of rGeo also have similar
needs...

[Kyle Drake]

Secondly, Kyle: I think I have a valid use of EM in that I'm pulling
real time event data of a TCP Socket, and processing, manipulating and
storing the data, and streaming it out again in a better format, a
pipeline of sorts. As an aside there be a web app with a restfull API
that can deliver snapshot or historical data in either an admin UI or
as XML/JSON/ODATA as clients need it.

I would recommend implementing the TCPSocket with Celluloid::IO. You could then implement the ActiveRecord component using the native threading environment (the way ActiveRecord was designed to be used), while using a reactor pattern to manage the TCPSocket connections. Then, you will avoid the problem of having to replace every Ruby IO-related library with a special EM monkey-patched, spaghetti-code version (and the corresponding bizarre, hard to debug, stranded island problems that will result).

I went down this route myself, and got really burned in the process. So did a lot of other people. Raggi did (an EM maintainer), and he tells people not to use EM for new projects. Mike Perham did, and he now deprecates the EM patching in some of his code. Tony did, and he didn't think it was the right way, and made Cool.io (and finally Celluloid) to deal with it. Ryan Dahl didn't like it so much, he left and started a new programming language. Difference was, Javascript is designed to be callback oriented.. Ruby is not! Ruby uses threads for concurrency. They're really not that bad (unless you use something like Thin, which turns them off, and then Ruby -does- block on IO.. see Puma). And when we start seeing 128+ core machines, then threads are pretty much going to be the only game in town.

The source data has a significant amount of geo data hence the desire
to use rGeo.

The "processing, manipulating and storing the data, and streaming it
out again" is where the need for a concurrent model is required. I
also want to leverage some rails functionality for the admin interface
(active scaffold), and that then becomes a constraint as well. It
would not surprise me if other applications of rGeo also have similar
needs...

I would recommend separating the administration component from the socket component, so that you can take down and manage the admin structure without taking down the corresponding TCPSocket system. You certainly don't want to have to drop connections every time you update your admin component, and of course, the admin component probably doesn't require a super fast TCPSocket. So that will save you a lot of trouble, especially if you decide to use EM for that component, because then the Rails admin code won't block things.

Best of luck!

-Kyle

[SimonS]

Kyle

I value your input...

In the world of Ruby it tough to find out what the best path is for a
relative newbie...

I'll take a look at Celluloid::IO, and see what this path takes me...

Another crook in the tail it the desire to deploy to Heroku...and not
get unnecessarily twisted in workers...and keep a uniform code base.

Thanks again for the advice
-Simon

[SimonS]

Hey Kyle

Don't suppose you can point me to some docs or samples that combines
Celluloid:io and Rails in the same app code base so I can share the
models...? but use a procfile for the different parts of the
process..socket vs. admin/API

Thanks
-Simon