Expectations - what should they be?

[Sujal Shah]

Hey folks,

So, I'm generally new to coding with EventMachine & Synchrony but have used other async dev tools in the past.

I'm now testing an API we've built here, and I want to make sure I understand the performance thresholds of this before we push it up to our production setup.

My question is simply this: how do I tell that the code is properly async? 

The rule of thumb I'm using is that the server should be able to handle more requests per second as concurrency goes up especially once it passes the maximum a single thread could do. Seems obvious, right? I would expect this to go up as concurrency goes up until we hit a backlog limit or whatever.

For example, if the request takes 25ms to process (and that 25ms is mostly a WAN roundtrip to mongodb), I would expect to sustain at least 40rps, and ideally much much more since this is all IO.

However, in my tests, I'm usually seeing only the 40rps.

So, either my expectations are wrong and I've forgotten something, or my code is blocking somewhere. 

Also, on that note, is there a good example of a Goliath or em-synchrony app that uses Mongoid? Want to make sure I'm understanding the require order correctly.

Thanks,

Sujal

[Sujal Shah]

So, in my case, it seems like Mongoid is only making a single connection back to the DB. This seems like a possible bottleneck. Is there a recommended way to create a connection pool? Or does the driver understand how to multiplex queries over the same connection?

Sujal

[dan sinclair]

EM-Synchrony provides a connection pool class that you'll want to wrap around the driver. Take a look at: https://github.com/postrank-labs/goliath/blob/master/examples/config/http_log.rb for an example using mongo.

dan

[Adam Watson]

Hey Sujal,

Here's how I do it: https://gist.github.com/1371170

Cheers,

Adam

On Wed, Nov 16, 2011 at 11:58 AM, Sujal Shah <code...@gmail.com> wrote:

[Sujal Shah]

Gah, how did I miss that example??

Both of you, thanks. 

Sorry about the noise. :)

Sujal

[Joshua Warchol]

Hey Sujal, cool to see you working with Goliath.

There is a plugin for logging the reactor loop delay. Non-async code
would start to see that get all out of whack. I think its
Goliath::Plugin::Latency

[Sujal Shah]

Josh, thanks - I was running that already, saw some jitter but that's it. That's what confused me. It was like it was being async but throttled. Just took me a while to get around to the DB connection.

Adam, Dan - thanks again. Any specific glue needed to hook it up to Mongoid? Or should it just work? The examples I've seen access the driver directly instead of the connection pool. I'm looking at this now, but if you happen to know...

Sujal

[dan sinclair]

Sorry, never used mongoid myself, always just used the driver directly though the connection pool.

dan

[Adam Watson]

Hey Sujal,

You can use whatever variable you assigned to the connection pool to
for issuing your queries. The em-synchrony connection pool handles the
underlying logic of assigning an available connection to the executing
fiber.

Cheers,
Adam

[Adam Watson]

Oh, hey, I partially misunderstood your question... for Mongoid
connection pooling, you'll want to do something like this:
https://gist.github.com/1371577

Then you'll just issue queries via your model objects as usual.

Adam


On Nov 16, 1:39 pm, Sujal Shah <codesu...@gmail.com> wrote:

[Sujal Shah]

Adam, thanks.

From what I can tell, this doesn't actually create a pool. My testing shows that the setting seems to be ignored, and looking through the code (for the mongo ruby driver) shows that this setting is actually ignored unless the app is multithreaded, which EventMachine apps are not.

So... I think this means that neither solution we've discussed will create a connection pool compatible with both EM-Synchrony & mongoid. My best guess is that we should retrofit Mongoid to use the EventMachine::Synchrony::ConnectionPool solution.

What am I missing?

Sujal

[Adam Watson]

Hi Sujal,

Interesting...  after initializing a Mongoid connection pool, I see this when I inspect  Mongoid.master['primary_pool']:

=> #<Mongo::Collection:0x0000010319d710 @name="primary_pool", @db=#<Mongo::DB:0x000001033c4e80 @name="development", @connection=#<Mongo::Connection:0x000001033c7cc0 @host_to_try=["localhost", 27017], @port=nil, @host=nil, @slave_ok=nil, @max_bson_size=16777216, @ssl=false, @socket_class=TCPSocket, @auths=[], @id_lock=#<Mutex:0x000001033c7c48>, @pool_size=20, @timeout=5.0, @op_timeout=nil, @connect_timeout=nil, @connection_mutex=#<Mutex:0x000001033c7c20>, @safe=false, @safe_mutexes={#<TCPSocket:(closed)>=>#<Mutex:0x000001033c5858>, #<TCPSocket:fd 47>=>#<Mutex:0x000001033c3eb8>}, @queue=#<ConditionVariable:0x000001033c7b58 @waiters=[], @waiters_mutex=#<Mutex:0x000001033c7b08>>, @primary=["localhost", 27017], @primary_pool=#<Mongo::Pool:0x819e286c @host=localhost @port=27017 @ping_time= 0/20 sockets available.>, @logger=nil, @read_primary=true>, @strict=nil, @pk_factory=nil, @safe=false, @read_preference=:primary, @cache_time=300>, @connection=#<Mongo::Connection:0x000001033c7cc0 @host_to_try=["localhost", 27017], @port=nil, @host=nil, @slave_ok=nil, @max_bson_size=16777216, @ssl=false, @socket_class=TCPSocket, @auths=[], @id_lock=#<Mutex:0x000001033c7c48>, @pool_size=20, @timeout=5.0, @op_timeout=nil, @connect_timeout=nil, @connection_mutex=#<Mutex:0x000001033c7c20>, @safe=false, @safe_mutexes={#<TCPSocket:(closed)>=>#<Mutex:0x000001033c5858>, #<TCPSocket:fd 47>=>#<Mutex:0x000001033c3eb8>}, @queue=#<ConditionVariable:0x000001033c7b58 @waiters=[], @waiters_mutex=#<Mutex:0x000001033c7b08>>, @primary=["localhost", 27017], @primary_pool=#<Mongo::Pool:0x819e286c @host=localhost @port=27017 @ping_time= 0/20 sockets available.>, @logger=nil, @read_primary=true>, @cache_time=300, @cache={}, @safe=false, @read_preference=:primary, @pk_factory=BSON::ObjectId, @hint=nil>

I may be misinterpreting this, but it looks like mongoid has, in fact, created a connection pool. I assumed that Goliath would be able to take advantage of this since it can perform multiple concurrent requests.

However, now that I take another look at Ilya's EM-Synchrony readme, I'm noting that the Mongoid support states "all functions synchronous".

Hmm, now I'm a bit confused about this, too :)

Ilya, would you mind helping to clarify?

Cheers,

Adam

[Sujal Shah]

The interesting thing is that I see the same thing, BUT I don't see any more than 1 connection open to mongod. Since mongod logging to the console, I can watch the connections come in. I only see the "is_master" connection, then the master collection connection. Nothing more - I've tried pool size values from 5-40.

Contrast that to the methods you & Dan suggested, using the fiber aware connection pool classes and em-mongo: I see all the connections on the mongod side when the pool is initialized. I've also tried to force high parallel loads to see if maybe it's smarter than I think and is somehow only opening additional connections when needed, but no success so far.

So... several mysteries. 

I've decided to table this work know that I have a backup plan ( rewrite the few essential queries directly against em-mongo), but I'm planning on continuing my dive through the code to see what I'm missing.  I'll also see if I can figure out a solution.

Sujal

[Ilya Grigorik]

Hey guys, not intimately familiar with the mongoid pool logic, but based on your description it sounds like it's creating a single connection -- which is fine. Simply wrap that connection within the em-synchrony pool and you should be good to go (although I would still recommend using em-mongo directly). 

ig

[Sujal Shah]

As a little exercise while at my in-laws for the holidays, I attempted to get Mongoid to use em-mongo inside em-synchrony. I've got the first chunk of this up as a pull request on the synchrony project. If anyone else might want to use this, please take a look - I'm still working on writing tests, so assume there are bugs still.

https://github.com/igrigorik/em-synchrony/pull/98

Would love feedback on the approach and style.

The changes to break some compatibility with the existing em-mongo functionality to get it closer to the original non-synchrony em-mongo and mongo-ruby-driver.

Sujal

[Sujal Shah]

Just wanted to mention, patch is updated. Our app (still in dev) is using this now and this has improved throughput on Heroku quite a bit for us.

Still some work to do.

https://github.com/igrigorik/em-synchrony/pull/98

Sujal