Mongoid 3 Pool Size

[Meskyanichi]

Hey,

I've been trying to figure out where I should be putting the `pool_size` option in the `mongoid.yml`. I tried to search through the Mongoid source to find where the `pool` or `pool_size` comes in to play but was unable to find anything.

Does this configuration properly set the pool size on the `default` and `mydb2` sessions databases?

production:

  sessions:

    default:

      database: mydb

      hosts:

        - localhost:27017

      options:

        pool_size: 100

    mydb2:

      uri: mongodb://user:pass@host:port/mydb2

      options:

        pool_size: 100

Or is this incorrect?

Thanks!

[Durran Jordan]

There is no pool_size option anymore as we do not use a connection pool. It's an unnecessary complexity for the design route that we took (there's one Moped session on each thread that Mongoid manages) and we read and write from the same socket on each thread.

2012/7/17 Meskyanichi <mesky...@gmail.com>

[Meskyanichi]

Thanks for the quick reply!

I am currently using Sidekiq (a multi-threaded worker library) that queues/performs background jobs through Redis. However, the operations that I perform are querying two mongo databases through Mongoid 3 itself.

Some queries are directly on `Mongoid.session(:default)` and `Mongoid.session(:mydb2)` (which is the Moped session I assume?) that don't go through the models to update certain collections. I store them in to global variables from an initializer in Rails, like so:

$mongo_default = Mongoid.session(:default)

$mongo_mydb2 = Mongoid.session(:mydb2)

Then I use these from within about a hundred threads to perform a lot of queries. I also do a bunch of queries through actual models (model.save/update/destroy/etc).

Now, I'm not sure but is it safe to store both sessions in these global variables and accessing them from 100 different threads? Or should I be calling `Mongoid.session(:default)` and `Mongoid.session(:mydb2)` from within each thread instead? I take it that models will just handle the thread/session thing own their own when performing operations on the databases.

Also, even though they read/write from/to the same socket, this doesn't hinder a threaded environment in terms of performance/blocking requests, does it?

Thanks again!

[Durran Jordan]

As for putting the sessions in global variables to access from different threads, *definitely do not do this*. :) What you risk with that is some threads getting the return results of others when querying... Ie thread one tells the session to insert something, thread 2 is asking for a response before thread 1 does the same. In order to maintain thread safety each thread needs to have its own session (and thus its own connection) to the database.

This is why Mongoid manages this for you - so you are correct in thinking you should just be calling Mongoid.session fresh each time instead of making them global, all the thread management will be handled under the covers for you.

As for performance, whether you are using multiple threads and multiple sockets or connection pools the IO is still going to block. Sidekiq I believe is using Celluloid under the hood so it should be an implementation of actors on top of fibers, which will alleviate this slightly by allowing another fiber to run while another is IO blocked. But given enough database operations you will see a point where adding more workers wont do you any good. I think that's up to your app and finding where the sweet spot is. Depending on the operations you are performing, I would venture to say you could put Mongo itself at 100% CPU before the worker farm pegged it's server. (The Mongoid benchmarks can put MongoDB in certain cases at 100% CPU with a single Ruby process.. Hint, don't do a large series of ops that increase document sizes.)

Hope this helps... If you get into specific performance related issues and can provide benchmarks as well in a large worker farm like this that would be great to see, and maybe us help tune things in the future.

2012/7/17 Meskyanichi <mesky...@gmail.com>

[Meskyanichi]

Ha! Glad I asked. :)

That could've ended pretty bad (and hard to debug). I didn't realize that Mongoid 3 manages all of this for you. Pretty slick!

The majority of the queries (in the worker farm) are `update`-type queries so the document size won't really increase.

I'm currently using JRuby on a pretty good dedicated server, so the bottleneck will likely be Mongo (whenever the time comes) rather than my worker farm on the server.

However, it'll probably be a while before it even gets to that point since it seems that Mongo can handle quite a good amount of load.

Thanks for the detailed info! Very helpful.

Cheers

[K-Liss]

Does this mean, there is one socket per thread, or one socket per process? I have a sidekiq processor with 130 workers and something is holding it back. It would be unfortunate if those workers had to fight over a single socket :)

[Durran Jordan]

This is one socket per thread, so Sidekiq can go full throttle... Just be sure to be using Kiqstand (https://github.com/mongoid/kiqstand) so the open connections don't get overloaded.

2012/11/1 K-Liss <sergei.t...@gmail.com>