Java asynchronous driver: too many threads waiting for a connection

[Noam Ben Ami]

Hi,

I am testing the async driver from https://github.com/jyemin/mongo-java-driver/tree/3.0.x-netty (using NIO2). My application needs to handle thousands of inserts per second. 

I noticed that on every call to asyncInsert() a new I/O thread is created and waits for an available connection. I see that the maximum number of possibly waiting threads is set to the max connections multiplied by a factor of 5.

During my test I get many of the following exception:

MongoWaitQueueFullException: Too many threads are already waiting for a connection. Max number of threads (maxWaitQueueSize) of 500 has been exceeded.

I tried to set the max connection option to a relatively high value but that did not help and the above exception is still thrown. Furthermore, the driver spawned a huge amount of threads waiting for connections.  

Since it is NIO2 based I wonder why there is a need to spawn so many I/O threads, whereas a single thread can handle multiple connections?

I would be very grateful for any feedback.

[Rob Moore]

I don't work for MongoDB Inc. but Jeff and company are still working on what the asynchronous interface is going to look like and how to graft that onto the synchronous core. I expect there will be many changes between now and when it is finally released.

Have you thought about using the Asynchronous Java Driver (which I lead the development of) instead of using a pre-release branch of the 3.0 driver from MongoDB Inc? It is asynchronous to the core and we released the 2.0.0 version a couple of weeks ago.

Rob.

[Noam Ben Ami]

Hi Rob,

Actually I did came across your driver on its previous version where batching was not supported. I just saw that there is a new version with batching support and I will definitely give it a try.

Could you please explain how the application threads are balanced across all connections? Is there some kind of off loading to another group of threads where each such thread operates on a specific connection?

Thanks in advance,

Noam.

[Rob Moore]

On Wednesday, July 9, 2014 10:50:00 AM UTC-4, Noam Ben Ami wrote:

Could you please explain how the application threads are balanced across all connections? Is there some kind of off loading to another group of threads where each such thread operates on a specific connection?

The application thread sending a request will try and reuse the most recently used connection if it is idle. If it is not idle is will choose the most lightly used connection between the most recently used and the connection used the longest ago. This simple heuristic for balancing requests to connections has proven to provide a good balance between speed (pick a connection quickly) and avoiding connections with long running requests.

As for threads: The pre-2.0 driver used a pair of threads for each connection (one reader and one writer). For the 2.0 release we switched (the default) to a single reader thread per connection and use the application's thread to perform the writes to the sockets. For either model the hand-off of messages between threads is an optimized queue based on the design of the Disruptor but making the assumption that there is a single reader. (Which, for the driver, is always true.)

If you are looking for absolute maximum data throughput performance (and can afford the extra threads) you can switch the connection model in the configuration back to the two thread model. We are also looking at implementing a selector based connection model that will drop the thread count to 1 per MongoClient (or a configurable count) but that work has not started and will likely not offer near the performance of the 1 or 2 thread per connection models. This new model will mainly be useful for resource constrained applications.

Rob.