Closed sockets on servers

[trav]

I've written a performance testing framework to validate the usage of
Voldemort and have come up with some interesting data. I'll post the
findings and framework later today, but wanted to ask a couple
questions about some things I'm seeing:

- It seems that the server is closing a lot of socket connections --
are the not persisted across accesses? It appears that the
maxConnsPerNode setting in the client would dictate that they would
remain open. Certainly acquiring a connection is a cost I'd rather
not pay for each access.

[2009-03-31 14:04:40,999] INFO Client /10.248.54.127:40204
disconnected. (voldemort.server.socket.SocketServer)
[2009-03-31 14:04:41,001] INFO Client /10.248.54.127:40424 connected.
(voldemort.server.socket.SocketServer)

Subsequently, I'm getting a few failures on the client side:
[2009-03-31 14:06:16,318] WARN Get operation timed out after 5000 ms.
(voldemort.store.routed.RoutedStore)
[2009-03-31 14:06:21,045] WARN Could not connect to node 1 at host1
marking as unavailable for 10000 ms.
(voldemort.store.routed.RoutedStore)

I'm currently only porking the server via a multi-threaded client, and
don't think I have settings such that it would exhaust the pool of
connections, so I'm a little confused why I would be seeing this.

-- Jon

[ijuma]

Hi Jon,

What version of Voldemort are you using?

Best,
Ismael

[Jon Travis]

I'm checked out on HEAD from the project page's QuickStart section:

$ svn checkout http://project-voldemort.googlecode.com/svn/trunk/
project-voldemor

-- Jon

[Jay Kreps]

Hi Jon,

We use commons pool for connection pooling and the behavior of commons
pool is a little unusual. There are two settings: the maximum number
of total connections and the maximum number of connections per server.
When the maximum total number is hit, the client will close a
connection to one of the other servers to steal that slot for the
connection it needs. This means that under heavy load, if the max
total connections is low, it will always try to steal and will always
close connections rather than waiting. To avoid this just set the
maximum total connections to some high number.

-Jay

[ijuma]

On Mar 31, 8:49 pm, Jay Kreps <jay.kr...@gmail.com> wrote:
> Hi Jon,
>
> We use commons pool for connection pooling

By the way, now that you mention this in the old TODO file, there was:

"-- Rewrite socket pool to not use commons-pool because it sucks"

Did you change your mind about this since then? ;)

Best,
Ismael

[Dain Sundstrom]

I've hacked on commons pool, and strongly suggest that no one uses
it. It is very^3 buggy.

Have you looked at mina 2.x (FWIU 1.x was not fast, but 2.x is)?

-dain

[Jay Kreps]

No it still sucks :-). But I haven't seen any issues after I figured
out the way max_total_connections was handled. We should consider just
removing the max total connection param because the behavior it has is
almost always wrong.

-Jay

[Jon Travis]

FWIW: changing that param fixed all the socket issues I was seeing before..

-- Jon

[Jay Kreps]

Yeah there was a prior discussion about MINA a few weeks. The original
implementation used it, and I would not be opposed to going back if
someone is up for doing i and can get to a point of having something
stable, fast, and well-tested. At the time the performance I got was
not that good, but I suspect that was as much me as anything. Not
having to tune the number of connections is an advantage for sure.

-Jay