Client thread safety, memory leaks?

[Maarten Koopmans]

Hi,

The list is a bit quiet - I hope somebody can shine light on this -
even if I'm totally off track here.

I have a multi-threaded server that acts as a client to Voldemort, The
threads are more like actors/processes, in that they share nothing
except the Voldemort clients. These are obtained from a singleton, and
are shared across threads.

1) Is it safe to do so (i.e. is Voldemort's client thread safe)? I
read and assumed it was.

2) I am seeing small memory leaks on the server, while all my "server"
threads are cleaned up nicely according to my profiler. Is there a
known leak/issue here with the Voldemort clients in 0.90.1? Or should
I give every thread its own client instance from a factory object (but
how do I close them explicitly then)?

3) Of course the behavior differs across JDKs: OS X seems to have
almost no problems, Linux with OpenJDK6 is the worst, Sun JDK6 on
Linux also has the problem, but the leak is smaller per request. Just
to make life easier.... But the leak exists. The difference between OS
X and Linux is striking in the sense that I've also had to up ulimit
for processes on Linux (not OS X) to a high value ( I chose 51200,
just to get rid of out of file descriptors).

FWIW, I create the SocketStoreClientFactory with this client config
(in Scala) - clueless why, but these settings seem to work:

clientconfig = new ClientConfig()
clientconfig.setBootstrapUrls(bootlist)
clientconfig.setMaxBootstrapRetries(1)
clientconfig.setMaxThreads(512)
clientconfig.setMaxQueuedRequests(5000)
//clientconfig.setMaxTotalConnections(25000)
//clientconfig.setMaxConnectionsPerNode(9000)
clientconfig.setSelectors(512)

And a typical factory method for the singleton looks like this:

var requestCounterClient: VMClient[String,Long] = null

def getRequestMeterer() = {
if (requestCounterClient == null) {
debug("Getting VM request client")
val tmp: StoreClient[String,Long]= factory getStoreClient("requests")
requestCounterClient = new VMClient[String,Long](tmp)
}
debug("requestCounterClient - null?" + (requestCounterClient == null))

requestCounterClient
}

VMClient is a scala wrapper around LazyStoreClient, and the
requestCounterClient is null until the first request comes in.

If I should take a different approach or am missing something .... any
help/insights/advice greatly appreciated.

Thanks,

Maarten

[Carlos Tasada]

Hi Maarten,

Answering your questions:

1) AFAIK it's safe. At least I'm running a multithread server and I've not seen any problem.

2) As you already commented in another thread of the group, I've already found and fixed some memory and socked leaks. You can find the proposed patch in  https://github.com/voldemort/voldemort/pull/70 

3) I also needed to increase my ulimit to avoid "Too Many Open File" errors, but the previous patch seems to be solving the issues

I hope it helps.

Carlos

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[Maarten Koopmans]

Hi Carlos,

Many thanks, I thought it was me. Did you apply the patch against 0.90.1 or do I need to build against the current master branch?

Thanks again, I'll report my findings tomorrow or early next week ( I am in GMT+1).

And I hope the core team will be a bit more responsive, if only to incorporate patches and make releases....

Thanks again,

Maarten

[Carlos Tasada]

Hi Marteen,

Even when the patch is for the master branch, you can apply it to 0.90.1 easily. In fact I've it applied there in my production code.

AFAIK there are no other leaks in the connection code, but I known there're some points that could be improved (some of them are already modified in the master branch)

Regards,

Carlos

[Maarten Koopmans]

Hi!

Late reply, but the patch worked. With a subtle memory leak caused by
the Scala compiler in my software all runs smoothly. Only thing that I
find slightly worrisome is that the community seems to slowly get to a
halt.

Sure Voldemort largely just works, but if nobody asks questions then
that suggest not many new users - the lack of roadmap for releases
etc. is also not helping. Or is this a quiet period in the US in terms
of holidays or so?

Best regards, Maarten

[Carlos Tasada]

Hi Maarten,

I'm happy to heard the patch worked. Those fixes will be available in the next Voldemort version.

As you say the newsgroup is really quiet, but if you check in the project github page, there is some activity there. Seems that version 0.95 is already in the oven ;)

And regarding the roadmap, I think it's also a problem related with the maintenance of the opened issues. The impression is that the project is not really transparent with the community, or at least that's my impression. A different thing is the feedback from the developers that so far has been great, at least in my personal experience.

Regards,

Carlos.

[Chinmay Soman]

Thanks Carlos for helping out with the patch and for the knowledge
sharing.

@ Maarten : I can understand the general frustration regarding late
replies in the Voldemort community. There would be a release out soon
including the patch submitted by Carlos. And yes, we will also try to
better communicate with respect to releases.

The reality is that there are a lot of open issues in the LinkedIn
world keeping the few devs busy. FYI: we're now aggressively looking
into the pending pull requests on github (since that is of higher
priority). In time, we hope to catch up with the Google group
questions as well.

[Maarten Koopmans]

Good to read, catching up.... Voldemort is such an exceptional tedhnology, that it would be painful to watch it go out. I'm happy that's not happening!