Voldemort Open Source release 1.6.0 out

[Siddharth Singh]

Hi Everyone,

Voldemort 1.6.0 has been released : http://engineering.linkedin.com/voldemort/announcing-voldemort-160-open-source-release

Take it for a spin and as always, please let us know of any issues.

Note :  Voldemort now uses BDB-JE 5. Please read the blog post about how to upgrade.

Thanks
Siddharth

[Ryan Barker]

We are in the middle of upgrading and consolidating an array of voldemort servers and clients and had a few questions.

1) Can the 0.80 client talk to a 1.6 server?
2) Can the 1.3 servers be updated 1 at a time to 1.6 while clients (1.3, 0.96 and 0.8) are connecting to it?
3) What are the recommended configurations for flash drive based servers vs. spinning disk servers in this version?
4) Is the performance of the thin client and thick client comparable?

Our current plan is to upgrade and consolidate servers on 1.3, update to 1.6 on the server side, then update the client side to 1.6.

Any issues?

[Vinoth C]

Inline..

On Friday, January 31, 2014 10:58:51 PM UTC-8, Ryan Barker wrote:

We are in the middle of upgrading and consolidating an array of voldemort servers and clients and had a few questions.

1) Can the 0.80 client talk to a 1.6 server?

Should be able to.. i.e we have not changed any client protocol messages. But strongly recommend moving to 1.6 client since there have been quite a critical bug fixes in 0.96,1.3 and 1.6.. 

2) Can the 1.3 servers be updated 1 at a time to 1.6 while clients (1.3, 0.96 and 0.8) are connecting to it?

yeah.. Thats how we did it.. Our client were running 1.3 I think.. 

3) What are the recommended configurations for flash drive based servers vs. spinning disk servers in this version?

In general, it should not change your config tht much.. (we did not reconfigure anything)..  We have not really run this on spinning disks. So you tell us how it goes :)
Of course, post in the forums and I will work with you on getting it configured..
 

4) Is the performance of the thin client and thick client comparable?

There are two angles to this.. In terms of latency, I don't think the thin client -> coordinator -> servers path will add more than say 1ms more to the average latency.. But then, if you are now doing X operations/sec at 1 ms latency, then to get the same throughput you may need to double the threads.. So basically, you can get comparable latency, and throw a little more resources to achieve the same throughput.. makes sense? 

Our current plan is to upgrade and consolidate servers on 1.3, update to 1.6 on the server side, then update the client side to 1.6.

Any issues?

yes..  It seems like you have a some very old clients. So, only thing I recommend is to test out the 0.8 client to 1.6  server compatibility before moving on..

[Ryan Barker]

Yes we do have some very old clients ;) and servers for that matter. We are in the process of upgrading and consolidating most of them onto a cluster of flash based servers. Unfortunately we have some other extremely high risk projects going on so I was hoping to sneak in an upgrade all the way to 1.6 before upgrading the clients. We obviously were planning on testing this before hand, was just hoping for some guidance. We have had some issues (specifically with compressed stores) with using the 1.3 client to talk to the 0.8 and 0.9X servers so the plan had been to finish the upgrades to 1.3 first on the server side then the client side.  The 1.6 release obviously throws a wrinkle into this, and doubling the QA effort is not feasible at this time.

Separately, is there a pom.xml for this release? If not, I will once again make one for you. You really should switch to maven or gradle and separate the client to its own code base already ;)

Ryan

[Vinoth C]

>> Separately, is there a pom.xml for this release? If not, I will once again make one for you. You really should switch to maven or gradle and separate the client to its own code base already ;)

Nope not yet.. :) . But, I think this will happen soon as all the vold devs in Linkedin have agreement on this.

Anyways, keep the forum posted on how the 1.6 release goes..

[Ryan Barker]

I received very poor reception with my plan to update to 1.6 any time soon unfortunately :(. That said I am definitely willing to help with the pom.xml and refactorings on my own time needed to get the distribution into a good shape for nexus central.  I have quite a few ideas (in fact its hard to shut me up as my coworkers will tell you) on how to separate the server/thick client/shared/example/thin client projects with maven in a quick and safe manner. I've done this 3 times with the voldemort code base and even more with various internal projects at this point so I feel I have some experience in the matter ;) Make a git branch and send me an email and I'll help get it done. There are quite a few maven plugins/settings that will make everything run much smoother from a release standpoint. Maven has its issues but its not that bad honestly.

Send me an email at my work or personal email.

Ryan

[Vinoth C]

My first reaction to spliting up client and servers is : it could be tricky for us given the massive amount of clients inside Linkedin that depend on the single voldemort jar.. That upgrade process could take months and we cannot afford to be blocked on development in the meantime.. Technically, there is no reason why it should not be done.. Let me take this back to the team and get back to you.. 

May be a good start would be to simply mavenize the project for starters, atleast giving us dependency management and help integrate with other maven projects.. 

[Justin Mason]

On Tue, Feb 4, 2014 at 3:12 PM, Vinoth C <mail.vino...@gmail.com> wrote:

May be a good start would be to simply mavenize the project for starters, atleast giving us dependency management and help integrate with other maven projects.. 

+1, agreed.

BTW, if gradle is an option, it may provide an easier migration path -- I've found it very scriptable and flexible, compared to Maven's much stricter model.

--j.

[Ryan Barker]

To handle this, you have 2 options. 1 is to simply create a voldemort-all artifact in addition to the voldemort-client, voldemort-server, etc artifacts and people can choose what they want. This causes major issues in the long run however.

Id recommend simply not creating a voldemort-all artifact and including all of the sub project jars in any project that both needs a server and a client inside of it.. There is really no difference between having 1 large jar and X small jars with the exact same files as the 1 large jar. The benefit of not having your server dependencies on your clients makes your 'thick client' implementations far less thick.

All that said, creating an voldemort-all artifact pom is super trivial to do and would take me about 1/10 the time of actually splitting up the dependencies properly, so up to you.

I can honestly get you a pom.xml for a simple voldemort-all within a day or 2 if need be, and proper sub modules in about a week.. just let me know. Its fairly trivial with eclipse to move things around and I already know the correct maven dependencies to add to auto generate the protobuf code, know what files need to be moved into src/main/resources and src/test/resources etc from the work I did on 0.96...

[Johannes Alkjaer]

Great :)

Are you going to make an offical binary release?
"Download Latest" on http://www.project-voldemort.com/voldemort/ point to the google-code repo which only has the yesteryear releases.

Regards
Johannes

[Vinoth C]

Thanks for catching this Johannes.. 

The binaries are here

https://github.com/voldemort/voldemort/releases/tag/release-1.6.0-cutoff

looks like someone forgot to update the site.. 

[Vinoth C]

This is now fixed!! 

[Dário Nascimento]

Ryan, did you move forward with this idea of splitting the client/server and create independent maven dependencies? At least for me, this would be useful. I'm deploying a client in Wildfly 8... 

[Ryan Barker]

I did not move forward sorry. I have been short on time and more importantly, the voldemort team seems to want to use gradle.

[Dário Nascimento]

No worries :) Problem solved, manual pom dependencies... I will try to split my modified voldemort into client and server. However, I will not commit since my version has a different interface and voldemort team don't want it...

Java client uses serializing, start, delegating, pipeline routed and socket Stores, maybe that's why (I have 2 days experience only, maybe I'm wrong)...

Thanks ;)

[Curtis Ruck]

Is there going to be any formal releases for the 1.8.x versions currently on github?  Or should I just clone and build locally?