Upgrade to BDB 4.0
ijuma
BDB 4.0 has been out for a while and I've been using it with Voldemort
for some time without issues. If you use BDB 4.0 instead of BDB 3.x
with Voldemort, the fetchKeys operation of AdminClient is a lot more
efficient as it doesn't retrieve the value from disk. Bhupesh seems
interested in this for the work that he and Alex are doing for
rebalancing (see http://code.google.com/p/project-voldemort/issues/detail?id=167
for details).
The change itself is trivial[1], and users can easily continue to use
BDB 3.x by including the old version as a library, so I plan to merge
it to master on Sunday, if there are no objections. What does this
mean to you?
- Once you move to a version of Voldemort that uses BDB 4.0, you won't
be able to rollback to an older version _with_ BDB 3.x as it upgrades
the disk format. You _can_ rollback to an old version of Voldemort and
still use the BDB 4.0 jar as long as you compile Voldemort yourself.
- If you use other applications in the _same_ process as Voldemort who
also use BDB JE directly (not via Voldemort), you will need to
recompile them with BDB JE 4.0 as it's not binary compatible (it is
source compatible though).
If any of this is a problem to you, now is the time to let us know. :)
Best,
Ismael
[1] http://github.com/ijuma/voldemort/commit/7d73231f072b92848c6a56129c25bf0c998c3f64
Geir Magnusson Jr.
On Dec 18, 2009, at 4:10 AM, ijuma wrote:
> Hey all,
>
> BDB 4.0 has been out for a while and I've been using it with Voldemort
> for some time without issues.
In production, I assume? What kind of usecase? I'm using Voldemort in 3 places in our core online transaction processing at Gilt (shopping cart, checkout and order processing) - our $300MM business *depends* on V for the core revenue generation. If there's a problem, we don't make money, and presumably I have to post a note to this list looking for consulting work :)
> If you use BDB 4.0 instead of BDB 3.x
> with Voldemort, the fetchKeys operation of AdminClient is a lot more
> efficient as it doesn't retrieve the value from disk. Bhupesh seems
> interested in this for the work that he and Alex are doing for
> rebalancing (see http://code.google.com/p/project-voldemort/issues/detail?id=167
> for details).
>
> The change itself is trivial[1], and users can easily continue to use
> BDB 3.x by including the old version as a library, so I plan to merge
> it to master on Sunday, if there are no objections. What does this
> mean to you?
>
> - Once you move to a version of Voldemort that uses BDB 4.0, you won't
> be able to rollback to an older version _with_ BDB 3.x as it upgrades
> the disk format. You _can_ rollback to an old version of Voldemort and
> still use the BDB 4.0 jar as long as you compile Voldemort yourself.
>
> - If you use other applications in the _same_ process as Voldemort who
> also use BDB JE directly (not via Voldemort), you will need to
> recompile them with BDB JE 4.0 as it's not binary compatible (it is
> source compatible though).
>
> If any of this is a problem to you, now is the time to let us know. :)
This got my antennas up, as this is a big deal for me. I'm trying to get my own production use locked into more recent versions, so the fact you can't go back is a problem.
Putting on my marketing hat :
I know we're a mere v0.6, but I'm utterly convinced that this is a big mistake and we should be 1.x at this point given the maturity of the code and core line-of-business uses it's getting. Maybe I'm alone in the following position, but I'd like for us to position ourselves as 'production ready'. Only crazy people would throw a 0.6 version of anything into production :) A 1.1 is an entirely different story. I know that there are a lot of features in progress that *we* perceive as being big holes that don't make us worthy of a 1.0, but a) that's our perception and V *is* solidly providing incredible value for lots of people and b) MSFT proved that you can ship just about any crap you have lying around as a 1.0 and it's fine....
(removing hat)
So to that end, maybe we start supporting 2 versions of V, one with BDB3, one with BDB4, until we can be really sure that going to BDB4 version is 'safe', and then put vBDB3 in maintenance.
I'm not going to suggest how we version it right now - I have ideas, but I'm still savoring the great "add +0.02 event" that got us to 0.60 instead of 0.58 :)
>
> Best,
> Ismael
>
> [1] http://github.com/ijuma/voldemort/commit/7d73231f072b92848c6a56129c25bf0c998c3f64
>
> --
>
> You received this message because you are subscribed to the Google Groups "project-voldemort" group.
> To post to this group, send email to project-...@googlegroups.com.
> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.
>
>
ijuma
> I know we're a mere v0.6, but I'm utterly convinced that this is a big mistake and we should be 1.x at this point given the maturity of the code and core line-of-business uses it's getting.
I don't want to get into a versioning debate, but a distributed store
without online rebalancing doesn't deserve to be 1.0 in my opinion.
Voldemort doesn't even have automated offline rebalancing. ;) So,
maybe sometime next year after the work from Bhupesh and Alex is
merged to master, we can start thinking about it. Personally, I'd like
to see some work to make it possible to upgrade internal data formats
(things like vector clocks) transparently before 1.0.
> So to that end, maybe we start supporting 2 versions of V, one with BDB3, one with BDB4, until we can be really sure that going to BDB4 version is 'safe', and then put vBDB3 in maintenance.
Did you look at the diff? You can continue to use BDB3 if you prefer.
You just have to keep BDB3 in your lib folder instead of BDB4 and
build it with that. I will leave the decision on whether two different
binaries should be provided to the LinkedIn guys as they are the ones
who build the releases.
Best,
Ismael
bhupesh bansal
I think I agree with you that "not able to use bdb 3.xx" is a big deal.
My question is we have to do the plunge some time and what features/plans we should have to give folks confidence ?
In my opinion there are few things we can do
1) Formalize a rollback plan in case 4.0 did fail on us: One plan is to use bdb dump/bdb Load tools which takes some
time 24/48 hours but can work if needed. We should talk to bdb guys to see if there are better options.
2) A adminClient based data backup feature : we can stream data to hadoop everynight for daily backups etc ?
3) Wait for bdb 4.1xx
Best
Bhupesh
ijuma
> 3) Wait for bdb 4.1xx
I don't see how this would help. It seems like people here are taking
the release number too seriously. BDB4 is not a rewrite. As far as I
understand, they bumped the release number mostly because of the newly
introduced HA stuff, which Voldemort does not use.
Best,
Ismael