Dynamic store creation

[Bruce Ritchie]

All,

One of the last major hurdles I have to overcome with using Voldemort
as an advanced caching server is to allow for dynamic creation of
stores. It seems that the AdminClient.updateRemoteStoreDefList(nodeId,
storesList) might almost work except I can't see anything in the code
that calls into StorageService.openStore(StoreDefinition) to register
a new store, schedule cleanup, etc. Am I missing something? If not,
I've love to work out a solution that would work for everyone to allow
new stores to be created dynamically through the admin client.

Regards,

Bruce Ritchie

[bhupesh bansal]

Bruce, 

 AdminClient.updateRemoteStoreDefList(nodeId,storesList) doesn't handle the case of adding new stores on the fly for now, It would be good to have it though ..

Best

Bhupesh

--

You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[Rob Adams]

Note that you can get around this by just using one store and adding a prefix to your keys to differentiate different "stores".  You won't be able to use the json serialization unless you have a common schema though.

[Bruce Ritchie]

Unfortunately that trick will not work for us because we're (mis)using
voldemort as a cache server, not just a key-value store. Being a cache
server we occasionally need to clear the cache in a single operation.

On Dec 22, 1:16 pm, Rob Adams <read...@readams.net> wrote:
> Note that you can get around this by just using one store and adding a
> prefix to your keys to differentiate different "stores".  You won't be able
> to use the json serialization unless you have a common schema though.
>

> On Mon, Dec 21, 2009 at 11:28 AM, bhupesh bansal <bbansal....@gmail.com>wrote:
>
>
>
> > Bruce,
>
> >  AdminClient.updateRemoteStoreDefList(nodeId,storesList) doesn't handle the
> > case of adding new stores on the fly for now, It would be good to have it
> > though ..
>
> > Best
> > Bhupesh
>

> > On Mon, Dec 21, 2009 at 11:21 AM, Bruce Ritchie <bruce.ritc...@gmail.com>wrote:
>
> >> All,
>
> >> One of the last major hurdles I have to overcome with using Voldemort
> >> as an advanced caching server is to allow for dynamic creation of
> >> stores. It seems that the AdminClient.updateRemoteStoreDefList(nodeId,
> >> storesList) might almost work except I can't see anything in the code
> >> that calls into StorageService.openStore(StoreDefinition) to register
> >> a new store, schedule cleanup, etc.  Am I missing something?  If not,
> >> I've love to work out a solution that would work for everyone to allow
> >> new stores to be created dynamically through the admin client.
>
> >> Regards,
>
> >> Bruce Ritchie
>
> >> --
>
> >> You received this message because you are subscribed to the Google Groups
> >> "project-voldemort" group.
> >> To post to this group, send email to project-...@googlegroups.com.
> >> To unsubscribe from this group, send email to

> >> project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>

> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/project-voldemort?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "project-voldemort" group.
> > To post to this group, send email to project-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>

[Jay Kreps]

Yeah I think this would be a very useful feature and not too hard to
implement as a api on admin client. If any one wants to take a shot, a
patch would definitely be accepted...

-Jay

> To unsubscribe from this group, send email to project-voldem...@googlegroups.com.

[Bruce Ritchie]

I'm sure if someone doesn't beat me to it I'll be adding that
functionality early January.

Bruce

[Bruce Ritchie]

I've taken some steps to implement this (http://github.com/Omega1/
voldemort/commit/38ba35813b3ad0e1f1635b122e208a247d5cea4c) however I
can't say I'm terribly happy with the solution, especially in that
store removal is completely unimplemented. If anyone could review I'd
appreciate it.

Regards,

Bruce Ritchie

[Jay Kreps]

Hey Bruce,

This is great. Couple of quick things:

I think you removed the fillInStackTrace on ObsoleteVersionException,
was this just an accident?
(http://github.com/Omega1/voldemort/commit/38ba35813b3ad0e1f1635b122e208a247d5cea4c#diff-5)

Do you think it might be easier to add a add/remove store to the
adminClient api directly? One concern I have about the
handleUpdateStoreMetadata is that it takes it upon itself to figure
out what the diff between the stores is. This is a bit complicated to
do, and it is unclear to the caller which changes are handled
correctly and which are not--for example if I change the replication
factor on an existing store that will not really do anything, but that
is a little unintuitive. If we had an addStore API on the admin client
which internally read, updated, and wrote the new stores.xml, it would
be clear what action was taking place and we could remove the code
that has to calculate the diff (though we would need to add a new PB
definition for that request, though that is fairly easy).

-Jay

[Bruce Ritchie]

On Jan 31, 11:18 pm, Jay Kreps <jay.kr...@gmail.com> wrote:

> I think you removed the fillInStackTrace on ObsoleteVersionException,
> was this just an accident?

> (http://github.com/Omega1/voldemort/commit/38ba35813b3ad0e1f1635b122e2...)

I think so, yes. Not sure why I did that :(

> Do you think it might be easier to add a add/remove store to the
> adminClient api directly?

Yes, I do. I just wasn't sure if that was more desirable then fleshing
out the existing api. I'm actually for removing the
AdminClient.updateRemoteStoreDefList(..) method as it's not currently
used and I don't really think it's something that can really be
properly implemented without more support for reloading engines/
stores.

> One concern I have about the
> handleUpdateStoreMetadata is that it takes it upon itself to figure
> out what the diff between the stores is. This is a bit complicated to
> do, and it is unclear to the caller which changes are handled
> correctly and which are not--for example if I change the replication
> factor on an existing store that will not really do anything, but that
> is a little unintuitive. If we had an addStore API on the admin client
> which internally read, updated, and wrote the new stores.xml, it would
> be clear what action was taking place and we could remove the code
> that has to calculate the diff (though we would need to add a new PB
> definition for that request, though that is fairly easy).

Yep. Let me go ahead and re-implement this, shouldn't take much time
at all.

Regards,

Bruce Ritchie

[Bruce Ritchie]

> Yep. Let me go ahead and re-implement this, shouldn't take much time
> at all.

I've implemented the addStore functional which can be seen @
http://github.com/Omega1/voldemort/commit/8033cf0a4e3ec864dd480bf89b39f1a2c5d21abd.
I haven't attempted to implement removeStore yet since none of the
existing server api's support that functionality and I'm not familiar
enough yet with all the nuances of the code to know what will require
touching and what won't.

Regards,

Bruce Ritchie

[Bruce Ritchie]

I forgot I had not created an issue in the bug tracker for this, so
issue 217 now encompasses this enhancement.

On Feb 2, 12:07 pm, Bruce Ritchie <bruce.ritc...@gmail.com> wrote:
> > Yep. Let me go ahead and re-implement this, shouldn't take much time
> > at all.
>

> I've implemented the addStore functional which can be seen @http://github.com/Omega1/voldemort/commit/8033cf0a4e3ec864dd480bf89b3....

[Jay Kreps]

Awesome, thanks! we will get it applied.

-Jay