swapping a newly added (via admin tool) read-only store

[Avani]

I created new stores in my production cluster using admin tool add-
store utility.
when creating and swapping the stores, the swap step failed with the
following error:
11/01/05 15:55:08 INFO readonly.StoreSwapper: Attempting swap for node
0 dir = /voldemort/hdfs-fetcher/test_add_store_1294271682512/node-0
11/01/05 15:55:08 ERROR readonly.StoreSwapper: Exception thrown during
swap operation on node 0:
voldemort.VoldemortException: Error while performing operation:
'test_add_store' is not a registered read-only store.
at
voldemort.store.readonly.StoreSwapper.invokeSwap(StoreSwapper.java:
129)
at
voldemort.store.readonly.StoreSwapper.swapStoreData(StoreSwapper.java:
65)
at
voldemort.store.readonly.StoreSwapper.main(StoreSwapper.java:196)
11/01/05 15:55:08 INFO readonly.StoreSwapper: Attempting swap for node
1 dir = /voldemort/hdfs-fetcher/test_add_store_1294271684809/node-1
11/01/05 15:55:08 ERROR readonly.StoreSwapper: Exception thrown during
swap operation on node 1:
voldemort.VoldemortException: Error while performing operation:
'test_add_store' is not a registered read-only store.
at
voldemort.store.readonly.StoreSwapper.invokeSwap(StoreSwapper.java:
129)
at
voldemort.store.readonly.StoreSwapper.swapStoreData(StoreSwapper.java:
65)
at
voldemort.store.readonly.StoreSwapper.main(StoreSwapper.java:196)
11/01/05 15:55:08 INFO readonly.StoreSwapper: Attempting swap for node
2 dir = /voldemort/hdfs-fetcher/test_add_store_1294271683164/node-2
11/01/05 15:55:08 ERROR readonly.StoreSwapper: Exception thrown during
swap operation on node 2:
voldemort.VoldemortException: Error while performing operation:
'test_add_store' is not a registered read-only store.
at
voldemort.store.readonly.StoreSwapper.invokeSwap(StoreSwapper.java:
129)
at
voldemort.store.readonly.StoreSwapper.swapStoreData(StoreSwapper.java:
65)
at
voldemort.store.readonly.StoreSwapper.main(StoreSwapper.java:196)
Exception in thread "main" voldemort.VoldemortException:
voldemort.VoldemortException: Error while performing operation:
'test_add_store' is not a registered read-only store.
at
voldemort.store.readonly.StoreSwapper.invokeSwap(StoreSwapper.java:
138)
at
voldemort.store.readonly.StoreSwapper.swapStoreData(StoreSwapper.java:
65)
at
voldemort.store.readonly.StoreSwapper.main(StoreSwapper.java:196)
Caused by: voldemort.VoldemortException: Error while performing
operation: 'test_add_store' is not a registered read-only store.
at
voldemort.store.readonly.StoreSwapper.invokeSwap(StoreSwapper.java:
129)
... 2 more

[Roshan Sumbaly]

Hi Avani,
I'm assuming your using the 0.81 jar. This is a known bug where-in the servlet based push job doesn't pick up newly added stores. You need to bounce the cluster so as to pick it up.

Best,
Roshan

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To post to this group, send email to project-...@googlegroups.com.
To unsubscribe from this group, send email to project-voldem...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/project-voldemort?hl=en.

[Avani]

yes, it is 0.81. Any easy fix or work around. For obvious reasons, I
do not want to bounce my production cluster everytime I add a new
store to support a new application. Bouncing will affect all running
applications.

On Jan 5, 5:09 pm, Roshan Sumbaly <rsumb...@gmail.com> wrote:
> Hi Avani,
> I'm assuming your using the 0.81 jar. This is a known bug where-in the
> servlet based push job doesn't pick up newly added stores. You need to
> bounce the cluster so as to pick it up.
>
> Best,
> Roshan
>

> > project-voldem...@googlegroups.com<project-voldemort%2Bunsu...@googlegroups.com>
> > .

[Roshan Sumbaly]

The fix is here - https://github.com/voldemort/voldemort/blob/06579fa08deea6c4d86a2b6337a9ec3c4d854ab4/src/java/voldemort/server/http/gui/ReadOnlyStoreManagementServlet.java

Best,
Roshan

You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To post to this group, send email to project-...@googlegroups.com.

To unsubscribe from this group, send email to project-voldem...@googlegroups.com.

[Avani]

hello Roshan,

So I was able to do the swap successfully, but that required the
latest (git) version of Voldemort. Is there a particular way to
upgrade Voldemort ? Any documentation?

It seems the new version for the store is now symlinked as latest and
it is no more version-0 (but version-1). I am not sure if this is
because I did not use --push-version option correctly for swap-
store.sh.

I am now concerned about how Voldemort would handle the latest data. I
hope my application that is integrated with Voldemort would continue
to behave the same.

My command :
=============
bin/swap-store.sh --cluster config/cluster.xml --file
hdfs://box.com:9020/user/test_add_store --name test_add_store --push-
version 0

Resulting store:
================
ll data/read-only/test_add_store/
total 15
drwxr-xr-x 2 hadoop other 4 Jan 5 15:17 version-0
drwxr-xr-x 2 hadoop other 4 Jan 5 18:47 version-1
lrwxrwxrwx 1 hadoop other 72 Jan 5 18:47 latest ->
<absolute_path>/version-1


On Jan 5, 5:21 pm, Roshan Sumbaly <rsumb...@gmail.com> wrote:
> The fix is here -https://github.com/voldemort/voldemort/blob/06579fa08deea6c4d86a2b633...
>
> Best,
> Roshan

> > <project-voldemort%2Bunsu...@googlegroups.com<project-voldemort%252Buns...@googlegroups.com>

[Roshan Sumbaly]

The latest version requires you to run a grandfathering script on your data because we've changed the way we version the folders. We still haven't documented this anywhere and would suggest that you not upgrade yet.

Best,
Roshan

You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To post to this group, send email to project-...@googlegroups.com.

To unsubscribe from this group, send email to project-voldem...@googlegroups.com.

[Avani]

Thanks a lot,Roshan, In that case, I'll stick with the older version
(and the bug) for production for now.
I did copy the latest libs dist and bin after compiling new git code
into my dev cluster to get the code below to work.

Thanks again.

On Jan 5, 7:08 pm, Roshan Sumbaly <rsumb...@gmail.com> wrote:
> The latest version requires you to run a grandfathering script on your data
> because we've changed the way we version the folders. We still haven't
> documented this anywhere and would suggest that you not upgrade yet.
>
> Best,
> Roshan
>

> > > > <project-voldemort%2Bunsu...@googlegroups.com<project-voldemort%252Buns...@googlegroups.com>
> > <project-voldemort%252Buns...@googlegroups.com<project-voldemort%25252Bun...@googlegroups.com>

[Ross MacKinnon]

Does this mean that dropping in the fixed
ReadOnlyStoreManagementServlet.java, mentioned above, is not enough to
fix the problem? i.e., store swapping will still fail unless you move
to the latest build of Voldemort?

Ross

> > > > > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>
> > > <project-voldemort%2Bunsu...@googlegroups.com<project-voldemort%252Buns ubsc...@googlegroups.com>

>
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/project-voldemort?hl=en.
>
> > > --
> > > You received this message because you are subscribed to theGoogleGroups
> > > "project-voldemort" group.
> > > To post to this group, send email to project-...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > > project-voldem...@googlegroups.com<project-voldemort%2Bunsubscr i...@googlegroups.com>

[Roshan Sumbaly]

Hi Ross,
Nup, we removed the fix from the latest version since it doesn't play well with our internal system. ( Also because we've internally migrated to the new new "admin" based swapper ) But if you're interested you can pick up the fix from here -
https://github.com/voldemort/voldemort/blob/06579fa08deea6c4d86a2b6337a9ec3c4d854ab4/src/java/voldemort/server/http/gui/ReadOnlyStoreManagementServlet.java

Best,
Roshan

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To post to this group, send email to project-...@googlegroups.com.

To unsubscribe from this group, send email to project-voldem...@googlegroups.com.

[Apurva]

Hi Community,

Just want to verify is the above problem fixed? As I am still seeing the same issue in new 1.10+ cluster

Thanks,

Apurva

[Félix GV]

Hi Apurva,

The ReadOnlyManagementServlet is deprecated and not maintained anymore.

Arbitrary swapping is still supported, but only from the admin tool (callable via the shell or java).

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Apurva Thomas]

Thanks Felix.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "project-voldemort" group.

To unsubscribe from this group and stop receiving emails from it, send an email to project-voldemort+unsubscribe@googlegroups.com.