EOFException while set-metadata for stores.xml in v1.10+

[Mohanjot Singh]

Hi Community,

I am facing EOFException while using --set-metadata option for setting stores.xml using the admin client.  Please note --set-metadata for cluster.xml works fine.

Scenario: Added a zone in stores.xml - hence bumped up the Replication-factor accordingly.

bin/voldemort-admin-tool.sh --set-metadata stores.xml --set-metadata-value <path_to_stores.xml/stores.xml> --url tcp://localhost:<admin_port>

** I tried using vadmin.sh as well as voldemort-admin-tool.sh is deprecated in V 1.10+

Exception :

voldemort.VoldemortException: java.io.EOFException

at voldemort.client.protocol.admin.AdminClient$RPCOperations.sendAndReceive(AdminClient.java:646)

at voldemort.client.protocol.admin.AdminClient$RPCOperations.access$1500(AdminClient.java:591)

at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.updateRemoteMetadata(AdminClient.java:1138)

at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.updateRemoteMetadata(AdminClient.java:1180)

at voldemort.VoldemortAdminTool.executeSetMetadata(VoldemortAdminTool.java:1393)

at voldemort.VoldemortAdminTool.main(VoldemortAdminTool.java:601)

Caused by: java.io.EOFException

at java.io.DataInputStream.readInt(DataInputStream.java:392)

at voldemort.client.protocol.pb.ProtoUtils.readToBuilder(ProtoUtils.java:216)

at voldemort.client.protocol.admin.AdminClient$RPCOperations.innerSendAndReceive(AdminClient.java:600)

at voldemort.client.protocol.admin.AdminClient$RPCOperations.sendAndReceive(AdminClient.java:641)

... 5 more

java.io.EOFException

NPE in server logs:

voldemort.server.niosocket.AsyncRequestHandler] ERROR Caught throwable from Socket[addr=/<clientIP>,port=34270,localport=<admin_port>] [voldemort-admin-server-t14]

java.lang.NullPointerException

at voldemort.store.configuration.ConfigurationStorageEngine.put(ConfigurationStorageEngine.java:146)

at voldemort.store.configuration.ConfigurationStorageEngine.put(ConfigurationStorageEngine.java:50)

at voldemort.store.metadata.MetadataStore.put(MetadataStore.java:355)

at voldemort.store.metadata.MetadataStore.put(MetadataStore.java:493)

at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleSetMetadata(AdminServiceRequestHandler.java:1440)

at voldemort.server.protocol.admin.AdminServiceRequestHandler.handleRequest(AdminServiceRequestHandler.java:197)

at voldemort.server.niosocket.AsyncRequestHandler.read(AsyncRequestHandler.java:192)

at voldemort.common.nio.SelectorManagerWorker.run(SelectorManagerWorker.java:105)

at voldemort.common.nio.AbstractSelectorManager.run(AbstractSelectorManager.java:243)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Only way to fix the issue /get rid of the error was to bounce the cluster and let the nodes pick metadata changes automatically.

Can someone please advice or has faced this issue and suggest a solution to this?

[Arunachalam]

Are you running your Voldemort Server on windows or linux ? Seems like listFiles can return null which is not accounted for.

Using Set-Metadata to add or remove stores regardless of this bug, will require the server bounce.

You should use Set-Metadata only to modify the store.

you should use 'vadmin.sh store add|update|delete' to do the store related operation.

set-metadata is only intended when everything else fails and you have no other option to force override it. You should see the logs in your server logs, if you do that.

https://github.com/voldemort/voldemort/blob/master/src/java/voldemort/store/metadata/MetadataStore.java#L306

Thanks,

Arun.

Thanks,

Arun.

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Mohanjot Singh]

Hi Arun,

I am running Voldemort Server on Linux.

I am using set-metadata for applying the new stores.xml with changes <replication-factor>, <replication-factor zone-id> values. This is being done for Zone expansion test.

which is part of modification of store.

As I had mentioned set-metadata works fine while setting the new value for cluster.xml.

Thanks

Mohanjot

[Arunachalam]

I see.

We will eventually merge this patch. In the mean time, if you have a fork, please apply this patch and retry.

https://github.com/voldemort/voldemort/pull/393

Thanks,

Arun.

[Brendan Harris (a.k.a. stotch on irc.oftc.net)]

Another thing to note, set-metadata (meta set) is a bit of a blunt instrument and can have problems. It should only be used when making mass changes across all existing stores on a non-live node or non-live cluster. Otherwise it should only ever be used for modifying cluster.xml The best solution for updating existing stores is to use "vadmin.sh store update" or "voldemort-admin-tool.sh --update-store-defs". This will iterate over the stores in the input file, compare between them and the original, sanity check the changes and apply the difference in changes as well as update the store's metadata version. If you use "meta set" or "set-metadata", the store's metadata version number will not be incremented and any clients reading or writing the stores may not notice the changes until you restart them.

Brendan

[Arunachalam]

As Brendan pointed out, you should only use the 'vadmin.sh store update' command with the full stores.xml which should achieve the same purpose. Let us know if that is not the case.

Thanks,

Arun.

On Fri, Mar 25, 2016 at 4:37 PM, Brendan Harris (a.k.a. stotch on irc.oftc.net) <dre...@gmail.com> wrote:

Another thing to note, set-metadata (meta set) is a bit of a blunt instrument and can have problems. It should only be used when making mass changes across all existing stores on a non-live node or non-live cluster. Otherwise it should only ever be used for modifying cluster.xml The best solution for updating existing stores is to use "vadmin.sh store update" or "voldemort-admin-tool.sh --update-store-defs". This will iterate over the stores in the input file, compare between them and the original, sanity check the changes and apply the difference in changes as well as update the store's metadata version. If you use "meta set" or "set-metadata", the store's metadata version number will not be incremented and any clients reading or writing the stores may not notice the changes until you restart them.

Brendan

--