Hadoop bnp job complains "not a SequenceFile"

[Apurva]

Hi all,

I am trying to run the hadoop bnp job and encountering the following exception while running ./bin/run-bnp.sh config.txt

16/05/11 22:04:36 ERROR azkaban.VoldemortBuildAndPushJobRunner: Exception while running BnP job!

voldemort.VoldemortException: An exception occurred during Build and Push !!

at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:539)

at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJobRunner.main(VoldemortBuildAndPushJobRunner.java:34)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: hdfs://host1:9000/user/hadoop/textinputformat/part-r-00000 not a SequenceFile

at voldemort.store.readonly.mr.utils.HadoopUtils.getSchemaFromPath(HadoopUtils.java:184)

at voldemort.store.readonly.mr.utils.HadoopUtils.getSchemaFromPath(HadoopUtils.java:116)

at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.verifyOrAddJsonStore(VoldemortBuildAndPushJob.java:578)

at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:471)

... 7 more

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: hdfs://host1:9000/user/hadoop/textinputformat/part-r-00000 not a SequenceFile

at voldemort.store.readonly.mr.utils.HadoopUtils.getSchemaFromPath(HadoopUtils.java:184)

at voldemort.store.readonly.mr.utils.HadoopUtils.getSchemaFromPath(HadoopUtils.java:161)

... 10 more

Caused by: java.lang.RuntimeException: java.io.IOException: hdfs://host1:9000/user/hadoop/textinputformat/part-r-00000 not a SequenceFile

at voldemort.store.readonly.mr.utils.HadoopUtils.getMetadataFromSequenceFile(HadoopUtils.java:111)

at voldemort.store.readonly.mr.utils.HadoopUtils.getSchemaFromPath(HadoopUtils.java:137)

... 11 more

Caused by: java.io.IOException: hdfs://host1:9000/user/hadoop/textinputformat/part-r-00000 not a SequenceFile

at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1920)

at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1880)

at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1829)

at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1843)

at voldemort.store.readonly.mr.utils.HadoopUtils.getMetadataFromSequenceFile(HadoopUtils.java:101)

... 12 more

BnP run script finished!

config.txt:

type=java

job.class=voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob

hadoop.job.ugi=hadoop

build.input.path=/user/hadoop/textinputformat

build.output.dir=/wordcount/output

push.store.name=wordcount

push.cluster=tcp://localhost:

push.store.description="test store"

push.store.owners=myemail@myworkplace.com

build.replication.factor=1

Could someone please help me out?

Thanks,

Apurva

[Félix GV]

What is the format of the files in your input directory?

The best-supported format to use with Voldemort RO is avro, but you do need to configure it in your BnP job. You can see those configs here:

http://www.project-voldemort.com/voldemort/build-and-push.html

Otherwise, without any specific indication, the BnP job assumes a default format of binary json, which is not super standard/straightforward to serialize correctly.

--
Félix

push.store.owners=mye...@myworkplace.com

build.replication.factor=1

Could someone please help me out?

Thanks,

Apurva

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at https://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.