Import data from hdfs

[Thiago Henrique dos Santos Bento]

Hi!

I'm trying to import data from hdfs with Build and Push plugin.I installed the Azkaban and Voldemort project, but i don't know what i have to do now. 

I have a batch view table in hdfs, but it is not stored as json or avro. Do I need to store this in avro or json?

Can somebody help me with this process? i have no idea how do i do this.

Thanks!

[Felix GV]

The HadoopStoreJobRunner accepts an 'inputformat' parameter that should accept any org.apache.hadoop.mapred.InputFormat implementation. I would suggest that as a first step in your investigation.

If you go down this route, you may also need to pass in other parameters specifying implementations of other components of the BnP job, but I'm not sure about that. For example, there is a 'writer.class' parameter which would be an implementation of voldemort.store.readonly.disk.HadoopStoreWriter but currently both the AvroStoreBuilderReducer and HadoopStoreBuilderReducer use the same provided implementation so it would seem it is generic enough to accomodate both.

We don't use anything else than Avro at LinkedIn, so we may not be able to provide much help if you run into issues. If that happens, please feel free to post follow up questions, but reading the code may end up being your best ally.

Good luck! And if you get a chance, please report back here (or in a blog post/GitHub branch, etc.) to let us know if you manage to make BnP work with something else than JSON or Avro (: !

--
 
Felix GV
Data Infrastructure Engineer
Distributed Data Systems
LinkedIn
 
f...@linkedin.com
linkedin.com/in/felixgv

---

From: project-...@googlegroups.com [project-...@googlegroups.com] on behalf of Thiago Henrique dos Santos Bento [thiago...@gmail.com]
Sent: Friday, November 14, 2014 8:12 AM
To: project-...@googlegroups.com
Subject: [project-voldemort] Import data from hdfs

--
You received this message because you are subscribed to the Google Groups "project-voldemort" group.
To unsubscribe from this group and stop receiving emails from it, send an email to project-voldem...@googlegroups.com.
Visit this group at http://groups.google.com/group/project-voldemort.
For more options, visit https://groups.google.com/d/optout.

[Thiago Henrique dos Santos Bento]

I created a batch view in avro format.

I installed Azkaban project with plugins(HDFS Browser, Jobtypes e etc).

My HDFS Browser: Error: Cannot get FileSystem.

It doesn't metter, because i use Hue to HDFS Browser.

I'm trying to run a BnP but i'm getting these errors:

19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample INFO - WARN Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - Exception in thread "main" java.lang.reflect.InvocationTargetException
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at java.lang.reflect.Method.invoke(Method.java:606)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.runMethod(JavaJobRunnerMain.java:176)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.(JavaJobRunnerMain.java:125)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.main(JavaJobRunnerMain.java:72)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: No files found in path pattern /apps/hive/warehouse/desenv.db/test
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.utils.AvroUtils.getSchemaFromPath(AvroUtils.java:112)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.utils.AvroUtils.getAvroSchemaFromPath(AvroUtils.java:118)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.verifyOrAddStoreAvro(VoldemortBuildAndPushJob.java:694)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:271)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	... 7 more
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.IllegalArgumentException: No files found in path pattern /apps/hive/warehouse/desenv.db/test
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.utils.AvroUtils.getSchemaFromPath(AvroUtils.java:80)
19-11-2014 04:17:09 PST hadoop-hello-world-push-new-sample ERROR - 	... 10 more

There is an avro file and avro schema file in this place /apps/hive/warehouse/desenv.db/test.
Any idea how to fix this error?

Thanks

[Thiago Henrique dos Santos Bento]

now this:

19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Exception in thread "main" java.lang.reflect.InvocationTargetException
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at java.lang.reflect.Method.invoke(Method.java:606)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.runMethod(JavaJobRunnerMain.java:176)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.(JavaJobRunnerMain.java:125)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.main(JavaJobRunnerMain.java:72)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.ExceptionInInitializerError
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VAdminProto.(VAdminProto.java:27508)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VAdminProto$VoldemortAdminRequest.(VAdminProto.java:26393)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteMetadata(AdminClient.java:1232)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteStoreDefList(AdminClient.java:1362)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.findAndVerifyAvro(VoldemortBuildAndPushJob.java:808)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.verifyOrAddStoreAvro(VoldemortBuildAndPushJob.java:743)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:271)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	... 7 more
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.RuntimeException: Generated message class "voldemort.client.protocol.pb.VProto$Versioned$Builder" missing method "getVersionBuilder".
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1417)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage.access$1300(GeneratedMessage.java:57)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage$FieldAccessorTable$SingularMessageFieldAccessor.(GeneratedMessage.java:1854)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage$FieldAccessorTable.ensureFieldAccessorsInitialized(GeneratedMessage.java:1511)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage$FieldAccessorTable.(GeneratedMessage.java:1467)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VProto$1.assignDescriptors(VProto.java:6363)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:305)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VProto.(VProto.java:6476)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	... 14 more
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.NoSuchMethodException: voldemort.client.protocol.pb.VProto$Versioned$Builder.getVersionBuilder()
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at java.lang.Class.getMethod(Class.java:1665)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1415)
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	... 21 more

Em quarta-feira, 19 de novembro de 2014 10h23min58s UTC-2, Thiago Henrique dos Santos Bento escreveu:

I created a batch view in avro format.

I installed Azkaban project with plugins(HDFS Browser, Jobtypes e etc).

My HDFS Browser: Error: Cannot get FileSystem.

now

 

19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Exception in thread "main" java.lang.reflect.InvocationTargetException 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at java.lang.reflect.Method.invoke(Method.java:606) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at azkaban.jobtype.JavaJobRunnerMain.runMethod(JavaJobRunnerMain.java:176) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at azkaban.jobtype.JavaJobRunnerMain.(JavaJobRunnerMain.java:125) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at azkaban.jobtype.JavaJobRunnerMain.main(JavaJobRunnerMain.java:72) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.ExceptionInInitializerError 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.pb.VAdminProto.(VAdminProto.java:27508) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.pb.VAdminProto$VoldemortAdminRequest.(VAdminProto.java:26393) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteMetadata(AdminClient.java:1232) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteStoreDefList(AdminClient.java:1362) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.findAndVerifyAvro(VoldemortBuildAndPushJob.java:808) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.verifyOrAddStoreAvro(VoldemortBuildAndPushJob.java:743) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:271) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - ... 7 more 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.RuntimeException: Generated message class "voldemort.client.protocol.pb.VProto$Versioned$Builder" missing method "getVersionBuilder". 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1417) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage.access$1300(GeneratedMessage.java:57) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage$FieldAccessorTable$SingularMessageFieldAccessor.(GeneratedMessage.java:1854) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage$FieldAccessorTable.ensureFieldAccessorsInitialized(GeneratedMessage.java:1511) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage$FieldAccessorTable.(GeneratedMessage.java:1467) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.pb.VProto$1.assignDescriptors(VProto.java:6363) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.Descriptors$FileDescriptor.internalBuildGeneratedFileFrom(Descriptors.java:305) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at voldemort.client.protocol.pb.VProto.(VProto.java:6476) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - ... 14 more 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.NoSuchMethodException: voldemort.client.protocol.pb.VProto$Versioned$Builder.getVersionBuilder() 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at java.lang.Class.getMethod(Class.java:1665) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - at com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1415) 19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - ... 21 more

[Arunachalam]

Please do not create multiple post  for the same discussion.

The error message seems like you have a missing dependency, but not sure what is that. Is there anyone else outside LinkedIn using the Hadoop BuildAndPush and can help Thiago here?

Caused by: java.lang.NoSuchMethodException: voldemort.client.protocol.pb.VProto$Versioned$Builder.getVersionBuilder()
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at java.lang.Class.getMethod(Class.java:1665) 
19-11-2014 09:11:32 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage.getMethodOrDie(GeneratedMessage.java:1415)  

--

[Thiago Henrique dos Santos Bento]

26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - Exception in thread "main" java.lang.reflect.InvocationTargetException
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at java.lang.reflect.Method.invoke(Method.java:606)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.runMethod(JavaJobRunnerMain.java:176)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.(JavaJobRunnerMain.java:125)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at azkaban.jobtype.JavaJobRunnerMain.main(JavaJobRunnerMain.java:72)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - Caused by: java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VAdminProto$GetMetadataRequest.getSerializedSize(VAdminProto.java:196)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.VAdminProto$VoldemortAdminRequest.getSerializedSize(VAdminProto.java:24665)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.pb.ProtoUtils.writeMessage(ProtoUtils.java:204)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$RPCOperations.innerSendAndReceive(AdminClient.java:598)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$RPCOperations.getSocketAndStreams(AdminClient.java:619)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$RPCOperations.sendAndReceive(AdminClient.java:640)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$RPCOperations.access$1500(AdminClient.java:591)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteMetadata(AdminClient.java:1237)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.client.protocol.admin.AdminClient$MetadataManagementOperations.getRemoteStoreDefList(AdminClient.java:1362)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.findAndVerifyAvro(VoldemortBuildAndPushJob.java:808)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.verifyOrAddStoreAvro(VoldemortBuildAndPushJob.java:743)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	at voldemort.store.readonly.mr.azkaban.VoldemortBuildAndPushJob.run(VoldemortBuildAndPushJob.java:271)
26-11-2014 07:35:40 PST hadoop-hello-world-push-new-sample ERROR - 	... 7 more
26-11-2014 07:35:41 PST hadoop-hello-world-push-new-sample INFO - Process completed unsuccessfully in 3 seconds.
26-11-2014 07:35:41 PST hadoop-hello-world-push-new-sample ERROR - Job run failed!

Em sexta-feira, 14 de novembro de 2014 14h12min05s UTC-2, Thiago Henrique dos Santos Bento escreveu:

[Felix GV]

Hi Thiago,

What version of Hadoop are you using? And what version of protobuf is on your classpath when executing this job? In fact, if you can post the entirety of the classpath you use for running the BnP job, that would be useful.

Thanks!

--
 
Felix GV
Data Infrastructure Engineer
Distributed Data Systems
LinkedIn
 
f...@linkedin.com
linkedin.com/in/felixgv

---

From: project-...@googlegroups.com [project-...@googlegroups.com] on behalf of Thiago Henrique dos Santos Bento [thiago...@gmail.com]
Sent: Wednesday, November 26, 2014 7:43 AM
To: project-...@googlegroups.com
Subject: [project-voldemort] Re: Import data from hdfs

--