Re: gobblin-hive registration(Wrong FS: hdfs://localhost:9083, expected: file:///)

[Issac Buenrostro]

It seems that you don't have the correct Hadoop configuration in your system. There should be a file hdfs-site.xml under /etc/hadoop or the environment variable HADOOP_CONF_DIR should be set.

‪On Mon, Aug 29, 2016 at 6:52 AM ‫דניאל מזרחי‬‎ <daniel....@gmail.com> wrote:‬

hi, 

i am new to Gobblin, and running into some difficulties with the kafka-hdfs with hive registration.

i want to take events from Kafka and save them into HDFS according to partition from 

one of the fields (this works fine), and then register the table and the data to HIVE.

i am trying to use     "data.publisher.type=gobblin.publisher.BaseDataPublisherWithHiveRegistration" according to this explanation (here : http://gobblin.readthedocs.io/en/latest/user-guide/Hive-Registration/) .

my relevant configurations are 

    hive.registration.fs.uri=hdfs://localhost:9083

    hive.registration.policy=gobblin.hive.policy.HiveRegistrationPolicyBase

    hive.database.name=default

    hive.table.name=test

    avro.schema.literal={"namespace": "test","name": "event","type": "record","fields":[{"name":"t", "type":"int"},{"name":"timestamp","type":"string"},{"name":"type","type":"string"},{"name":"level","type":"int"},{"name":"message","type":"string"}]

but when i am trying to run it i am getting this error

2016-08-29 16:25:50 IDT INFO  [Commit-thread-0] gobblin.publisher.HiveRegistrationPublisher  94 - Number of paths to be registered in Hive: 1

2016-08-29 16:25:50 IDT INFO  [Commit-thread-0] gobblin.publisher.HiveRegistrationPublisher  107 - **************************** path  To  Publish  : 1

2016-08-29 16:25:50 IDT INFO  [Commit-thread-0] gobblin.publisher.HiveRegistrationPublisher  116 - Failed to generate HiveSpec

java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9083, expected: file:///

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)

at gobblin.publisher.HiveRegistrationPublisher.publishData(HiveRegistrationPublisher.java:111)

at gobblin.publisher.DataPublisher.publish(DataPublisher.java:61)

at gobblin.publisher.BaseDataPublisherWithHiveRegistration.publish(BaseDataPublisherWithHiveRegistration.java:47)

at gobblin.runtime.SafeDatasetCommit.commitDataset(SafeDatasetCommit.java:140)

at gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:91)

at gobblin.runtime.SafeDatasetCommit.call(SafeDatasetCommit.java:43)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://localhost:9083, expected: file:///

at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:647)

at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:82)

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:601)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)

at gobblin.hive.avro.HiveAvroSerDeManager.addSchemaProperties(HiveAvroSerDeManager.java:110)

at gobblin.hive.avro.HiveAvroSerDeManager.addSerDeProperties(HiveAvroSerDeManager.java:87)

at gobblin.hive.HiveRegistrationUnit.setSerDeProps(HiveRegistrationUnit.java:336)

at gobblin.hive.policy.HiveRegistrationPolicyBase.getTable(HiveRegistrationPolicyBase.java:249)

at gobblin.hive.policy.HiveRegistrationPolicyBase.getTables(HiveRegistrationPolicyBase.java:229)

at gobblin.hive.policy.HiveRegistrationPolicyBase.getHiveSpecs(HiveRegistrationPolicyBase.java:291)

at gobblin.publisher.HiveRegistrationPublisher$1.call(HiveRegistrationPublisher.java:100)

at gobblin.publisher.HiveRegistrationPublisher$1.call(HiveRegistrationPublisher.java:96)

... 6 more

can someone please tell me what am i missing ? 

thanks ! 

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/e2f5a094-a61f-4071-a7ad-dfd26d7674a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.