WebHCat and Camus

[Jonathan Hodges]

Hi,

We are attempting to kick off Camus jobs via the HCatalog REST API, WebHCat.  Has anyone ever tried this?

The following is our Camus command-line

hadoop jar /home/hadoop/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar com.linkedin.camus.etl.kafka.CamusJob -P /home/hadoop/camus_non_avro.properties

curl -s -d user.name=hadoop \

        -d jar=/tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar \

        -d class=com.linkedin.camus.etl.kafka.CamusJob \

        -d arg=-P \

        -d arg=/tmp/camus_non_avro.properties \

        'http://internal-daalt-hcatalog-1507773817.us-east-1.elb.amazonaws.com/templeton/v1/mapreduce/jar'

The issue is the properties file param.

Exception in thread "main" java.io.FileNotFoundException: /tmp/camus_non_avro.properties (No such file or directory)

        at java.io.FileInputStream.open(Native Method)

        at java.io.FileInputStream.<init>(FileInputStream.java:146)

        at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:592)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)

        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)

        at com.linkedin.camus.etl.kafka.CamusJob.main(CamusJob.java:562)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:187)

Is there a way to specify an HDFS directory for the properties file instead of local filesystem?

Thanks for the help!

Jonathan