Missing required property writer.staging.dir

[Greg Zhang]

Hi

I got the following error message when running gobblin in mr mode, i checked that  gobblin-mapreduce.properties has a configuration  writer.staging.dir=${env:GOBBLIN_WORK_DIR}/task-staging
what is missing?

LF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Exception in thread "Thread-57" java.lang.IllegalArgumentException: Missing required property writer.staging.dir
    at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)
    at gobblin.util.WriterUtils.getWriterStagingDir(WriterUtils.java:63)
    at gobblin.util.JobLauncherUtils.cleanTaskStagingData(JobLauncherUtils.java:169)
    at gobblin.runtime.mapreduce.GobblinOutputCommitter.abortJob(GobblinOutputCommitter.java:99)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)


Thanks
GZ

[Issac Buenrostro]

What command are you using for launching Gobblin?

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/27cbe804-9d41-4a37-a27a-04b45d69996f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Greg Zhang]

Hi,

The command is as following and job_mr.pull is on hdfs. 

bin/gobblin-mapreduce.sh --conf /Users/gzhang/job_conf/job-mr.pull


Thanks
GZ

[Greg Zhang]

Hi,

I try out placing the gobblin-mapreduce.properties at different directory on HDFS in addition to the local file system gobblin-dist/conf/, that is on hdfs, put it under
 ${GOBBLIN_WORK_DIR}/conf, uploading the local conf to hdfs, still not luck getting it work, same error, i assume that  gobblin-mapreduce.properties can be either on local filesystem or hdfs.

Any input on this  is appreciated.

[Issac Buenrostro]

Hi Greg,

I don't believe the configuration files can be in HDFS (i.e. both gobblin-mapreduce.properties and any *.pull files need to be in local file system). Can you try that?

Issac

--

You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/e6402e79-201a-49e3-92f1-f975700f7253%40googlegroups.com.

[Greg Zhang]

Hi Iassac,

Thanks for your quick response.

The conf/gobblin-mapreduce.properties is pretty much what is from the goblin build tar ball, and it is under  ~/gobblin-dist/conf/,  i change it bit from fs.uri=hdfs://localhost:8020 to fs.uri=hdfs://localhost:9000, and i checked that writer.staging.dir=${env:GOBBLIN_WORK_DIR}/task-staging is in this property file.  the job file is on local file system now ~/job_config/job-mr.pull now. i actually also add writer.staging.dir=/Users/gzhang/staging/ in the job-mr.pull to override 

that one set from gobblin-mapreduce.properties file. 

with both property file and job pull file on local file system.  i run with the command line

gzhang-2243:gobblin-dist gzhang$ bin/gobblin-mapreduce.sh  --conf /Users/gzhang/job_conf/job-mr.pull

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/Users/gzhang/gobblin-dist/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/usr/local/Cellar/hadoop/2.7.2/libexec/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

Exception in thread "Thread-57" java.lang.IllegalArgumentException: Missing required property writer.staging.dir

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93)

at gobblin.util.WriterUtils.getWriterStagingDir(WriterUtils.java:63)

at gobblin.util.JobLauncherUtils.cleanTaskStagingData(JobLauncherUtils.java:169)

at gobblin.runtime.mapreduce.GobblinOutputCommitter.abortJob(GobblinOutputCommitter.java:99)

at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:549)

btw, i build the goblin against hadoop2.7.2.    i can have it run on standlone mode with separate job configuration without issue. 

one question what is the search path of goblin MR to locate gobblin-mapreduce.properties, since it is  not specified in the command line.

-GZ

[Sahil Takiar]

The "bin/gobblin-mapreduce.sh" script assumes that the "gobblin-mapreduce.properties" file is under "confgobblin-mapreduce.properties".

It the exception above the only one you see in the logs? This exception is only getting printed out because the Local MR job failed for some other reason, and the GobblinOutputCommitter is trying to run some cleanup code. So, the root cause of the failure is somewhere else.

--Sahil