Unable to execute gobblin-mapreduce.sh command

[Kunal Gupta]

I executed following command on terminal with new job configuration of kafka similar as described on github site

./bin/gobblin-mapreduce.sh --conf /Users/kunalgupta/Desktop/Backup/Kunal/Projects/DataScience/gobblin/gobblin-example/src/main/resources/kafkaextract.pull

  kafkaextract.pull

    

job.name=GobblinKafkaQuickStart

job.group=GobblinKafka

job.description=Gobblin quick start job for Kafka

job.lock.enabled=false

kafka.brokers=localhost:9092

source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource

extract.namespace=gobblin.extract.kafka

writer.builder.class=gobblin.writer.SimpleDataWriterBuilder

writer.file.path.type=tablename

writer.destination.type=HDFS

writer.output.format=txt

data.publisher.type=gobblin.publisher.BaseDataPublisher

mr.job.max.mappers=1

metrics.reporting.file.enabled=true

metrics.log.dir=/gobblin-kafka/metrics

metrics.reporting.file.suffix=txt

bootstrap.with.offset=earliest

fs.uri=hdfs://localhost:9001

writer.fs.uri=hdfs://localhost:9001

state.store.fs.uri=hdfs://localhost:9001

topic.whitelist=test

mr.job.root.dir=/gobblin-kafka/working

state.store.dir=/gobblin-kafka/state-store

task.data.root.dir=/jobs/kafkaetl/gobblin/gobblin-kafka/task-data

data.publisher.final.dir=/gobblintest/job-output

I got the following error:-

Error: java.io.IOException: Not all tasks running in container attempt_1447911777385_0012_m_000000_0 completed successfully

at gobblin.runtime.AbstractJobLauncher.runWorkUnits(AbstractJobLauncher.java:585)

at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.run(MRJobLauncher.java:584)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Error: java.io.IOException: Not all tasks running in container attempt_1447911777385_0012_m_000000_1 completed successfully

at gobblin.runtime.AbstractJobLauncher.runWorkUnits(AbstractJobLauncher.java:585)

at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.run(MRJobLauncher.java:584)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Error: java.io.IOException: Not all tasks running in container attempt_1447911777385_0012_m_000000_2 completed successfully

at gobblin.runtime.AbstractJobLauncher.runWorkUnits(AbstractJobLauncher.java:585)

at gobblin.runtime.mapreduce.MRJobLauncher$TaskRunner.run(MRJobLauncher.java:584)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Please help me to resolve it.

[Issac Buenrostro]

Hi Kunal,

These messages only mean that the tasks in the MR jobs fail. You would need to look at the logs of the mappers to figure out what went wrong. If you need help interpreting those logs, let us know.

Issac

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/7e312a3e-c79a-41d1-b391-7593f3db9470%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.