YARN Gobblin runtime error.

188 views

Skip to first unread message

Vicky Kak

unread,

Sep 20, 2016, 2:37:14 AM9/20/16

to gobblin-users

I have installed Gobblin with YARN, and geting the following error.

********************************************************************************************************

2016-09-19 20:53:16 IST ERROR [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1  - Failed to start container container_1474298323450_0001_01_000003
java.io.IOException: Filesystem closed
	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
	at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
	at gobblin.yarn.YarnService.addContainerLocalResources(YarnService.java:359)
	at gobblin.yarn.YarnService.newContainerLaunchContext(YarnService.java:337)
	at gobblin.yarn.YarnService.access$800(YarnService.java:99)
	at gobblin.yarn.YarnService$AMRMClientCallbackHandler$1.run(YarnService.java:563)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

********************************************************************************************************

I see the similar issues being reported from the MR
https://github.com/linkedin/gobblin/issues/1219

Another thread out there is with similar issue
https://groups.google.com/forum/#!topic/gobblin-users/IbEGs2zOgwY

I am not sure what is going wrong. I have tried multiple configurations changes in the system but it giving the same error.
I have followed the documentation here http://gobblin.readthedocs.io/en/latest/user-guide/Gobblin-on-Yarn/#deployment-on-a-unsecured-yarn-cluster.

Here is the configurations at my end
application.conf
*********************************************************************************************************
# Sample configuration properties for the Gobblin Yarn application launcher

# Yarn/Helix configuration properties
gobblin.yarn.app.name=GobblinYarn
gobblin.yarn.app.master.memory.mbs=256
gobblin.yarn.initial.containers=2
gobblin.yarn.container.memory.mbs=512
#gobblin.yarn.conf.dir=<directory where Gobblin on Yarn related configuration files are located>
gobblin.yarn.conf.dir=/home/vickey/development/gobblin/gobblin-dist/conf/yarn
#gobblin.yarn.lib.jars.dir=<directory where Gobblin on Yarn lib jars are located>
gobblin.yarn.lib.jars.dir=/home/vickey/development/gobblin/gobblin-dist/lib
gobblin.yarn.app.master.files.local=${gobblin.yarn.conf.dir}"/log4j-yarn.properties,"${gobblin.yarn.conf.dir}"/application.conf,"${gobblin.yarn.conf.dir}"/reference.conf"
gobblin.yarn.container.files.local=${gobblin.yarn.app.master.files.local}
gobblin.yarn.logs.sink.root.dir=/home/vickey/development/gobblin/gobblin-dist/applogs
# Cluster configuration properties
gobblin.cluster.helix.cluster.name=GobblinYarn
#gobblin.cluster.job.conf.path=<path where Gobblin job configuration file are located>
gobblin.cluster.job.conf.path=/home/vickey/development/gobblin/gobblin-dist/job-config

# File system URIs
writer.fs.uri=${fs.uri}
state.store.fs.uri=${fs.uri}

# Writer related configuration properties
writer.destination.type=HDFS
writer.output.format=AVRO
writer.staging.dir=${gobblin.yarn.work.dir}/task-staging
writer.output.dir=${gobblin.yarn.work.dir}/task-output

# Data publisher related configuration properties
data.publisher.type=gobblin.publisher.BaseDataPublisher
data.publisher.final.dir=${gobblin.yarn.work.dir}/job-output
data.publisher.replace.final.dir=false

# Directory where job/task state files are stored
state.store.dir=${gobblin.yarn.work.dir}/state-store

# Directory where error files from the quality checkers are stored
qualitychecker.row.err.file=${gobblin.yarn.work.dir}/err

# Disable job locking for now
job.lock.enabled=false

# Directory where job locks are stored
job.lock.dir=${gobblin.yarn.work.dir}/locks

# Directory where metrics log files are stored
metrics.log.dir=${gobblin.yarn.work.dir}/metrics

# Interval of task state reporting in milliseconds
task.status.reportintervalinms=1000

# Enable metrics / events
metrics.enabled=true
#admin.server.enabled=true
#job.execinfo.server.enabled=true
*********************************************************************************************************
reference.conf
*********************************************************************************************************
# Sample configuration properties with default values

# Yarn/Helix configuration properties
gobblin.yarn.app.queue=default
gobblin.yarn.app.name=GobblinYarn
gobblin.yarn.app.master.memory.mbs=512
gobblin.yarn.app.master.cores=1
gobblin.yarn.app.report.interval.minutes=5
gobblin.yarn.max.get.app.report.failures=4
gobblin.yarn.email.notification.on.shutdown=false
gobblin.yarn.initial.containers=1
gobblin.yarn.container.memory.mbs=512
gobblin.yarn.container.cores=1
gobblin.yarn.container.affinity.enabled=true
gobblin.yarn.helix.instance.max.retries=2
gobblin.yarn.keytab.login.interval.minutes=1440
gobblin.yarn.token.renew.interval.minutes=720
gobblin.yarn.work.dir=/gobblin

# Cluster configuration properties
gobblin.cluster.helix.cluster.name=GobblinYarn
gobblin.cluster.zk.connection.string="localhost:2181"

fs.uri="hdfs://localhost:9000"

job.execinfo.server.enabled=false
*********************************************************************************************************

log4j-yarn.properties, I modified it to provide more details in nohup.out but it is not giving it

*********************************************************************************************************
#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

# log4j configuration used during build and unit tests

log4j.rootLogger=info,stdout
log4j.threshhold=ALL
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss z} %-5p [%t] %C %X{tableName} - %m%n

# Suppressed loggers
#log4j.logger.org.apache.helix.controller.GenericHelixController=INFO
#log4j.logger.org.apache.helix.controller.stages=INFO
#log4j.logger.org.apache.helix.controller.strategy.AutoRebalanceStrategy=INFO
#log4j.logger.org.apache.helix.manager.zk=INFO
#log4j.logger.org.apache.helix.monitoring.mbeans.ClusterStatusMonitor=INFO
#log4j.logger.org.apache.helix.store.zk.AutoFallbackPropertyStore=INFO
*********************************************************************************************************

Here is the nohup.out
*********************************************************************************************************
WARN [NativeCodeLoader] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'admin'
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:147)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159)
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164)
    at com.typesafe.config.impl.SimpleConfig.getBoolean(SimpleConfig.java:174)
    at gobblin.yarn.GobblinYarnAppLauncher.launch(GobblinYarnAppLauncher.java:285)
    at gobblin.yarn.GobblinYarnAppLauncher.main(GobblinYarnAppLauncher.java:794)
*********************************************************************************************************

Here is the conainer logs with the error
*********************************************************************************************************
2016-09-20 11:54:55 IST INFO [main] gobblin.cluster.GobblinClusterManager - Using ZooKeeper connection string: localhost:2181
2016-09-20 11:54:55 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Create a zk-based cluster manager. zkSvr: localhost:2181, clusterName: GobblinYarn, instanceName: GobblinClusterManager, type: CONTROLLER
2016-09-20 11:54:55 IST INFO [main] org.apache.helix.HelixManagerProperties - load helix-manager properties: {minimum_supported_version.batch_message=0.6.1, clustermanager.version=0.6.6-SNAPSHOT, minimum_supported_version.participant=0.4}
2016-09-20 11:54:55 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: TASK_REPLY, poolSize: 40, factory: org.apache.helix.messaging.handling.AsyncCallbackService@14cd1699, pool: java.util.concurrent.ThreadPoolExecutor@77888435[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:54:56 IST WARN [main] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-09-20 11:54:57 IST INFO [main] org.quartz.impl.StdSchedulerFactory - Using default implementation for ThreadExecutor
2016-09-20 11:54:57 IST INFO [main] org.quartz.simpl.SimpleThreadPool - Job execution threads will use class loader of thread: main
2016-09-20 11:54:58 IST INFO [main] org.quartz.core.SchedulerSignalerImpl - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2016-09-20 11:54:58 IST INFO [main] org.quartz.core.QuartzScheduler - Quartz Scheduler v.2.2.3 created.
2016-09-20 11:54:58 IST INFO [main] org.quartz.simpl.RAMJobStore - RAMJobStore initialized.
2016-09-20 11:54:58 IST INFO [main] org.quartz.core.QuartzScheduler - Scheduler meta-data: Quartz Scheduler (v2.2.3) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'
Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
NOT STARTED.
Currently in standby mode.
Number of jobs executed: 0
Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2016-09-20 11:54:58 IST INFO [main] org.quartz.impl.StdSchedulerFactory - Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2016-09-20 11:54:58 IST INFO [main] org.quartz.impl.StdSchedulerFactory - Quartz scheduler version: 2.2.3
2016-09-20 11:54:58 IST INFO [main] org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl - Upper bound of the thread pool size is 500
2016-09-20 11:54:58 IST INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy - yarn.client.max-nodemanagers-proxies : 500
2016-09-20 11:54:58 IST INFO [main] gobblin.cluster.GobblinClusterManager - Starting the Gobblin Cluster Manager
2016-09-20 11:54:59 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - ClusterManager.connect()
2016-09-20 11:54:59 IST INFO [main] org.apache.helix.controller.GenericHelixController - createDefaultRegistry
2016-09-20 11:54:59 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController$ClusterEventProcessor - START ClusterEventProcessor thread
2016-09-20 11:54:59 IST INFO [ZkClient-EventThread-34-localhost:2181] org.I0Itec.zkclient.ZkEventThread - Starting ZkClient event thread.
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:host.name=vickey.kak
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.version=1.8.0_65
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.vendor=Oracle Corporation
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.home=/home/vickey/Installable/jdk/jdk1.8.0_65/jre
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.class.path=/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/libthrift-0.9.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-core-2.6.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/geronimo-annotation_1.0_spec-1.1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-exec-1.0.1-core.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/pegasus-common-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-common-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/helix-core-0.6.6-SNAPSHOT.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jersey-json-1.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-hdfs-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-jdbc-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/curator-framework-2.10.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/metrics-core-2.2.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jdo2-api-2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-cli-1.3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/maven-scm-provider-svnexe-1.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-yarn-api-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/datanucleus-core-3.2.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-server-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/curator-recipes-2.10.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/javax.servlet-api-3.1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/calcite-linq4j-1.2.0-incubating.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/janino-2.7.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/c3p0-0.9.1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/mockito-core-1.10.19.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/xmlenc-0.52.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/velocity-1.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-0.20-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-configuration-1.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-email-1.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/servlet-api-2.5-20081211.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-test-harness-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jets3t-0.9.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-azkaban-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-annotations-2.6.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/kafka-schema-registry-client-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-math3-3.5.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-cluster-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-0.23-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/bcpg-jdk15on-1.52.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/influxdb-java-2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-metastore-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-rest-server-0.8.0-20-gd59a25e.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/ant-launcher-1.9.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-config-core-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-all-7.6.0.v20120127.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/kafka-clients-0.8.2.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/maven-scm-api-1.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-example-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jpam-1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-kafka-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-lang-2.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/lombok-1.16.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-logging-1.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/javassist-3.18.2-GA.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/snappy-0.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/geronimo-jta_1.1_spec-1.1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/metrics-jvm-3.1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-compress-1.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/scala-library-2.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/calcite-core-1.2.0-incubating.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/stax-api-1.0-2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-autoscaling-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/asm-commons-3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-httpclient-3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/slf4j-api-1.7.21.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/parseq-1.3.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-applicationautoscaling-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-mapreduce-client-common-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-aws-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-codec-1.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jta-1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/bsh-2.0b4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gson-2.6.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/scala-xml_2.11-1.0.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-yarn-common-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-mapper-asl-1.9.13.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/httpcore-4.4.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/kafka_2.11-0.8.2.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/datanucleus-api-jdo-3.2.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/guava-retrying-2.0.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/servlet-api-2.5.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/guava-15.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/plexus-utils-1.5.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/httpclient-4.5.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jline-0.9.94.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-servlet-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/mysql-connector-java-5.1.38.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jasper-compiler-5.5.23.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-hive-registration-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hamcrest-core-1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jsch-0.1.53.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/generator-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-core-asl-1.9.13.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/reflections-0.9.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-dbcp-1.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/azkaban-2.5.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-sts-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/config-1.2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/flyway-core-3.2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/snappy-java-1.1.1.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/guice-4.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-netty-standalone-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/regexp-1.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-s3-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-ec2-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-rest-api-rest-client-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/kafka-json-serializer-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-core-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/scala-parser-combinators_2.11-1.0.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-yarn-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/netty-3.2.3.Final.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/avro-ipc-1.7.7-tests.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/json-20070829.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-rest-api-data-template-0.8.0-20-gd59a25e.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/avro-ipc-1.7.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jettison-1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/geronimo-jpa_3.0_spec-1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/calcite-avatica-1.2.0-incubating.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jcommander-1.48.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/antlr-runtime-3.5.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-annotations-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/metrics-core-3.1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/groovy-all-2.1.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-lang3-3.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-jaxrs-1.8.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/mail-1.4.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/netty-3.7.0.Final.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/zkclient-0.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jaxb-impl-2.2.3-1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-0.20S-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jersey-server-1.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-common-secure-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/apache-log4j-extras-1.2.17.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jersey-guice-1.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-core-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/kafka-avro-serializer-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-compiler-2.7.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/maven-scm-provider-svn-commons-1.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/data-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/guice-servlet-3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/javax.mail-1.5.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/zookeeper-3.4.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-yarn-server-common-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/opencsv-2.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/paranamer-2.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/log4j-1.2.17.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-util-6.1.26.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/li-jersey-uri-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/objenesis-2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-admin-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/common-utils-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-shims-common-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-mapreduce-client-core-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-server-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-databind-2.6.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/derby-10.12.1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/stax-api-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/protobuf-java-2.5.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-kms-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jasper-runtime-5.5.23.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-xc-1.8.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/data-transform-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/joda-time-2.9.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-data-management-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/java-xmlbuilder-0.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/lz4-1.2.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/javax.inject-1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/degrader-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/junit-3.8.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-compaction-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-client-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jersey-core-1.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aws-java-sdk-iam-1.11.8.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/geronimo-jaspic_1.0_spec-1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/annotations-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-metastore-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-pool2-2.4.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/asm-3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/common-config-2.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-metrics-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/codemodel-2.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jansi-1.11.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/cglib-nodep-2.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/bonecp-0.8.0.RELEASE.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/bcprov-jdk15on-1.52.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/curator-client-2.10.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-service-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-net-3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/quartz-2.2.3.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/avro-1.7.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-vfs2-2.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-http-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/ST4-4.0.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-runtime-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-config-client-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-collections-3.2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-util-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-io-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/metrics-graphite-3.1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/activation-1.1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-common-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/testng-6.9.10.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-rest-api-0.8.0-20-gd59a25e.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jsr305-3.0.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/retrofit-1.9.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/d2-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-serde-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jsp-api-2.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-docgen-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-security-9.2.14.v20151106.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jdo-api-3.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/avro-mapred-1.7.7-hadoop2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-rest-client-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/okio-1.4.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-utility-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-daemon-1.0.13.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/okhttp-2.4.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-tools-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/restli-common-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/libfb303-0.9.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/eigenbase-properties-1.1.5.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/slf4j-log4j12-1.7.21.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/transaction-api-1.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/aopalliance-1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jopt-simple-3.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jackson-dataformat-cbor-2.6.6.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hive-ant-1.0.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jaxb-api-2.2.2.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-pool-1.5.4.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-auth-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-io-2.5.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/asm-tree-3.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/commons-el-1.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-api-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/ant-1.9.1.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/hadoop-yarn-client-2.3.0.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/mina-core-1.1.7.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jetty-6.1.26.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/datanucleus-rdbms-3.2.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/gobblin-tunnel-0.6.2-862-g342d20b.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/r2-1.15.9.jar:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/jasypt-1.9.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/etc/hadoop:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/hadoop-common-2.3.0-tests.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/hadoop-common-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/hadoop-nfs-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jersey-json-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/activation-1.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/xmlenc-0.52.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jets3t-0.9.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-io-2.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-lang-2.6.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-logging-1.1.3.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/stax-api-1.0-2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/httpclient-4.2.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-httpclient-3.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-configuration-1.6.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-math3-3.1.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/zookeeper-3.4.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/servlet-api-2.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jsr305-1.3.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-compress-1.4.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jettison-1.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/httpcore-4.2.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/hadoop-annotations-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-cli-1.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jersey-server-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/guava-11.0.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/mockito-all-1.8.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jsch-0.1.42.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/junit-4.8.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/avro-1.7.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/paranamer-2.3.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/log4j-1.2.17.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jetty-util-6.1.26.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/xz-1.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-codec-1.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jersey-core-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/asm-3.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-net-3.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-collections-3.2.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/netty-3.6.2.Final.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jsp-api-2.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-digester-1.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/hadoop-auth-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/commons-el-1.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/jetty-6.1.26.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-nfs-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/hadoop-hdfs-2.3.0-tests.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-io-2.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/guava-11.0.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/asm-3.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/commons-el-1.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-api-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-server-tests-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-common-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-server-common-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-client-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jersey-json-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/activation-1.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-io-2.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-lang-2.6.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-httpclient-3.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/zookeeper-3.4.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/servlet-api-2.5.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jline-0.9.94.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jsr305-1.3.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jettison-1.1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-cli-1.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jersey-server-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/guava-11.0.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/log4j-1.2.17.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/xz-1.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/javax.inject-1.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/commons-codec-1.4.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jackson-jaxrs-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jersey-client-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jersey-core-1.9.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/asm-3.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jackson-mapper-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jackson-xc-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/aopalliance-1.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jackson-core-asl-1.8.8.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/guice-3.0.jar:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/yarn/lib/jetty-6.1.26.jar
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.library.path=:/home/vickey/development/gobblin/hadoop-2.3.0/lib/native:/home/vickey/development/gobblin/hadoop-2.3.0/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.io.tmpdir=/tmp
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:java.compiler=<NA>
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:os.name=Linux
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:os.arch=amd64
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:os.version=3.13.0-88-generic
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:user.name=vickey
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:user.home=/home/vickey
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.Environment - Client environment:user.dir=/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001
2016-09-20 11:54:59 IST INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.helix.manager.zk.ZkClient@5ee2b6f9
2016-09-20 11:54:59 IST INFO [main] org.I0Itec.zkclient.ZkClient - Waiting for keeper state SyncConnected
2016-09-20 11:54:59 IST INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn$SendThread - Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
2016-09-20 11:54:59 IST INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn$SendThread - Socket connection established to localhost/127.0.0.1:2181, initiating session
2016-09-20 11:54:59 IST INFO [main-SendThread(localhost:2181)] org.apache.zookeeper.ClientCnxn$SendThread - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1574270e4930050, negotiated timeout = 30000
2016-09-20 11:54:59 IST INFO [main-EventThread] org.I0Itec.zkclient.ZkClient - zookeeper state changed (SyncConnected)
2016-09-20 11:54:59 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/MasterSlave
2016-09-20 11:54:59 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/LeaderStandby
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/STORAGE_DEFAULT_SM_SCHEMATA
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/OnlineOffline
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/SchedulerTaskQueue
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.ZkBaseDataAccessor - Node already exists. path: /GobblinYarn/STATEMODELDEFS/Task
2016-09-20 11:55:00 IST INFO [main] org.I0Itec.zkclient.ZkClient - Waiting for keeper state SyncConnected
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - KeeperState: SyncConnected, zookeeper:State:CONNECTED Timeout:30000 sessionid:0x1574270e4930050 local:/127.0.0.1:40458 remoteserver:localhost/127.0.0.1:2181 lastZxid:2094 xid:7 sent:7 recv:7 queuedpkts:0 pendingresp:0 queuedevents:0
2016-09-20 11:55:00 IST INFO [main] org.I0Itec.zkclient.ZkClient - Waiting for keeper state SyncConnected
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Handling new session, session id: 1574270e4930050, instance: GobblinClusterManager, instanceTye: CONTROLLER, cluster: GobblinYarn, zkconnection: State:CONNECTED Timeout:30000 sessionid:0x1574270e4930050 local:/127.0.0.1:40458 remoteserver:localhost/127.0.0.1:2181 lastZxid:2094 xid:7 sent:7 recv:7 queuedpkts:0 pendingresp:0 queuedevents:0
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.manager.zk.DistributedLeaderElection
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER, listener: org.apache.helix.manager.zk.DistributedLeaderElection@a776e
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.DistributedLeaderElection - GobblinClusterManager is trying to acquire leadership for cluster: GobblinYarn
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.DistributedLeaderElection - GobblinClusterManager acquired leadership for cluster: GobblinYarn
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER/MESSAGES, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Init HelixTaskExecutor
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.monitoring.mbeans.MessageQueueMonitor - Register MBean: ClusterStatus: cluster=GobblinYarn,messageQueue=GobblinClusterManager
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Skip init a new thread pool for type: TASK_REPLY, already existing pool: java.util.concurrent.ThreadPoolExecutor@77888435[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], isShutdown: false
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - No Messages to process
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 40ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349 for type: MESSAGES_CONTROLLER to path: /GobblinYarn/CONTROLLER/MESSAGES
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: CONTROLLER_MSG, poolSize: 40, factory: org.apache.helix.manager.zk.DefaultControllerMessageHandlerFactory@5b43fbf6, pool: java.util.concurrent.ThreadPoolExecutor@1080b026[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: SCHEDULER_MSG, poolSize: 40, factory: org.apache.helix.manager.zk.DefaultSchedulerMessageHandlerFactory@dcfda20, pool: java.util.concurrent.ThreadPoolExecutor@6d304f9d[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: PARTICIPANT_ERROR_REPORT, poolSize: 40, factory: org.apache.helix.manager.zk.DefaultParticipantErrorMessageHandlerFactory@51cd7ffc, pool: java.util.concurrent.ThreadPoolExecutor@30d4b288[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/CONFIGS/PARTICIPANT listener:org.apache.helix.controller.GenericHelixController
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONFIGS/PARTICIPANT, listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - START: GenericClusterController.onConfigChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - END: GenericClusterController.onConfigChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/CONFIGS/PARTICIPANT listener:org.apache.helix.controller.GenericHelixController Took: 46ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a for type: CONFIGS to path: /GobblinYarn/CONFIGS/PARTICIPANT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/LIVEINSTANCES listener:org.apache.helix.controller.GenericHelixController
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/LIVEINSTANCES, listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.monitoring.mbeans.ClusterStatusMonitor - Register MBean: ClusterStatus: cluster=GobblinYarn
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - START: Invoking controller pipeline for event: configChange
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - START ReadClusterDataStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - START: ClusterDataCache.refresh()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - START: Generic GenericClusterController.onLiveInstanceChange()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - END: ClusterDataCache.refresh(), took 108 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - END ReadClusterDataStage.process(). took: 117 ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - END: Generic GenericClusterController.onLiveInstanceChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/LIVEINSTANCES listener:org.apache.helix.controller.GenericHelixController Took: 147ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a for type: LIVEINSTANCES to path: /GobblinYarn/LIVEINSTANCES
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/IDEALSTATES listener:org.apache.helix.controller.GenericHelixController
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/IDEALSTATES, listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - START BestPossibleStateCalcStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - END BestPossibleStateCalcStage.process(). took: 3 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - START TaskAssignmentStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - END TaskAssignmentStage.process(). took: 1 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - END: Invoking controller pipeline for event: configChange, took 162 ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - START: Generic GenericClusterController.onIdealStateChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - END: GenericClusterController.onIdealStateChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/IDEALSTATES listener:org.apache.helix.controller.GenericHelixController Took: 45ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a for type: IDEALSTATES to path: /GobblinYarn/IDEALSTATES
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.controller.GenericHelixController
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER, listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - START: Invoking controller pipeline for event: liveInstanceChange
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - START ReadClusterDataStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - START: ClusterDataCache.refresh()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - START: GenericClusterController.onControllerChange()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - END: ClusterDataCache.refresh(), took 64 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - END ReadClusterDataStage.process(). took: 66 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - START BestPossibleStateCalcStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - END BestPossibleStateCalcStage.process(). took: 1 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - START TaskAssignmentStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - END TaskAssignmentStage.process(). took: 1 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ExternalViewComputeStage - START ExternalViewComputeStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ExternalViewComputeStage - END ExternalViewComputeStage.process(). took: 12 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - END: Invoking controller pipeline for event: liveInstanceChange, took 84 ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.controller.GenericHelixController - END: GenericClusterController.onControllerChange()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.controller.GenericHelixController Took: 96ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a for type: CONTROLLER to path: /GobblinYarn/CONTROLLER
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager$StatusDumpTask - Start StatusDumpTask
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - START: Invoking controller pipeline for event: idealStateChange
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.monitoring.ZKPathDataDumpTask - Init ZKPathDataDumpTask for cluster: GobblinYarn, thresholdNoChangeMsForStatusUpdates: 900000, thresholdNoChangeMsForErrors: 86400000, maxLeafCount: 10000
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - START ReadClusterDataStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - START: ClusterDataCache.refresh()
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.manager.zk.DistributedLeaderElection Took: 575ms
2016-09-20 11:55:00 IST INFO [StatusDumpTimerTask] org.apache.helix.monitoring.ZKPathDataDumpTask - Scan statusUpdates and errors for cluster: GobblinYarn, by controller: org.apache.helix.manager.zk.ZKHelixManager@7a3b82d
2016-09-20 11:55:00 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 START:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.manager.zk.DistributedLeaderElection
2016-09-20 11:55:00 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER, listener: org.apache.helix.manager.zk.DistributedLeaderElection@a776e
2016-09-20 11:55:00 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.DistributedLeaderElection - GobblinClusterManager is trying to acquire leadership for cluster: GobblinYarn
2016-09-20 11:55:00 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 END:INVOKE /GobblinYarn/CONTROLLER listener:org.apache.helix.manager.zk.DistributedLeaderElection Took: 84ms
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.CallbackHandler - Skip processing callbacks for listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349, path: /GobblinYarn/CONTROLLER/MESSAGES, expected types: [CALLBACK, FINALIZE] but was INIT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - init handler: /GobblinYarn/CONTROLLER/MESSAGES, org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.CallbackHandler - Skip processing callbacks for listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a, path: /GobblinYarn/CONFIGS/PARTICIPANT, expected types: [CALLBACK, FINALIZE] but was INIT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - init handler: /GobblinYarn/CONFIGS/PARTICIPANT, org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.CallbackHandler - Skip processing callbacks for listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a, path: /GobblinYarn/LIVEINSTANCES, expected types: [CALLBACK, FINALIZE] but was INIT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - init handler: /GobblinYarn/LIVEINSTANCES, org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.CallbackHandler - Skip processing callbacks for listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a, path: /GobblinYarn/IDEALSTATES, expected types: [CALLBACK, FINALIZE] but was INIT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - init handler: /GobblinYarn/IDEALSTATES, org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST WARN [main] org.apache.helix.manager.zk.CallbackHandler - Skip processing callbacks for listener: org.apache.helix.controller.GenericHelixController@4cc6fa2a, path: /GobblinYarn/CONTROLLER, expected types: [CALLBACK, FINALIZE] but was INIT
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - init handler: /GobblinYarn/CONTROLLER, org.apache.helix.controller.GenericHelixController@4cc6fa2a
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ClusterDataCache - END: ClusterDataCache.refresh(), took 124 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.ReadClusterDataStage - END ReadClusterDataStage.process(). took: 129 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - START BestPossibleStateCalcStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.BestPossibleStateCalcStage - END BestPossibleStateCalcStage.process(). took: 0 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - START TaskAssignmentStage.process()
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.stages.TaskAssignmentStage - END TaskAssignmentStage.process(). took: 0 ms
2016-09-20 11:55:00 IST INFO [Thread-7] org.apache.helix.controller.GenericHelixController - END: Invoking controller pipeline for event: idealStateChange, took 132 ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 START:INVOKE /GobblinYarn/LIVEINSTANCES listener:gobblin.cluster.GobblinClusterManager.GobblinLiveInstanceChangeListener
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/LIVEINSTANCES, listener: gobblin.cluster.GobblinClusterManager$GobblinLiveInstanceChangeListener@19fb8826
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.CallbackHandler - 1 END:INVOKE /GobblinYarn/LIVEINSTANCES listener:gobblin.cluster.GobblinClusterManager.GobblinLiveInstanceChangeListener Took: 37ms
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.manager.zk.ZKHelixManager - Added listener: gobblin.cluster.GobblinClusterManager$GobblinLiveInstanceChangeListener@19fb8826 for type: LIVEINSTANCES to path: /GobblinYarn/LIVEINSTANCES
2016-09-20 11:55:00 IST INFO [main] org.apache.helix.messaging.DefaultMessagingService - registering msg factory for type SHUTDOWN
2016-09-20 11:55:01 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: SHUTDOWN, poolSize: 40, factory: gobblin.cluster.GobblinClusterManager$ControllerShutdownMessageHandlerFactory@55b5f5d2, pool: java.util.concurrent.ThreadPoolExecutor@5bfa8cc5[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:55:01 IST INFO [main] org.apache.helix.messaging.DefaultMessagingService - registering msg factory for type USER_DEFINE_MSG
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 START:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER/MESSAGES, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349
2016-09-20 11:55:01 IST INFO [main] org.apache.helix.messaging.handling.HelixTaskExecutor - Registered message handler factory for type: USER_DEFINE_MSG, poolSize: 40, factory: gobblin.yarn.GobblinApplicationMaster$ControllerUserDefinedMessageHandlerFactory@5316e95f, pool: java.util.concurrent.ThreadPoolExecutor@3f053c80[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.messaging.handling.HelixTaskExecutor - Dropping NO-OP message. mid: 5f0a8b63-eb08-40ed-be1e-fce90cdc0ef0, from: GobblinClusterManager
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 END:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 125ms
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 START:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER/MESSAGES, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349
2016-09-20 11:55:01 IST INFO [main] gobblin.runtime.app.ServiceBasedAppLauncher - Starting the Gobblin application and all its associated Services
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.messaging.handling.HelixTaskExecutor - Dropping NO-OP message. mid: 9c62cc71-8811-40fc-9d3e-267cbef3cc96, from: GobblinClusterManager
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 END:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 58ms
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 START:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - GobblinClusterManager subscribes child-change. path: /GobblinYarn/CONTROLLER/MESSAGES, listener: org.apache.helix.messaging.handling.HelixTaskExecutor@6f8f9349
2016-09-20 11:55:01 IST INFO [JobConfigurationManager STARTING] gobblin.cluster.JobConfigurationManager - Loading job configurations from /tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.messaging.handling.HelixTaskExecutor - No Messages to process
2016-09-20 11:55:01 IST INFO [ZkClient-EventThread-34-localhost:2181] org.apache.helix.manager.zk.CallbackHandler - 34 END:INVOKE /GobblinYarn/CONTROLLER/MESSAGES listener:org.apache.helix.messaging.handling.HelixTaskExecutor Took: 33ms
2016-09-20 11:55:01 IST INFO [YarnService STARTING] gobblin.yarn.YarnService - Starting the YarnService
2016-09-20 11:55:01 IST INFO [GobblinHelixJobScheduler STARTING] gobblin.scheduler.JobScheduler - Starting the job scheduler
2016-09-20 11:55:01 IST INFO [GobblinHelixJobScheduler STARTING] org.quartz.core.QuartzScheduler - Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2016-09-20 11:55:01 IST INFO [YarnService STARTING] org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at /0.0.0.0:8030
2016-09-20 11:55:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stderr to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stderr
java.nio.channels.ClosedChannelException
    at org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1353)
    at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
    at sun.nio.cs.StreamEncoder.implClose(StreamEncoder.java:316)
    at sun.nio.cs.StreamEncoder.close(StreamEncoder.java:149)
    at java.io.OutputStreamWriter.close(OutputStreamWriter.java:233)
    at java.io.BufferedWriter.close(BufferedWriter.java:266)
    at com.google.common.io.Closer.close(Closer.java:214)
    at gobblin.util.logs.LogCopier$LogCopyTask.copyChangesOfLogFile(LogCopier.java:541)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:484)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:55:02 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stdout to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stdout
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:55:03 IST INFO [YarnService STARTING] gobblin.yarn.YarnService - ApplicationMaster registration response: maximumCapability { memory: 8192 virtual_cores: 32 } queue: "default"
2016-09-20 11:55:03 IST INFO [YarnService STARTING] gobblin.yarn.YarnService - Requesting initial containers
2016-09-20 11:55:06 IST INFO [AMRM Callback Handler Thread] gobblin.yarn.YarnService$AMRMClientCallbackHandler - Container container_1474352584304_0001_01_000002 has been allocated
2016-09-20 11:55:06 IST INFO [AMRM Callback Handler Thread] gobblin.yarn.YarnService$AMRMClientCallbackHandler - Container container_1474352584304_0001_01_000003 has been allocated
2016-09-20 11:55:06 IST INFO [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1 - Starting container container_1474352584304_0001_01_000003
2016-09-20 11:55:06 IST ERROR [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1 - Failed to start container container_1474352584304_0001_01_000003
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.yarn.YarnService.addContainerLocalResources(YarnService.java:359)
    at gobblin.yarn.YarnService.newContainerLaunchContext(YarnService.java:337)
    at gobblin.yarn.YarnService.access$800(YarnService.java:99)
    at gobblin.yarn.YarnService$AMRMClientCallbackHandler$1.run(YarnService.java:563)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:55:06 IST INFO [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1 - Starting container container_1474352584304_0001_01_000002
2016-09-20 11:55:06 IST ERROR [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1 - Failed to start container container_1474352584304_0001_01_000002
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.yarn.YarnService.addContainerLocalResources(YarnService.java:359)
    at gobblin.yarn.YarnService.newContainerLaunchContext(YarnService.java:337)
    at gobblin.yarn.YarnService.access$800(YarnService.java:99)
    at gobblin.yarn.YarnService$AMRMClientCallbackHandler$1.run(YarnService.java:563)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:56:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stderr to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stderr
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:56:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stdout to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stdout
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:57:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stderr to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stderr
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:57:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stdout to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stdout
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:58:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stderr to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stderr
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:58:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stdout to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stdout
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:59:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stderr to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stderr
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-09-20 11:59:01 IST ERROR [pool-7-thread-1] gobblin.util.logs.LogCopier$LogCopyTask - Failed while copying logs from file:/home/vickey/development/gobblin/hadoop-2.3.0/logs/userlogs/application_1474352584304_0001/container_1474352584304_0001_01_000001/GobblinApplicationMaster.stdout to hdfs://localhost:9000/user/vickey/GobblinYarn/application_1474352584304_0001/_applogs/container_1474352584304_0001_01_000001/container_1474352584304_0001_01_000001.GobblinApplicationMaster.stdout
java.io.IOException: Filesystem closed
    at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:689)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1744)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
    at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1399)
    at gobblin.util.logs.LogCopier$LogCopyTask.createNewLogFileIfNeeded(LogCopier.java:492)
    at gobblin.util.logs.LogCopier$LogCopyTask.runOneIteration(LogCopier.java:482)
    at gobblin.util.concurrent.ScheduledExecutorServiceTaskScheduler$RunnableTask.run(ScheduledExecutorServiceTaskScheduler.java:110)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

*********************************************************************************************************

Container std.err
*********************************************************************************************************
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/filecache/173/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/vickey/development/gobblin/hadoop-2.3.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/vickey/development/gobblin/hadoop-2.3.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
Sep 20, 2016 11:55:01 AM com.google.common.util.concurrent.ServiceManager$ServiceListener failed
SEVERE: Service JobConfigurationManager [FAILED] has failed in the STARTING state.
java.lang.RuntimeException: The specified job configurations directory was not found: /tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz
	at gobblin.util.SchedulerUtils.loadGenericJobConfigsRecursive(SchedulerUtils.java:181)
	at gobblin.util.SchedulerUtils.loadGenericJobConfigs(SchedulerUtils.java:83)
	at gobblin.cluster.JobConfigurationManager.startUp(JobConfigurationManager.java:80)
	at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
	at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
	at java.lang.Thread.run(Thread.java:745)

Exception in thread "main" Exception in thread "JobConfigurationManager STARTING" java.lang.IllegalStateException: Expected to be healthy after starting
	at com.google.common.base.Preconditions.checkState(Preconditions.java:150)
	at com.google.common.util.concurrent.ServiceManager.awaitHealthy(ServiceManager.java:278)
	at gobblin.runtime.app.ServiceBasedAppLauncher.start(ServiceBasedAppLauncher.java:147)
	at gobblin.cluster.GobblinClusterManager.start(GobblinClusterManager.java:151)
	at gobblin.yarn.GobblinApplicationMaster.main(GobblinApplicationMaster.java:199)
java.lang.RuntimeException: The specified job configurations directory was not found: /tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz
	at gobblin.util.SchedulerUtils.loadGenericJobConfigsRecursive(SchedulerUtils.java:181)
	at gobblin.util.SchedulerUtils.loadGenericJobConfigs(SchedulerUtils.java:83)
	at gobblin.cluster.JobConfigurationManager.startUp(JobConfigurationManager.java:80)
	at com.google.common.util.concurrent.AbstractIdleService$2$1.run(AbstractIdleService.java:54)
	at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
	at java.lang.Thread.run(Thread.java:745)
*********************************************************************************************************

I can see the files here 
*********************************************************************************************************
vickey@vickey:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz$ ll
total 12
drwx------ 3 vickey vickey 4096 Sep 20 11:54 ./
drwxr-xr-x 3 vickey vickey 4096 Sep 20 11:54 ../
drwx------ 2 vickey vickey 4096 Sep 20 11:54 job-config/
vickey@vickey:/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz$ pwd
/tmp/hadoop-vickey/nm-local-dir/usercache/vickey/appcache/application_1474352584304_0001/container_1474352584304_0001_01_000001/job-config.tar.gz
*********************************************************************************************************

What could be wrong here?

Thanks,
Vicky

Vicky Kak

unread,

Sep 21, 2016, 10:53:42 PM9/21/16

to gobblin-users

Looks like the message is too long and some contents in the earlier mail had got deleted, I am getting a error like this

2016-09-19 20:53:16 IST ERROR [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1  - Failed to start container container_1474298323450_0001_01_000003
java.io.IOException: Filesystem closed

Any thoughts about it?

Abhishek Tiwari

unread,

Sep 22, 2016, 2:15:53 AM9/22/16

to Vicky Kak, gobblin-users

Most of the configs look ok. Did you verify if the permissions for various directories are open enough and accessible to user running gobblin-yarn?

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/2518b860-3aea-47d4-8975-24ae621f81a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Vicky Kak

unread,

Sep 22, 2016, 10:28:43 PM9/22/16

to gobblin-users, vick...@gmail.com

I am going to take a deeper look at it in few days time. I am getting my first job working with the LocalJobLauncher implementation.

On Thursday, September 22, 2016 at 11:45:53 AM UTC+5:30, Abhishek Tiwari wrote:

Most of the configs look ok. Did you verify if the permissions for various directories are open enough and accessible to user running gobblin-yarn?

On Wed, Sep 21, 2016 at 7:53 PM, Vicky Kak <vick...@gmail.com> wrote:

Looks like the message is too long and some contents in the earlier mail had got deleted, I am getting a error like this

2016-09-19 20:53:16 IST ERROR [ContainerLaunchExecutor] gobblin.yarn.YarnService$AMRMClientCallbackHandler$1 - Failed to start container container_1474298323450_0001_01_000003 java.io.IOException: Filesystem closed

Any thoughts about it?

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.

Reply all

Reply to author

Forward

0 new messages