Gobblin failed to launch one or more job out of multiple jobs.

[Kunal Gupta]

I tried to run standalone gobblin with 3 job configuration file. These 3 jobs tries to read data from kafka server and put into one consumer HDFS in different directory but sometime all 3 jobs did not run, in some cases one job get failed or sometime two etc. I have attached 3 job configuration file too.

I got the following error when one job get failed. I have also attached the gobblin log file.

2015-11-19 18:56:26 IST ERROR [JobScheduler-2] gobblin.scheduler.JobScheduler$NonScheduledJobRunner  533 - Failed to run job GobblinKafkaQuickStart

gobblin.runtime.JobException: Failed to run job GobblinKafkaQuickStart

at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:285)

at gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:531)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: gobblin.runtime.JobException: Failed to launch and run job GobblinKafkaQuickStart

at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:332)

at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:283)

... 4 more

Caused by: java.util.ConcurrentModificationException

at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)

at java.util.ArrayList$Itr.next(ArrayList.java:851)

at gobblin.metrics.GobblinMetrics.startMetricReporting(GobblinMetrics.java:338)

at gobblin.runtime.AbstractJobLauncher.launchJob(AbstractJobLauncher.java:187)

at gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:326)

[Issac Buenrostro]

Hi Kunal,

The problem is that all of your jobs have the same name (i.e. the value of job.name is the same), which prevents the Gobblin runtime from correctly identifying the attributes of different jobs. This is a race condition, as each job gets a timestamp appended to the name, but if some jobs with the same name are launched at the exact same time, you will encounter this kind of exceptions.

Just use a different job.name for each job. Let me know if this works.

Best,

Issac

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/dceebb6d-a18c-454c-a2cf-5ba14bb3203e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.