"java.lang.IllegalStateException: zip file closed" in mapper tasks when running gobblin pull (0.8.0)

[Chris Kelly]

Hi all! I'm running into an error where I am encountering this zip file closed error on a few mapper tasks.  I'm having a very difficult time diagnosing the issue.  Does anyone know what may be causing it?  

2017-04-10 17:23:26,083 ERROR [TaskExecutor-0] gobblin.runtime.Task: Task task_autotable-pull_1491844939807_11 failed
java.lang.IllegalStateException: zip file closed
        at java.util.zip.ZipFile.ensureOpen(ZipFile.java:634)
        at java.util.zip.ZipFile.access$200(ZipFile.java:56)
        at java.util.zip.ZipFile$1.hasMoreElements(ZipFile.java:487)
        at java.util.jar.JarFile$1.hasMoreElements(JarFile.java:241)
        at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:30)
        at org.reflections.vfs.ZipDir$1$1.computeNext(ZipDir.java:26)
        at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
        at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
        at org.reflections.Reflections.scan(Reflections.java:240)
        at org.reflections.Reflections.scan(Reflections.java:204)
        at org.reflections.Reflections.<init>(Reflections.java:129)
        at org.reflections.Reflections.<init>(Reflections.java:170)
        at org.reflections.Reflections.<init>(Reflections.java:143)
        at gobblin.util.ClassAliasResolver.<init>(ClassAliasResolver.java:52)
        at gobblin.source.extractor.extract.kafka.UniversalKafkaSource.getExtractor(UniversalKafkaSource.java:29)
        at gobblin.runtime.TaskContext.getExtractor(TaskContext.java:119)
        at gobblin.runtime.Task.run(Task.java:126)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

[Abhishek Tiwari]

Hi Chris, 

I haven't seen this before, but on a quick search it seems that a few jar(s) might be corrupted in the classpath. Can you please try to download the distribution again and try? 

Abhishek

--
You received this message because you are subscribed to the Google Groups "gobblin-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.
To post to this group, send email to gobbli...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/a9bfbbf1-f1e1-40e7-9b7c-19f8db6115fb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris Kelly]

Hi Abhishek, I actually believe I figured out the problem.  As noted here, I think Reflections don't behave super well across threads.  Currently the task executer pool is set to be 10 by default.  Because of this, if two tasks are executing on the same mapper, the zipfile used in reflection may get closed in the middle of search the classpath, causing this error.  By setting taskexecutor.threadpool.size to 1, I ended up fixing my issue.  

-Chris

Abhishek

To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-user...@googlegroups.com.

[Abhishek Tiwari]

Interesting, internally however, we do use a larger threadpool but never run into this issue. 

Abhishek

To unsubscribe from this group and stop receiving emails from it, send an email to gobblin-users+unsubscribe@googlegroups.com.

To post to this group, send email to gobbli...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/gobblin-users/7898bc91-85c1-4204-90aa-58db6358c29b%40googlegroups.com.