We recently upgraded to storm 0.8.0 and we experience a new issue when we
had classpath issues. With a new version of a dependency on the classpath
causing a java.lang.VerifyError in our code, we would see the expected:
storm brought down the JVM that experienced the error and started again.
This however exposed another issue: zeromq would blow up and we kept
running the disk out of space with core dump files. This was where the core
dump file took us:
#0 0x00000031de232885 in raise () from /lib64/libc.so.6
#1 0x00000031de234065 in abort () from /lib64/libc.so.6
#2 0x00000031de22b9fe in __assert_fail_base () from /lib64/libc.so.6
#3 0x00000031de22bac0 in __assert_fail () from /lib64/libc.so.6
#4 0x00007fa4cd69fbc2 in get_socket (env=0x7fa4d88271d0,
obj=0x7fa48f1f0688, do_assert=1) at Socket.cpp:543
#5 0x00007fa4cd69fe85 in Java_org_zeromq_ZMQ_00024Socket_send
(env=0x7fa4d88271d0, obj=<value optimized out>, msg=0x7fa48f1f0680,
flags=2) at Socket.cpp:365
I found a few posts in zeromq related threads online, but it was all
questions, no answers. It feels like a potential race condition with the
JVM shutting down and something trying to re-use that socket?
The interesting part is when we resolved our classpath issue we quit seeing
the issue. Thoughts?