Xuggler enabled app sometimes hangs beyond recovery

[Stas Oskin]

Hi.

Something quite strange that I encountered since starting using Xuggler, and it recently start appearing in increasing amounts:
In a very rare fashion, the Java app just hangs, and the only way it can be released, is kill -9.

There are no errors or anything in the logs.

The only point connecting this to Xuggle I have, is that the app, when not doing any video related operations (like opening  and processing video, and so on), is working fine.

Does anyone every encountered this, or has any idea what this could be?

Thanks!

[Stas Oskin]

Hi again.

I mean by "not doing any video operations", is when the app never actually uses the Xuggler library, even that the library is included in the project.

Regards.

[Art Clarke]

Have you tried attaching a Java debugger to the process and seeing where it is?  If the stack trace points to xuggler, then detach and build a debug version of Xuggler, then attach with GDB to see where the native stack trace is.  It could be a FFmpeg blocking operation that is getting it stuck.

- Art

--

You received this message because you are subscribed to the Google Groups "xuggler-users" group.
To post to this group, send email to xuggle...@googlegroups.com.
To unsubscribe from this group, send email to xuggler-user...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/xuggler-users?hl=en.

--
http://www.xuggle.com/
xu‧ggle (zŭ' gl) v. To freely encode, decode, and experience audio and video.

Use Xuggle to get the power of FFmpeg in Java.

[Stas Oskin]

Hi Art.

Actually, I tried to use only jstack so far.

Problem that in this specific case, it reports an error, and suggests to force the attachment via -F flag. Once I do it, it just prints a list of exceptions and exists.

By java debugger, you probably mean jdb?

Regards.

[Art Clarke]

jstack will do the same thing; if you're blocked in a Xuggler call in any of your threads, then attach with GDB.  There is a handy script of GDB commands to print a C/C++ stack trace across threads in mk/buildtools.

- Art

[Stas Oskin]

Hi.

You mean the stacktrace.gdb?

Regards.

[Art Clarke]

That's the one.

- Art

[Stas Oskin]

Hi Art.

I finally managed to catch the issue again, and produced the log according to your instructions:
http://pastebin.com/ma42a74d

I see from it that Xuggler stack in getLogger() stage, waiting for some synchronizing object to become available.

Thread 11 (Thread 0x40eb2940 (LWP 7360)):

#0  0x0000003c7160ad09 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

#1  0x00002b0ca166c9f7 in os::PlatformEvent::park ()

   from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#2  0x00002b0ca16456f9 in Monitor::ILock () from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#3  0x00002b0ca1645e30 in Monitor::lock_without_safepoint_check ()

   from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#4  0x00002b0ca16d6bfa in SafepointSynchronize::block ()

   from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#5  0x00002b0ca175389b in JavaThread::check_safepoint_and_suspend_for_native_trans ()

   from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#6  0x00002b0ca14aa6f6 in jni_NewStringUTF () from /usr/java/jdk1.6.0_16/jre/lib/amd64/server/libjvm.so

#7  0x00002aaadd6d8a63 in com::xuggle::ferry::Logger::getLogger () from /usr/lib/libxuggle-ferry.so.3.2.879

#8  0x00002aaadd6d9276 in com::xuggle::ferry::Logger::getStaticLogger ()

   from /usr/lib/libxuggle-ferry.so.3.2.879

#9  0x00002aaaddc70ed4 in ?? () from /usr/lib/libxuggle-xuggler.so.3.2.879

#10 0x00002aaaddca1ec6 in ?? () from /usr/lib/libxuggle-xuggler.so.3.2.879

#11 0x0000000000000000 in ?? ()


Any idea why it hangs at this stage?

Thanks in advance!

[Art Clarke]

This looks really odd; What JVM are you using? Xuggler is parked at
that point in time, but so are most threads. It's allocating a Java
string...

The other interesting note is the hotspot compiler seems to be
separately compiling in two threads -- that may be related. Can you
search around in the Sun (or whoever makes your JVM) to see if people
have seem similar issues with JNI-based code being executed while
HotSpot is compiling.

- Art

[Stas Oskin]

Hi.

This is Sun indeed, version 1.6.0_16.

I will look if anything similar was reported in the past.

Any chance this something on Xuggler side?

Regards.

[Art Clarke]

Always possible, but looking at your stack trace I'm not sure. I've
seen bugs in the JVM before though when making JNI calls and having
threading get stuck (for example, we work around one such issue on
Xuggler start-up by allocating a IBuffer object of size one because
JNI has a thread-bug if the first Java direct byte buffer is not
allocated on a single thread).

The best way to convince me it's a Xuggler bug is to make a test-case
that you can reproduce, and then I can reproduce. Until then, I have
to ask you to hunt the Java forums.

- Art

[Stas Oskin]

Hi Art.

As I started encountering this problem more often, I put some time into researching it.

It appears a fairy common issue, due to bug in JVM:
http://bugs.sun.com/view_bug.do?bug_id=6791815

I presume it's the same one you mentioned being fixed at Xuggler?

Any way to verify the fix is actually working?
Perhaps I should allocate a ByteBuffer of 1 myself, to ensure it works across any other JNI libraries as well?

Thanks.

[Art Clarke]

Xuggler allocates a ByteBuffer of size 1 when it is loaded the first time.  The JVM ensure that loading of a class the first time is a thread-protected method (i.e. other attempts to load the class at the same instance block until that's done).  The only way I had to replicate the issue thoroughly was to create a many-threaded test, and we did that, and it runs in our test suite automatically.

If you'd like to verify that indeed every single Xuggler class does the right workaround, you can look in generated/src/...  The insertion of the code for doing this is automated, so will be in 100% of classes.

- Art

[Stas Oskin]

Hi Art.

I asked around on JVM forums.

The problem seem to be related to dlsym() blocking, happening due to libraries hooks, which as I remember, Xuggler uses.

You can see more details here:

http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2010-April/001025.html

Any chance Xuggler has anything to do with this?

Or it might be another library which I use?

Thanks.

[Art Clarke]

We do not use dlsym.

- Art

[Stas Oskin]

Hi.

I see, any idea how to find from thread the used library?

Thanks again.

[Stas Oskin]

BTW, resurrecting this (very) long thread, the issue was solved by allocating a single packet in the beginning, via single thread, before continuing to multi-threaded allocations.

I presume it due to logger initialization done by Xuggler on first video operation / FFMPEG binding, which is not completely thread-safe.