OSv VM crashes after CNF exception

16 views
Skip to first unread message

Roman Shaposhnik

unread,
Jun 18, 2014, 5:05:33 PM6/18/14
to Osv Dev
Hi!

the Java process that I'm trying to bootstrap dies
in CNF exception that is related to logging (see
the stacktrace below). Two things are weird about
this:
* the app itself doesn't require slf4j, and since I see
io.osv.jul.LogManagerWrapper smack in the middle
of that trace I assume that's some kind of handshake
going badly on the OSv logging side of things

* the entire OSv VM crashes at this point. This is definitely
not the behavior I see if I just have a ClassNotFound anywhere
else. What gives and how can I avoid the entire VM crashing
as a result of something like this?

Thanks,
Roman.

Failed to load handler: org.slf4j.bridge.SLF4JBridgeHandler
java.lang.ClassNotFoundException: org.slf4j.bridge.SLF4JBridgeHandler
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at io.osv.jul.LogManagerWrapper.reconfigureRootHandlers(LogManagerWrapper.java:59)
at io.osv.jul.LogManagerWrapper$LogManagerInitializer$2.intercept(LogManagerWrapper.java:117)
at $java.util.logging.Logger$$EnhancerByCGLIB$$35bc9e1f.getResourceBundleName(<generated>)
at java.util.logging.Logger.getEffectiveResourceBundleName(Logger.java:1672)
at java.util.logging.Logger.doLog(Logger.java:626)
at java.util.logging.Logger.log(Logger.java:654)
at java.util.logging.Logger.info(Logger.java:1172)
at com.gemstone.gemfire.management.internal.cli.LogWrapper.info(LogWrapper.java:206)
at com.gemstone.gemfire.management.internal.cli.shell.Gfsh.<init>(Gfsh.java:168)
at com.gemstone.gemfire.management.internal.cli.shell.Gfsh.getInstance(Gfsh.java:261)
at com.gemstone.gemfire.management.internal.cli.Launcher.launchGfsh(Launcher.java:156)
at com.gemstone.gemfire.management.internal.cli.Launcher.executeTarget(Launcher.java:144)
at com.gemstone.gemfire.management.internal.cli.Launcher.main(Launcher.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at io.osv.ContextIsolator.runMain(ContextIsolator.java:220)
at io.osv.ContextIsolator.access$400(ContextIsolator.java:29)
at io.osv.ContextIsolator$3.run(ContextIsolator.java:107)
Exception in thread "Gfsh Launcher"
java.lang.IllegalArgumentException: java.lang.ClassNotFoundException:
com.cloudius.cli.OSvTerminal
at jline.Terminal.setupTerminal(Terminal.java:65)
at jline.Terminal.getTerminal(Terminal.java:26)
at jline.ConsoleReader.<init>(ConsoleReader.java:174)
at jline.ConsoleReader.<init>(ConsoleReader.java:169)
at jline.ConsoleReader.<init>(ConsoleReader.java:157)
at org.springframework.shell.core.JLineShell.createConsoleReader(JLineShell.java:213)
at com.gemstone.gemfire.management.internal.cli.shell.Gfsh.createConsoleReader(Gfsh.java:484)
at org.springframework.shell.core.JLineShell.run(JLineShell.java:95)
at java.lang.Thread.run(Thread.java:744)
FATAL exit ..

Pekka Enberg

unread,
Jun 19, 2014, 2:47:14 AM6/19/14
to Roman Shaposhnik, Osv Dev, Tomasz Grabiec
On Thu, Jun 19, 2014 at 12:05 AM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> the Java process that I'm trying to bootstrap dies
> in CNF exception that is related to logging (see
> the stacktrace below). Two things are weird about
> this:
> * the app itself doesn't require slf4j, and since I see
> io.osv.jul.LogManagerWrapper smack in the middle
> of that trace I assume that's some kind of handshake
> going badly on the OSv logging side of things
>
> * the entire OSv VM crashes at this point. This is definitely
> not the behavior I see if I just have a ClassNotFound anywhere
> else. What gives and how can I avoid the entire VM crashing
> as a result of something like this?

Tomek, thoughts on this?

- Pekka
> --
> You received this message because you are subscribed to the Google Groups "OSv Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Tomasz Grabiec

unread,
Jun 19, 2014, 5:42:53 AM6/19/14
to Roman Shaposhnik, Osv Dev
On Wed, Jun 18, 2014 at 11:05 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
> Hi!
>
> the Java process that I'm trying to bootstrap dies
> in CNF exception that is related to logging (see
> the stacktrace below). Two things are weird about
> this:
> * the app itself doesn't require slf4j, and since I see
> io.osv.jul.LogManagerWrapper smack in the middle
> of that trace I assume that's some kind of handshake
> going badly on the OSv logging side of things
>
> * the entire OSv VM crashes at this point. This is definitely
> not the behavior I see if I just have a ClassNotFound anywhere
> else. What gives and how can I avoid the entire VM crashing
> as a result of something like this?

What do you mean by a crash? Does it segfault and core dump or just
exits as if System.exit() was called?
Our mgmt module sets "jline.terminal" property to
OSvTerminal.class.getName(), so I'm assuming you are starting
mgmt.shell along with gemfire. Is that right? Can you please share how
do you build and run the image? What modules are involved, what are
the command lines?

Our multi-java app isolation is not perfect. The errors you see could
be explained by system property isolating proxy being removed. That
could happen if gemfire did something like System.setProperties().

> at jline.Terminal.setupTerminal(Terminal.java:65)
> at jline.Terminal.getTerminal(Terminal.java:26)
> at jline.ConsoleReader.<init>(ConsoleReader.java:174)
> at jline.ConsoleReader.<init>(ConsoleReader.java:169)
> at jline.ConsoleReader.<init>(ConsoleReader.java:157)
> at org.springframework.shell.core.JLineShell.createConsoleReader(JLineShell.java:213)
> at com.gemstone.gemfire.management.internal.cli.shell.Gfsh.createConsoleReader(Gfsh.java:484)
> at org.springframework.shell.core.JLineShell.run(JLineShell.java:95)
> at java.lang.Thread.run(Thread.java:744)
> FATAL exit ..
>

Roman Shaposhnik

unread,
Jun 20, 2014, 1:15:32 AM6/20/14
to Tomasz Grabiec, Osv Dev
On Thu, Jun 19, 2014 at 2:42 AM, Tomasz Grabiec
<tgra...@cloudius-systems.com> wrote:
> On Wed, Jun 18, 2014 at 11:05 PM, Roman Shaposhnik <ro...@shaposhnik.org> wrote:
>> Hi!
>>
>> the Java process that I'm trying to bootstrap dies
>> in CNF exception that is related to logging (see
>> the stacktrace below). Two things are weird about
>> this:
>> * the app itself doesn't require slf4j, and since I see
>> io.osv.jul.LogManagerWrapper smack in the middle
>> of that trace I assume that's some kind of handshake
>> going badly on the OSv logging side of things

Btw, for those following this: I figured out why it was the case.
The properties governing the logger (set by CRaSH) were
messing me up (since I was using CRaSH's java "command"
with a different cp).

>> * the entire OSv VM crashes at this point. This is definitely
>> not the behavior I see if I just have a ClassNotFound anywhere
>> else. What gives and how can I avoid the entire VM crashing
>> as a result of something like this?
>
> What do you mean by a crash? Does it segfault and core dump or just
> exits as if System.exit() was called?

It appears like it just exits (as though I typed exit in CRaSH).

> Our mgmt module sets "jline.terminal" property to
> OSvTerminal.class.getName(), so I'm assuming you are starting
> mgmt.shell along with gemfire. Is that right? Can you please share how
> do you build and run the image? What modules are involved, what are
> the command lines?

The image is a off-the-shelf cloudius/osv image with a bunch
of custom files cpiod'ed in. When image boots it starts CRaSH.
From that point on I use:
ssh admin@address java ...
to start my stuff as additional java threads.

> Our multi-java app isolation is not perfect.

Understood. And I actually need multi-java app isolation. What
I need is my app being reasonably isolated from CRaSH (and
whatever other threads it may have started).

Or to put it another way -- I don't really even need CRaSH, but
I do need an ability to interactively start additional java threads
from outside of the running container.

> The errors you see could
> be explained by system property isolating proxy being removed. That
> could happen if gemfire did something like System.setProperties().

Hm. Let me play with this a bit.

Thanks,
Roman.

Tomasz Grabiec

unread,
Jun 20, 2014, 4:11:39 AM6/20/14
to Roman Shaposhnik, Osv Dev
So I looked a bit closer at this and the problem is probably simpler
than I thought. When new isolated context is started it inherits
system properties from parent context (CRaSH in this case). This
should be easy to fix, I'll send a patch soon.

Tomasz Grabiec

unread,
Jun 20, 2014, 8:10:31 AM6/20/14
to Roman Shaposhnik, Osv Dev
On Fri, Jun 20, 2014 at 10:11 AM, Tomasz Grabiec
It could be that because of the exceptions we get gemfire is calling
System.exit which brings the JVM down and since it's the main
application, the whole VM is down.

>>
>>> Our mgmt module sets "jline.terminal" property to
>>> OSvTerminal.class.getName(), so I'm assuming you are starting
>>> mgmt.shell along with gemfire. Is that right? Can you please share how
>>> do you build and run the image? What modules are involved, what are
>>> the command lines?
>>
>> The image is a off-the-shelf cloudius/osv image with a bunch
>> of custom files cpiod'ed in. When image boots it starts CRaSH.
>> From that point on I use:
>> ssh admin@address java ...
>> to start my stuff as additional java threads.
>
> So I looked a bit closer at this and the problem is probably simpler
> than I thought. When new isolated context is started it inherits
> system properties from parent context (CRaSH in this case). This
> should be easy to fix, I'll send a patch soon.

The patch has been posted.

Roman Shaposhnik

unread,
Jun 23, 2014, 1:04:58 AM6/23/14
to Tomasz Grabiec, Osv Dev
On Fri, Jun 20, 2014 at 5:10 AM, Tomasz Grabiec
<tgra...@cloudius-systems.com> wrote:
>> So I looked a bit closer at this and the problem is probably simpler
>> than I thought. When new isolated context is started it inherits
>> system properties from parent context (CRaSH in this case). This
>> should be easy to fix, I'll send a patch soon.
>
> The patch has been posted.

Perfect! Thank you so much. Perhaps the only other thing that would've
saved me some time would be an option that tells you from which thread
a System.exit() was called -- IOW, what part of an application terminated
the jvm and made OSv app exit.

Thanks,
Roman.
Reply all
Reply to author
Forward
0 new messages