JVM Crash When Compiling Native Images
35 views
Skip to first unread message
Tom Gregory
Nov 19, 2020, 12:25:42 PM11/19/20
to Quarkus Development mailing list
Hi All,
I have reported this to Oracle and they've asked if anyone from the Quarkus team has looked into this.
Recently, and it may have been after I upgraded my Linux kernel, occasionally the JVM will crash when compiling native images. It will compile fine and then it will crash and it won't compile anything until I reboot and then it will compile again a few times and then start crashing.
Here is the information:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fac4a6fb5ff, pid=27, tid=101
#
# JRE version: OpenJDK Runtime Environment GraalVM CE 20.2.0 (11.0.8+10) (build 11.0.8+10-jvmci-20.2-b03)
# Java VM: OpenJDK 64-Bit Server VM GraalVM CE 20.2.0 (11.0.8+10-jvmci-20.2-b03, mixed mode, tiered, jvmci, jvmci compiler, compressed oops, parallel gc, linux-amd64)
# Problematic frame:
# v ~BufferBlob::vtable chunks
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /project/core.27)
#
# An error report file with more information is saved as:
# /project/hs_err_pid27.log
Compiled method (JVMCI) 22938 10755 4 jdk.vm.ci.meta.ResolvedJavaType::resolveConcreteMethod (26 bytes)
total in heap [0x00007fac52157790,0x00007fac52158068] = 2264
relocation [0x00007fac521578f8,0x00007fac52157960] = 104
constants [0x00007fac52157960,0x00007fac52157980] = 32
main code [0x00007fac52157980,0x00007fac52157ca0] = 800
stub code [0x00007fac52157ca0,0x00007fac52157cb0] = 16
oops [0x00007fac52157cb0,0x00007fac52157cc0] = 16
metadata [0x00007fac52157cc0,0x00007fac52157d48] = 136
scopes data [0x00007fac52157d48,0x00007fac52157e80] = 312
scopes pcs [0x00007fac52157e80,0x00007fac52157fe0] = 352
dependencies [0x00007fac52157fe0,0x00007fac52157fe8] = 8
nul chk table [0x00007fac52157fe8,0x00007fac52158058] = 112
JVMCI data [0x00007fac52158058,0x00007fac52158068] = 16
Compiled method (JVMCI) 22938 12125 4 com.oracle.graal.pointsto.meta.AnalysisUniverse::collectMethodImplementations (98 bytes)
total in heap [0x00007fac52687510,0x00007fac52688858] = 4936
relocation [0x00007fac52687678,0x00007fac52687730] = 184
constants [0x00007fac52687740,0x00007fac52687760] = 32
main code [0x00007fac52687760,0x00007fac52687fe0] = 2176
stub code [0x00007fac52687fe0,0x00007fac52688040] = 96
oops [0x00007fac52688040,0x00007fac52688050] = 16
metadata [0x00007fac52688050,0x00007fac526880e0] = 144
scopes data [0x00007fac526880e0,0x00007fac52688588] = 1192
scopes pcs [0x00007fac52688588,0x00007fac526887b8] = 560
dependencies [0x00007fac526887b8,0x00007fac526887c8] = 16
nul chk table [0x00007fac526887c8,0x00007fac52688848] = 128
JVMCI data [0x00007fac52688848,0x00007fac52688858] = 16
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled
#
# If you would like to submit a bug report, please visit:
# https://github.com/oracle/graal/issues
#
Error: Image build request failed with exit status 134
Unfortunately, I cannot constantly reproduce. If I reboot my system, it will work for a while and then start eliciting the same error.
GraalVM 20.2.0
Quarkus 1.9.2
OS Ubuntu: 5.8.17-050817-generic
Architecture: AMD64
Stack Trace:
[ERROR] Failed to execute goal io.quarkus:quarkus-maven-plugin:1.9.2.Final:native-image (default) on project codigo-provider-sendback-lda: Failed to generate native image: io.quarkus.builder.BuildException: Build failure: Build failed due to errors [ERROR] [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: java.lang.RuntimeException: Failed to build native image [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:307) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:566) [ERROR] at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:936) [ERROR] at io.quarkus.builder.BuildContext.run(BuildContext.java:277) [ERROR] at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) [ERROR] at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046) [ERROR] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578) [ERROR] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452) [ERROR] at java.base/java.lang.Thread.run(Thread.java:834) [ERROR] at org.jboss.threads.JBossThread.run(JBossThread.java:479) [ERROR] Caused by: java.lang.RuntimeException: Image generation failed. Exit code: 134 [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:520) [ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:276) [ERROR] ... 12 more [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:I'm also including the log.
Any thoughts would be great.
Thanks,
Tom
Sanne Grinovero
Nov 19, 2020, 3:30:57 PM11/19/20
to fathero...@gmail.com, Quarkus Development mailing list
Hi Tom,
I happen to be running the same GraalVM build as local default, and
have been building quite a bit of (different) native images recently
for various tests, but have never seen it crash.
Is there any chance you could share a project in which it happens with
reasonable frequency?
Thanks
> --
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/614d6277-b4c0-4ca6-84d3-73a4421ca5can%40googlegroups.com.
I happen to be running the same GraalVM build as local default, and
have been building quite a bit of (different) native images recently
for various tests, but have never seen it crash.
Is there any chance you could share a project in which it happens with
reasonable frequency?
Thanks
> You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/614d6277-b4c0-4ca6-84d3-73a4421ca5can%40googlegroups.com.
Tom Gregory
Nov 19, 2020, 4:48:20 PM11/19/20
to Sanne Grinovero, Quarkus Development mailing list
Hi Sanne,
Unfortunately, it is all my projects. It is not limited to any one of them. What I will do is create a very simple project and see if it happens. Please keep in mind that if I reboot, then it works fine for a while and then it starts crashing again. Perhaps it is this version of Linux or something environmental. I recently built this new computer. On my old computer, it was never an issue. So, I don't know if there is a kernel parameter that I need to tune, etc. I'm not sure.
Thanks,
Tom
Sanne Grinovero
Nov 19, 2020, 4:55:33 PM11/19/20
to fathero...@gmail.com, Quarkus Development mailing list
On Thu, 19 Nov 2020 at 21:48, Tom Gregory <fathero...@gmail.com> wrote:
>
> Hi Sanne,
>
> Unfortunately, it is all my projects. It is not limited to any one of them. What I will do is create a very simple project and see if it happens. Please keep in mind that if I reboot, then it works fine for a while and then it starts crashing again. Perhaps it is this version of Linux or something environmental. I recently built this new computer. On my old computer, it was never an issue. So, I don't know if there is a kernel parameter that I need to tune, etc. I'm not sure.
Might be useful to start with memtest86 ? Check the temperatures, and
>
> Hi Sanne,
>
> Unfortunately, it is all my projects. It is not limited to any one of them. What I will do is create a very simple project and see if it happens. Please keep in mind that if I reboot, then it works fine for a while and then it starts crashing again. Perhaps it is this version of Linux or something environmental. I recently built this new computer. On my old computer, it was never an issue. So, I don't know if there is a kernel parameter that I need to tune, etc. I'm not sure.
things like that? The native-image compiler can certainly stress a
machine.
Stuart Douglas
Nov 19, 2020, 5:05:16 PM11/19/20
to Sanne Grinovero, fathero...@gmail.com, Quarkus Development mailing list
On Fri, 20 Nov 2020 at 08:55, Sanne Grinovero <sa...@hibernate.org> wrote:
On Thu, 19 Nov 2020 at 21:48, Tom Gregory <fathero...@gmail.com> wrote:
>
> Hi Sanne,
>
> Unfortunately, it is all my projects. It is not limited to any one of them. What I will do is create a very simple project and see if it happens. Please keep in mind that if I reboot, then it works fine for a while and then it starts crashing again. Perhaps it is this version of Linux or something environmental. I recently built this new computer. On my old computer, it was never an issue. So, I don't know if there is a kernel parameter that I need to tune, etc. I'm not sure.
Might be useful to start with memtest86 ? Check the temperatures, and
things like that? The native-image compiler can certainly stress a
machine.
My old desktop would do something similar, except the whole machine would lock up. Increasing the fan speed to keep temperatures down seemed to alleviate it somewhat (I think it was an issue with the CPU, it was not hitting the sort of temps that should cause instability, but every time the temp went up the machine would crash).
Stuart
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2oq-ODruFC6bTOtuTzML4O5Ma38f1aWax5eKXOrJFeAg%40mail.gmail.com.
Tom Gregory
Nov 19, 2020, 5:16:09 PM11/19/20
to Stuart Douglas, Sanne Grinovero, Quarkus Development mailing list
Thanks to the both of you. I'll check the temperatures, etc and see if there is something there. It is a new Ryzen 3900X, so unless it is a temperature thing, I can't imagine anything else going on. I have heard the fan spin up, but we'll see.
Thanks for the ideas.
Tom
Georgios Andrianakis
Nov 19, 2020, 5:20:33 PM11/19/20
to fathero...@gmail.com, Stuart Douglas, Sanne Grinovero, Quarkus Development mailing list
FWIW, on my Ryzen 3950x, I almost never hear the fan spin faster, even when running native builds. Of course that CPU doesn't come with a stock cooler and one needs to buy (a rather expensive) cooling solution.
Also, you might want to make sure that the airflow of the tower is proper.
Or else maybe there is some kind of memory issue?
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAGsuPth%3DXLJWdUze%2BJvukM77hfSC81YFht8pes03Js8YT144hg%40mail.gmail.com.
Stuart Douglas
Nov 19, 2020, 5:28:04 PM11/19/20
to Georgios Andrianakis, fathero...@gmail.com, Sanne Grinovero, Quarkus Development mailing list
On Fri, 20 Nov 2020 at 09:20, Georgios Andrianakis <gand...@redhat.com> wrote:
FWIW, on my Ryzen 3950x, I almost never hear the fan spin faster, even when running native builds. Of course that CPU doesn't come with a stock cooler and one needs to buy (a rather expensive) cooling solution.Also, you might want to make sure that the airflow of the tower is proper.
The stock cooler on the 3900X is very loud, I can hear it pretty much all the time.
Stuart
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nAC81081rO%3DEnZTRnh5uJJ0R9BdYwiiUjGVW3Jxy%2BCDA%40mail.gmail.com.
Peter Palaga
Nov 26, 2020, 6:02:16 AM11/26/20
to Quarkus Development mailing list
I saw strange failures on my TR 1920X system when I was experimenting
with setting the RAM frequency to the RAM's factory max value which was
higher than the motherboard's default value. Strange enough, the
symptoms were rather CPU related. I guess the root cause was the
shortage of energy at peak times (the box has only a 650W power supply).
-- P
> >> > To view this discussion on the web visit
>
https://groups.google.com/d/msgid/quarkus-dev/614d6277-b4c0-4ca6-84d3-73a4421ca5can%40googlegroups.com
>
<https://groups.google.com/d/msgid/quarkus-dev/614d6277-b4c0-4ca6-84d3-73a4421ca5can%40googlegroups.com>.
>
https://groups.google.com/d/msgid/quarkus-dev/614d6277-b4c0-4ca6-84d3-73a4421ca5can%40googlegroups.com
>
> >
> > --
> > You received this message because you are subscribed
> to the Google Groups "Quarkus Development mailing list"
> group.
> > To unsubscribe from this group and stop receiving
> emails from it, send an email to
> quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.
> > --
> > You received this message because you are subscribed
> to the Google Groups "Quarkus Development mailing list"
> group.
> > To unsubscribe from this group and stop receiving
> emails from it, send an email to
> quarkus-dev...@googlegroups.com
> > To view this discussion on the web visit
>
https://groups.google.com/d/msgid/quarkus-dev/CAGsuPtj3zNcWjQFmvTA6Z-hGxRJi-DmkGn4%3Dc1cEeFJhsK5cCA%40mail.gmail.com
>
>
<https://groups.google.com/d/msgid/quarkus-dev/CAGsuPtj3zNcWjQFmvTA6Z-hGxRJi-DmkGn4%3Dc1cEeFJhsK5cCA%40mail.gmail.com>.
>
> -- You received this message because
you are subscribed to
> the Google Groups "Quarkus Development mailing list"
group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev%2Bunsu...@googlegroups.com>.
> -- You received this message because
you are subscribed to
> the Google Groups "Quarkus Development mailing list"
group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> quarkus-dev...@googlegroups.com
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2oq-ODruFC6bTOtuTzML4O5Ma38f1aWax5eKXOrJFeAg%40mail.gmail.com
>
>
<https://groups.google.com/d/msgid/quarkus-dev/CAFm4XO2oq-ODruFC6bTOtuTzML4O5Ma38f1aWax5eKXOrJFeAg%40mail.gmail.com>.
>
> -- You received this message because you are
subscribed to the
> Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev...@googlegroups.com>.
> -- You received this message because you are
subscribed to the
> Google Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from
> it, send an email to quarkus-dev...@googlegroups.com
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/quarkus-dev/CAGsuPth%3DXLJWdUze%2BJvukM77hfSC81YFht8pes03Js8YT144hg%40mail.gmail.com
>
>
<https://groups.google.com/d/msgid/quarkus-dev/CAGsuPth%3DXLJWdUze%2BJvukM77hfSC81YFht8pes03Js8YT144hg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> -- You received this message because you are subscribed to
the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to quarkus-dev...@googlegroups.com
> <mailto:quarkus-dev...@googlegroups.com>.
> -- You received this message because you are subscribed to
the Google
> Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to quarkus-dev...@googlegroups.com
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nAC81081rO%3DEnZTRnh5uJJ0R9BdYwiiUjGVW3Jxy%2BCDA%40mail.gmail.com
>
>
<https://groups.google.com/d/msgid/quarkus-dev/CALeTM-nAC81081rO%3DEnZTRnh5uJJ0R9BdYwiiUjGVW3Jxy%2BCDA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>
> --
> You received this message because you are subscribed to the Google
Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
send an email to quarkus-dev...@googlegroups.com
<mailto:quarkus-dev...@googlegroups.com>.
> --
> You received this message because you are subscribed to the Google
Groups "Quarkus Development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it,
send an email to quarkus-dev...@googlegroups.com
> To view this discussion on the web visit
https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cyUy-yV-iWnwitNpc1ivQN%3Dai7VbNN5bzSwcw1Aee6-pg%40mail.gmail.com
<https://groups.google.com/d/msgid/quarkus-dev/CAD%2BL2cyUy-yV-iWnwitNpc1ivQN%3Dai7VbNN5bzSwcw1Aee6-pg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Tom Gregory
Nov 26, 2020, 4:40:18 PM11/26/20
to ppa...@redhat.com, Quarkus Development mailing list
Thanks Peter,
I think I've narrowed it down to the memory. It appears as though the memory is not 100% compatible with the motherboard. I've just installed new memory that has been certified by Asus and we'll see how that goes. I find it strange that it works okay for a while after booting and then starts failing. A simple reboot fixes the problem. I'm not that conversant on the in's and out's of the hardware, but I would have expected the same failures consistently. Anyway, we'll see how this new memory plays. So far, my tests have been working. Next week I'll perform enough development to test native builds to see if there are any failures.
Tom
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/5423a4da-ba7d-4586-526e-af65a2f1065a%40redhat.com.
Emmanuel Bernard
Nov 27, 2020, 4:52:28 AM11/27/20
to fathero...@gmail.com, Peter Palaga, Quarkus Development mailing list
You will find this read very useful https://shipilev.net/jvm/test-your-memory/
It explains why the JVM is one of the best memory testing tool out there :) and native compilation ironically is run as a JVM process, not a native one.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/CAGsuPtjYjYfRkdFemLHYth4-uj9EoG-A%2BESfHMd_5JJgt517KQ%40mail.gmail.com.
Reply all
Reply to author
Forward
0 new messages