*** buffer overflow detected ***: java terminated

[Madhu Muchukota]

Hi Team - I am seeing frequently my Jenkins controller going down with below error message

*** buffer overflow detected ***: /opt/apps/java/current_java/bin/java terminated
======= Backtrace: =========
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f3a41b6a7a7]
/lib64/libc.so.6(+0x116922)[0x7f3a41b68922]
/lib64/libc.so.6(+0x118707)[0x7f3a41b6a707]
/lib64/libglib-2.0.so.0(g_spawn_sync+0x230)[0x7f39bc703130]
/lib64/libglib-2.0.so.0(g_spawn_command_line_sync+0x75)[0x7f39bc703805]
/lib64/libgio-2.0.so.0(+0xb2100)[0x7f39bd095100]
/lib64/libgio-2.0.so.0(g_dbus_address_get_for_bus_sync+0x340)[0x7f39bd096ad0]
/lib64/libgio-2.0.so.0(+0xbf2fe)[0x7f39bd0a22fe]
/lib64/libgio-2.0.so.0(g_bus_get_sync+0x56)[0x7f39bd0a7c46]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7b00)[0x7f39bb529b00]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7bed)[0x7f39bb529bed]
/lib64/libglib-2.0.so.0(+0x48d47)[0x7f39bc6b8d47]
/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x159)[0x7f39bc6bc119]
/lib64/libglib-2.0.so.0(+0x4c478)[0x7f39bc6bc478]
/lib64/libglib-2.0.so.0(g_main_context_iteration+0x2c)[0x7f39bc6bc52c]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7d6d)[0x7f39bb529d6d]
/lib64/libglib-2.0.so.0(+0x735b0)[0x7f39bc6e35b0]
/lib64/libpthread.so.0(+0x7ea5)[0x7f3a4223cea5]
/lib64/libc.so.6(clone+0x6d)[0x7f3a41b50b0d]
======= Memory map: ========
680000000-80161f000 rw-p 00000000 00:00 0
80161f000-840000000 ---p 00000000 00:00 0
5568a8c67000-5568a8c68000 r-xp 00000000 fd:04 17591637                   /opt/apps/java/jdk-11.0.16+8/bin/java
5568a8e68000-5568a8e69000 r--p 00001000 fd:04 17591637                   /opt/apps/java/jdk-11.0.16+8/bin/java

I am not sure what is causing Jenkins to get crashed. 

Any help on this is highly appreciated please.

Regards,

Madhu M

[Ivan Fernandez Calvo]

You do it provide any information about Jenkins version, OS version of JDK version so it is difficult to know the cause. Checking the stack trace your JDK has about two years so I guess all the rest of your system is as old as the JDK so update it at leas to the lates JDK 11 and  update your system libraries to the lates security updates that probably resolv yourissue

[Dirk Heinrichs]

Am Freitag, dem 02.12.2022 um 13:41 -0800 schrieb Madhu Muchukota:

5568a8c67000-5568a8c68000 r-xp 00000000 fd:04 17591637                   /opt/apps/java/jdk-11.0.16+8/bin/java

You need to update to 11.0.17, it contains a fix.

HTH...

Dirk

-- 

Dirk Heinrichs

Senior Systems Engineer, Delivery Pipeline

OpenText ™ Discovery | Recommind

Phone: +49 2226 15966 18

Email: dhei...@opentext.com

Website: www.recommind.de

Recommind GmbH, Von-Liebig-Straße 1, 53359 Rheinbach

Vertretungsberechtigte Geschäftsführer Gordon Davies, Madhu Ranganathan, Christian Waida, Registergericht Amtsgericht Bonn, Registernummer HRB 10646

This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden

Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail sind nicht gestattet.

[M.Madhusudana Reddy]

Thanks Ivan and Dirk.

My java version was jdk-11.0.16+8 and I updated this to the latest one in JAVA 11 this morning which is jdk-11.0.17+8 and I still see the same error.

Tomcat is of version 9.0.0.0

Jenkins is of version 2.375.1

Thanks

M.Madhu

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/db34e239bf19b96d52f34e4974decbb06dc974ab.camel%40opentext.com.

[Mark Waite]

On Monday, December 5, 2022 at 10:29:49 AM UTC-7 Madhu Muchukota wrote:

Thanks Ivan and Dirk.

My java version was jdk-11.0.16+8 and I updated this to the latest one in JAVA 11 this morning which is jdk-11.0.17+8 and I still see the same error.

Tomcat is of version 9.0.0.0

Jenkins is of version 2.375.1

The message that you're seeing indicates that the Java process was killed by the operating system when the Java process performed an unsafe operation (buffer overflow).  The Java virtual machine works very hard to prevent Java programs from being able to perform unsafe operations.  There are some areas where Java allows unsafe operations (Java Native Interface, Java Native Runtime project), but those are exceptional cases.  In almost all cases, buffer overflow failures should not happen in a Linux Java process.

If you can provide instructions so that others can duplicate the problem, the OpenJDK team and others are likely very interested in the conditions that cause the failure.

If your goal is to avoid the issue rather than investigate why it is failing, you might try running Jenkins without Tomcat.  If that still shows the issue, then you might try bisecting your most recent plugin updates to see if one of the recent plugin updates is associated with the change of behavior.

A process killed by buffer overflow is very rare in the Jenkins world and in the Java world.

Mark Waite

 

[M.Madhusudana Reddy]

Thanks Mark 

All I'm doing is just triggering/running a multi branch pipeline job from the controller UI.

I will be happy to share more information needed from my end. 

I also looked at the compute resources of my controller node and don't see any issues over there. 

###############################

*** buffer overflow detected ***: /opt/apps/java/current_java/bin/java terminated
======= Backtrace: =========

/lib64/libc.so.6(__fortify_fail+0x37)[0x7f829c70b7a7]
/lib64/libc.so.6(+0x116922)[0x7f829c709922]
/lib64/libc.so.6(+0x118707)[0x7f829c70b707]
/lib64/libglib-2.0.so.0(g_spawn_sync+0x230)[0x7f821b6ec130]
/lib64/libglib-2.0.so.0(g_spawn_command_line_sync+0x75)[0x7f821b6ec805]
/lib64/libgio-2.0.so.0(+0xb2100)[0x7f821c07e100]
/lib64/libgio-2.0.so.0(g_dbus_address_get_for_bus_sync+0x340)[0x7f821c07fad0]
/lib64/libgio-2.0.so.0(+0xbf2fe)[0x7f821c08b2fe]
/lib64/libgio-2.0.so.0(g_bus_get_sync+0x56)[0x7f821c090c46]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7b00)[0x7f821a512b00]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7bed)[0x7f821a512bed]
/lib64/libglib-2.0.so.0(+0x48d47)[0x7f821b6a1d47]
/lib64/libglib-2.0.so.0(g_main_context_dispatch+0x159)[0x7f821b6a5119]
/lib64/libglib-2.0.so.0(+0x4c478)[0x7f821b6a5478]
/lib64/libglib-2.0.so.0(g_main_context_iteration+0x2c)[0x7f821b6a552c]
/usr/lib64/gio/modules/libdconfsettings.so(+0x7d6d)[0x7f821a512d6d]
/lib64/libglib-2.0.so.0(+0x735b0)[0x7f821b6cc5b0]
/lib64/libpthread.so.0(+0x7ea5)[0x7f829cdddea5]
/lib64/libc.so.6(clone+0x6d)[0x7f829c6f1b0d]
======= Memory map: ========
680000000-80163f000 rw-p 00000000 00:00 0
80163f000-840000000 ---p 00000000 00:00 0
55c431546000-55c431547000 r-xp 00000000 fd:04 23498504                   /opt/apps/java/jdk-11.0.17+8/bin/java
55c431747000-55c431748000 r--p 00001000 fd:04 23498504                   /opt/apps/java/jdk-11.0.17+8/bin/java
55c431748000-55c431749000 rw-p 00002000 fd:04 23498504                   /opt/apps/java/jdk-11.0.17+8/bin/java
55c433331000-55c433755000 rw-p 00000000 00:00 0                          [heap]
7f8219509000-7f821950a000 ---p 00000000 00:00 0
7f821950a000-7f8219d0a000 rw-p 00000000 00:00 0
7f8219d0a000-7f8219d0b000 ---p 00000000 00:00 0
7f8219d0b000-7f821a50b000 rw-p 00000000 00:00 0
7f821a50b000-7f821a517000 r-xp 00000000 fd:00 33776740                   /usr/lib64/gio/modules/libdconfsettings.so
7f821a517000-7f821a716000 ---p 0000c000 fd:00 33776740                   /usr/lib64/gio/modules/libdconfsettings.so
7f821a716000-7f821a717000 r--p 0000b000 fd:00 33776740                   /usr/lib64/gio/modules/libdconfsettings.so
7f821a717000-7f821a718000 rw-p 0000c000 fd:00 33776740                   /usr/lib64/gio/modules/libdconfsettings.so
7f821a718000-7f821a71c000 r-xp 00000000 fd:00 102553411                  /usr/lib64/libuuid.so.1.3.0
7f821a71c000-7f821a91b000 ---p 00004000 fd:00 102553411                  /usr/lib64/libuuid.so.1.3.0
7f821a91b000-7f821a91c000 r--p 00003000 fd:00 102553411                  /usr/lib64/libuuid.so.1.3.0
7f821a91c000-7f821a91d000 rw-p 00004000 fd:00 102553411                  /usr/lib64/libuuid.so.1.3.0
7f821a91d000-7f821a959000 r-xp 00000000 fd:00 102553418                  /usr/lib64/libblkid.so.1.1.0
7f821a959000-7f821ab58000 ---p 0003c000 fd:00 102553418                  /usr/lib64/libblkid.so.1.1.0
7f821ab58000-7f821ab5b000 r--p 0003b000 fd:00 102553418                  /usr/lib64/libblkid.so.1.1.0
7f821ab5b000-7f821ab5c000 rw-p 0003e000 fd:00 102553418                  /usr/lib64/libblkid.so.1.1.0
7f821ab5c000-7f821ab5d000 rw-p 00000000 00:00 0
7f821ab5d000-7f821ab72000 r-xp 00000000 fd:00 103879708                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f821ab72000-7f821ad71000 ---p 00015000 fd:00 103879708                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f821ad71000-7f821ad72000 r--p 00014000 fd:00 103879708                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f821ad72000-7f821ad73000 rw-p 00015000 fd:00 103879708                  /usr/lib64/libgcc_s-4.8.5-20150702.so.1
7f821ad73000-7f821adb3000 r-xp 00000000 fd:00 102553420                  /usr/lib64/libmount.so.1.1.0
7f821adb3000-7f821afb3000 ---p 00040000 fd:00 102553420                  /usr/lib64/libmount.so.1.1.0
7f821afb3000-7f821afb4000 r--p 00040000 fd:00 102553420                  /usr/lib64/libmount.so.1.1.0
7f821afb4000-7f821afb5000 rw-p 00041000 fd:00 102553420                  /usr/lib64/libmount.so.1.1.0
7f821afb5000-7f821afb6000 rw-p 00000000 00:00 0
7f821afb6000-7f821afcc000 r-xp 00000000 fd:00 103211987                  /usr/lib64/libresolv-2.17.so

#############################################################

Thanks

M.Madhu

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/b2444171-95b2-45ed-8e9a-146407bc41dcn%40googlegroups.com.

[Mark Waite]

On Monday, December 5, 2022 at 1:02:18 PM UTC-7 Madhu Muchukota wrote:

Thanks Mark 

All I'm doing is just triggering/running a multi branch pipeline job from the controller UI.

I will be happy to share more information needed from my end. 

Unfortunately, somehow you have a failure mode that is different than the 100,000+ Jenkins controllers around the world that are running with Java 11.  I don't have any suggestions about what could be different.

Things that you could explore:

• Does the same failure happen
  • on a different computer running the same configuration?
  • on a different computer running without Tomcat?
  • on the same computer running without Tomcat?
    
• Were there recent upgrades to any components?
  • Core upgrade
  • Plugin upgrades
  • Operating system upgrades
• Were there any recent system configuration changes?
  • Were settings increased on the system make it more secure or to better defend it?

[M.Madhusudana Reddy]

Hi Mark - 

Looks like the Tomcat upgrade from 9.0.0 to 9.0.70 fixed the issue. I am no longer seeing that error/Jenkins controller getting crashed. 

Thanks all for all your help.

Regards,

M.Madhu

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/7d818b09-f66f-4681-8163-2c1b26070e88n%40googlegroups.com.

[Dirk Heinrichs]

Am Montag, dem 05.12.2022 um 11:28 -0600 schrieb M.Madhusudana Reddy:

My java version was jdk-11.0.16+8 and I updated this to the latest one in JAVA 11 this morning which is jdk-11.0.17+8 and I still see the same error.

Tomcat is of version 9.0.0.0

Ah, I see. We have different conditions, then. We're not running Jenkins in Tomcat here. And I checked again what our problem was with Java 11.0.16. It was a memory leak which led to Jenkins (or better: its JVM) being OOM-killed by the Linux kernel. THAT was fixed with 11.0.17.

Bye...

[M.Madhusudana Reddy]

I think I spoke too early.

Even with the newer version of Tomcat, I am seeing the issue.

Thanks

M.Madhu

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/dba5491ad09b0ca4eb8ba76c80ecf6dcb663316d.camel%40opentext.com.