HELP! Wildfly won't start up
2,489 views
Skip to first unread message
Scott Savarese
Nov 17, 2021, 9:34:59 AM11/17/21
to WildFly
Having problems getting Wildfly started up. I'm trying to install it for EJBCA following these instructions (https://download.primekey.se/docs/EJBCA-Enterprise/latest/WildFly_18___JBoss_EAP_7.3.html) but once I get to the part where it wants me to start Wildfly and create a credential store, I have problems. This isn't an EJBCA issue, as I haven't even gotten to the part where I put that on. This is simply Wildfly.
So, the instructions are basically: 1. Install Java 2. Download and unzip Wildfly 3. add memory to the JVM heap and 4. start wildfly. And that is all I've done. Its a brand new OS instance so nothing special to it.
What winds up happening when WIldfly starts up is that it simply goes into a futex or a nanosleep wait (according to strace) and doesn't actually start up (just consumes a ton of CPU cycles). Sometimes I'll get a line of output from Java. Sometimes it'll give me a bit. Once in a blue moon wildfly will actually start listening on the management ports. But when I run jboss-cli to create the credential store, it also spins in the nanosleep/futex wait eating up CPU resources.
I've tried a few things:
- Different OS platforms: Debian Buster and Rocky 8.4
- Different openjdk versions: 8 and 11. Both the headless package.
- Different wildfly versions: 10 (minimum ejbca supported), 18 (current EJBCA supported) and 25 (current wildfly version).
- Starting it up differently: Running standalone.sh from the command line, using systemd to start it, and then grabbing the command line systemd uses to run it from the command line.
- Increasing CPU and RAM from 2 core 4gb RAM to 4 core 8 gb RAM.
Why is this happening? Am I the only one seeing this? What am I missing? It has to be simple as I don't see anyone else complaining of this problem.
Thanks,
Scott
Note; This may be a repost. I don't see my original message in the groups messages currently. Not sure if it is being moderated or not. Figured I'd try again.
James Perkins
Nov 17, 2021, 9:38:40 AM11/17/21
to WildFly
Do you see error messages or anything like that? What step to you get to where it starts to fail?
Scott Savarese
Nov 17, 2021, 9:47:40 AM11/17/21
to WildFly
No error messages. It fails at wildfly startup. Sorry, my other post may be more descriptive.
Sometimes I get a single line of output from java. Sometimes it'll start up and try to load all the wildfly components but never finish. Very rarely do I get a message that the management port even comes up. Never any error messages or indications of a problem. The only thing abnormal I see (other than waiting forever and not getting a startup completed message) is that Java consumes an entire core (or more) and strace shows continuous nanosleep and/or futex waits. Even killing it in that state is hard. Ctrl-C (when started from command line) or systemctl stop don't work. I have kill -9 it to get it to die.
I'm not a Java guy. Systems Admin, yes. Python yes. But I know very little about JVM's and nothing about Wildfly.
Thanks,
Scott
James Perkins
Nov 17, 2021, 10:28:54 AM11/17/21
to WildFly
If you just download WIldFly and start it with ${JBOSS_HOME}/bin/standalone.sh does it work?
Scott Savarese
Nov 17, 2021, 10:41:43 AM11/17/21
to WildFly
No, here is some output that will help explain what I see... In this case, I remove my existing wildfly and untar from the source I downloaded. I then run a background date loop so you can see how it actually hanging... Then I start up wildfly. The first time it starts up fine. I control-C to stop it and start it again. The second time it hangs there and won't accept my control-C at all. Nor does it start up.
[root@ejbca1 ~]# cd /opt
[root@ejbca1 opt]# ls
new old wildfly wildfly-25.0.1.Final wildfly-25.0.1.Final.tar.gz
[root@ejbca1 opt]# rm -rf wildfly wildfly-25.0.1.Final
[root@ejbca1 opt]# tar -xzvf wildfly-25.0.1.Final.tar.gz > /dev/null
[root@ejbca1 opt]# ln -s wildfly-25.0.1.Final wildfly
[root@ejbca1 opt]# chown -Rh wildfly:wildfly wildfly-25.0.1.Final
[root@ejbca1 opt]# su - wildfly
Last login: Tue Nov 16 22:11:58 UTC 2021 on pts/0
[wildfly@ejbca1 ~]$ cd /opt/wildfly
[wildfly@ejbca1 ~]$ while :; do date; sleep 60; done &
[1] 1399
[wildfly@ejbca1 ~]$ Wed Nov 17 15:36:10 UTC 2021
[wildfly@ejbca1 ~]$ /opt/wildfly/bin/standalone.sh
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/wildfly
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:36:30,210 INFO [org.jboss.modules] (main) JBoss Modules version 1.12.0.Final
15:36:34,432 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.13.Final
15:36:34,543 INFO [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
15:36:35,513 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) starting
15:36:43,079 INFO [org.wildfly.security] (ServerService Thread Pool -- 26) ELY00001: WildFly Elytron version 1.17.1.Final
15:36:51,344 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
15:36:51,616 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.8.4.Final
15:36:51,700 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.8.4.Final
15:36:52,243 INFO [org.wildfly.extension.elytron.oidc._private] (ServerService Thread Pool -- 52) WFLYOIDC0001: Activating WildFly Elytron OIDC Subsystem
15:36:52,354 INFO [org.jboss.as.jaxrs] (ServerService Thread Pool -- 56) WFLYRS0016: RESTEasy version 4.7.2.Final
15:36:52,374 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 54) WFLYCLINF0001: Activating Infinispan subsystem.
15:36:52,388 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 74) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.
15:36:52,522 INFO [org.wildfly.extension.microprofile.opentracing] (ServerService Thread Pool -- 66) WFLYTRACEXT0001: Activating MicroProfile OpenTracing Subsystem
15:36:52,524 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 67) WFLYNAM0001: Activating Naming Subsystem
15:36:52,547 INFO [org.wildfly.extension.health] (ServerService Thread Pool -- 53) WFLYHEALTH0001: Activating Base Health Subsystem
15:36:52,728 INFO [org.wildfly.extension.microprofile.config.smallrye] (ServerService Thread Pool -- 64) WFLYCONF0001: Activating MicroProfile Config Subsystem
15:36:52,792 INFO [org.wildfly.extension.microprofile.jwt.smallrye] (ServerService Thread Pool -- 65) WFLYJWT0001: Activating MicroProfile JWT Subsystem
15:36:52,963 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 76) WFLYWS0002: Activating WebServices Extension
15:36:52,950 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 55) WFLYIO001: Worker 'default' has auto-configured to 4 IO threads with 32 max task threads based on your 2 available processors
15:36:53,354 INFO [org.wildfly.extension.metrics] (ServerService Thread Pool -- 63) WFLYMETRICS0001: Activating Base Metrics Subsystem
15:36:53,331 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 61) WFLYJSF0007: Activated the following Jakarta Server Faces Implementations: [main]
15:36:53,504 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 44) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
15:36:53,792 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 5.0.23.Final
15:36:54,567 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 2.2.12.Final starting
15:36:54,779 INFO [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting Jakarta Connectors Subsystem (WildFly/IronJacamar 1.5.2.Final)
15:36:55,410 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
15:36:55,461 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2
15:36:55,831 INFO [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
15:36:55,896 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 75) WFLYUT0014: Creating file handler for path '/opt/wildfly/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
15:36:56,432 INFO [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 32 (per class), which is derived from thread worker pool sizing.
15:36:56,515 INFO [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 8 (per class), which is derived from the number of CPUs on this host.
15:36:57,041 WARN [org.wildfly.extension.elytron] (MSC service thread 1-4) WFLYELY00023: KeyStore file '/opt/wildfly/standalone/configuration/application.keystore' does not exist. Used blank.
15:36:57,088 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
15:36:57,311 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) Queuing requests.
15:36:57,316 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0018: Host default-host starting
15:36:57,490 WARN [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY01084: KeyStore /opt/wildfly/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
15:36:58,902 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 127.0.0.1:8080
15:37:01,694 INFO [org.jboss.as.ejb3] (MSC service thread 1-3) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
15:37:02,606 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTPS listener https listening on 127.0.0.1:8443
15:37:02,679 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
15:37:03,300 INFO [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: WildFly Full cumulative patch ID is: base, one-off patches include: none
15:37:03,421 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/wildfly/standalone/deployments
15:37:03,931 INFO [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022052: Starting JBossWS 5.4.4.Final (Apache CXF 3.4.4)
15:37:05,196 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
15:37:05,229 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) started in 39104ms - Started 298 of 538 services (337 services are lazy, passive or on-demand)
15:37:05,245 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
15:37:05,247 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
Wed Nov 17 15:37:10 UTC 2021
^C15:37:12,700 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0272: Suspending server
15:37:12,716 INFO [org.jboss.as.ejb3] (Thread-2) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
15:37:12,734 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
15:37:13,051 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
15:37:13,071 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
15:37:13,091 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0019: Host default-host stopping
15:37:13,127 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow HTTPS listener https suspending
15:37:13,148 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTP listener default suspending
15:37:13,197 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
15:37:13,206 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 127.0.0.1:8443
15:37:13,223 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 2.2.12.Final stopping
15:37:13,294 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
15:37:13,461 INFO [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) stopped in 687ms
[wildfly@ejbca1 ~]$ /opt/wildfly/bin/standalone.sh
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/wildfly
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:37:20,821 INFO [org.jboss.modules] (main) JBoss Modules version 1.12.0.Final
Wed Nov 17 15:38:10 UTC 2021
Wed Nov 17 15:39:10 UTC 2021
^C^C^C^C^C^C^C^C
Wed Nov 17 15:40:10 UTC 2021[root@ejbca1 opt]# ls
new old wildfly wildfly-25.0.1.Final wildfly-25.0.1.Final.tar.gz
[root@ejbca1 opt]# rm -rf wildfly wildfly-25.0.1.Final
[root@ejbca1 opt]# tar -xzvf wildfly-25.0.1.Final.tar.gz > /dev/null
[root@ejbca1 opt]# ln -s wildfly-25.0.1.Final wildfly
[root@ejbca1 opt]# chown -Rh wildfly:wildfly wildfly-25.0.1.Final
[root@ejbca1 opt]# su - wildfly
Last login: Tue Nov 16 22:11:58 UTC 2021 on pts/0
[wildfly@ejbca1 ~]$ cd /opt/wildfly
[wildfly@ejbca1 ~]$ while :; do date; sleep 60; done &
[1] 1399
[wildfly@ejbca1 ~]$ Wed Nov 17 15:36:10 UTC 2021
[wildfly@ejbca1 ~]$ /opt/wildfly/bin/standalone.sh
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/wildfly
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:36:30,210 INFO [org.jboss.modules] (main) JBoss Modules version 1.12.0.Final
15:36:34,432 INFO [org.jboss.msc] (main) JBoss MSC version 1.4.13.Final
15:36:34,543 INFO [org.jboss.threads] (main) JBoss Threads version 2.4.0.Final
15:36:35,513 INFO [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) starting
15:36:43,079 INFO [org.wildfly.security] (ServerService Thread Pool -- 26) ELY00001: WildFly Elytron version 1.17.1.Final
15:36:51,344 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
15:36:51,616 INFO [org.xnio] (MSC service thread 1-2) XNIO version 3.8.4.Final
15:36:51,700 INFO [org.xnio.nio] (MSC service thread 1-2) XNIO NIO Implementation Version 3.8.4.Final
15:36:52,243 INFO [org.wildfly.extension.elytron.oidc._private] (ServerService Thread Pool -- 52) WFLYOIDC0001: Activating WildFly Elytron OIDC Subsystem
15:36:52,354 INFO [org.jboss.as.jaxrs] (ServerService Thread Pool -- 56) WFLYRS0016: RESTEasy version 4.7.2.Final
15:36:52,374 INFO [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 54) WFLYCLINF0001: Activating Infinispan subsystem.
15:36:52,388 WARN [org.jboss.as.txn] (ServerService Thread Pool -- 74) WFLYTX0013: The node-identifier attribute on the /subsystem=transactions is set to the default value. This is a danger for environments running multiple servers. Please make sure the attribute value is unique.
15:36:52,522 INFO [org.wildfly.extension.microprofile.opentracing] (ServerService Thread Pool -- 66) WFLYTRACEXT0001: Activating MicroProfile OpenTracing Subsystem
15:36:52,524 INFO [org.jboss.as.naming] (ServerService Thread Pool -- 67) WFLYNAM0001: Activating Naming Subsystem
15:36:52,547 INFO [org.wildfly.extension.health] (ServerService Thread Pool -- 53) WFLYHEALTH0001: Activating Base Health Subsystem
15:36:52,728 INFO [org.wildfly.extension.microprofile.config.smallrye] (ServerService Thread Pool -- 64) WFLYCONF0001: Activating MicroProfile Config Subsystem
15:36:52,792 INFO [org.wildfly.extension.microprofile.jwt.smallrye] (ServerService Thread Pool -- 65) WFLYJWT0001: Activating MicroProfile JWT Subsystem
15:36:52,963 INFO [org.jboss.as.webservices] (ServerService Thread Pool -- 76) WFLYWS0002: Activating WebServices Extension
15:36:52,950 INFO [org.wildfly.extension.io] (ServerService Thread Pool -- 55) WFLYIO001: Worker 'default' has auto-configured to 4 IO threads with 32 max task threads based on your 2 available processors
15:36:53,354 INFO [org.wildfly.extension.metrics] (ServerService Thread Pool -- 63) WFLYMETRICS0001: Activating Base Metrics Subsystem
15:36:53,331 INFO [org.jboss.as.jsf] (ServerService Thread Pool -- 61) WFLYJSF0007: Activated the following Jakarta Server Faces Implementations: [main]
15:36:53,504 INFO [org.jboss.as.connector.subsystems.datasources] (ServerService Thread Pool -- 44) WFLYJCA0004: Deploying JDBC-compliant driver class org.h2.Driver (version 1.4)
15:36:53,792 INFO [org.jboss.remoting] (MSC service thread 1-4) JBoss Remoting version 5.0.23.Final
15:36:54,567 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0003: Undertow 2.2.12.Final starting
15:36:54,779 INFO [org.jboss.as.connector] (MSC service thread 1-4) WFLYJCA0009: Starting Jakarta Connectors Subsystem (WildFly/IronJacamar 1.5.2.Final)
15:36:55,410 INFO [org.jboss.as.naming] (MSC service thread 1-4) WFLYNAM0003: Starting Naming Service
15:36:55,461 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0018: Started Driver service with driver-name = h2
15:36:55,831 INFO [org.jboss.as.mail.extension] (MSC service thread 1-1) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
15:36:55,896 INFO [org.wildfly.extension.undertow] (ServerService Thread Pool -- 75) WFLYUT0014: Creating file handler for path '/opt/wildfly/welcome-content' with options [directory-listing: 'false', follow-symlink: 'false', case-sensitive: 'true', safe-symlink-paths: '[]']
15:36:56,432 INFO [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 32 (per class), which is derived from thread worker pool sizing.
15:36:56,515 INFO [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 8 (per class), which is derived from the number of CPUs on this host.
15:36:57,041 WARN [org.wildfly.extension.elytron] (MSC service thread 1-4) WFLYELY00023: KeyStore file '/opt/wildfly/standalone/configuration/application.keystore' does not exist. Used blank.
15:36:57,088 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
15:36:57,311 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) Queuing requests.
15:36:57,316 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0018: Host default-host starting
15:36:57,490 WARN [org.wildfly.extension.elytron] (MSC service thread 1-1) WFLYELY01084: KeyStore /opt/wildfly/standalone/configuration/application.keystore not found, it will be auto generated on first use with a self-signed certificate for host localhost
15:36:58,902 INFO [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener default listening on 127.0.0.1:8080
15:37:01,694 INFO [org.jboss.as.ejb3] (MSC service thread 1-3) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
15:37:02,606 INFO [org.wildfly.extension.undertow] (MSC service thread 1-3) WFLYUT0006: Undertow HTTPS listener https listening on 127.0.0.1:8443
15:37:02,679 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-2) WFLYJCA0001: Bound data source [java:jboss/datasources/ExampleDS]
15:37:03,300 INFO [org.jboss.as.patching] (MSC service thread 1-1) WFLYPAT0050: WildFly Full cumulative patch ID is: base, one-off patches include: none
15:37:03,421 INFO [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/wildfly/standalone/deployments
15:37:03,931 INFO [org.jboss.ws.common.management] (MSC service thread 1-4) JBWS022052: Starting JBossWS 5.4.4.Final (Apache CXF 3.4.4)
15:37:05,196 INFO [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0212: Resuming server
15:37:05,229 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) started in 39104ms - Started 298 of 538 services (337 services are lazy, passive or on-demand)
15:37:05,245 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0060: Http management interface listening on http://127.0.0.1:9990/management
15:37:05,247 INFO [org.jboss.as] (Controller Boot Thread) WFLYSRV0051: Admin console listening on http://127.0.0.1:9990
Wed Nov 17 15:37:10 UTC 2021
^C15:37:12,700 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0272: Suspending server
15:37:12,716 INFO [org.jboss.as.ejb3] (Thread-2) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
15:37:12,734 INFO [org.jboss.as.server] (Thread-2) WFLYSRV0220: Server shutdown has been requested via an OS signal
15:37:13,051 INFO [org.jboss.as.connector.subsystems.datasources] (MSC service thread 1-1) WFLYJCA0010: Unbound data source [java:jboss/datasources/ExampleDS]
15:37:13,071 INFO [org.jboss.as.mail.extension] (MSC service thread 1-4) WFLYMAIL0002: Unbound mail session [java:jboss/mail/Default]
15:37:13,091 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0019: Host default-host stopping
15:37:13,127 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0008: Undertow HTTPS listener https suspending
15:37:13,148 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0008: Undertow HTTP listener default suspending
15:37:13,197 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0007: Undertow HTTP listener default stopped, was bound to 127.0.0.1:8080
15:37:13,206 INFO [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0007: Undertow HTTPS listener https stopped, was bound to 127.0.0.1:8443
15:37:13,223 INFO [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0004: Undertow 2.2.12.Final stopping
15:37:13,294 INFO [org.jboss.as.connector.deployers.jdbc] (MSC service thread 1-2) WFLYJCA0019: Stopped Driver service with driver-name = h2
15:37:13,461 INFO [org.jboss.as] (MSC service thread 1-3) WFLYSRV0050: WildFly Full 25.0.1.Final (WildFly Core 17.0.3.Final) stopped in 687ms
[wildfly@ejbca1 ~]$ /opt/wildfly/bin/standalone.sh
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /opt/wildfly
JAVA: java
JAVA_OPTS: -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
15:37:20,821 INFO [org.jboss.modules] (main) JBoss Modules version 1.12.0.Final
Wed Nov 17 15:38:10 UTC 2021
Wed Nov 17 15:39:10 UTC 2021
^C^C^C^C^C^C^C^C
James Perkins
Nov 17, 2021, 10:49:50 AM11/17/21
to WildFly
On Wednesday, November 17, 2021 at 7:41:43 AM UTC-8 googl...@scottsavarese.com wrote:
No, here is some output that will help explain what I see... In this case, I remove my existing wildfly and untar from the source I downloaded. I then run a background date loop so you can see how it actually hanging... Then I start up wildfly. The first time it starts up fine. I control-C to stop it and start it again. The second time it hangs there and won't accept my control-C at all. Nor does it start up.
No changes on the second start? That definitely seems odd.
Scott Savarese
Nov 17, 2021, 10:53:42 AM11/17/21
to WildFly
Nope. That output is exactly what was on my screen... An exact cut and paste. Ctrl-C to stop it the first then and then up arrow return to run it again the same way.
If other people had the same problem I'd say it is an issue with Wildfly. But I'm the only one having this problem. It has to be a me issue. I'm clearly doing something wrong or missing something. Did stopping it with Ctrl-C break it? Is there a prerequisite that is missing?
Java version, by the way, is currently 1.8:
[wildfly@ejbca1 ~]$ rpm -qa | grep java
tzdata-java-2021e-1.el8.noarch
java-1.8.0-openjdk-headless-1.8.0.312.b07-1.el8_4.x86_64
javapackages-filesystem-5.3.0-2.module+el8.3.0+125+5da1ae29.noarch
tzdata-java-2021e-1.el8.noarch
java-1.8.0-openjdk-headless-1.8.0.312.b07-1.el8_4.x86_64
javapackages-filesystem-5.3.0-2.module+el8.3.0+125+5da1ae29.noarch
James Perkins
Nov 17, 2021, 4:52:37 PM11/17/21
to WildFly
Very strange indeed. What OS are you on? I could possibly try booting up a docker container or VM and see if I can reproduce it. I use CTRL+C all the time to stop WildFly.
If run jps to see which JVM's are running then if it's just one called "jboss-modules.jar" you could run kill -3 <PID> to get a thread dump.
Scott Savarese
Nov 17, 2021, 5:20:03 PM11/17/21
to WildFly
So, kill -3 isn't working (even if the signal is sent as root). Which makes sense actually... If I can't ctrl-c the process when it is hung/spinning like this, then it is not listening to interrupts properly. Kill -9 always works because that is how linux handles emergency interrupts. Sad we can't see where this thing is hanging though.
As for the OS... I'm using Rocky Linux 8.4. Literally a fresh install with a single drive and one big disk. In terms of packages I always do the bare minimum. No servers, tools, or x-windows. Once booted up I patch to current (in case the installer doesn't). Then I install openjdk (see the RPM version from the prior message), drop wildfly into /opt and run the commands you saw above.
The hardware platform is KVM on x86_64. I can send you the VM's config file if that helps. Not sure why it would though.
Thanks for helping,
Scott
James Perkins
Nov 17, 2021, 5:43:23 PM11/17/21
to WildFly
Ah okay. You could try something like jstack -l -e <PID>, but that would require installing the full JDK.
I did try with the rockylinux/rockylinux:8.4, in podman, but I was able to start WildFly as many times as I wanted. I'm not too sure what the issue could be really.
Scott Savarese
Nov 17, 2021, 6:38:55 PM11/17/21
to WildFly
Thanks, no dice with jstack either. Tried it from the wildfly and root user. Looks like it tries to take over the process completely (pid 3069 no longer appears in top output and the jstack pid eats CPU like crazy. Had to kill -9 it. I did however, strace the java process. If I grep out the futex and clock looping junk, there is nothing really there... I'm including that info as well at the very bottom.
[root@ejbca1 ~]# ps -ef | grep java
wildfly 3069 3021 99 23:29 pts/0 00:01:34 java -D[Standalone] -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/wildfly/standalone/configuration/logging.properties -jar /opt/wildfly/jboss-modules.jar -mp /opt/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/wildfly -Djboss.server.base.dir=/opt/wildfly/standalone
root 3105 3085 0 23:30 pts/3 00:00:00 grep --color=auto java
[root@ejbca1 ~]# su - wildfly
Last login: Wed Nov 17 23:29:21 UTC 2021 on pts/0
[wildfly@ejbca1 ~]$ jstack -l -e 3069
Usage:
jstack [-l] <pid>
(to connect to running process)
jstack -F [-m] [-l] <pid>
(to connect to a hung process)
jstack [-m] [-l] <executable> <core>
(to connect to a core file)
jstack [-m] [-l] [server_id@]<remote server IP or hostname>
(to connect to a remote debug server)
Options:
-F to force a thread dump. Use when jstack <pid> does not respond (process is hung)
-m to print both java and native frames (mixed mode)
-l long listing. Prints additional information about locks
-h or -help to print this help message
[wildfly@ejbca1 ~]$ jstack -l 3069
3069: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
[wildfly@ejbca1 ~]$ jstack -l -F 3069
Attaching to process ID 3069, please wait...
^C^C^C^C^C
wildfly 3069 3021 99 23:29 pts/0 00:01:34 java -D[Standalone] -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true -Dorg.jboss.boot.log.file=/opt/wildfly/standalone/log/server.log -Dlogging.configuration=file:/opt/wildfly/standalone/configuration/logging.properties -jar /opt/wildfly/jboss-modules.jar -mp /opt/wildfly/modules org.jboss.as.standalone -Djboss.home.dir=/opt/wildfly -Djboss.server.base.dir=/opt/wildfly/standalone
root 3105 3085 0 23:30 pts/3 00:00:00 grep --color=auto java
[root@ejbca1 ~]# su - wildfly
Last login: Wed Nov 17 23:29:21 UTC 2021 on pts/0
[wildfly@ejbca1 ~]$ jstack -l -e 3069
Usage:
jstack [-l] <pid>
(to connect to running process)
jstack -F [-m] [-l] <pid>
(to connect to a hung process)
jstack [-m] [-l] <executable> <core>
(to connect to a core file)
jstack [-m] [-l] [server_id@]<remote server IP or hostname>
(to connect to a remote debug server)
Options:
-F to force a thread dump. Use when jstack <pid> does not respond (process is hung)
-m to print both java and native frames (mixed mode)
-l long listing. Prints additional information about locks
-h or -help to print this help message
[wildfly@ejbca1 ~]$ jstack -l 3069
3069: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
[wildfly@ejbca1 ~]$ jstack -l -F 3069
Attaching to process ID 3069, please wait...
^C^C^C^C^C
[root@ejbca1 ~]# strace -fp 3069 2>&1 | grep -v sched | grep -v clock | grep -v futex
strace: Process 3069 attached with 17 threads
[pid 3143] [ Process PID=3143 runs in x32 mode. ]
[pid 3080] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out)
strace: Process 3069 attached with 17 threads
[pid 3143] [ Process PID=3143 runs in x32 mode. ]
[pid 3080] <... restart_syscall resumed>) = -1 ETIMEDOUT (Connection timed out)
Scott Savarese
Nov 17, 2021, 6:40:33 PM11/17/21
to WildFly
So, it works for you and not for me... OK. That's interesting... What java are you using? I was using headless and then installed the devel package, but maybe a different environment is correct? Did you do anything else to your VM? Any non-default settings? Maybe there are packages I'm missing?
James Perkins
Nov 17, 2021, 6:45:40 PM11/17/21
to WildFly
On Wednesday, November 17, 2021 at 3:40:33 PM UTC-8 googl...@scottsavarese.com wrote:
So, it works for you and not for me... OK. That's interesting... What java are you using? I was using headless and then installed the devel package, but maybe a different environment is correct? Did you do anything else to your VM? Any non-default settings? Maybe there are packages I'm missing?
I tried to replicate your environment as close as I could. I suppose the kernel would be different since I was using podman though. I did just install the headless Java package for testing. The only other thing I installed was wget, but I doubt that changed anything :)
This is my list of what I have installed https://gist.github.com/jamezp/f7d2ae89bc9470ad656d25ee8e3afc51.
Scott Savarese
Nov 18, 2021, 8:07:53 AM11/18/21
to WildFly
So, your list makes sense. Even a minimal install on my end had more packages than you. But I didn't see anything that you had that I didn't, so I'm not missing anything.
The whole purpose of Java is to abstract the kernel and OS from the application. After 25 years of Java being around, I'm sure that they solved the problem and it isn't kernel version specific. That leaves 1 thought... What about timeouts? KVM isn't the fastest performing hypervisor around. I'm rebuilding my VM now and it should only take a few minutes, but for some reason KVM is taking a very long time to do any task. I'm wondering if Java or Wildfly have timeouts that if they are missed cause Wildfly not to start properly. If so, is there a way to increase timeouts in Java?
Thanks,
Scott
Scott Savarese
Nov 18, 2021, 10:59:07 AM11/18/21
to WildFly
So this is definitely something related to system performance. Either that is a timeout getting missed and forcing the wait loop, or because the hardware is slower things get into an eternal lock state, or something similar.
I rebuilt my OS, this time focused on performance. My underlying hypervisor didn't have virtualization features enabled properly and the VM's all were slow and groggy. Once that was fixed and things moved 100x faster, Wildfly installation works and I can start/stop it as many times as I want. Things happened in less time.
There is definitely something going on related to slow performance... Happy to help investigate more, but now at least y'all have a definitive place to look and can reproduce the problem better.
James Perkins
Nov 18, 2021, 12:31:00 PM11/18/21
to WildFly
Awesome to hear, thank you for the information.
With regards to the wait loop. It looks like the entry point that it stalls on. Why it does I'm not too sure, but the fact that it's only printing the first log line it makes me thing it must be there.
Scott Savarese
Nov 18, 2021, 1:09:20 PM11/18/21
to WildFly
OK, sounds good. I'm closing this out now... at least for my part. I'll let y'all look at cleaning up the issue with wildfly so that someone else with a slow computer (raspberry pi) won't have issues too!
THanks for your help getting this worked out for me. I appreciate it!
Scott
Reply all
Reply to author
Forward
0 new messages