Rundeck starts and goes into Dead state

[Jabraj G]

root@app101:~
11:18:19 # service rundeckd status

rundeckd dead but pid file exists

My Java version is

11:18:57 # java -version
openjdk version "1.8.0_262"
OpenJDK Runtime Environment (build 1.8.0_262-b10)
OpenJDK 64-Bit Server VM (build 25.262-b10, mixed mode)

[rac...@rundeck.com]

Hi Jabraj,

Which Rundeck version and kind of installation are you using (RPM/DEB/WAR)? Which install/upgrade process are you followed? Which Operating System (and version) are you using? Can you test with a clean install on a virtual machine? Any clue if you do `journalctl -u rundeckd.service`?

Regards!

[Jabraj G]

Rundeck was working fine earlier, stopped working after patching few packages.

11:41:31 # rpm -qa | grep rundeck
rundeck-config-3.0.22.20190512-1.201905130059.noarch
rundeck-2.6.9-1.21.GA.noarch

11:42:33 # cat /etc/os-release
NAME="Oracle Linux Server"
VERSION="6.10"
ID="ol"
VERSION_ID="6.10"
PRETTY_NAME="Oracle Linux Server 6.10"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:6:10:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 6"
ORACLE_BUGZILLA_PRODUCT_VERSION=6.10
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=6.10

[rac...@rundeck.com]

Hi Jabrag,

It seems that you've installed rundeck 2.6.9 and rundeck 3.0.22-config packages (if you're using rundeck 2.6 you need rundeck-2.6-config, if you're using rundeck 3.0.22 you need rundeck-3.0.22-config, try to keep the same version of rundeck and rundeck-config packages (that's not necessary if you like to upgrade to 3.1/3.2 or 3.3 branches).

You can obtain the rpm's from here.

Hope it helps!

[Jabraj G]

Downgraded the config to 2.6.9 on both app nodes.

00:59:16 # rpm -qa | grep runde
rundeck-config-2.6.9-1.21.GA.noarch
rundeck-2.6.9-1.21.GA.noarch

01:05:35 # rpm -qa | grep rund
rundeck-config-2.6.9-1.21.GA.noarch
rundeck-2.6.9-1.21.GA.noarch

Below are the logs.

ro...@app101.com:/var/log/rundeck ( app101.com )

01:12:10 # tail service.log
    at org.eclipse.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:173)
    at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:311)
    at org.eclipse.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:251)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)
    at org.eclipse.jetty.server.Server.doStart(Server.java:272)
    at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:59)
    at com.dtolabs.rundeck.RunServer.run(RunServer.java:120)
    at com.dtolabs.rundeck.RunServer.main(RunServer.java:80)
2020-08-28 01:11:19.831:INFO:/:Closing Spring root WebApplicationContext

2020-08-28 01:11:19.835:INFO:oejsh.ContextHandler:stopped o.e.j.w.WebAppContext{/,file:/var/lib/rundeck/exp/webapp/},/var/lib/rundeck/exp/webapp

ro...@app101.com:/var/log/rundeck ( app101.com )
01:12:15 # tail rundeck.log
    ... 134 more
2020-08-28 01:00:33,640 [main] ERROR com.dtolabs.rundeck.core.plugins.ScriptPluginProviderLoader - Plugin not loaded: Found no py-winrm-plugin-2.0.1/plugin.yaml within: /var/lib/rundeck/libext/py-winrm-plugin-2.0.1.zip
2020-08-28 01:00:33,640 [main] ERROR com.dtolabs.rundeck.core.plugins.ScriptPluginScanner - Skipping plugin file: metadata was invalid: /var/lib/rundeck/libext/py-winrm-plugin-2.0.1.zip
2020-08-28 01:00:36,370 [main] INFO  grails.app.conf.BootStrap - Starting Rundeck 2.6.9-1...
2020-08-28 01:00:36,370 [main] INFO  grails.app.conf.BootStrap - using rdeck.base config property: /var/lib/rundeck
2020-08-28 01:00:36,379 [main] INFO  grails.app.conf.BootStrap - loaded configuration: /etc/rundeck/framework.properties
2020-08-28 01:00:36,451 [main] INFO  grails.app.conf.BootStrap - RSS feeds disabled
2020-08-28 01:00:36,451 [main] INFO  grails.app.conf.BootStrap - Preauthentication is disabled
2020-08-28 01:00:36,476 [main] INFO  grails.app.conf.BootStrap - Rundeck is ACTIVE: executions can be run.

2020-08-28 01:00:36,678 [main] INFO  grails.app.conf.BootStrap - Rundeck startup finished in 342ms

Rundeck starts on both app nodes and goes to dead state after a while.

[rac...@rundeck.com]

Hi Jabraj,

Can you post the full service.log output at the moment of launch rundeck? (tail -f /var/log/rundeck/service.log). Please hide or change any potentially sensitive information.

Thanks!

[Jabraj G]

Please find the log here from since the issue started occurring.

https://gist.github.com/gjabraj/471383ccf128539da4645d331a2eb34d

[Rahul Saxena]

May be you need to increase JVM xms value. It can be related to memory.

--
You received this message because you are subscribed to the Google Groups "rundeck-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rundeck-discu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rundeck-discuss/48f60e6f-a9e6-4988-b5b8-b04fe8798bc2n%40googlegroups.com.

[Jabraj G]

05:12:12 # ./jpstat.sh
=====  ==============================  =======  =======  ======  ======  =====
 PID                Name               CurHeap  MaxHeap  CurRAM  MaxRAM  %_CPU
=====  ==============================  =======  =======  ======  ======  =====
2443   RunServer                           101      101     128     128      0

[rac...@rundeck.com]

Hi Jabraj,

Checking your service.log you can see the following line:

java.net.BindException: Address already in use

This means that Rundeck can’t launch because can’t listen on the default TCP port (4440 by default) perhaps another service is using it, you’re using some security software which blocks that port or you need to reconfigure Rundeck to listen on another TCP port, you can identify the process with lsof -i :4440.

Hope it helps!

​

[Jabraj G]

I downgraded the java version to 1.7.0 and the rundeckd service runs fine now.

Thank you so much to everyone.

My next roadblock is to make the haproxy service run fine.

[rac...@rundeck.com]

Great news Jabraj :-)

[Jabraj G]

Even though the Rundeck service is running fine i see 503 error.

05:08:05 # curl -v localhost:4440

* About to connect() to localhost port 4440 (#0)
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 4440 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: localhost:4440
> Accept: */*
>
< HTTP/1.1 503 Service Unavailable
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 1287
< Server: Jetty(7.6.0.v20120127)
<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Service Unavailable</title>
</head>
<body>
<h2>HTTP ERROR: 503</h2>
<p>Problem accessing /. Reason:
<pre>    Service Unavailable</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>                                     
                                                
</body>
</html>
* Connection #0 to host localhost left intact

* Closing connection #0

[rac...@rundeck.com]

Hi Jabraj,

Can you share the service.log content at the moment of accessing to Rundeck (usually at /var/log/rundeck/ path) to take a look?

Regards!

​

[Jabraj G]

Please find the service.log here - https://gist.github.com/gjabraj/471383ccf128539da4645d331a2eb34d

[rac...@rundeck.com]

Hi Jabraj,

Checking the logs I can see this: Error requesting URL Resource Model Source: http://127.0.0.1:9980//gamma_continuous_patching: com.dtolabs.rundeck.core.common.FileUpdaterException: java.net.ConnectException: Connection refused (Connection refused) which means that some URL Resource Model Source is unreachable by Rundeck, check your Resource Models, and make sure that’s Rudneck can reach them (for example testing your remote Resource Model providers from Rundeck server).

Hope it helps!

PD: Post typo fixed :-)

​