Internal Server Error when using dispatch command, after upgrading from 1.3.0 to 1.4.4

[Maxime Lemanissier]

Hi,

I've just upgraded Rundeck from 1.3.0 to 1.4.4 (and I've followed http://rundeck.org/docs/upgrading/index.html)

My deployment script, which uses Rundeck dispatch command now fails :

rundeck@factory-server-01:/data/workspaces/rundeck/tools/bin$ ./dispatch --project Cougar --nodes=name=iprod-test-sa-02 --scriptfile /home/rundeck/deploy/temp/12655334.sh
error: Unable to queue the execution: Response content unexpectedly empty. Internal Server Error

Here is what I have in Rundeck log file :

2013-01-02 11:28:32,282 ERROR GrailsExceptionResolver - Exception occurred when processing request: [POST] /scheduledExecution/uploadAndExecute.xml
Stacktrace follows:
java.lang.NullPointerException: Cannot get property 'argString' on null object
        at ExecutionService.selectSecureOptionInput(ExecutionService.groovy:1084)
        at ExecutionService.selectSecureOptionInput(ExecutionService.groovy)
        at ExecutionService$selectSecureOptionInput.callCurrent(Unknown Source)
        at ExecutionService.executeScheduledExecution(ExecutionService.groovy:974)
        at ExecutionService$$FastClassByCGLIB$$38316c9d.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        at ExecutionService$$EnhancerByCGLIB$$c927e3d.executeScheduledExecution(<generated>)
        at ExecutionService$executeScheduledExecution.call(Unknown Source)
        at ScheduledExecutionController$_closure20_closure100.doCall(ScheduledExecutionController.groovy:1028)
        at ScheduledExecutionController$_closure20.doCall(ScheduledExecutionController.groovy:1021)
        at ScheduledExecutionController$_closure20.doCall(ScheduledExecutionController.groovy)

I've tried the most basic usage of dispatch command, just 1 node with 1 simple command and it always fails with this error.

Have I missed something during the upgrade?

[Greg Schueler]

Hi Maxime,

can you check your framework.properties file for the "framework.centraldispatcher.classname"? The classname value changed between 1.2 and 1.3, so if you have upgraded since 1.2 you may still have the old value.

It should be this:

framework.centraldispatcher.classname = com.dtolabs.client.services.RundeckAPICentralDispatcher

If it is the old value, please try updating it and restarting rundeck

On Wed, Jan 2, 2013 at 10:05 AM, Alex-SF <aho...@users.sourceforge.net> wrote:

Hi,

Was this a JAR install?

Thanks

[Maxime Lemanissier]

Hi Greg,

I had indeed the old value:

rundeck@factory-server-01:/data/workspaces/rundeck/etc$ grep 'framework.centraldispatcher.classname' framework.properties
framework.centraldispatcher.classname = com.dtolabs.client.services.RundeckCentralDispatcher

I replaced it with com.dtolabs.client.services.RundeckAPICentralDispatcher, but now Rundeck dispatch command fails with

error: Unable to queue the execution: Response content unexpectedly empty. Method Not Allowed

And no error in Rundeck log.
Is this related to ACL configuration? During the upgrade, I replaced my admin.aclpolicy by the one provided at http://rundeck.org/docs/upgrading/index.html#example-admin.aclpolicy

I tried then to rollback to 1.3 and I got the same error.
Had to revert framework.centraldispatcher.classname to its previous value

[Maxime Lemanissier]

Yes, it's a JAR install
Rundeck process started with :

#!/bin/bash
export RUNDECK_HOME='/data/workspaces/rundeck'
nohup java -Xmx1024m -XX:NewSize=1024k -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256m -jar ${RUNDECK_HOME}/rundeck-launcher.jar &> rundeck.log &

[Maxime Lemanissier]

Or is it related to https://github.com/dtolabs/rundeck/issues/228 ?

[Maxime Lemanissier]

I've just installed a fresh Rundeck to test my deployment script and it also fails with the same error, for version 1.4.4 and even 1.3.0.
I tried with this last version to use com.dtolabs.client.services.RundeckCentralDispatcher as dispatcher but it also fails with "error: Unable to queue the execution: Response content unexpectedly empty. OK"

This means that dispatch command has only ever worked from a Rundeck 1.2 installation?

[Maxime Lemanissier]

Problem fixed locally on maint-1.4.5 branch after cherry picked commit https://github.com/dtolabs/rundeck/commit/a166a30120bcee476f71ddbf035d1791c51c9283

Could this commit be merged to maint-1.4.5 branch?

[Greg Schueler]

Hi Maxime, 

Yes, I can add that to the maint-1.4.5 branch. Sorry for the trouble.

[Maxime Lemanissier]

Hi,

Ok, I've checked out maint-1.4.5 branch and rebuilt a new version. I don't have the previous problem anymore. But I have 2 new problems :-s

Rundeck fails with

Failed dispatching to node cupid@cerberus-19: com.dtolabs.rundeck.core.execution.service.FileCopierException: [jsch-scp] Failed copying the file: no remote path in cupid@cerberus-19:/tmp/135781279...@cerberus-19-dispatch-script.sh

when I try to dispatch (from Rundeck command line or from Rundeck webapp) a script to a remote node which contains a '@' in its name (here, cupid@cerberus-19). As the name of the node is used in the remote script name and @ is a special character, the remote copy of the script fails.
It works if I replace @ by - in the name of the node.
This was working with Rundeck 1.3.
Is this a bug or a new restriction (so I should then remove all @ in nodes name)?

I also get an error message in Rundeck webapp when displaying an execution output (for a job being run, or previously run) :
Error performing request (loadMoreOutputTail): /rundeck/execution/tailExecutionOutput.json
I can download the execution output, but not display it in Rundeck.

[Maxime Lemanissier]

For the last problem, I don't know if it's related, but in addition to upgrading from 1.3 to 1.4.5-dev, Rundeck is also running now on /rundeck context path (http://myServer:4440 => http://myServer:4440/rundeck) behind a front-end Apache using mod_proxy (=> http://myServer/rundeck)
I also have the same error if I bypass Apache by connecting to Rundeck with http://myServer:4440/rundeck, but it might be related to the usage of context /rundeck

[Greg Schueler]

Hi Maxime,

if you change the URL for the rundeck host, set the "grails.serverURL" value in the rundeck-config.properties: it should be set to the URL you expect to see in the browser (so include "/rundeck" if you have added that, and remove the port number if you are going to proxy from 80 to 4440)

Using @ in the hostname of a node is a shortcut for user@hostname.  This may not be a useful feature anymore.  If you can, avoid using that character, and if you think that is undesirable behavior please log it as a bug :).  (I think it is a holdover from controltier days).  It may be that the behavior changed between 1.3 and 1.4 around that usage that makes it now more strict. 

For the "loadMoreOutputTail" issue, is there any log information in the server log?

[Maxime Lemanissier]

Hi Greg,

I have removed '@' in node name as I'm already specifying the user with the appropriate tag (actually I have several users/connections on these servers, so I added user@.* in nodes names to distinguish them), now working fine ;)

To run Rundeck on /rundeck context, I modified startup script to add -Dserver.web.context=/rundeck and etc/framework.properties (framework.server.port and framework.rundeck.url properties) but indeed, I forgot to update server/config/rundeck-config.properties. I've done so but unfortunately I'm still facing the "loadMoreOutputTail" issue.
In addition to this error message, I also have to reload the page to update the status of a running job. Otherwise, the job keeps running endlessly (in the GUI).
I then rolled back all these modifications to run Rundeck on / context, and the issue is still here so it may not be related to Rundeck context.

The only clue I have found in Rundeck logs is
2013-01-11 10:16:45,928 [main] INFO  org.apache.commons.httpclient.HttpMethodDirector - Redirect requested but followRedirects is disabled
in var/logs/command.log

No others log trace that could relate to this issue.

[Greg Schueler]

Ok, are there any errors in the javascript console in your web browser?

[Maxime Lemanissier]

Hi Greg,

There is no javascript error but an HTTP 403.

[16:21:54,295] POST http://factory-server-01/rundeck/execution/tailExecutionOutput.json [HTTP/1.1 403 Forbidden 125ms]

{"error":"Not authorized for action \"read\" for Execution 9194","id":"9194","offset":"0","completed":false,"entries":[]}

I'm logged with admin account, using ACL http://rundeck.org/docs/upgrading/index.html#example-admin.aclpolicy

[Maxime Lemanissier]

Hi,

I reinstalled a fresh version of Rundeck, based on the same JAR used in production, and it's working fine.
I then copied the file etc/admin.aclpolicy of this fresh install to the production install. It's also working fine there.

Comparing the 2 ACL files :
rundeck@factory-server-01:/data/workspaces/rundeck/etc$ diff -y admin.aclpolicy.from-upgrade admin.aclpolicy
context:                                                        context:
  project: '.*' # all projects                                    project: '.*' # all projects
for:                                                            for:
  resource:                                                       resource:
 ...
  adhoc:                                                          adhoc:
    - allow: [run,kill] # allow running/killing adhoc jobs    |     - allow: '*' # allow read/running/killing adhoc jobs
  ...

It seems http://rundeck.org/docs/upgrading/index.html#example-admin.aclpolicy is missing the read permission on adhoc

[Greg Schueler]

Aha! thanks for tracking down the culprit there.  

[Greg Schueler]

FYI: I filed a bug https://github.com/dtolabs/rundeck/issues/280