UNAUTHORIZED for api/invalid

[JMAO]

Rundeck community, 

I am looking to use job forecase feature and experiencing an issue described as below.

The installation is Rundeck 4.3.0-20220602 and RD cli 2.0.0 on Ubuntu 18.04 LTS. Without any customization, rd cli tool works if RD_URL points to http://localhost:4440/api/41 to get job forecast information. Web UI works as expected too.

However, after a customization to the server URL for sitting behind an https reverse proxy, this approach stopped working.

The highlight of the customization includes:

grails.serverURL=httpS://SERVERNAME/rundeck

framework.server.url = httpS://SERVERNAME/rundeck

By putting an Apache web server in the front, users can access the server with httpS://SERVERNAME/rundeck. This part works as expected.

But, rd does not seem to like it. With RD_URL=https://SERVERNAME/rundeck/api/41, it complains:

$ rd projects list -v
java.lang.IllegalStateException: Password Authentication failed, expected a successful response.
        at org.rundeck.client.util.FormAuthInterceptor.authenticate(FormAuthInterceptor.java:82)
        at org.rundeck.client.util.FormAuthInterceptor.intercept(FormAuthInterceptor.java:59)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
        at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:204)
        at org.rundeck.client.util.Client.checkErrorDowngradable(Client.java:166)
        at org.rundeck.client.util.Client.apiCallDowngradable(Client.java:606)
        at org.rundeck.client.tool.commands.RdToolImpl.apiCallDowngradable(RdToolImpl.java:107)
        at org.rundeck.client.tool.commands.RdToolImpl.apiCall(RdToolImpl.java:78)
        at org.rundeck.client.tool.extension.BaseCommand.apiCall(BaseCommand.java:24)
        at org.rundeck.client.tool.commands.Projects.list(Projects.java:58)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1972)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2358)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2314)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2316)
        at picocli.CommandLine.execute(CommandLine.java:2078)
        at org.rundeck.client.tool.Main.main(Main.java:154)

Apache log shows a 403 error

"POST /rundeck/api/41/j_security_check HTTP/1.1" 403 534 "-" "rd-cli-tool/2.0.0 (rundeck-cli/2.0.0)"

Rundeck log shows:

ERROR interceptors.AuthorizationInterceptor - (unauthenticated) UNAUTHORIZED for api/invalid

Does anyone have experience with this? 

Thank you.

[rac...@rundeck.com]

Hi JMAO,

Did you try this configuration from the RD CLI side?

Regards.

[JMAO]

I have an Apache reverse proxy in the front of rundeck. I tried to put :4440 back to accommodate this bypass URL but no lock.

Rundeck is configured with:

server.servlet.context-path=/rundeck
server.useForwardHeaders=true

grails.serverURL=httpS://SERVERNAME/rundeck

framework.server.url = httpS://SERVERNAME/rundeck

Apache proxy running on the same server with Rundeck is configured as:

ProxyPass /rundeck http://localhost:4440/rundeck

ProxyPassReverse /rundeck http://localhost:4440/rundeck

I feel the SSL and reverse proxy made this setup more complicated for rd to work with a URL like https://SERVERNAME/rundeck/api/41.

Some further attempts using RD_URL like http://SERVERNAME:4440/rundeck/api/41 or  httpS://SERVERNAME/rundeck/api/41 still show "java.lang.IllegalStateException: Password Authentication failed, expected a successful response." Otherwise, I see 'jobs forecast: requires API >= 31 (current: 29)"

Is there a way to change the server default API to 41?

Thanks!

[JMAO]

Used RD_DEBUG=3 to generate some debug info from the server itself.

$ export RD_URL=http://localhost:4440/rundeck/api/41

$ rd jobs forecast -i d937431a-f621-4b50-ad56-5ac69a68daac

# Including extension: org.rundeck.client.tool.commands.repository.Plugins
# Including extension: org.rundeck.client.tool.commands.enterprise.license.License
# Including extension: org.rundeck.client.tool.commands.enterprise.cluster.Cluster
# Including extension: org.rundeck.client.ext.acl.Acl
Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: --> GET http://localhost:4440/rundeck/ http/1.1

...

Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: --> POST http://localhost:4440/rundeck/api/41/j_security_check http/1.1

...

INFO: Cookie: JSESSIONID=node01df6ff2n7b43lxs7axeg4pund11.node0
Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO:
Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: j_username=USER&j_password=SECRET
Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: --> END POST (43-byte body)
Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: <-- 403 Forbidden http://localhost:4440/rundeck/api/41/j_security_check (10ms)

...

Jul 13, 2022 4:30:45 PM okhttp3.internal.platform.Platform log
INFO: {"error":true,"apiversion":41,"errorCode":"unauthorized","message":"(unauthenticated) is not authorized for: /rundeck/api/41/j_security_check"}

...

When RD_URL=httpS://SERVERNAME/rundeck/api/41, the result is the same as above (except localhost -> SERVERNAME).

Where is rundeck handling the authentication for this use case? Maybe an alternative is to permit the access like this.

Thanks.

[rac...@rundeck.com]

Hi JMAO,

Currently, the only way to switch to API 41 is to move to the latest version, you can test that version on a non-prod env to ensure that works in your environment.

Regards!

[rac...@rundeck.com]

And here, you can see the minimal API version of that feature. In that site you can see that value for all endpoints :-)

[JMAO]

I am running Rundeck server 4.3.0 and RD 2.0.0. Are you saying once I upgrade to the latest Rundeck 4.3.1 and RD 2.0.1, the server side API will be 41 for RD?

Thank you.

[JMAO]

Ok, I found a workaround which uses RD_TOKEN instead of RD_USERNAME/RD_PASSWORD. Then for the same server side configuration, rd jobs forecast works with RD_URL=httpS://SERVERNAME/rundeck/api/41.

It will still be good to know why api/41 is not the default for rd.

Thanks.

[rac...@rundeck.com]

Hey JMAO,

I built a similar SSL reverse proxy test environment to understand what is happening and I see the same behavior:

Forecast using user/password way.

Forecast using token auth way (it works).

Regarding the API version issue, I saw that weird error when the RD_URL env var is set as "https://yourhost/rdeck" instead of "https://yourhost/rdeck/api/41" and set the user/password auth from the RD CLI side, so, it seems a part of the same bug.

So, could you open a new issue here? Thanks *a lot* for your feedback!

Greetings!

[JMAO]

Opened an issue https://github.com/rundeck/rundeck-cli/issues/465. Thanks a lot for looking into it.