What should we do with the CLI?

[Daniel Beck]

Hi everyone,

We've now had not one, but two unauthenticated remote code execution vulnerabilities in Jenkins:

- SECURITY-218 in https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2015-11-11
- SECURITY-360 in https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2016-11-16

Clearly, the current state of the CLI is a problem. Its Java serialization based protocol without prior authentication has now repeatedly resulted in remote code execution. For something like Jenkins, keeper of all the secrets, this is just unacceptable.

For now, we have a new system property that allows administrators to disable all of the CLI. I don't think this is good enough, but it's all we had time to do for SECURITY-360.
https://jenkins.io/doc/upgrade-guide/2.19/#disabling-the-cli

So, what are we going to do with the CLI? Remove it altogether? Only allow access via SSHD? Only allow authenticated use? Don't use remoting? Limit CLI remoting to specific users? WDYT?

Daniel

[Robert Collins]

Writing the CLI purely in terms of the REST API would be good -
security wise you then have a single path to secure, and no vulns
unique to the CLI; obviously some work is entailed :P.

-Rob

[Jesse Glick]

On Mon, Dec 12, 2016 at 11:03 PM, Daniel Beck <m...@beckweb.net> wrote:
> Only allow authenticated use?

Well the problems in these cases related to vulnerabilities in the
Remoting layer preceding the check for authentication. We could
probably make this safer by using a whitelist during the
authentication phase, and then later switch to the current blacklist
after we have confirmed user identity. The design problem at the root
of JENKINS-12543 may make this trickier to do for username/password
authentication.

> Don't use remoting?

This would be my preferred approach. The Remoting protocol has some
obvious power—you can do special operations on the client side—but it
seems inherently hard to secure. The performance is also poor—every
command has heavy startup overhead.

In practice most commands (or selectable command modes) can run just
fine using a simple protocol that only allows arguments, nonstreaming
stdin, and possibly streaming stdout/stderr. You can already use SSH
protocol, but to make things easier for both users and administrators
I would advocate an HTTP(S)-based protocol, normally authenticated
using API tokens just like Jenkins’ current “REST” APIs. Most commands
would be usable using a generic client like `wget`/`curl`, though we
could also provide a custom client, built perhaps in Go, which would
offer a similar interface to the current `java -jar jenkins-cli.jar`
and `ssh` styles. Commands like `build -v` would just need to hold
open an HTTP response, which is less of an issue if we can assume
Servlet 3.x. There would be no need for a special port, and standard
proxies or even SSO systems would work transparently.

Alternately we could just deprecate the whole CLI (ideally via
JENKINS-26463) and make sure that every command (or command mode) has
a complete replacement using existing REST endpoints. Most already do,
but not all. Again we would have the option of building a custom
client to make it easier to invoke the most commonly needed commands
(`build`, etc.), though there is no good way to make it extensible by
plugins in that case: you would need to hard-code command names and
options in the client, since it would need to map them to URL patterns
and form/data submission protocols. From a plugin developer standpoint
this would be the better option since you only need to understand and
think about one kind of non-GUI interaction with your features,
whereas currently you need to at least consider providing two.

[James Dumay]

There is a REST API in Blue Ocean that could be repurposed for something like this if someone is keen.

[Oliver Gondža]

I would say we have to keep the client (jenkins-cli.jar) and Java API
more or less untouched to keep things compatible for clients and
plugins. As already noted on CERT list, I lean towards removing the
remoting/channel support in favor of sshd/ssl (provided we do not have
to reimplement all the commands and clients) - which seems doable.
Breaking the handful of implementations that rely on channel seems
tolerable to me.

I strongly argue against command rewrite, have a look at our effort to
move from @CLIMethod to CLICommand (which only involves a subset of
commands). It took years, was a risky and heavy change and it is still
not done.

The idea of bridging REST API and CLI commands sounds appealing (write
once, invoke in 2 different ways!) but it seem to follow the pattern of
@CLIMethod. It is extra easy/convenient but is exactly as easy to break
it with a seemingly innocent change (especially if you only care for,
say, REST and your client depend on CLI). Putting aside than noone want
to add 5 more tests to make sure one annotation does what we hope it
does so we end up exposing use-cases noone ever bothered to cover by
tests. This is an old CLI struggle. Also, I do not think we can take
advantage of all the features of REST/CLI.

--
oliver

[Robert Sandell]

Disable CLI by default for new installations, like we enable security by default.

Perhaps most importantly change all the documentation that mentions CLI and replace with ssh.

We should also look into the commands that require a remoting channel to the client and see if we can rewrite them to not need one, or provide some other alternative.

Then, later, we can look into if we can guard the cli behind some permission so that paranoid admins can selectively enable it for some trusted humans or bots.

/B

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/c4792149-cccb-7c11-9c0b-3341ac945283%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Jesse Glick]

On Thu, Dec 15, 2016 at 2:58 AM, Oliver Gondža <ogo...@gmail.com> wrote:
> I would say we have to keep the client (jenkins-cli.jar) and Java API more
> or less untouched to keep things compatible for clients and plugins.

If JENKINS-26463 is implementable, this just means it all becomes a
plugin, which we could deprecate. With the Remoting transport deleted,
most of the security maintenance burden goes away.

> The idea of bridging REST API and CLI commands sounds appealing (write once,
> invoke in 2 different ways!) but it seem to follow the pattern of
> @CLIMethod. It is extra easy/convenient but is exactly as easy to break it
> with a seemingly innocent change (especially if you only care for, say, REST
> and your client depend on CLI).

So I think you are arguing against any kind of automatic exposure of
REST endpoints as CLI commands, which I do not think anyone was
proposing to begin with. At least, my two proposals were, briefly:

· retain the CLI API as is, but provide an alternate HTTP-based
transport (and, optionally, corresponding generic client)
· deprecate the CLI altogether, and only do REST (again, optionally,
creating a dedicated client for the most common REST operations in
core)

[Jesse Glick]

On Mon, Dec 12, 2016 at 11:03 PM, Daniel Beck <m...@beckweb.net> wrote:

> So, what are we going to do with the CLI?

Please read and comment (you can fork my gist if you just want to
suggest edits):

https://gist.github.com/jglick/9721427da892a9b2f75dc5bc09f8e6b3

[gsim...@netflix.com]

REST, REST, REST, REST.  Toss the CLI.  Don't worry about a replacement.  Don't just bolt onto the existing REST api.  Design a modern restful interface.  If you do that we can all make a client work with it.

Please and thank you!

[Jesse Glick]

On Thu, Mar 16, 2017 at 1:50 PM, gsimpson via Jenkins Developers
<jenkin...@googlegroups.com> wrote:
> Don't just bolt onto the existing REST api. Design a modern restful
> interface.

No objection to someone taking on such an effort, but as mentioned in
the “Alternatives” section, that is way out of scope for a near-term
solution to the problems we face—which require a quick development
effort and minimal disruption to existing CLI users.

[Daniel Beck]

> On 15.03.2017, at 23:40, Jesse Glick <jgl...@cloudbees.com> wrote:
>
> Please read and comment (you can fork my gist if you just want to
> suggest edits):
>
> https://gist.github.com/jglick/9721427da892a9b2f75dc5bc09f8e6b3

I'm very happy with this proposal. It's well thought out and discusses the pros and cons of the various approaches (and makes it clear I underestimated the necessary changes quite a bit…)

FWIW I agree with your response to the REST proposal -- this is about fixing an existing API and allowing existing users to migrate to a safer implementation with minimal effort, not about adding a new one.

[Jesse Glick]

Reminder that https://github.com/jenkinsci/jenkins/pull/2795 is
functionally complete and I would like reviews (also on a few small
linked PRs).

[Jesse Glick]

Merged toward 2.54.