no isolation between workspaces of different users (security problem)

[Cristian Zamfir]

Hi, 

I setup Jenkins 1.431 with project-based matrix authorization, LDAP authentication, and Tomcat 6. 

Most users have their own jobs and it is important to isolate users from each other. For instance, they cannot read or delete the codebase of another user.

User have only read and build access to their jobs, but they are not allowed to configure a job. 

However, it turns out users can read each other's workspace quite easily from within one of the tests triggered by a build (for instance a malicious test). If we configure a shell build step that does `make test`, this would execute the malicious script.  

For instance, if Jenkins is located in /var/lib/jenkins/ and one user has /var/lib/jenkins/workspace/user1-job and the other has /var/lib/jenkins/workspace/user2-job, user1 can write a test that does cat ../user2-job/* and see the entire codebase of user2 or do rm -rf ../../ and delete the entire Jenkins directory. This is possible because all these files are owned by tomcat6

This means that Jenkins does not provide the required isolation, so we cannot use Jenkins to run tests, which was the whole point of using it in the first place.

I searched the mailing list, but did not find an answer. Is there a known lightweight solution to this problem? 

The alternative seems to be to have a different slave/VM for each user, or a chroot, but we have more than 100 users, so this is will add a lot of overhead. 

Thank you very much for your help, 

Cristi

[Christoph Kutzinski]

Jenkins does not (and cannot I guess) provide a user isolation on that
level.
If you need it, you probably will have to provide a slave node for each
user.

[Cristian Zamfir]

Thank you for your answer, I realized that as well, I will have to setup a slave/user. However, I am still pondering which is the most lightweight solution that requires the least hardware resources. I could either setup a chroot with and a ramdisk or run a KVM virtual machine for each user. 

Does anyone have experience with setting up many (around 100) lightweight sandboxes?

Thank you, 

Cristi

[Les Mikesell]

You'll probably want some way to let Jenkins spin virtual machines up
and down on demand. That has been discussed here regarding VMware
ESXi guests but I'm not sure about the resolution or how it would work
with KVM.

--
Les Mikesell
lesmi...@gmail.com

[Cristian Zamfir]

Hi Les, 

Thank you for your answer. 

It looks like I could use the Libvirt Slaves Plugin https://wiki.jenkins-ci.org/display/JENKINS/Libvirt+Slaves+Plugin for that. 

Do people use it? I see the latest release is from more than an year ago. 

It will also be quite challenging to spin VMs up and down, even if I use suspend/resume.

 Thanks, 

Cristi

[Mark Waite]

---

From: Cristian Zamfir <zamfi...@gmail.com>
To: jenkins...@googlegroups.com
Sent: Thursday, October 6, 2011 2:14 PM
Subject: Re: no isolation between workspaces of different users (security problem)

My understanding was that chroot is lighter than virtual machines because it accepts that all the processes are executing in the same kernel.  "http://aplawrence.com/Drag/kvm_virtualization.html" describes containers with the phrase "Since they share the same kernel and such they are very efficient, very fast. Almost no overhead associated with this sort of 'VM'. Fast I/O, fast disk access, efficient memory usage, etc etc. You can run dozens and dozens of these things with the same resource it takes just to run a few Xen hosts."

I confess however, that my attempt to configure a chroot jail to host a Jenkins slave node was a failure.  It was my first and only attempt to create a chroot jail environment, so the failure was likely due to my inexperience more than lack of capacity in the core technology.  

I was able to configure my own VirtualBox virtual machines on first attempt, which left me feeling that a virtual machine setup was easier for a novice like me.

Mark Waite

[Udo Rader]

Thanks for bringing this entire thing up. We are having exactly the same
concerns,
http://groups.google.com/group/jenkinsci-users/browse_thread/thread/b301845e0274790d#

Having 100 virtual machines around for just the purpose of being jenkins
slaves sees to be an incredible overhead to me.

The probable easiest and most lightweight way to achieve seperated
sandboxes is to have multiple tomcat instances running on the build
machine, each of them running as a seperate user.

IMHO this is a major security flaw, because it is not uncommon having
tomcat running as root and thus exposing the entire build box to
malicious tests injected via jenkins. There should be a way to
explicitly configure the *system* user that builds and tests run under.

[Jesse Farinacci]

Greetings,

On Fri, Oct 7, 2011 at 3:37 AM, Udo Rader <lis...@bestsolution.at> wrote:
> The probable easiest and most lightweight way to achieve seperated
> sandboxes is to have multiple tomcat instances running on the build
> machine, each of them running as a seperate user.

This is the best way, if you actually require isolated environments
and Tomcat. I provide what I think is an even easier mechanism below.

> IMHO this is a major security flaw, because it is not uncommon having
> tomcat running as root and thus exposing the entire build box to
> malicious tests injected via jenkins. There should be a way to
> explicitly configure the *system* user that builds and tests run under.

Uhhh... so, because you're too lazy to configure Tomcat properly, then
it's a Jenkins security flaw? Sorry, no. If you run any service as
root, then you're opening yourself up to just about anything; and
where is it common to run Tomcat as root? This isn't the default.
There's simply no way for Jenkins to alter capabilities for the
container which it is running under.. that job is up to you. If you
can't be bothered, I'd suggest you use one of the pre-assembled binary
packages (.rpm, .deb, etc) which already drop permissions by running
non-privileged.

The best way to achieve the kind of thing you're looking for, right
now, is to simply create a new user account for each Jenkins instance,
copy the .war file into its home directory, and execute it directly
passing in hostname and port parameters. It is easily scriptable and
provides the discussed levels of isolation.

-Jesse

--
There are 10 types of people in this world, those
that can read binary and those that can not.

[Udo Rader]

On 10/07/2011 11:50 AM, Jesse Farinacci wrote:
> Uhhh... so, because you're too lazy to configure Tomcat properly, then
> it's a Jenkins security flaw? Sorry, no. If you run any service as
> root, then you're opening yourself up to just about anything; and
> where is it common to run Tomcat as root? This isn't the default.
> There's simply no way for Jenkins to alter capabilities for the
> container which it is running under.. that job is up to you. If you
> can't be bothered, I'd suggest you use one of the pre-assembled binary
> packages (.rpm, .deb, etc) which already drop permissions by running
> non-privileged.

Thanks for your response.

No, I am neither too lazy to configure tomcat properly nor am I
blaming/flaming anyone.

I just would like to see a possibility to run "jobs" as any configurable
system user. "A job" in that sense can be anything build related.
Invocing commands such as mvn or shell scripts in general as a user
different from the tomcat user can be easily achieved using sudo on unix
systems for example.

And I did not say that it was common running tomcat as root, I said that
it is not uncommon (note the fine difference). I am very aware that most
preconfigured servlet engines shipped by the distributions are running
under a dedicated user, yet most of them not are not chroot'ed (which is
absolutely not trivial to configure ie. for tomcat).

> The best way to achieve the kind of thing you're looking for, right
> now, is to simply create a new user account for each Jenkins instance,
> copy the .war file into its home directory, and execute it directly
> passing in hostname and port parameters. It is easily scriptable and
> provides the discussed levels of isolation.

Yes, this sounds like the most reasonable alternative so far, thanks for
this idea.

Yet still, in a default Jenkins installation users are managed within
Jenkins and those users can access each other's workspace without any
limitation. I would call this a security flaw, but please correct me if
I am wrong :)

[Jesse Farinacci]

Greetings,

On Fri, Oct 7, 2011 at 6:14 AM, Udo Rader <lis...@bestsolution.at> wrote:
> I just would like to see a possibility to run "jobs" as any configurable
> system user. "A job" in that sense can be anything build related.
> Invocing commands such as mvn or shell scripts in general as a user
> different from the tomcat user can be easily achieved using sudo on unix
> systems for example.

This topic comes up about once every 6 months. I wouldn't say it is
common, but it isn't uncommon ;-) Can you describe a general method
for a Java process to do sudo-like activities? I'm not aware of
anything like, or if it is possible in a Java framework.

> Yet still, in a default Jenkins installation users are managed within
> Jenkins and those users can access each other's workspace without any
> limitation. I would call this a security flaw, but please correct me if
> I am wrong :)

I wouldn't call it a security flaw, but some language is required;
maybe gap? Jenkins security mechanisms all deal with the
administration and operation of Jenkins itself, not with what Jenkins
executes. The only thing that comes to mind is enabling a Java
security manager, but I know next to nothing about this technology,
and I suspect it isn't going to fit with a general user-based
security.

Maybe you could investigate creating a general Jenkins security
manager which, when typically gets enabled at startup via -D or some
such. I do know that the security manager could restrict file access
via checkRead / checkWrite and perhaps veto any attempt to do a file
io against something outside its workspace? I don't know.

This kind of security is evidently not a high (or medium) priority for
anyone actively working on Hudson/Jenkins. You must be the change you
wish to see in the project.

[Udo Rader]

On 10/07/2011 03:15 PM, Jesse Farinacci wrote:
> Greetings,
>
> On Fri, Oct 7, 2011 at 6:14 AM, Udo Rader <lis...@bestsolution.at> wrote:
>> I just would like to see a possibility to run "jobs" as any configurable
>> system user. "A job" in that sense can be anything build related.
>> Invocing commands such as mvn or shell scripts in general as a user
>> different from the tomcat user can be easily achieved using sudo on unix
>> systems for example.
>
> This topic comes up about once every 6 months. I wouldn't say it is
> common, but it isn't uncommon ;-) Can you describe a general method
> for a Java process to do sudo-like activities? I'm not aware of
> anything like, or if it is possible in a Java framework.

I must admit that don't know too much about the internals of how Jenkins
operates. For that part, I know that Jenkins already spawns shell
processes in order to perform arbitrary tasks:

https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/tasks/Shell.java

The same applies for maven:

https://github.com/jenkinsci/jenkins/blob/master/core/src/main/java/hudson/tasks/Maven.java

It may not be a "general method for a Java process to do sudo-like
activities" but for those two parts it should be sufficient if those
executables were run by the logged in Jenkins-user using his/her Jenkins
credentials, e.g. by invoking the shell using su - $USER -c $COMMAND or
similar. Like I said, my knowledge about Jenkins is limited (we are only
in the process of evaluating if it will eventually fit our
requirements), so I might just miss something important.

But I understand that Jenkins is working differently right now.

> I wouldn't call it a security flaw, but some language is required;
> maybe gap? Jenkins security mechanisms all deal with the
> administration and operation of Jenkins itself, not with what Jenkins
> executes. The only thing that comes to mind is enabling a Java
> security manager, but I know next to nothing about this technology,
> and I suspect it isn't going to fit with a general user-based
> security.

Yes, I can hardly see where the Java security manager would be helpful
when build tasks and tests tasks are started via the (shell) command
line. Running shell commands requires restriction on the shell level, so
at least I can only think about either running them sudo like or
chroot'ing the shell processes.

> Maybe you could investigate creating a general Jenkins security
> manager which, when typically gets enabled at startup via -D or some
> such. I do know that the security manager could restrict file access
> via checkRead / checkWrite and perhaps veto any attempt to do a file
> io against something outside its workspace? I don't know.
>
> This kind of security is evidently not a high (or medium) priority for
> anyone actively working on Hudson/Jenkins. You must be the change you
> wish to see in the project.

Thanks for that insight, but I am definitively not sure if that task
would fit my limited shoes ;)

And if the OP is still listening: sorry for hijacking your thread ...

[Mark Waite]

Cristian Zamfir wrote:
>
> Most users have their own jobs and it is important to isolate users from
> each other. For instance, they cannot read or delete the codebase of
> another
> user.
>
> User have only read and build access to their jobs, but they are not
> allowed
> to configure a job.
>
> However, it turns out users can read each other's workspace quite easily
> from within one of the tests triggered by a build (for instance a
> malicious
> test). If we configure a shell build step that does `make test`, this
> would
> execute the malicious script.
>

> I searched the mailing list, but did not find an answer. Is there a known
> lightweight solution to this problem?
>
> The alternative seems to be to have a different slave/VM for each user, or
> a
> chroot, but we have more than 100 users, so this is will add a lot of
> overhead.
>

I think you've understood the issue and the alternatives very well. Jenkins
Jobs are executed with the userid of the process on the node where they are
running. It seems like what you're envisioning is that the Jenkins slave
process would switch to a different user before executing the job on behalf
of that user. Jenkins doesn't do that.

You correctly noted that one solution is to have slave agents which are user
specific. That has the benefit that you could run the slave agent as that
user in order to take advantage of operating system access checks for that
user. It has the disadvantage you noted, overhead required to configure
slave nodes for each user.

You could also consider "pooling" slave agents and assigning users to pools.
The users within a "pool" would still be able to see each other's work, but
not outside their pool (assuming you limited jobs to execute within only a
specific pool of users). Pooling would probably allow load balancing of
jobs between slave agents more effective and would likely reduce your
overhead to managing several pools instead of 100+ independent slave
entities.

You could also consider shifting responsibility for the problem, accepting
that if a malicious developer performs this sort of attack then it is a
management problem rather than a continuous integration problem. I'm sure
there are places where that is not acceptable (government security work,
military research work, etc.), but it is an alternative to consider if
you're not in one of those places.

Thanks,
Mark Waite

--
View this message in context: http://jenkins.361315.n4.nabble.com/no-isolation-between-workspaces-of-different-users-security-problem-tp3853930p3865252.html
Sent from the Jenkins users mailing list archive at Nabble.com.

[Cristian Zamfir]

Hi, thank you very much for the ideas. 

I realized that the only way to do this is to have an isolated environment for each project. I was still a bit surprised by the fact that Jenkins does not at least run builds as a separate user. I guess it is harder to `su` in a portable way. However, I did not understand this lack of isolation from the documentation until I actually tried it out. I still think it is risky because one project could accidentally `rm -rf ../../` and that is enough to delete the entire Jenkins install. 

In choosing the best solution, the main problem was the tradeoff between security/isolation and allocated resources (disk space, RAM, CPU, network isolation, etc). There are many solutions out there, ranging from chroot to OS containers (e.g., LXC, OpenVZ), and VM-based (KVM, Xen, etc.). 

I now have a reasonably secure setup for Jenkins for running 80+ independent projects in reasonably good isolation without wasting a lot of resources. Since others may be interested, I will briefly describe my setup here.

I setup a pool of machines that are *not* Jenkins slaves. Instead, each such machine has a unix user for each project.  

Each user is chroot-ed using the ChrootDirectory option of SSH. I created a single chroot jail (using debootstrap) for each machine and chmod-ed 700 each /home/$user.  

Each project is build by first rsync-ing the project src to the machine where it will be built . The jobs execute remotely with the uid associated with each project. Don't forget to mount -bind /proc in the chroot if you want Java to run. 

This means there is virtually no overhead from doing all this isolation, but the isolation is not exactly perfect, the network is shared, etc. 

Note that this only works if your Jenkins users are not allowed to configure the jobs (this can be done if you allow project-based security). This is because all the setup (ssh-ing into the remote chroot-ed accounts, rsync-ing the source code, etc.) is done inside the shell build script.  If users can change that, they can do pretty much anything they want with your server. In this case, the only solution is to have a Jenkins slave machine per project. However, in my case that would have wasted a lot of resources and would not really scale. 

Cristi

[K96]

Hi everyone,

I created a wiki sub page under 'Securing Jenkins'.
https://wiki.jenkins-ci.org/display/JENKINS/Security+Problems+relating+to+Job+Execution

I'd like to summarize and discuss security issues relating to this
topic on this page. Any kind of opinions and suggestions are very
welcome.

Regards,
@kuhcrow

> View this message in context:http://jenkins.361315.n4.nabble.com/no-isolation-between-workspaces-o...

[Cristian Zamfir]

Hi, 

Thanks for looking at this. I added some notes and also a thought on using the Java Security Manager to make sure all these problems go away. I will also quote it here: 

"A better idea than chroots, OS containers, or VMs may be to use the Java Security Manager for enabling isolation between projects. AFAIK it enables accesses to specific files/paths in the file system, it ensures users cannot kill other processes and can restrict access to specific network ports. This solution has the advantage that it is portable. It may also be quite easy to allow the admin of the job to configure the security settings of a job via a plugin. Then the users that "own" the job can run any shell script under the constraints imposed by the security manager. "

I am only incidentally familiar with the Java Security Manager, but at first sight it looks possible to use it. 

Best, 

Cristi

[K96]

Hi,
Very thanks for follwing up.

> A job can also read/write Jenkins configuration files because jobs are executed with thee same user id (e.g., tomcat6).

Do you mean "build on master" ?

We strongly recommend our developers not to perform build on master
(configure system as # of executors = 0),
rather create slaves and perform build on slaves even when these slave
nodes are on the same master machine,
because we can easily separate OS accounts for slave nodes from that
of master, and control thier resources and privileges.

> I am only incidentally familiar with the Java Security Manager,

Using Java Security Manager looks great,
but I and many Jenkins users are not so familiar with that.
So if you attach your sample settings for Java Security Manager,
it will be very useful to us.

Regards,
@kuhcrow (Baba Takesh, Tokyo, Japan)

On 10月21日, 午前7:08, Cristian Zamfir <zamfir.e...@gmail.com> wrote:
> Hi,
> Thanks for looking at this. I added some notes and also a thought on using
> the Java Security Manager to make sure all these problems go away. I will
> also quote it here:
>
> "A better idea than chroots, OS containers, or VMs may be to use the Java

> Security Manager<http://download.oracle.com/javase/1,5.0/docs/api/java/lang/SecurityMa...> for

[Cristian Zamfir]

On Oct 21, 2011, at 7:11 AM, K96 wrote:

> Hi,
> Very thanks for follwing up.
>
>> A job can also read/write Jenkins configuration files because jobs are executed with thee same user id (e.g., tomcat6).
>
> Do you mean "build on master" ?
>
> We strongly recommend our developers not to perform build on master
> (configure system as # of executors = 0),
> rather create slaves and perform build on slaves even when these slave
> nodes are on the same master machine,
> because we can easily separate OS accounts for slave nodes from that
> of master, and control thier resources and privileges.

I did not understand that one can bound a slave to an OS account. Can you please point me to the documentation where this is mentioned? It would not solve all the problems, but maybe it is good enough for some setups. The Java Security Manager seems more configurable.

>
>> I am only incidentally familiar with the Java Security Manager,
>
> Using Java Security Manager looks great,
> but I and many Jenkins users are not so familiar with that.
> So if you attach your sample settings for Java Security Manager,
> it will be very useful to us.

I do not have much experience with the Java Security Manager, but I understand that it can be used programmatically, so it could be used internally by Jenkins to enforce a configuration provided by a user. For instance, the user could select that a specific job is only allowed to access a certain port range, this setting would be parsed by Jenkins, which would get a reference to the security manager (e.g., System.getSecurityManager()) and enforce the user-provided policy setting. I did not test this, but I believe it is possible.

Thanks,
Cristi