Securely obtain the Jenkins package and public key

[Abhijith Chandrashekar]

Hello all,

I work with a tech company where we're trying to establish a pristine build environment for all of our products. As part of this, we are looking to create a Jenkins CI server from scratch using the most secure methods possible. This would be on an underlying CentOS 6.2 machine. From reading the guide on installing Jenkins on CentOS/RedHat I see that the package and the key are both obtained over http as - 

wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo

and 

rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key

This raises possibilities of a Man-in-the-middle attack compromising the integrity of the repo or the key or both. To avoid this, is there a way to obtain the package and the key securely? This could either be over HTTPS, SFTP or by exchanging PGP keys with the owner and then transporting it over email.

If there's a better place to post this question, please inform.

Thanks,

Abhijith

[Johannes Wienke]

Hi,

I'd like to underline this issue. With the increasing use of Jenkins, it
might actually become an interesting target for attacks, as in some
environments the jenkins installation is tighly integrated into the
system infrastructure, e.g. generating binary packages for linux
distributions etc.

Cheers,
Johannes

[Daniel Beck]

On 10.01.2014, at 16:56, Johannes Wienke <jwi...@techfak.uni-bielefeld.de> wrote:

> an interesting target for attacks

Jenkins security is a joke. You can find security issues without trying, even in core. And the process to resolve them seems to be really broken.

[teilo]

> an interesting target for attacks

Jenkins security is a joke. You can find security issues without trying, even in core. And the process to resolve them seems to be really broken.

Have you helped to improve this situation by actually reporting them via the proper channels?

https://wiki.jenkins-ci.org/display/JENKINS/Security+Advisories

 /James 

[Daniel Beck]

On 10.01.2014, at 18:11, teilo <teilo+...@teilo.net> wrote:

> Have you helped to improve this situation by actually reporting them via the proper channels?

Yes. That's why I consider the resolution process to be broken. The "proper channels" don't work.

The first security issue I reported was SECURITY-35 in email-ext (installed on 30% of all instances) which I re-filed publicly as JENKINS-15213 after getting no response for three months. The email-ext author informed me he didn't receive any information from those with access to the private issue tracker and quickly fixed the problem. Another five months later, a response to SECURITY-35 arrived, explaining that, because the process was broken, some issues were overlooked.

Then there's the ongoing SECURITY-87: I reported that anyone can trivially DoS any Jenkins instance (including those where anonymous has no permissions) on 13 Aug 2013. AFAICT the problem persists. Sure, it's not privilege escalation, but still annoying if you're running a public instance.

Another example of a security issue in current LTS is JENKINS-20800, which I originally reported to Cloudbees Enterprise support in a non-security context (so it was filed publicly). I only later found it to be trivially exploitable on any Jenkins instance by anyone. Four weeks ago, the fix was backported early to LTS, likely because I asked for it on the dev list. But 1.532.2 still doesn't even have an RC. Should I have reported it separately as a security issue? Maybe, but the developers were aware of this issue, and by then I'd mostly given up on "proper channels".

[Daniel Beck]

On 08.01.2014, at 23:08, Abhijith Chandrashekar <abhijith.ch...@gmail.com> wrote:

> This raises possibilities of a Man-in-the-middle attack compromising the integrity of the repo or the key or both.

The war packages themselves are signed by Kohsuke. You can use the tool 'jarsigner' to verify.

Of course, you'd need a secure way to make sure it's actually his signature, but that should be easier than changing the entire distribution chain.

[abhijith chandrashekar]

> Of course, you'd need a secure way to make sure it's actually his signature, but that should be easier than changing the entire distribution chain.

That's exactly the problem. Any ideas on how I can do that?

Thanks,
Abhijith

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/3O8vpxrWZH8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[teilo]

On Sunday, 12 January 2014 22:20:17 UTC, Abhijith Chandrashekar wrote:

> Of course, you'd need a secure way to make sure it's actually his signature, but that should be easier than changing the entire distribution chain.

That's exactly the problem. Any ideas on how I can do that?

Thanks,
Abhijith

http://kohsuke.org/about/pgp/

But if you are that security paranoid then you should download the sources, inspect them (and the history)  and then compile them yourself every release (like you do for all the plugins right!?).

/James 

[abhijith chandrashekar]

Thanks Tielo. Although I do think downloading sources and inspecting them would not only be overkill, but also not a foolproof way of ensuring security. What if the source files are mangled during download?

The only few ways I can think of are

1. to get the binaries and keys/hashes over PGP email from somebody on the inside
2. obtain it over HTTPS on their website
3. using SFTP or
4. by having someone on the inside ship a DVD that contains the binary + public key.

I'm trying other avenues (CloudBees, for example) but there do not seem to be any provisions to do this.

Thanks,
Abhijith

--

[Richard Mortimer]

Hi Abhijith,

I think you need to read about chains of trust. Everything that you
suggest below is at best hiding what you are downloading from an
observer. It doesn't stop man in the middle attacks or guarantee that
the contents were not corrupted during transit.

As James suggested all you need to do is convince yourself that
Kohsuke's public PGP key is actually his. Once you have done that you
can verify that the signed release is actually what Kohsuke meant to
release.

How paranoid you are is really up to you. You could just trust the key
that is publicly posted by Kohsuke, attempt to verify the key via a web
of trust, or even try to convince Kohsuke to meet you in person to
verify the key.

Really I suspect that most of that is just overkill. You just need to
ensure that you have a repeatable build environment where you have a
reasonable level of trust that the software you downloaded is fit for
purpose. Many Linux OS distributions (Debian for example) have all of
the individual packages are linked back to a widely published central
key allowing you to detect any tampering in transit.

The same applies to any commercial arrangements with appropriate
software suppliers. You would need to get & trust a public key from them
to allow you to verify that you receive exactly what was intended to be
sent.

Regards

Richard

> <mailto:jenkinsci-users%2Bunsu...@googlegroups.com>.

> For more options, visit https://groups.google.com/groups/opt_out.
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send

[Kohsuke Kawaguchi]

I've modified the server infrastructure so that the PGP public key is available at https://jenkins-ci.org/jenkins-ci.org.key. These keys are used to sign native packages. In the next release I'll update the those package documentaion to refer to this key location in HTTPS.

Aside from that, regardless of the packaging, the war file is signed with my X509 certificate. That certificate is itself signed by CoMoDo, so you can establish a known trust chain back up to the well-trusted root CAs.

2014/1/8 Abhijith Chandrashekar <abhijith.ch...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Kohsuke Kawaguchi

[Kohsuke Kawaguchi]

OK, that's embarassing.

Indeed we haven't figured out how to communicate security problems to plugin developers. Sometimes it's not obvious who to talk to, and even when it is obvious, we haven't configured JIRA to let us grant read access on issue-by-issue basis.

Issues not getting a timely enough attention is unfortunate, but aside from trying to add more people to the jenkinsci-cert group (which we are always trying), I'm not sure how to resolve that.

Daniel, given the level of activity you commit in the core, I feel like you could help us fixing those issues, in addition to finding them.

2014/1/10 Daniel Beck <m...@beckweb.net>

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Kohsuke Kawaguchi

[Marius Gedminas]

On Wed, Jan 15, 2014 at 06:57:07PM -0800, Kohsuke Kawaguchi wrote:
> OK, that's embarassing.
>
> Indeed we haven't figured out how to communicate security problems to
> plugin developers. Sometimes it's not obvious who to talk to, and even when
> it is obvious, we haven't configured JIRA to let us grant read access on
> issue-by-issue basis.
>
> Issues not getting a timely enough attention is unfortunate, but aside from
> trying to add more people to the jenkinsci-cert group (which we are always
> trying), I'm not sure how to resolve that.
>
> Daniel, given the level of activity you commit in the core, I feel like you
> could help us fixing those issues, in addition to finding them.

As an outsider, I'm glad to see you care about security.


What about packaging? The Debian packages available from upstream start
the web server on the public IP right from the post-inst script, before
you get a chance to set up any security. This allows remote code
execution as user jenkins.

(The Jenkins package in the Ubuntu archive is configured to start the
web server on localhost only, which sounds like a sane mitigation
strategy for me.)

Should I file a bug? (I have filed a few bugs about packaging issues
(one involving data loss), but haven't seen any response at all after
many months, which made me stop caring. I have some small expertise
with Debian packaging, and I'm willing to post patches, if I think they
will not be silently ignored for months.)

Marius Gedminas
--
Open Source Software: There are days when I can't figure out whether I'm living
in a Socialist utopia or a Libertarian one.
-- Alex Future Bokov

[Kohsuke Kawaguchi]

On 01/16/2014 06:19 AM, Marius Gedminas wrote:
> On Wed, Jan 15, 2014 at 06:57:07PM -0800, Kohsuke Kawaguchi wrote:
>> OK, that's embarassing.
>>
>> Indeed we haven't figured out how to communicate security problems to
>> plugin developers. Sometimes it's not obvious who to talk to, and even when
>> it is obvious, we haven't configured JIRA to let us grant read access on
>> issue-by-issue basis.
>>
>> Issues not getting a timely enough attention is unfortunate, but aside from
>> trying to add more people to the jenkinsci-cert group (which we are always
>> trying), I'm not sure how to resolve that.
>>
>> Daniel, given the level of activity you commit in the core, I feel like you
>> could help us fixing those issues, in addition to finding them.
>
> As an outsider, I'm glad to see you care about security.
>
>
> What about packaging? The Debian packages available from upstream start
> the web server on the public IP right from the post-inst script, before
> you get a chance to set up any security. This allows remote code
> execution as user jenkins.
>
> (The Jenkins package in the Ubuntu archive is configured to start the
> web server on localhost only, which sounds like a sane mitigation
> strategy for me.)

Yeah, this is a good idea.

Perhaps a slight variation of it is to keep listening to these ports
like we do today but to show the message saying non-local access is
blocked until you acknowledge what you are doing?

That way, it would work across platforms and it's less confusing to
those who are less experienced in the system administration?

Something like that can be packaged as a plugin, which makes the
integration process easy --- you can just go create code yourself and
other people can try/use it, and we'd only have to bundle it in the core.

> Should I file a bug? (I have filed a few bugs about packaging issues
> (one involving data loss), but haven't seen any response at all after
> many months, which made me stop caring. I have some small expertise
> with Debian packaging, and I'm willing to post patches, if I think they
> will not be silently ignored for months.)

OK, our apologies. Do you know which ones they are? We want to take a look.

--
Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/
Try Jenkins Enterprise, our professional version of Jenkins

[Marius Gedminas]

On Thu, Jan 16, 2014 at 09:33:34AM -0800, Kohsuke Kawaguchi wrote:
> On 01/16/2014 06:19 AM, Marius Gedminas wrote:
> >What about packaging? The Debian packages available from upstream start
> >the web server on the public IP right from the post-inst script, before
> >you get a chance to set up any security. This allows remote code
> >execution as user jenkins.
> >
> >(The Jenkins package in the Ubuntu archive is configured to start the
> >web server on localhost only, which sounds like a sane mitigation
> >strategy for me.)
>
> Yeah, this is a good idea.

I filed https://issues.jenkins-ci.org/browse/JENKINS-21417 for this.

> Perhaps a slight variation of it is to keep listening to these ports
> like we do today but to show the message saying non-local access is
> blocked until you acknowledge what you are doing?
>
> That way, it would work across platforms and it's less confusing to
> those who are less experienced in the system administration?
>
> Something like that can be packaged as a plugin, which makes the
> integration process easy --- you can just go create code yourself
> and other people can try/use it, and we'd only have to bundle it in
> the core.

Hm. Yes, a page explaining what to do would be more user-friendly.
E.g. it could tell the admin how to use SSH port-forwarding to connect
to the Jenkins and get full access to the configuration pages, or what
config files to edit to set a password.

(I'm not volunteering to implement this. My familiarity with Java ended
with a brief university course in 2000.)

> >Should I file a bug? (I have filed a few bugs about packaging issues
> >(one involving data loss), but haven't seen any response at all after
> >many months, which made me stop caring. I have some small expertise
> >with Debian packaging, and I'm willing to post patches, if I think they
> >will not be silently ignored for months.)
>
> OK, our apologies. Do you know which ones they are? We want to take a look.

https://issues.jenkins-ci.org/browse/JENKINS-18797 and
https://issues.jenkins-ci.org/browse/JENKINS-18798 were the most
annoying, with the 1st one clobbering my configuration (luckily, I had a
backup).

https://issues.jenkins-ci.org/browse/JENKINS-19329 is related to 18798.

Perhaps I picked the wrong component? It was the only one that
mentioned Debian packaging.

Marius Gedminas
--
6 out of 7 dwarves are not Happy.