Square Hack Week project: secure updates for RubyGems

[Tony Arcieri]

Hi there. I've talked to some people within Square and we're interested in
creating a system for providing end-to-end integrity of RubyGems, as well
as being able to revoke known compromised RubyGems while still surviving
the compromise of system keys.

While the specific design goals are up for debate, we'd probably try to do
a prototype implementation of The Update Framework on top of the existing
RubyGems X.509 certificate system (with perhaps a few modifications):

http://www.updateframework.com/projects/project

The main goals would be:

- Try to leverage as much of the existing work on signed RubyGems as
possible
- Depend only on the Ruby standard library and try not to pull in any
additional dependencies that RubyGems doesn't already depend on
- Produce a system with minimum (i.e. "zero") cost and operational
overhead which would still provide practical security guarantees and could
ensure all gems are signed (and also provide a way to retroactively sign
all existing gems)

If this sounds good to you, I'd love to talk more about fleshing out what
we would actually implement during Hack Week so we can have a plan that
lets us hit the ground running and get as much done as possible in a week,
with the goal of having something worthwhile that can be merged into the
upstream projects.

We also have Dan Boneh as a staff cryptographer and can probably rope him
in to review our design ;)

--
Tony Arcieri
_______________________________________________
RubyGems-Developers mailing list
http://rubyforge.org/projects/rubygems
RubyGems-...@rubyforge.org
http://rubyforge.org/mailman/listinfo/rubygems-developers

[Justin Cappos]

We (from the TUF project) would be also happy to review and discuss the
designs you guys are thinking about. Especially anything you are thinking
about diverging from TUF on is worth a discussion.

Even if you implement TUF verbatim, there are also some subtleties to key /
role setup that can make a big difference.

For example, we set up PyPI's metadata in such a way that even if the repo
is compromised users requesting packages from most projects would not be at
risk. The design still allows developers to create new projects with zero
lag / extra overhead / manual intervention. So it's much better security
with the same usability.

Another minor change with PyPI's setup resulted in an absolutely huge
metadata overhead reduction. A straightforward metadata signing setup
required several MB of data to be downloaded for each install. We
implemented hash-based delegations and reduced this by more than a factor
of 10x.

So let us know what you're thinking and we can maybe suggest some security
/ performance tweaks.

By the way, it is excellent that you have Dan Boneh to look over potential
issues as well. He's a really world-class crypto guy and I'm sure will
give great feedback.

Thanks,
Justin

[Nick Quaranto]

Sounds awesome, Tony. When is Square Hack Week, for those not inside of
Square? :)

Nick

On Sat, Sep 14, 2013 at 2:22 PM, Tony Arcieri <bas...@gmail.com> wrote:

[Trishank Karthik Kuppusamy]

On 09/14/2013 05:30 PM, Justin Cappos wrote:
> We (from the TUF project) would be also happy to review and discuss the
> designs you guys are thinking about. Especially anything you are thinking
> about diverging from TUF on is worth a discussion.
>

I am one of the TUF developers, and though I write more Python these
days, I have written my fair share of Ruby. Let us know how we can help
from New York!

[Tony Arcieri]

On Sat, Sep 14, 2013 at 2:30 PM, Justin Cappos <jca...@poly.edu> wrote:

> For example, we set up PyPI's metadata in such a way that even if the repo
> is compromised users requesting packages from most projects would not be at
> risk. The design still allows developers to create new projects with zero
> lag / extra overhead / manual intervention. So it's much better security
> with the same usability.
>

Awesome! It would be great it we didn't impact the system usability. In my
opinion, if we do a good job, people shouldn't notice the signature system
is even there unless something is actually compromised ;)

> Another minor change with PyPI's setup resulted in an absolutely huge
> metadata overhead reduction. A straightforward metadata signing setup
> required several MB of data to be downloaded for each install. We
> implemented hash-based delegations and reduced this by more than a factor
> of 10x.

Nice, I would love to hear more about that, as well as more general
practical advice about implementing TUF. So far I have just read the
"Survivable Key Compromise..." paper and can see how it could be
potentially mapped to tools like RubyGems and Gemcutter. At Square we have
a lot of experience with PKI, but mostly for TLS purposes, and a system
like RubyGems where there are many publishers is definitely quite the
special case versus what we're used to.

If there's any particularly awesome resources on what you've done in PyPI
I'd love to check them out.

[Tony Arcieri]

On Sat, Sep 14, 2013 at 12:30 PM, Nick Quaranto <ni...@quaran.to> wrote:

> Sounds awesome, Tony. When is Square Hack Week, for those not inside of
> Square? :)

The next Hack Week will be sometime in Q4, probably November I'm guessing.

[Jordi Massaguer Pla]

Hi! I'll be very happy to know about the progress of this project and to
eventually help on it.

Is there a mailing list I should join or will you use this one for
communication?

thanks for taking this project

jordi

[Tony Arcieri]

On Wed, Sep 18, 2013 at 3:05 AM, Jordi Massaguer Pla
<jmassa...@suse.de>wrote:

> Is there a mailing list I should join or will you use this one for
> communication?

This ML should probably be fine

--
Tony Arcieri

[Benjamin Fleischer]

Any update on this?

On Thursday, September 19, 2013 1:32:23 AM UTC-5, Tony Arcieri wrote:
>
> On Wed, Sep 18, 2013 at 3:05 AM, Jordi Massaguer Pla

> <jmassa...@suse.de <javascript:>>wrote:

>
> > Is there a mailing list I should join or will you use this one for
> > communication?
>
>
> This ML should probably be fine
>
> --
> Tony Arcieri
> _______________________________________________
> RubyGems-Developers mailing list
> http://rubyforge.org/projects/rubygems

> RubyGems-...@rubyforge.org<http://rubyforge.org/projects/rubygemsRubyG...@rubyforge.org>
> http://rubyforge.org/mailman/listinfo/rubygems-developers
>

[Jordi Massaguer Pla]

On 10/03/2013 12:24 PM, Jordi Massaguer Pla wrote:
> On 10/03/2013 05:26 AM, Tony Arcieri wrote:
>> On Wed, Oct 2, 2013 at 6:22 PM, Benjamin Fleischer <bflei...@gmail.com
>> <mailto:bflei...@gmail.com>> wrote:
>>
>> Any update on this?
>>
>>
>> Personally I've been thinking about this bundler proposal a lot and
>> whether it and a few accompanying mechanisms can solve most of the
>> problems in the TUF threat model:
>>
>> https://github.com/bundler/bundler-features/issues/27
>>
>> Specifically have a look at this response of mine, which transcends the
>> goals of the actual Bundler issue, but I think covers most of the bases
>> from an attack perspective:
>>
>> https://github.com/bundler/bundler-features/issues/27#issuecomment-25510553
>>
>> --
>> Tony Arcieri
> Having a checksum of the gems in bundler Gemfile.lock file sounds very
> good. This could help you knowing if a gem has changed.
>
> However, if the gem got compromised before you created your
> Gemfile.lock, you won't know that you are using a bad gem.
>
> In order to solve the latter issue, I agree that having the checksum
> signed will solve this issue, as long as the key used to signed it is
> trustable.
>
> And there comes my question: how do we manage the trust on the keys?
>
> A similar problem has been fixed long time ago on linux distributions.
> On a distribution like SUSE, SUSE employers will review the packages and
> sign them with the SUSE key, which its public part is distributed in the
> SUSE DVDs.
>
> However I don't think this model can be used in rubygems.org which
> stores thousands of gems, and is mostly run on a community effort. Thus
> I would go more on a trust model like gpg, I mean trusting individuals
> in a chain of trust and having a revoke list.
>
> This is what debian does:
>
> http://www.debian.org/doc/manuals/developers-reference/new-maintainer.html
>
>
> Is this the direction you think we should go / are we going?
>
> regards
>
> jordi

>

_______________________________________________
RubyGems-Developers mailing list
http://rubyforge.org/projects/rubygems
RubyGems-...@rubyforge.org

http://rubyforge.org/mailman/listinfo/rubygems-developers

[Jordi Massaguer Pla]

I am answering myself :) .

I've seen rubygems.org is going in this direction:

http://rubygems.rubyforge.org/rubygems-update/Gem/Security.html

though there is still some work in the TODO list, I think that is the
right direction. In my opinion, the most important missing piece is the
revokation part.

[Justin Cappos]

Revocation is a big issue, as is appropriately separating trust. For
example, who do I trust to know what key corresponds to a project? I
think that RubyGems will do this.

Furthermore, I assume the repo would keep that online to sign new projects
as they are registered. If so, then if the repo is compromised, can I
just say that some attacker controlled key is responsible for signing for
any project? That would seem to make the impact of a compromise of the
repo be that they can hack all users of RubyGEMS even if the devels are
secure (a bad outcome!).

We're working with the Python folks to try to address this issue (and a
bunch of others). Specifically, we are working on a PEP that has a layout
/ use that will make it so that:

1) PyPI can immediately create accounts for new projects
2) even if an attacker compromises the repo and all keys on PyPI are
stolen, very few projects / users will be at risk

We could help to create a similar configuration / protections for RubyGems
as well. However, to make this work RubyGems will need to have richer
mechanisms than SUSE, RedHat, Debian, etc. use --- for instance, a way of
performing selective trust delegation.

The situation with RubyGems is more complex than for Linux package managers
because you don't trust your Gems devels and your users shouldn't trust the
RubyGems repo more than they have to.)

We are interested in working with you guys to plan a setup that is very
secure and has minimal impact on usability for users, developers, and repo
admins.

Justin



On Thu, Oct 3, 2013 at 6:39 AM, Jordi Massaguer Pla
<jmassa...@suse.de>wrote:

[Tony Arcieri]

On Thu, Oct 3, 2013 at 3:24 AM, Jordi Massaguer Pla <jmassa...@suse.de> wrote:

However, if the gem got compromised before you created your

Gemfile.lock, you won't know that you are using a bad gem.

In order to solve the latter issue, I agree that having the checksum
signed will solve this issue, as long as the key used to signed it is
trustable.

While having a signature on every single gem will provide end-to-end integrity and is an admirable goal, we can solve the problem of detecting tainted gems in our apps without mandating a digital signature on every single gem.

Instead, RubyGems can maintain a community revocation list of known tainted gems, signed by a RubyGems public key which can ship with RubyGems itself (and whose private key can be airgapped). RubyGems or Bundler could check this revocation list before installing gems, and fail closed by default (with a flag to override) if a malicious gem is detected.

This way we can detect tainted gems after-the-fact, even if they weren't signed in the first place. That's not to say that we shouldn't sign the gems as well: using a combination thereof would be great, and is the exact sort of "survivable" multi-layered approach TUF recommends.

However I don't think this model can be used in rubygems.org which
stores thousands of gems, and is mostly run on a community effort. Thus
I would go more on a trust model like gpg, I mean trusting individuals
in a chain of trust and having a revoke list.

I don't think GPG is a good solution here whatsoever. The basic features of "trusting individuals in a chain of trust" are already provided by the X.509-based signature system that's already available in RubyGems. The other things that GPG provides, such as the basic mechanics of generating/managing keys and checking signatures are already present in RubyGems as well.

While GPG is a nice tool for something like Debian, because it provides a self-contained tool for code signing and key management, with Ruby it duplicates features that are already present in the Ruby standard library via the OpenSSL extension, and indeed features that are already implemented in RubyGems.

Ruby, unlike a Linux distribution, has to ship to multiple targets, including Windows and the JVM (and the JVM on Windows). Requiring a third party tool to shell out to is not a good solution, especially when the basic functions it provides are already present in a cross-platform manner via the Ruby standard library.

In addition, if we're to fully consider the TUF threat model: what happens when an individual "maintainer"'s key is compromised? How do we handle revoking only the gems that were signed after the key compromise while still maintaining trust in ones that were known good? For this to work, we need some mechanism of validating gems even in the event of a key compromise. Looking at Debian's keysigning policies I'm not sure how (or if) they handle this at all:

https://wiki.debian.org/Keysigning#Step_6:_Send_your_signed_key_to_the_server

I would invite you to read the TUF paper and study its threat model and namely look at what problems digital signatures used in the model you describe do/don't solve:

http://freehaven.net/~arma/tuf-ccs2010.pdf

--
Tony Arcieri

--
Tony Arcieri