Ongoing IHG work to improve Hackage security

[Duncan Coutts]

All,

The IHG members identified Hackage security as an important issue some
time ago and myself and my colleague Austin have been working on a
design and implementation.

The details are in this blog post:

http://www.well-typed.com/blog/2015/04/improving-hackage-security

We should have made more noise earlier about the fact that we're working
on this. We saw that it was important to finally write this up now
because other similar ideas are under active discussion and we don't
want to cause too much unnecessary duplication.

The summary is this:

We're implementing a system to significantly improve Hackage security.
It's based on a sensible design (The Update Framework) by proper crypto
experts. The basic system is fully automatic and covers all packages on
Hackage. A proposed extension would give further security improvements
for individual packages at the cost of a modest effort from package
authors.

http://theupdateframework.com/

It will also allow the secure use of untrusted public Hackage mirrors,
which is the simplest route to better Hackage reliability. As a bonus
we're including incremental index downloads to reduce `cabal update`
wait times. And it's all fully backwards compatible.


I should also note that our IHG funding covers the first phase of the
design, and for the second phase we would very much welcome others to
get involved with the detailed design and implementation (or join the
IHG and contribute further funding).

--
Duncan Coutts, Haskell Consultant
Well-Typed LLP, http://www.well-typed.com/

[Michael Snoyman]

I've read the blog post, and am still trying to understand the implications of TUF. However, it's incredibly difficult to give solid review of a system which is stated to be "based on TUF," without knowing what the delta implied by that is. For now, I can only ask simple questions:

1. Is there any timeline for the changes needed to Hackage and cabal-install?

2. Is there any idea of how much extra code will need to be maintained going forward for this? This is an important point, given that both Hackage and Cabal are having trouble keeping up with demand already.

3. Is there any mitigation of eavesdropping attacks on the authorization headers sent to Hackage by uploaders for digest authentication?

4. Is there any mitigation against a compromise of the Hackage server itself, either the code base or the system?

Overall, I'm quite wary of a solution stated as "experts devised this, it's good, we'll implement it, everyone stop worrying." I take your point about crypto-humility, but I'm not confident that an approach based on TUF addresses that concern since it involves a new implementation (or copy-pasted implementation) of crypto primitives together with unspecified changes to TUF.

Note: wary is *not* a code word for opposed, but I strongly believe that anything we do here warrants far more discussion, and that can't happen until more details are explained.

--
You received this message because you are subscribed to the Google Groups "Commercial Haskell" group.
To unsubscribe from this group and stop receiving emails from it, send an email to commercialhask...@googlegroups.com.
To post to this group, send email to commerci...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/commercialhaskell/1429176835.25663.30.camel%40dunky.localdomain.
For more options, visit https://groups.google.com/d/optout.

[Mikhail Glushenkov]

Hi,

On 16 April 2015 at 11:33, Duncan Coutts <dun...@well-typed.com> wrote:
> All,
>
> The IHG members identified Hackage security as an important issue some
> time ago and myself and my colleague Austin have been working on a
> design and implementation.
>
> The details are in this blog post:
>
> http://www.well-typed.com/blog/2015/04/improving-hackage-security

Thank you, this is very exciting. But won't the post-release .cabal
update feature interfere with "package index as an append-only log"
concept? IIUC, right now it is implemented as a destructive update of
the corresponding package index entry, so making the package index
immutable will break backwards compatibility.

[Duncan Coutts]

Yes, we can use the tar file in an append-only way while allowing
metadata updates because that's the tar file format supports that.

The tar file format was originally designed for tape drives where
rewinding and updating old entries was far too expensive. So the tar
file format allows appending updated file entries to the end of the
archive. Compliant tar tools (including the standard unix tools, and
cabal-install) understand this and take the last entry in the archive as
the current file content.

[Mikhail Glushenkov]

Hi,

On 16 April 2015 at 15:34, Duncan Coutts <dun...@well-typed.com> wrote:
> Compliant tar tools (including the standard unix tools, and
> cabal-install) understand this and take the last entry in the archive as
> the current file content.

Thanks. I looked at the code again, and while this is not explicitly
mentioned in comments, we get this behaviour for free by relying on
Map.fromList.

[Duncan Coutts]

On Thu, 2015-04-16 at 16:08 +0200, Herbert Valerio Riedel wrote:
> On 2015-04-16 at 15:14:47 +0200, Mikhail Glushenkov wrote:
>
> [...]

>
> > Thank you, this is very exciting. But won't the post-release .cabal
> > update feature interfere with "package index as an append-only log"
> > concept? IIUC, right now it is implemented as a destructive update of
> > the corresponding package index entry, so making the package index
> > immutable will break backwards compatibility.
>

> Being historically a tape-originating archive format, `tar` does support
> such destructive updates as a standard-feature by appended content. See
> also the '--update' operation in the tar(1) man-page:
>
> | -u, --update
> | Append files which are newer than the corresponding copy in
> | the archive. Arguments have the same meaning as with -c and
> | -r options.
>
> Using this, we can actually have *all* historic .cabal revisions in the
> 00-index.tar file as hidden entries. This would in theory give
> `cabal-install` access to all cabal revisions (if we'd ever want to have
> a feature in cabal that requires access to previous cabal revisions)

Yes, we can choose to do that. That's sort-of what I was getting at when
I mentioned that you could then view the tar file as a transaction log.

> I actually find Duncan's approach rather elegant, as it simply builds on
> existing standard features of both .tar files and the HTTP protocol
> (resuming HTTP downloads). And they don't even require any custom HTTP
> server implementation to work, and there's a graceful fallback. And we
> don't need to introduce a completely new protocol/tool into the mix
> (such as Git, which has dump-http(s), smart-http(s), git, as
> transport-layers to chose from, and has various other moving parts such
> as a ~/.gitconfig that can affect how Git operates, like redirecting
> clone-urls transparently, also it avoids having to handle subtle
> platform-specific issues with shelling out to an external command; from
> this POV the HTTP/tar approach is the more KISS-compliant one, IMHO)
>
> Cheers,
> hvr
>

[Mikhail Glushenkov]

Hi,

On 16 April 2015 at 16:08, Herbert Valerio Riedel <hvri...@gmail.com> wrote:
> Being historically a tape-originating archive format, `tar` does support
> such destructive updates as a standard-feature by appended content.

Thanks, I understand this. My question was about whether cabal-install
supports this feature of tar (turns out, it does).

> Using this, we can actually have *all* historic .cabal revisions in the
> 00-index.tar file as hidden entries.

I don't think we do this right now, though.