Chunk checksums in base32/64, signed checksum files

[Anthony Bryan]

Tatsuhiro Tsujikawa, author of aria2, suggested allowing chunk
checksums in base32 or base64 which could make for smaller metalinks.
another thing we have talked about is "Accept-Encoding: gzip" for
compressing them for transfer.

also, looking at the Ubuntu release, they have an ISOs, MD5SUMS, and
MD5SUMS.gpg: http://releases.ubuntu.com/8.04/

if that style became "popular", do you think something like this is
too clunky? you could have a chain of trust..."MD5SUMS is signed and
ISO file checksum matches one listed there."

<file name="hardy-desktop-i386.iso">
..
<hash type="md5" file="MD5SUMS">508ff37c52bf0cebe3518bd655595913</hash>
..
<file name="MD5SUMS">
<signature type="pgp">

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[Nils]

On May 13, 8:04 am, "Anthony Bryan" <anthonybr...@gmail.com> wrote:
> Tatsuhiro Tsujikawa, author of aria2, suggested allowing chunk
> checksums in base32 or base64 which could make for smaller metalinks.

base32 is almiost unknown (as is base85 for example).

base64 has IIRC 33% overhead while the hexadecimal representation
doubles everything in size.
However I don't think it makes much sense to actually use it here,
weighting the added format/implementation complexity against the
benefits.
base64 wasn't designed to "compress" data, but to represent binary
data as "text" for systems choking on binary data like emails.
Furthermore there then should be some kind of type attribute to
distinguish between b64 and hex hashes, which will take away most
gains again.
Oh, and it isn't exactly backwards compatible.

> another thing we have talked about is "Accept-Encoding: gzip" for
> compressing them for transfer.

This would do better compared to b64. Clients which support gzip for
regular transfers likely support it already because they likely use
the same http implementation for both.
And server side things like multiviews/mod_deflate are your friends.
Even IIS seems to have a fairly decent implementation even caching the
gzipped results.

> also, looking at the Ubuntu release, they have an ISOs, MD5SUMS, and
> MD5SUMS.gpg:http://releases.ubuntu.com/8.04/
>
> if that style became "popular", do you think something like this is
> too clunky? you could have a chain of trust..."MD5SUMS is signed and
> ISO file checksum matches one listed there."
>
>    <file name="hardy-desktop-i386.iso">
>        ..
>        <hash type="md5" file="MD5SUMS">508ff37c52bf0cebe3518bd655595913</hash>
>        ..
>    <file name="MD5SUMS">
>      <signature type="pgp">

I kinda think this is doubles the efforts.
PGP signatures, being a hybrid encryption method, in fact only sign
the hash of the message (the download in our case). Usually sha1
hashes are used, but there are other algorithms supported as well.[1]
So a usually the computation goes like this: sign(hash(message))
But now it becomes: sign(hash(hash(message))

The proposed style is flawed as well.
MD5SUMS contains the output from md5sum for multiple files.

Nils

[1] http://tools.ietf.org/html/rfc4880#section-9.4

[Neil M.]

Nils wrote:
> On May 13, 8:04 am, "Anthony Bryan" <anthonybr...@gmail.com> wrote:
>> Tatsuhiro Tsujikawa, author of aria2, suggested allowing chunk
>> checksums in base32 or base64 which could make for smaller metalinks.
>
> base32 is almiost unknown (as is base85 for example).
>
> base64 has IIRC 33% overhead while the hexadecimal representation
> doubles everything in size.
> However I don't think it makes much sense to actually use it here,
> weighting the added format/implementation complexity against the
> benefits.
> base64 wasn't designed to "compress" data, but to represent binary
> data as "text" for systems choking on binary data like emails.

A hash IS binary data. A series of X bits. Hexadecimal is one way to
represent it. XML is a text system. base64 makes sense to me (base32
is much less common).

> Furthermore there then should be some kind of type attribute to
> distinguish between b64 and hex hashes, which will take away most
> gains again.

There is no need for this attribute. The hash is already tagged with a
"type" attribute. Based on the hash type and length of contained string
you can determine if it is hexadecimal or base64/32.

> Oh, and it isn't exactly backwards compatible.
>
>> another thing we have talked about is "Accept-Encoding: gzip" for
>> compressing them for transfer.
>
> This would do better compared to b64. Clients which support gzip for
> regular transfers likely support it already because they likely use
> the same http implementation for both.
> And server side things like multiviews/mod_deflate are your friends.
> Even IIS seems to have a fairly decent implementation even caching the
> gzipped results.
>

Agree this would be better. More backward compatible and you would see
gains for the whole .metalink file, like urls. Technically I'd say this
is outside of the scope of the Metalink Spec, there is nothing stopping
servers and clients from using gzip already.

>> also, looking at the Ubuntu release, they have an ISOs, MD5SUMS, and
>> MD5SUMS.gpg:http://releases.ubuntu.com/8.04/
>>
>> if that style became "popular", do you think something like this is
>> too clunky? you could have a chain of trust..."MD5SUMS is signed and
>> ISO file checksum matches one listed there."
>>
>> <file name="hardy-desktop-i386.iso">
>> ..
>> <hash type="md5" file="MD5SUMS">508ff37c52bf0cebe3518bd655595913</hash>
>> ..
>> <file name="MD5SUMS">
>> <signature type="pgp">
>
> I kinda think this is doubles the efforts.
> PGP signatures, being a hybrid encryption method, in fact only sign
> the hash of the message (the download in our case). Usually sha1
> hashes are used, but there are other algorithms supported as well.[1]
> So a usually the computation goes like this: sign(hash(message))
> But now it becomes: sign(hash(hash(message))
>

Yes, from a cryptographic standpoint this is a really ugly way to do
things. Also a pain to implement well on the client (why I haven't done
it for Metalink Checker yet!). You then also need to know something
about the file format of the MD5SUMS or some other format someone might
decide to use.

Note that this does make sense from a computational standpoint since
when you are dealing with files the size of ISOs it can take a long time
to compute a hash.

1. The cryptographically most efficient way, you must compute the iso
hash twice:

md5sum command: MD5SUMS = hash(iso)
gpg command: signature = encrypt(hash(iso))

* Note: You might be able to get gpg to output the hash value somehow
and skip the md5sum step, but I haven't been able to figure out how yet.

2. The computationally most efficient way, you only compute the iso hash
once:

md5sum command: MD5SUMS = hash(iso)
gpg command: signature = encrypt(hash(MD5SUMS))

> The proposed style is flawed as well.
> MD5SUMS contains the output from md5sum for multiple files.
>

I wouldn't call this a flaw, you just need to know the file format and
have the filenames match what is being downloaded, which should be the
case normally.

[Nils]

On May 14, 7:33 am, "Neil M." <nabbe...@gmail.com> wrote:
[...}

> A hash IS binary data.  A series of X bits.  Hexadecimal is one way to
> represent it.  XML is a text system.  base64 makes sense to me (base32
> is much less common).

Thanks for pointing out the obvious, again. Quiet frankly, do you
really think I don't know that?!

My point here was that base64 was brought to the discussion for some
reason, namely it "could make for smaller metalinks. "
And while this may hold true the downsides (yet another format to
parse, added complexity in spec and implementations, no backwards
compatibility) outweight the benefits (a damn few bytes saved in a
meta-file the size of which doesn't likely matter anyway compared to
the files it references).

> > Furthermore there then should be some kind of type attribute to
> > distinguish between b64 and hex hashes, which will take away most
> > gains again.
>
> There is no need for this attribute.  The hash is already tagged with a
> "type" attribute.  Based on the hash type and length of contained string
> you can determine if it is hexadecimal or base64/32.

There is need. Everything else would be inaccurate and messy.
Psychic guessing is not something a computer should do if it doesn't
have to.

[...]

> >> also, looking at the Ubuntu release, they have an ISOs, MD5SUMS, and
> >> MD5SUMS.gpg:http://releases.ubuntu.com/8.04/
>
> >> if that style became "popular", do you think something like this is
> >> too clunky? you could have a chain of trust..."MD5SUMS is signed and
> >> ISO file checksum matches one listed there."
>
> >>    <file name="hardy-desktop-i386.iso">
> >>        ..
> >>        <hash type="md5" file="MD5SUMS">508ff37c52bf0cebe3518bd655595913</hash>
> >>        ..
> >>    <file name="MD5SUMS">
> >>      <signature type="pgp">
>
> > I kinda think this is doubles the efforts.
> > PGP signatures, being a hybrid encryption method, in fact only sign
> > the hash of the message (the download in our case). Usually sha1
> > hashes are used, but there are other algorithms supported as well.[1]
> > So a usually the computation goes like this: sign(hash(message))
> > But now it becomes: sign(hash(hash(message))
>
> Yes, from a cryptographic standpoint this is a really ugly way to do
> things.  Also a pain to implement well on the client (why I haven't done
> it for Metalink Checker yet!).  You then also need to know something
> about the file format of the MD5SUMS or some other format someone might
> decide to use.

Agreed.

> Note that this does make sense from a computational standpoint since
> when you are dealing with files the size of ISOs it can take a long time
> to compute a hash.
>
> 1. The cryptographically most efficient way, you must compute the iso
> hash twice:
>
> md5sum command: MD5SUMS = hash(iso)
> gpg command: signature = encrypt(hash(iso))
>
> * Note: You might be able to get gpg to output the hash value somehow
> and skip the md5sum step, but I haven't been able to figure out how yet.
>
> 2. The computationally most efficient way, you only compute the iso hash
> once:
>
> md5sum command: MD5SUMS = hash(iso)
> gpg command: signature = encrypt(hash(MD5SUMS))
>
> > The proposed style is flawed as well.
> > MD5SUMS contains the output from md5sum for multiple files.
>
> I wouldn't call this a flaw, you just need to know the file format and
> have the filenames match what is being downloaded, which should be the
> case normally.

Kinda valid point, although I doubt that a bunch of more minutes
you'll have to spend on calculating those additional hashes would
really matter in a release process that likely takes hours anyway and
is often at least semi-automated.
And lets not forget that this has to be done once. Would be another
story if this was a client-side operation that would be carried out
millions of times. ;)

But then again implementors would need another parser for an arbitrary
format, it adds complexicity, etc.
And I'm not quiet sure how it should be represented...


If we want some signatures then the whole metalink should be signed
(that would include any file hashes and file signatures).
A serializer would serialize the contents for a canonicalized
represenation which then may get signed or checked against a
signature.
For example Mozilla does so for extension update manifests:
http://developer.mozilla.org/en/docs/Extension_Versioning%2C_Update_and_Compatibility#Update_RDF_Format

Benefits are that all would be self-contained and the whole metalink
would have an cryptographic identity instead of just the files it
points to.
That would, of course, require quiet a lot of work for implementors...
But so does signing anyway and/or parsing and processing MD5SUMS.

Nils

[Hampus Wessman]

I completely agree with you, Nils! Don't really have anything to add either...
Hampus Wessman