metalinks, torrents, and verification elements

[Matt Domsch]

I'm trying to understand common practice for including bittorrent URLs
in a metalink, either with version 3.0 or the IETF draft.

<file name="example.file">
<size>XXXX</size>
<verification>
<hash .../>
</verification>
<resources>
<url type="bittorrent">http://www.example.com/example.torrent</url>
</resources>
</file>

This makes sense, if each file has its own torrent. However, Fedora
and other torrent generator persons don't always do this. Instead,
they will have multiple files inside the torrent:

{'path': ['Fedora-10-x86_64-DVD.iso'], 'length': 4172283904L}
{'path': ['SHA1SUM'], 'length': 788}

In this instance, the metalink for the Fedora-10-x86_64-DVD.iso that
MirrorManager publishes has direct FTP/HTTP URLs to that ISO, and the
<verification> section has info on this single file.

But this single file isn't published in its own torrent, it's in a
torrent with those multiple pieces. The whole <verification> section
then is either meaningless, or would provide incorrect information, as
compared to the torrent URL.

Likewise, the CD downloads for the multi-CD set includes direct
HTTP/FTP URLs to each of the CDs, but the torrent includes all of them
in a single torrent.

Is the <verification> section completely ignored when torrents are
used, and instead the torrent metadata used? If so, can that be
documented in the specification?

Thanks,
Matt

[Neil M.]

I'm not familiar with how bittorrent works, but it seems to me:

1. The client could just download the file from the .torrent where the
name matches the file name in the metalink (in this case example.file).
No problem.

2. If you must download all files in a torrent, a client could still
download them all, but only run the checksum where the file names match.

Of course this depends on the names of the files matching, which isn't
ideal as things would break if they are not.

Neil

[Tatsuhiro]

In aria2, a metalink client, I just discard URLs in metalink if
torrent contains multi-file because it looks like current metalink
spec cannot accept multi-file torrent. No element for grouping files.
One file and its URLs. That's all.

The verification section is ignored if torrent is specified, either
single- or multi- file torrent, because .torrent file has piece hash
and they are enough to validate the integrity of file.
I'm not sure this behavior is documented or not.

[Hampus Wessman]

I hadn't thought about this before. It sure deserves a satisfactory
solution. Good that the issue was brought up!

I have no good ideas right now about how to best solve it.

When it comes to <verification> I think the intention is that you MAY
ignore it when a torrent is available, but a client can use it too. A
client without support for bittorrent would always use those hashes,
for example. Clients with support for bittorrent probably doesn't need
them when a torrent is available, but IMO it's up to the client
programmer to decide what to use in this case. A very pedantic and
somewhat paranoid client would of course check them all =)

Hampus

[Neil M.]

I think having clients ignore the "verification" section in the Metalink
is a bad idea. The file at the bittorrent URL could be changed to
deliver malicious content instead of the intended file. If the Metalink
verification is checked this can be prevented. Basically you use the
Metalink file as a trust anchor to ensure that the bittorrent file is
indeed what you really want. We assume that a user trusts the source of
the Metalink file, but not necessarily the source of the .torrent file
(the user doesn't open the metalink file and review the bittorrent URL).
If using GPG the metalink source can even be validated, something
bittorrent doesn't do.

Neil

[Hampus Wessman]

True. Checking all available file hashes doesn't take much time anyway
(you only need to compute the downloaded file's hashes once), so you
have very little to lose really.

One can safely ignore the chunk checksums in the metalink though,
unless you really want to be sure not to waste any time on bad torrents
(might not be worth all the extra hash checks, though). Perhaps it
would be useful with a way to specify a hash for a torrent? Then you
could easily and efficiently verify that it hasn't been replaced. A
lot better than waiting for a chunk checksum to fail (even after a
failed chunk checksum the client may not be able to tell from where
the bad data came!).

Not sure if it's easy to fit into the standard. Could change
metalink:metadata somehow... Would be good if one could specify which
file from the torrent to download too (doesn't need to be one with the
same name as the metalink <file>, necessarily). Would solve the initial
problem, below.

Hampus

[Nicolas Alvarez]

Hampus Wessman wrote:
> True. Checking all available file hashes doesn't take much time anyway
> (you only need to compute the downloaded file's hashes once), so you
> have very little to lose really.
>
> One can safely ignore the chunk checksums in the metalink though,
> unless you really want to be sure not to waste any time on bad torrents
> (might not be worth all the extra hash checks, though). Perhaps it
> would be useful with a way to specify a hash for a torrent? Then you
> could easily and efficiently verify that it hasn't been replaced. A
> lot better than waiting for a chunk checksum to fail (even after a
> failed chunk checksum the client may not be able to tell from where
> the bad data came!).

Why would it take more time to check more hashes? If both the metalink and
the torrent have chunk checksums with the same chunk size, just see if both
have the same checksums. Or, calculate chunk hash and compare it to both.

The bottleneck is I/O, not CPU. You can compute sets of chunk hashes for
different chunk sizes *and* the full-file hash "at the same time" (reading
the file only once).

[Hampus Wessman]

Checking multiple sets of chunk checksums, possibly having different
lenghts, adds complexity to the client regardless of speed. IF they
do have the same length, then of course you can compare each individual
chunk checksum in the metalink and the torrent. They don't have to be.

If they don't have the same length and you download the file randomly,
as bittorrent does, then it will get both complicated and most probably
take more time to calculate the additional hashes (while downloading).

I agree that reading the file once and computing all the hashes at the
same time (as you said) is a good strategy, once the file is finished.
The chunk checksums have hopefully been checked already at that time,
though.

If you have a slow or busy CPU it won't be I/O bound. It's obvious that
it takes more resources to check more hashes. If that translates to
more time depends on the system where it runs. What you say is probably
true for many modern desktop systems (not necessarily all, especially
not when compiling stuff in the background or similar).

Not that it's necessarily a bad idea to make those checks, but
IMO it might not be worth it (both when it comes to speed and
complexity). It's a lot easier when the lengths match, so a middle way
would be to only do it in those cases. As always I think it's up to the
client programmer. I'm not sure if I would have done it.

A hash for the whole torrent would make all this unnecessary and would
be a lot easier. It would be just as easy (and effective!) when the
chunks have different lengths too.

Hampus

[Anthony Bryan]

from what I remember when this came up before, the filename in a
multi-file torrent should match the filename in the metalink (here
<file name="example.file">). I know this isn't too flexible but I
think it can be made to work for most cases now.

Tatsuhiro, any ideas for improvements? perhaps a file or files
attribute for bittorrent URLs?

> Is the <verification> section completely ignored when torrents are
> used, and instead the torrent metadata used?  If so, can that be
> documented in the specification?

Tatsuhiro said aria2 behaves in this way:

"The verification section is ignored if torrent is specified, either
single- or multi- file torrent, because .torrent file has piece hash
and they are enough to validate the integrity of file.
I'm not sure this behavior is documented or not."

this becomes tricky - which is "authoritative", the metalink or the
torrent? linux distros probably roll their own, both torrents &
metalinks, so this isn't a problem. but others might not have control
over both.

then, basically all metalink clients will do full file checksum
verification. but few do chunk checksums, few do mirrors and torrents,
even fewer do both.

for instance, if DTA did chunk checksums, it still probably won't
support torrents any time soon.

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[Tatsuhiro]

On Jun 15, 6:24 am, Anthony Bryan <anthonybr...@gmail.com> wrote:
> from what I remember when this came up before, the filename in a
> multi-file torrent should match the filename in the metalink (here
> <file name="example.file">). I know this isn't too flexible but I
> think it can be made to work for most cases now.
>
> Tatsuhiro, any ideas for improvements? perhaps a file or files
> attribute for bittorrent URLs?

Yeah, if all file under files element matches to the content of
torrent file, adding torrent attribute to files element might make
things clear. But that seems very torrent specific.
Personally, I don't like the way of multi-file torrent which treats
many files as a single and hashing it into pieces. As said earlier in
this thread, it requires that to checking a piece hash, more than 1
file is needed sometimes. It makes things unnecessarily complicated.

>
> "The verification section is ignored if torrent is specified, either
> single- or multi- file torrent, because .torrent file has piece hash
> and they are enough to validate the integrity of file.
> I'm not sure this behavior is documented or not."
>
> this becomes tricky - which is "authoritative", the metalink or the
> torrent? linux distros probably roll their own, both torrents &
> metalinks, so this isn't a problem. but others might not have control
> over both.
>
> then, basically all metalink clients will do full file checksum
> verification. but few do chunk checksums, few do mirrors and torrents,
> even fewer do both.
>
> for instance, if DTA did chunk checksums, it still probably won't
> support torrents any time soon.

This is very difficult and I have no good idea at the moment.
I think checking all available checksums are waste of the time.
If we check chunk checksum, we don't need to check whole checksum.
I admit the possibility of replaced torrent file. So I think hashing
for the whole torrent is a good idea.

On Jun 15, 6:24 am, Anthony Bryan <anthonybr...@gmail.com> wrote:

[Anthony Bryan]

On Wed, Jun 17, 2009 at 7:45 AM, Tatsuhiro <tatsu...@gmail.com> wrote:
>
> On Jun 15, 6:24 am, Anthony Bryan <anthonybr...@gmail.com> wrote:
> > from what I remember when this came up before, the filename in a
> > multi-file torrent should match the filename in the metalink (here
> > <file name="example.file">). I know this isn't too flexible but I
> > think it can be made to work for most cases now.
> >
> > Tatsuhiro, any ideas for improvements? perhaps a file or files
> > attribute for bittorrent URLs?
>
> Yeah, if all file under files element matches to the content of
> torrent file, adding torrent attribute to files element might make
> things clear. But that seems very torrent specific.
> Personally, I don't like the way of multi-file torrent which treats
> many files as a single and hashing it into pieces. As said earlier in
> this thread, it requires that to checking a piece hash, more than 1
> file is needed sometimes. It makes things unnecessarily complicated.

here's what I think would work:

1) use a single file torrent.

2) the filename in a multi-file torrent should match the filename in

the metalink (here <file name="example.file">).

3) add a file attribute to <url>s of type="bittorrent" so you can
select one file out of a multi-file torrent

<url type="bittorrent" file="Fedora-11-i386-disc1.iso"
preference="100">http://torrent.fedoraproject.org/torrents//Fedora-11-i386-CDs.torrent</url>

[Matt Domsch]

On Sun, Jun 21, 2009 at 04:59:26PM -0400, Anthony Bryan wrote:
>
> On Wed, Jun 17, 2009 at 7:45 AM, Tatsuhiro <tatsu...@gmail.com> wrote:
> >
> > On Jun 15, 6:24 am, Anthony Bryan <anthonybr...@gmail.com> wrote:
> > > from what I remember when this came up before, the filename in a
> > > multi-file torrent should match the filename in the metalink (here
> > > <file name="example.file">). I know this isn't too flexible but I
> > > think it can be made to work for most cases now.
> > >
> > > Tatsuhiro, any ideas for improvements? perhaps a file or files
> > > attribute for bittorrent URLs?
> >
> > Yeah, if all file under files element matches to the content of
> > torrent file, adding torrent attribute to files element might make
> > things clear. But that seems very torrent specific.
> > Personally, I don't like the way of multi-file torrent which treats
> > many files as a single and hashing it into pieces. As said earlier in
> > this thread, it requires that to checking a piece hash, more than 1
> > file is needed sometimes. It makes things unnecessarily complicated.
>
> here's what I think would work:
>
> 1) use a single file torrent.

not feasible really. The convenience of multi-file torrents is too
much to pass up.

> 2) the filename in a multi-file torrent should match the filename in
> the metalink (here <file name="example.file">).
>
> 3) add a file attribute to <url>s of type="bittorrent" so you can
> select one file out of a multi-file torrent
>
> <url type="bittorrent" file="Fedora-11-i386-disc1.iso"
> preference="100">http://torrent.fedoraproject.org/torrents//Fedora-11-i386-CDs.torrent</url>

This could work for me.

-Matt

[Lloyd Prophet]

Hello fellow Metalinkers,

I believe the idea of Metalink is to allow swarming across multiple
protocols, not choose one over the other.
Can any current Metalink client swarm between BitTorrent & other
protocols?

The proper way to handle the situation, presented by Matt, is to use
file hashes (MD5, SHA-1, or similar).
Relying on filenames is not fail-safe, due to the fact that filenames
are arbitrary.
On the contrary, the file hash will never change unless the data
changes.

Specifying a MD5 or SHA-1 hash for the .torrent file is the best
solution.

[Tatsuhiro]

On 6月22日, 午前5:59, Anthony Bryan <anthonybr...@gmail.com> wrote:

> On Wed, Jun 17, 2009 at 7:45 AM, Tatsuhiro <tatsuhir...@gmail.com> wrote:
>
> > On Jun 15, 6:24 am, Anthony Bryan <anthonybr...@gmail.com> wrote:
> > > from what I remember when this came up before, the filename in a
> > > multi-file torrent should match the filename in the metalink (here
> > > <file name="example.file">). I know this isn't too flexible but I
> > > think it can be made to work for most cases now.
>
> > > Tatsuhiro, any ideas for improvements? perhaps a file or files
> > > attribute for bittorrent URLs?
>
> > Yeah, if all file under files element matches to the content of
> > torrent file, adding torrent attribute to files element might make
> > things clear. But that seems very torrent specific.
> > Personally, I don't like the way of multi-file torrent which treats
> > many files as a single and hashing it into pieces. As said earlier in
> > this thread, it requires that to checking a piece hash, more than 1
> > file is needed sometimes. It makes things unnecessarily complicated.
>
> here's what I think would work:
>
> 1) use a single file torrent.
>
> 2) the filename in a multi-file torrent should match the filename in
> the metalink (here <file name="example.file">).
>
> 3) add a file attribute to <url>s of type="bittorrent" so you can
> select one file out of a multi-file torrent
>
> <url type="bittorrent" file="Fedora-11-i386-disc1.iso"
> preference="100">http://torrent.fedoraproject.org/torrents//Fedora-11-i386-CDs.torrent</url>
>

If many files are listed in this way, same torrent URL is scattered in
metalink file and
this also requires an extra work for BitTorrent capable client: The
client has to look url element under every file element and gather
files which have same torrent file , because usually multi file
torrent is handled in the one torrent session.

[Tatsuhiro]

On 6月23日, 午前12:14, Lloyd Prophet <lloyd.prop...@googlemail.com> wrote:
> Hello fellow Metalinkers,
>
> I believe the idea of Metalink is to allow swarming across multiple
> protocols, not choose one over the other.
> Can any current Metalink client swarm between BitTorrent & other
> protocols?
>

aria2 can download a single file in HTTP/FTP/BitTorrent at the same
time.

[Anthony Bryan]

2009/6/23 Tatsuhiro <tatsu...@gmail.com>:

you mean a torrent capable client needs to go through the metalink &
grab all files in a mult-file torrent all at once?

I guess file="" isn't the best solution for all cases, but I'm having
trouble coming up with a better one for our paradigm of alternate URLs
for unique files. if we have another <metadata>/<torrent> section
under <files> (not <file)? I don't think that fits in with how we're
doing things tho.

let's keep thinking about/discussing this...

[Anthony Bryan]

On Mon, Jun 22, 2009 at 11:14 AM, Lloyd
Prophet<lloyd....@googlemail.com> wrote:
>
> Hello fellow Metalinkers,

thanks for joining us, LLoyd!

> I believe the idea of Metalink is to allow swarming across multiple
> protocols, not choose one over the other.
> Can any current Metalink client swarm between BitTorrent & other
> protocols?

as TT said, aria2 can. along with GetRight and Retriever. at some
point KGet might be able to as well.

> The proper way to handle the situation, presented by Matt, is to use
> file hashes (MD5, SHA-1, or similar).
> Relying on filenames is not fail-safe, due to the fact that filenames
> are arbitrary.
> On the contrary, the file hash will never change unless the data
> changes.
>
> Specifying a MD5 or SHA-1 hash for the .torrent file is the best
> solution.

good point! don't the torrents contain a SHA-1 hash of the whole file?
trying to remember...or is it just hashes of the chunks?