DownThemAll! & repairing downloads (chunk checksums)

[Anthony Bryan]

I don't have any numbers to back this up, but I'd guess that most
metalink users are taking advantage of them with DownThemAll!.

DTA is easy to use, pleasant looking, & just a Firefox extension install away.

the only metalink related feature it lacks is repairing a download if
the metalink has chunk checksums. it already verifies checksums for
the whole file.

I think this should be one of our higher priorities, because of the
amount of people that DTA reaches. for instance, openSUSE uses
metalinks with chunk checksums, but if a downloader uses a metalink
client that doesn't support them then the downloader could get
frustrated if there's an error in their big download...

we have this on our ideas page & here's the request in the DTA bugtracker

http://groups.google.com/group/metalink-discussion/web/gsoc-ideas
http://bugs.code.downthemall.net/trac/ticket/413

does anyone with JS skills want to work on this? Nils had said he
would accept a patch.
maybe someone from the openSUSE community could do it?

--
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
)) Easier, More Reliable, Self Healing Downloads

[Hampus Wessman]

That would be really nice. Chunk checksums is one of the coolest
metalink features, IMO!

I could do this after I've finished Metalink Editor 2.0, but that might
take some time. It would be even better if someone else did it before!

Hampus Wessman

[Ant Bryan]

thanks Hampus! I think it's one of the most useful too! so much so,
that I think it may be more important than ME2. not to knock ME2! :)
but ME1 is great enough and gets us by fine (not that the new features
would be good too).

I wonder how much work this actually is?

On May 9, 4:29 am, Hampus Wessman <hampus.wess...@gmail.com> wrote:
> That would be really nice. Chunk checksums is one of the coolest
> metalink features, IMO!
>
> I could do this after I've finished Metalink Editor 2.0, but that might
> take some time. It would be even better if someone else did it before!
>
> Hampus Wessman
>
> On Fri, 8 May 2009 17:31:15 -0400
>

[Hampus Wessman]

True. I'll look into it! Shouldn't be that hard.
/ Hampus


Ant Bryan skrev:

[Anthony Bryan]

Nils was kind enough to provide details (and I imagine help along the
way as well): http://bugs.code.downthemall.net/trac/ticket/413#comment:4

Replying to ant:

Nils, will you still accept a patch for this?

Sure.

If so, could you provide some guidance on where to start?

The verification code lives in trunk/chrome/content/dta/manager/verificator.js

There is already whole file checksumming support, so is
checksumming each chunk the difficult part, or is it changing DTA to
re-request a chunk when the checksums don't match?

There are multiple parts needed to implemented:

* Parsing the metalink trunk/chrome/content/dta/manager/metalinker.js
* Storing stuff, so that is survives between sessions. This has to
be backwards-compatible, i.e. loading old queue items lacking the new
stuff has to be successful. See the .toSource methods
trunk/chrome/content/dta/manager.js
* Calculating the checksums trunk/chrome/content/dta/manager/verificator.js
* Asking the user what he'd like to do in case of error.
* Setting up the retry chunks in case of error and starting the
download again(in the quest of uncoupling please do not mess with
QueueItem? data from verificator or similar, but instead provide some
public methods in QueueItem? that can be used for this matter).

I know that esp. the verificator/metalinker code is not very well
designed and too tightly coupled. Any implementor should try not to
add even more coupling, however.

See CreatingPatches.

It would be great if a patch would do the following, however this is optional:

* split verificator into a re-usable js module and some "user" code.
* split metalinker into a re-usable js module (parser/validation),
interface code (the selection dialog) and manager.js stuff.

[Hampus Wessman]

Nice. Sounds doable.
/ Hampus

Anthony Bryan skrev:

[Anthony Bryan]

Yes, it's definitely nicer that it's defined now & whoever works on it
has info on where to start.

I'm moving this to the top of the ideas page
http://groups.google.com/group/metalink-discussion/web/gsoc-ideas
because it seems like relatively low hanging fruit, important, &
potentially affecting many people (DTA has been downloaded 31 million
times!! & I'm guessing it's how most people use metalinks).