Re: Request for API to Support Multithreaded Download Manager

[Ant Bryan]

On Thursday, June 21, 2012 5:28:29 AM UTC-4, Tom Wardrop wrote:

Google Chrome currently has an experimental extensions API for downloads, but it doesn't seem to give the developer any control over the actual stream. It seems the extension allows management of downloads at a higher level which doesn't allow developers to, for example, support multithreaded downloads, where multiple HTTP requests are issued for a single file, each with a different HTTP range header specified. Multithreaded downloads is a popular request by users, and is personally one of the only reasons I still occasionally use Firefox, which is for large file downloads where multithreading is required to fully utilize my connection bandwidth.

I would love for Chrome to provide an API for issuing a webRequest that responds with some form of IO object for streaming the data, as well as an API File IO API for writing to the file system, with support for file offsets, etc. Given the lack of time spent on Chrome's built-in download manager, I doubt Google will ever implemented multithreading in their download manager, so the next best thing would be to provide an extension API which would allow someone else to build it. I find it surprising enough that to this day, no mainstream browser provides multithreaded downloads out of the box, given that it's almost impossible for anyone to saturate their connection over a single HTTP connection.

I just wanted to show that there's interest in such an API, and I urge Google to open up their extension API a little more to allow for more useful and creative extensions. Most of the extensions available at the moment, due to the currently restricted API, are mainly fluff - normally nothing more than minor shortcuts and convenience. There are no extensions that can actually build functionality onto the core web experience, unlike Firefox where the extension developers have access to pretty much everything the browser developers have access to.

 

hi Tom, I mirror your wants & expectations. internet connections are getting faster for some people but single server downloads don't seem to be increasing as proportionately.

I think this is a killer feature that's time has come. it would definitely improve things for many people, without them noticing.

some good news:

1) the downloads API is out of experimental in Chrome 22, so it should be seeing more use.

2) Metalink Downloader [ http://code.google.com/p/metalink-chrome-extension/ ] uses this API and supports Metalink - which if you haven't heard of it is a way (using HTTP headers or XML - RFC 6249 / RFC 5843) to get mirrors, checksums, & more info about downloads to download applications like browsers, FTP clients, P2P apps, etc.

it's no DTA for Firefox but it supports multithreaded downloads, checksum verification, and error repair among other things.

you can't currently edit/add which mirrors are used, they're gotten automatically from Metalinks. but if there's interest, we could add that feature. why don't you come by & check it out, give us some feedback?

these types of features are being watched in the extension for future inclusion natively in Chrome.

thanks,

ant

[Tom Wardrop]

Glad to know I'm not the only one. Really, it's surprising given the acceptance of multithreaded processors, that browsers don't have support for it out the box. It's basic computer science that serial processes can be accelerated by making them parallel (assuming the usually small overheads of doing so don't outweigh the typical performance improvement). Not all things can be made parallel, but HTTP downloads could be considered Embarrassingly Parallel, which is what makes it's omittance from all major browsers rather surprising.

It's to the same degree that I find surprising that the humble textarea element hasn't received any UI improvements (besides spell checkers and a resizeable corner) since their introduction, yet we're generating ever more content online - how is it still acceptable for someone to be able to accidentally lose hours of typed-work? Why don't browsers add the contents of textarea's to their history? It makes little sense.

It's times like these you wish you had a bit of sway in the world of browser development. With Chrome, perhaps unlike Firefox, I can't help but feel Google are letting their own corporate agenda get in the way of progress. That's the impression I get but I can't exactly put my finger on why it is I get that impression.

[Ant Bryan]

Metalink Downloader is, afaict, the first Chrome extension download manager. 

It just got on the Chrome Web Store a few days ago: https://chrome.google.com/webstore/detail/jnpljlobbiggcdikagmiepniibjdinap

Let us know if you have any comments or ideas on how to improve it!