Metalink is useful to Free and Open Source software. It was designed for community projects that use multiple mirrors and P2P. There is a whole ecosystem of applications, sites, and projects involved in Metalink.

 

We invite other Download Managers, Web browsers, BitTorrent clients, and other P2P programs to support Metalink.

If you're a developer, adding Metalink support to Open Source programs and getting projects to use it for their downloads will be a huge help. Examining the source code from current open source Metalink clients and generators could be useful.

 

Current open source Metalink clients (downloaders) 

• Appupdater (Cross Platform, Python)
• aria2 (Cross Platform, C++)
• DownThemAll! (Cross Platform Firefox Extension, Javascript)
• DLApplet (Cross Platform Browser Applet, Java)
  
• easyMule (Windows, C++)
  
• Free Download Manager (Windows, C++)
• GGet (Unix, Python) uses Metalink Checker
• KGet (Cross Platform, C++)
• libmetalink (C library)
• Metadl (Windows, C++)
• Metalink Checker (Cross Platform, Python). It can also be used as a library as it is in GGet
  
• Mini Downloader (Windows, C++)
• mulk (Unix, C) uses libmetalink and libcurl
  
• Phex (Cross Platform, Java)
• wxDownload Fast (Cross Platform, C++)
• Wubi (Python, previously C++)
  
• yum (Python)
  

 

Current open source Metalink generators:

• The MirrorBrain (C, Perl, Python)
  
• MirrorManager (Python)
  
• Metalink Editor (Cross Platform, GUI, Python).
• Metalink tools (Unix/Windows, C++).
• Metalink Library for PHP and Python (metalink-library) (Cross Platform, PHP, Python)
  
• Geo McFly (Unix/Windows, Python).
• Simba (Unix, Perl).
• RoPkg::Metalink / Metalink @ Packages Resources (Unix, Perl).
• cURL (Perl).

 

Ideas:

 

We didn't get accepted for GSoC 09, but these ideas could be good starting points for newcomers. They could also be used for GSoC for other projects. Please familiarize yourself with our project and software, and come up with ideas of your own. Please join our discussion list and post about your ideas!

Chunk checksum verification in DownThemAll! (Firefox Addon).

Importance: High.

Difficulty: Medium?

Skills: Javascript coding.

Bounty: $50

Details: DTA does full file verification right now. We just need it to use the chunk checksums to repair any errors. Nils Maier gives details on adding support for chunk checksums to DTA.

Google Chrome support.

Bounty: $300

Importance: High.

Difficulty: Easy to Medium?

Skills: C++ coding.

Bonus: Familiarity with libxml2, which Chrome already depends on. It uses NSS for checksums.

Details: Feature request. Chromium code. There's also libmetalink. Thread on chromium-dev.

Google Code, SourceForge, and Lauchpad support for Metalink

Importance: High.

Difficulty: Easy to Medium?

Skills: More advocacy than coding?

Details: Google Code, SourceForge, and Launchpad all serve up files for many open source projects. Some use mirrors and include checksums and signatures. These are all closed, except for Launchpad, so this involves more advocacy than coding, although maybe Metalink Library for PHP and Python (metalink-library) could be used. SourceForge bug.

Native Firefox support. 

Bounty: $500

Importance: High.

Difficulty: Hard. 

Skills: C++ coding.

Mentor: Ed Lee (Mozilla)

Bonus: Familiarity with Mozilla codebase.

Details: (This project was for Mozilla GSoC 2008, was accepted, but the student chose another organization). Firefox already handles XML & checksums so no added dependencies. The goal would be: Single source (ie no multi-source) downloads, checksum verification, and adding multiple files to the download queue.

Browser Based Metalink Download Client

Also see: openSUSE GSOC idea

Difficulty: Medium

Skills: Java, some Python coding

Mentor: Neil McNab (Appupdater / Metalink Checker author)

Progress: DLApplet (Java) has been integrated with Metalink Checker using Jython 2.5 RC 3.  It should support all features that Metalink Checker does, including multi-threaded download and checksum support.  It has been tested with Firefox 3.0.x and IE 8.  It is currently very memory intensive (around 80 MB) and a large download for a Java applet (almost 7 MB).  This can probably be improved by migrating the Python code to Java since this is likely due to the extra Jython classes.  Jython doesn't seem to adapt to applets very well.  Maybe adapting JDownloader (http://jdownloader.org/) is a better approach now.

Details: The idea is to create a "clientless" way to download Metalink files.  This way anyone with a browser with Java enabled can use and experience Metalinks without having to install a client that supports it.  It does not need to be a full featured client, probably only needs to be able to handle one metalink at a time, and has a minimal UI.  Can probably leverage reuse of existing Java or Python metalink code (Metalink Checker).
Older Notes: 1. Debian had a GSOC project which is similar, a Javascript Jigdo client. Presumably, it could be adapted to use .metalinks, but it has major limitations.  Source code is available here.  First, it only works with Firefox right now.  Second, because of limitations of javascript you can't grab files from a different domain anyway, limiting its usefulness for metalinks. 2. DLApplet (Java) has been added to the metalinks repository on sourceforge.  It is a proof of concept and only supports basic downloads right now (Metalink Lite), it needs multi-threaded download and checksum support.  It has been tested with Firefox 3.0.x and IE 7.  Maybe try adding JSpeedStreamer as a backend.

Native wget support.

Bounty: $100

Importance: High. 

Difficulty: Hard.

Skills: C coding. 

Details: Metalink support is planned in a future version of wget, but it could be added earlier for single source (ie not multi-source) downloads, with hopefully the ability to failover to other URLs if a mirror goes down. Co-ordinate with Micah Cowan (wget maintainer) on a patch.

wget GSoC list.

FlashGot support (Firefox Addon).

Importance: High.

Difficulty: Medium?

Skills: Javascript coding.

Details: FlashGot is a Firefox Addon that interfaces with external download managers.

Packaging for Linux Distributions!

Importance: High. If apps supporting Metalink exist, but people can't use them...

Details: Most apps could use packaging so people can easily use them. aria2 & KGet are heavily packaged. GGet, Metalink tools, Metalink Editor, Metalink Checker, metalink-library, libmetalink are not packaged as much.

 

Anything that makes it easier for distributions/projects to automate Metalink creation, like Apache or Drupal modules or plugins.

Difficulty: Easy to Medium.

Skills: Depends on project.

Details: Drupal proposal: posts, blogs, new thread. Peter Poeml has added metalink creation to the openSUSE download redirector! Matt Domsch added it to Fedora's MirrorManager.

 

Custom solutions for specific projects like Debian, CentOS, Eclipse, VLC, etc.

Difficulty: Easy to Medium.

Skills: Depends on project.

Details: We'd expect you to do this for multiple projects (5-8), as each will be a small amount of work

 

Native axel (download accelerator) support. 

Difficulty: Easy to Medium. 

Skills: C coding. 

Bounty: $50

Progress: Tatsuhiro has released libmetalink, a C Metalink Parser. But this will also require some axel specific code. Philipp Hagemeister, who works on Axel plans to do this.

Details: Metalink support might require adding XML & checksum library dependency.

High Definition video downloads made faster/error free with Metalinks

Difficulty: Medium to Hard.

Skills: JSpeedStreamer written in Java, Metalink Checker in Python.

Mentor: Neil McNab (Appupdater / Metalink Checker author)

Bounty: $200

Details: Could integrate with JSpeedStreamer or use Metalink Checker.

KGet: work on Metalink improvements and integrations with rest of KDE (Nepomuk, Kgpg)

Difficulty: Medium

Skills: C++ coding

Details:Add support for download repair, integration of BitTorrent/FTP/HTTP multisource downloads, pass digital signatures in Metalinks to KGpg, let Nepomuk acquire metadata for semantic desktop. KGet GSoC ideas list.

Progress: Accepted for GSOC 09!

 

Integration with Bouncer which serves up files for OpenOffice.org, Mozilla, & Gentoo

Difficulty: Medium to Hard.

Skills: PHP 

Progress?: Bram submitted a patch which has not been merged yet. 

Details:  "Bouncer is a database driven mirror management app that has three main components - a bounce script, a sentry script to check mirrors and files for integrity and a set of web-based administration tools."

  Bouncer already has a database filled with information: hash, urls, preferences, etc. However, the bouncer project is trying to keep it's mirrors in shape by using a spider and checking the downloads against the real files.

By serving up metalinks, you can keep the work of the spider down and give the users more security.

 

Integration with Arch Linux's Pacman updater

Difficulty: Medium to Hard.

Skills: C coding.

Mentor: Tatsuhiro Tsujikawa (aria2 author)

Details: Arch founder Judd Vinet provided an early Metalink script on their forums. A good start for using aria2 can be at Faster Pacman Downloads and Improve Pacman Performance. Mandriva already does something similar.

BitTorrent support for Metalink Checker

Difficulty: Medium to Hard.

Skills: Python coding.

Mentor: Neil McNab (Appupdater / Metalink Checker author)

Bounty: $200 

Details: Metalink Checker currently supports only HTTP and FTP downloads.  We should be able to reuse code from other Python based Bittorrent clients and add support to Metalink Checker.  BitTornado looks like the best Python option out there.  Note that this would then enable Bittorrent support in GGet as well.

 

Integration with FreeBSD ports infrastructure

Difficulty: Hard.

Contact: Unknown. Contact appropriate maintainers/mailing list to see if this is a possibility.

Details: "Metalink is a great concept which might be added to the ports infrastructure. Metalink files can be preprocessed in order to retain big mirror lists in bsd.sites.mk. Or an XML guru can convert bsd.sites.mk to an XML version and include it into metalinks in some way. Stub metalinks can be easily created for each port with a simple script.

At this time only www/aria2 has some Metalink support. Aria2 has a few dependencies, but it's completely unnecessary to cram it into the base system. FETCH_DEPENDS would work very nice.

Metalink support can be implemented simultaneously with P2P support, which is long overdue."

Metalink integration with Proxy/Cache

Progress: Kulbir Saini is working on this for squid with IntelligentMirror.

Bounty: $50

Importance: High.

Difficulty: ?

Details: A proxy could already have files in its cache that are listed in a metalink but they wouldn't be recognized if a different URL is used.

 

A Metalink download system similar to the BitTorrent client / server architecture

Difficulty: Ambitious and extremely hard. :)

Skills: Up to you.

Details: See Ad-Hoc Metalink Swarms?, Mirror URIs & MetaMirror Servers

 

 

 

COMPLETED!

 

Complete: Check out Appupdater 1.0!  

Design and create a friendly, easier to use GUI for Appupdater, which provides advanced functionality to Windows, similar to apt-get or yum on Linux.

Difficulty: Easy

Skills: Python coding, familiarity with UI/design issues.

Mentor: Neil McNab (Appupdater / Metalink Checker author)

Details: Get ideas from the better GUI package managers/updaters on Linux.

 

Complete: Neil has added .jigdo -> .metalink converter/downloader to Metalink Checker. 

Jigdo with .metalinks and Metalink downloader.

Difficulty: Easy.

Skills: Bash scripting for Jigdo, C++ for aria2 or other (if needed).

Mentor: Tatsuhiro Tsujikawa (aria2 author)

Complete: GGet uses Metalink Checker for metalink downloads!

GNOME/GTK GUI for Metalink Checker / Download Manager integrated into the GNOME Desktop

Difficulty: Medium ( Accepted for GNOME GSOC! )

Skills: Python coding, familiarity w/ UI/design issues

Mentor: Neil McNab (Appupdater / Metalink Checker author)

Details: Implement the features of a modern GUI download manager. Metalink Checker is written in Python and can be used as a library.

Native curl support.

In progress: COMPLETE - mulk, based on libcurl, supports metalink.

Tatsuhiro has failover working in curl but it wasn't included upstream. Patch and discussion.

Bounty: $500 ($200 from cURL Project, $300 from Metalink)

Importance: HIGHEST!

Difficulty: Hard?

Skills: C coding. 

Details: Daniel Stenberg, curl author, writes:

"A metalink file transfer library could be made as a layer on top of libcurl, and I think that is the only logical and sensible way.

Adding metalink support to the curl tool however, seems like a good idea to me…"

Requirements: (1) Single source (ie not multi-source) downloads from a single URL at a time. (2) Failover to other URLs listed in a Metalink if the current URL is unavailable. 

(3) Use chunk checksums and full file checksums so downloads retain integrity. (4) Code to curl's coding standards for quality and formatting so it can be incorporated into the curl sources.

 

Complete!: MirrorManager now supports Metalink, added by Matt Domsch.

Integrate Metalink generation into Fedora's MirrorManager

Difficulty: Medium

Skills: Python coding

Details: Coordinate with Matt Domsch, MirrorManager's maintainer. Discussion on fedora-devel & fedora-infrastructure.