Mirrors reflection using Metalink files proposal.

[Eliezer Croitoru]

The idea is to make Metalink or to use metalink files to reflect the
mirrors networks.
So I have been thinking of couple ways of using either metalinks as they
are now or adding a new feature to metalinks

The first idea I had is to use a DELTA metalink file that reflects a
mirror zones marker.

I took as an example output from FedoraProject mirrormanager and used it
to demonstrate how it works.
They do have a nice html output of the public mirror list with filters
but there is no option for a machine to take advantage of this network
information.

So they have couple repos and public mirrors which not all have the same
content and for an example fedora core 6 has only 8 public active mirrors.

The Red-Hat mirror is:
http://dl.fedoraproject.org/pub/archive/fedora/linux/core/6

The metalink for the ISO is in this url:
"https://mirrors.fedoraproject.org/metalink?path=pub/archive/fedora/linux/core/6/x86_64/iso/FC-6-x86_64-disc1.iso"

So I know that the base repo is at "pub/archive/fedora/linux/core/6" and
the delta file should be in the url:
"http://dl.fedoraproject.org/pub/archive/fedora/linux/core/6/delta_link"

A metalink for that file using the mirrormanager of FedoraProject at url:
"https://mirrors.fedoraproject.org/metalink?path=pub/archive/fedora/linux/core/6/delta_link"

The delta_link is relative to the mirror base directory and must contain
some info on the repo\mirror to be valid for metalink creation.

so a metalink file like this(cut down version):
##file start
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/"
type="dynamic" pubdate="Thu, 31 Jan 2013 23:08:59 GMT"
generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<files>
<file name="delta_link">
<mm0:timestamp>number</mm0:timestamp>
<size>number</size>
<verification>
<hash type="sha1">some_hash</hash>
</verification>
<resources maxconnections="1">
<url protocol="http" type="http" location="DE" preference="100"
>http://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/core/6/delta_link</url>
<url protocol="ftp" type="ftp" location="DE" preference="100"
>ftp://ftp-stud.hs-esslingen.de/pub/Mirrors/archive.fedoraproject.org/fedora/linux/core/6/delta_link</url>
<url protocol="ftp" type="ftp" location="IE" preference="99"
>ftp://ftp.heanet.ie/pub/fedora-archive/fedora/linux/core/6/delta_link</url>
<url protocol="rsync" type="rsync" location="IE"
preference="99"
>rsync://ftp.heanet.ie/pub/fedora-archive/fedora/linux/core/6/delta_link</url>
</resources>
</file>
</files>
</metalink>
##file end

Can be helpful to determine all the available mirrors on the network
with their protocols location preferences etc.

This is the most simple way to identify full mirrors and partial mirrors
using only the existing standards and without adding\planning any extra
parts of metalinks syntax.

Another way is to add another leaf that describes nodes in the mirror
network.
A sketch of the idea is in the next xml file:
under metalink node add a node like file\files named:
- "mirrors"
-- a "master" leaf which contains data on the official mirror or master
node in the mirror network.
--- the master uses couple leafs to maybe describe some exclusions or
inclusions rules separated by space.(exclusions and inclusions is not
mandatory)
--- A resources leaf can include the same url resources as in file\files.
These can describe the absolute path\url in the mirror related to the
master node specific path\url described in the parent master leaf.

This is a cut down list of the public fedora project 17 mirrors to give
an example of the idea.
##file start
<?xml version="1.0" encoding="utf-8"?>
<metalink version="99.9" xmlns="http://www1.ngtech.co.il/"
type="dynamic" pubdate="Thu, 31 Jan 2013 22:23:32 GMT"
generator="mirrormaster">
<!-- some comments from the server -->
<mirrors>
<!-- this section contains the informaition on the master server and
the list of mirrors to the relative master path -->
<!-- how about using master url instead of the master text I have
used?? -->
<master protocol="http" domain="dl.fedoraproject.org"
path="pub/fedora/releases/17">
<!-- maybe using a PRUNENAMES like syntax for the following include
or exclude?? -->
<exclude ext="" name="">
<include ext="" name="">
<resources maxconnections="1">
<url protocol="http" type="http" location="IL" preference="100"
>http://mirror.isoc.org.il/pub/fedora/releases/17</url>
<url protocol="ftp" type="ftp" location="IL" preference="100"
>ftp://mirror.isoc.org.il/fedora/releases/17</url>
<url protocol="http" type="http" location="GB" preference="99"
>http://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/17</url>
<url protocol="ftp" type="ftp" location="GB" preference="99"
>ftp://ftp.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/releases/17</url>
<url protocol="rsync" type="rsync" location="GB"
preference="99"
>rsync://rsync.mirrorservice.org/dl.fedoraproject.org/pub/fedora/linux/releases/17</url>
<url protocol="ftp" type="ftp" location="SE" preference="98"
>ftp://ftp.sunet.se/pub/Linux/distributions/fedora/linux/releases/17</url>
</resources>
</master>
</mirrors>
</metalink>
##file end

I will be happy to hear your thoughts about the ideas.
Note that these two solutions can be integrated into existing metalinks
ready systems without any changes to the existing clients and libs.
Smooth migration

--
Thanks,
Eliezer Croitoru