Mirroring apt.puppetlabs.com creates extra directories

[Aaron Hicks]

Hi there,

Using the recommended rsync command:

rsync -av --copy-links --del rsync://apt.puppetlabs.com/packages/apt/ apt/

this mirrors the repository, but seems to create a lot of extra directories in the dist directory. These generally take the form of a distribution name followed by a timestamp number, e.g. lucid-20131219145258860605811

are these directories real? If they do exist on the puppetlabs apt server are they made invisible with .htaccess or directives in the vhost configuration? (if they are could we see that please)

Regards,

Aaron Hicks

[Konrad Scherer]

On 13-12-19 09:12 PM, Aaron Hicks wrote:
> Hi there,
>
> Using the recommended rsync command:
>
> rsync -av --copy-links --del rsync://apt.puppetlabs.com/packages/apt/

> <http://www.google.com/url?q=http%3A%2F%2Fapt.puppetlabs.com%2Fpackages%2Fapt%2F&sa=D&sntz=1&usg=AFQjCNHk-fZHUKWgwdEHx7_08rxVGvJoEw> apt/

>
> this mirrors the repository, but seems to create a lot of extra
> directories in the dist directory. These generally take the form of a
> distribution name followed by a timestamp number,
> e.g. lucid-20131219145258860605811
>
> are these directories real? If they do exist on the puppetlabs apt
> server are they made invisible with .htaccess or directives in the vhost
> configuration? (if they are could we see that please)

The suggested rsync command works but the --copy-links option uses a lot
of disk space and bandwidth unnecessarily. I use rsync to mirror
puppetlabs, Fedora, EPEL, CentOS, Mint, OpenSuSE and I modified the
fantastic mirror-fedora [1] script to handle multiple rsync repos. If
you use the rsync options from the mirror-fedora script what you get is
symlinks from the distribution name to the current
distribution-timestamp directory.

> ll mirror/puppetlabs/apt/dists
Dec 20 00:09 lucid -> lucid-20131219145258860605811
Mar 1 2013 lucid-20130301133454158727269
Jul 29 12:31 lucid-20130729093049741633793
Dec 11 00:05 lucid-20131209174434251944132
Dec 19 17:56 lucid-20131219145258860605811
Dec 20 00:09 precise -> precise-20131219151742127528147
Dec 11 00:05 precise-20131209175838499943714
Dec 19 18:22 precise-20131219151742127528147
<snip>

This allows Puppetlabs to "atomically" update the repo and it has been
working well for me for over a year now.

I have uploaded my modifications to mirror-fedora [2], but I will warn
you that it is rough and I did not make any effort to make it usable by
others. That said I hope it is useful to you.

Puppetlabs should probably update the apt repo README to update the
rsync options. Next time I am in Jira I will create a bug report.

[1]: https://github.com/dlbewley/mirror-fedora/blob/master/mirror-fedora
[2]: https://github.com/kscherer/mirror-fedora
--
Konrad Scherer, MTS, Linux Products Group, Wind River