apt/yum.downloads.puppetlabs.com CDN & rsync deprecation

[Daniel Dreier]

On Wednesday we put yum.puppetlabs.com and apt.puppetlabs.com behind the CloudFront CDN in order to accelerate downloads for overseas users. Both repositories have historically been served from the Linode Fremont datacenter, and download performance from Australia and Asia in particular are dramatically faster with the CDN than without.

One side effect is that we no longer support rsync. Other tools like mrepo, reposync, and apt-mirror can sync to your local mirror via HTTP. Since you'll be accessing cached content from a local CloudFront edge location, HTTP mirroring should be very fast for most users.

--

Daniel Dreier

Technical Operations Engineer

GPG: BA4379FD

[Chris Kuehl]

Is there an alternative server we can rsync from? We've been using ftpsync (the recommended tool for mirroring apt repositories), which worked great until this was turned off. ftpsync appears to be at least somewhat superior to the other options.

Thanks,

Chris

[Andreas Paul]

We would also be interested in an alternative server to rsync from.

Best regards,
Andreas Paul

[Garrett Honeycutt]

+1 to an alternate rsync service.

A lot of us already have tooling built up around rsync, so this change
breaks work flows. It also comes without warning for a service that has
been around for ages.

Best regards,
-g

On 4/24/17 7:19 AM, Andreas Paul wrote:
> We would also be interested in an alternative server to rsync from.
>
> Best regards,
> Andreas Paul
>
> On Monday, April 24, 2017 at 2:46:24 PM UTC+2, Chris Kuehl wrote:
>
> Is there an alternative server we can rsync from? We've been using
> ftpsync (the recommended tool

> <https://www.debian.org/mirror/ftpmirror> for mirroring apt

> repositories), which worked great until this was turned off. ftpsync
> appears to be at least somewhat superior to the other options.
>
> Thanks,
> Chris
>
> On Friday, April 21, 2017 at 1:54:36 PM UTC-7, Daniel Dreier wrote:
>
> On Wednesday we put yum.puppetlabs.com

> <http://yum.puppetlabs.com> and apt.puppetlabs.com
> <http://apt.puppetlabs.com> behind the CloudFront CDN in order

> to accelerate downloads for overseas users. Both repositories
> have historically been served from the Linode Fremont
> datacenter, and download performance from Australia and Asia in
> particular are dramatically faster with the CDN than without.
>
> One side effect is that we no longer support rsync. Other tools
> like mrepo, reposync, and apt-mirror can sync to your local
> mirror via HTTP. Since you'll be accessing cached content from a
> local CloudFront edge location, HTTP mirroring should be very
> fast for most users.
>
> --
> Daniel Dreier
> Technical Operations Engineer
> GPG: BA4379FD
>

> --
> You received this message because you are subscribed to the Google
> Groups "Puppet Users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to puppet-users...@googlegroups.com
> <mailto:puppet-users...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/d1c9a550-cc51-48bb-8d5b-3c6d4c5477b8%40googlegroups.com
> <https://groups.google.com/d/msgid/puppet-users/d1c9a550-cc51-48bb-8d5b-3c6d4c5477b8%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.


--
Garrett Honeycutt
@learnpuppet
Puppet Training with LearnPuppet.com
Mobile: +1.206.414.8658

[Christopher Wood]

Not sure about the rest of you, but since I have no contract with Puppet Inc. specifying the maintenance of rsync services I would shrug and move to using apt-mirror. I've used it before and it's nearly drop-in.

https://apt-mirror.github.io/
https://packages.debian.org/jessie/apt-mirror
https://forge.puppet.com/jtopjian/apt_mirror

Pity about rsync service though.

> To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f49fa010-955c-e17d-2232-95d530d80c45%40garretthoneycutt.com.

[Brian Long]

I just cannot believe they would do this without any warning for the end user community, whether open-source or commercial customers.  Look at other open-source projects like Fedora and Ubuntu for an example; they would send warnings at least two weeks in advance before turning off a service that has been around for years.

/Brian/

> > <mailto:puppet-users+unsub...@googlegroups.com>.

[Daniel Dreier]

I'm sorry we broke people's rsync-based repository mirroring. I should have communicated this change well in advance. To get you all mirroring our repositories again, I've configured rsync.puppet.com. You can sync from it using a command like:

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/yum/ /tmp/yum/

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/apt/ /tmp/apt/

Please let me know if you have any trouble with the new rsync service.

[Andrew Elwell]

On Thursday, 27 April 2017 02:31:42 UTC+8, Daniel Dreier wrote:

I'm sorry we broke people's rsync-based repository mirroring. I should have communicated this change well in advance. To get you all mirroring our repositories again, I've configured rsync.puppet.com. You can sync from it using a command like:

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/yum/ /tmp/yum/

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/apt/ /tmp/apt/

Please let me know if you have any trouble with the new rsync service

Yes. 

aelwell@galaxy-1:~$ rsync  rsync://rsync.puppet.com/packages/apt/dists/

drwxrwxr-x          4,096 2017/01/18 09:05:57 .

drwxrwxr-x          4,096 2015/11/13 08:51:24 cumulus-20151112165122938545665

drwxrwxr-x          4,096 2015/11/18 03:21:38 cumulus-20151117105654400831046

drwxrwxr-x          4,096 2015/11/18 08:27:29 cumulus-20151117152326887946964

drwxrwxr-x          4,096 2017/05/11 21:58:14 cumulus

drwxrwxr-x          4,096 2017/05/11 21:58:14 huaweios

drwxrwxr-x          4,096 2015/11/13 08:51:22 jessie-20151112165118307861024

drwxrwxr-x          4,096 2015/11/18 03:21:54 jessie-20151117110350179683670

drwxrwxr-x          4,096 2015/11/18 08:28:03 jessie-20151117152857377466419

drwxrwxr-x          4,096 2017/05/11 21:58:14 jessie 

etc

it seems to be pulling in all your nightly builds which aren't visible in the web interface

http://apt.puppetlabs.com/dists/

[Melissa Stone]

Hi Andrew,

The nightly builds should be visible from the web interface (i.e., http://apt.puppetlabs.com/dists/trusty/puppet5-nightly/). We recently added these nightly builds in to allow folks to test the new puppet5 platform. You can check out Eric0's blog post on the nightly repos at https://puppet.com/blog/new-nightly-package-repositories-for-open-source-puppet

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0ebb5ec5-eaa3-414e-aeef-eb4eb7436687%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Melissa Stone

release engineering

mel...@puppet.com | @melissaapiedra

[Melissa Stone]

On Wed, May 24, 2017 at 10:49 AM Melissa Stone <mel...@puppet.com> wrote:

On Wed, May 24, 2017 at 4:16 AM Andrew Elwell <andrew...@gmail.com> wrote:

On Thursday, 27 April 2017 02:31:42 UTC+8, Daniel Dreier wrote:

I'm sorry we broke people's rsync-based repository mirroring. I should have communicated this change well in advance. To get you all mirroring our repositories again, I've configured rsync.puppet.com. You can sync from it using a command like:

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/yum/ /tmp/yum/

rsync -av --copy-links --del rsync://rsync.puppet.com/packages/apt/ /tmp/apt/

Please let me know if you have any trouble with the new rsync service

Yes. 

aelwell@galaxy-1:~$ rsync  rsync://rsync.puppet.com/packages/apt/dists/

drwxrwxr-x          4,096 2017/01/18 09:05:57 .

drwxrwxr-x          4,096 2015/11/13 08:51:24 cumulus-20151112165122938545665

drwxrwxr-x          4,096 2015/11/18 03:21:38 cumulus-20151117105654400831046

drwxrwxr-x          4,096 2015/11/18 08:27:29 cumulus-20151117152326887946964

drwxrwxr-x          4,096 2017/05/11 21:58:14 cumulus

drwxrwxr-x          4,096 2017/05/11 21:58:14 huaweios

drwxrwxr-x          4,096 2015/11/13 08:51:22 jessie-20151112165118307861024

drwxrwxr-x          4,096 2015/11/18 03:21:54 jessie-20151117110350179683670

drwxrwxr-x          4,096 2015/11/18 08:28:03 jessie-20151117152857377466419

drwxrwxr-x          4,096 2017/05/11 21:58:14 jessie 

etc

it seems to be pulling in all your nightly builds which aren't visible in the web interface

http://apt.puppetlabs.com/dists/

Hi Andrew,

The nightly builds should be visible from the web interface (i.e., http://apt.puppetlabs.com/dists/trusty/puppet5-nightly/). We recently added these nightly builds in to allow folks to test the new puppet5 platform. You can check out Eric0's blog post on the nightly repos at https://puppet.com/blog/new-nightly-package-repositories-for-open-source-puppet

Ah, I see now you were asking about the freight directories that aren't visible (i.e., jessie-20151112165118307861024). We're digging into this now.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/0ebb5ec5-eaa3-414e-aeef-eb4eb7436687%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Melissa Stone

release engineering

mel...@puppet.com | @melissaapiedra

[Morgan Rhodes]

Hi Andrew,

Looks like some files were accidentally synced out. They've been cleaned up now.

To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAHEe_kqgtRE3OcAzaTODZkXVm%2BO%2BUuNfqQh7XyvOADqNduYwSw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Morgan Rhodes

mor...@puppet.com

Release Engineer

PuppetConf 2017, 10-12 October
San Francisco, California
Save $328, register by 30 June w/ early adopter discount