apt, yum, downloads, and rsync infrastructure improvements

45 views
Skip to first unread message

Daniel Dreier

unread,
Jun 28, 2017, 2:25:57 PM6/28/17
to puppet...@googlegroups.com
Today we're making improvements to apt.puppetlabs.com, yum.puppetlabs.com, and downloads.puppetlabs.com. I don't anticipate any user-visible changes, and this notification is just to let folks know so that you can report problems to me.

Specifically, we're switching out the CDN backend - it's been running off a server in Linode, and we're switching to an S3 backend, which will be faster and more reliable.

If you see any problems with the apt/yum download infrastructure, please let me know.

--
Daniel Dreier
Technical Operations Engineer
GPG: BA4379FD

SCHAER Frederic

unread,
Jun 30, 2017, 5:14:47 AM6/30/17
to puppet...@googlegroups.com

Hi,

 

I  was (up to now) mirroring the puppetlabs repositories to both :

-          Make sure I have a local copy in case your repos are down, or our internet link is too weak

-          Not hammer on your infrastructure with our servers

 

Unfortunately, we just noticed our mirroring suddenly got broken just after your changes.

We use lftp, but I just tried with a recursive wget and see the same issue, depending on the max depth I ask.

 

What I see after a simple wget –r –l1 yum.puppetlabs.com is this :

 

# ll yum.puppetlabs.com/el/6Server/

total 28

-rw-r--r-- 1 root root 1635 Jun 28 20:01 dependencies

-rw-r--r-- 1 root root 1603 Jun 28 20:01 devel

-rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_lastModified.html

-rw-r--r-- 1 root root 1824 Jun 28 20:01 index_by_name_reverse.html

-rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_size.html

-rw-r--r-- 1 root root 1592 Jun 28 20:01 PC1

-rw-r--r-- 1 root root 1618 Jun 28 20:01 products

 

Symlinks or directories now are files, and those are html files with relative hrefs to the corresponding files/directories

 

Is there a « now recommended » way of mirroring the repos ?

(I’ve tried the lftp –dereference option to no avail…)

 

I see rsync is still sort of supported, but I have the impression you would prefer that we use your new CDN (hence, get the files through cloudfront using http …)

 

Thanks && regards

Frederic Schaer

 

De : puppet...@googlegroups.com [mailto:puppet...@googlegroups.com] De la part de Daniel Dreier
Envoyé : mercredi 28 juin 2017 20:26
À : puppet...@googlegroups.com
Objet : [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAGk8subLmT0QDSh79Jd9KOCxfzRXOTqK4i0Padvhk3Abq0iQTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Daniel Dreier

unread,
Jun 30, 2017, 2:57:42 PM6/30/17
to puppet...@googlegroups.com
On Fri, Jun 30, 2017 at 2:14 AM, SCHAER Frederic <frederi...@cea.fr> wrote:

Hi,

 

I  was (up to now) mirroring the puppetlabs repositories to both :

-          Make sure I have a local copy in case your repos are down, or our internet link is too weak

-          Not hammer on your infrastructure with our servers

Much appreciated!

 

Unfortunately, we just noticed our mirroring suddenly got broken just after your changes.

We use lftp, but I just tried with a recursive wget and see the same issue, depending on the max depth I ask.

 

What I see after a simple wget –r –l1 yum.puppetlabs.com is this :

 

# ll yum.puppetlabs.com/el/6Server/

total 28

-rw-r--r-- 1 root root 1635 Jun 28 20:01 dependencies

-rw-r--r-- 1 root root 1603 Jun 28 20:01 devel

-rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_lastModified.html

-rw-r--r-- 1 root root 1824 Jun 28 20:01 index_by_name_reverse.html

-rw-r--r-- 1 root root 1832 Jun 28 20:01 index_by_size.html

-rw-r--r-- 1 root root 1592 Jun 28 20:01 PC1

-rw-r--r-- 1 root root 1618 Jun 28 20:01 products

 

Symlinks or directories now are files, and those are html files with relative hrefs to the corresponding files/directories


Symlinks don't exist anymore because we're using S3 as the backend to CloudFront, and S3 has no concept of symlinks. We're still shipping symlinks to the rsync servers, so if you "rsync -a rsync://rsync.puppet.com/packages/yum /tmp/yum" you'll get the symlinks, which is a lot more space efficient than mirroring the fully-resolved filesystems via HTTPS.
 

 

Is there a « now recommended » way of mirroring the repos ?

(I’ve tried the lftp –dereference option to no avail…)


Each of the following should work:

Rsync: rsync -a rsync://rsync.puppet.com/packages/yum /tmp/yum
Directly from S3: aws s3 sync --exclude '*.html' s3://yum.puppetlabs.com /tmp/yum
HTTPS via CloudFront: wget -r https://yum.puppetlabs.com

Of them, rsync and aws s3 sync are both incremental, so they'll be more bandwidth efficient and faster compared to using wget. If any of those options aren't working, I'd like to know about it.

I'm seeing the same behavior you described when wget max depth is set, but I don't understand the use case where you'd limit depth on it; it seems like you'd end up with a broken repository if you didn't sync down pretty much everything. 
 

 

I see rsync is still sort of supported, but I have the impression you would prefer that we use your new CDN (hence, get the files through cloudfront using http …)


rsync is absolutely still supported. After we discontinued it last time and got a lot of pushback, I've been working on an HA rsync.
 

 

Thanks && regards

Frederic Schaer

 

De : puppet...@googlegroups.com [mailto:puppet-users@googlegroups.com] De la part de Daniel Dreier


Envoyé : mercredi 28 juin 2017 20:26
À : puppet...@googlegroups.com
Objet : [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements

 

Today we're making improvements to apt.puppetlabs.com, yum.puppetlabs.com, and downloads.puppetlabs.com. I don't anticipate any user-visible changes, and this notification is just to let folks know so that you can report problems to me.

 

Specifically, we're switching out the CDN backend - it's been running off a server in Linode, and we're switching to an S3 backend, which will be faster and more reliable.

 

If you see any problems with the apt/yum download infrastructure, please let me know.

 

--

Daniel Dreier

Technical Operations Engineer

GPG: BA4379FD

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAB586860327CB498EF79903967FEA2317B21A78%40E-EXDAGE-A0.extra.cea.fr.

For more options, visit https://groups.google.com/d/optout.

SCHAER Frederic

unread,
Jul 3, 2017, 4:33:55 AM7/3/17
to puppet...@googlegroups.com

Hi Daniel,

 

Many thanks for your answer, and for the detailed list of ways to mirror the repos.

The wget depth limit was just a way of testing without waiting for a whole tree sync whether this was just lftp on my side wich was failing, or really something that changed in the repos.

 

I tried rsync last week and it was awfully slow, and I ended up both upgrading lftp to a freshly compiled one + removing an “ignore-time” option, which fixed the issue for now – but I’m looking for a decent lftp replacement now ;).

 

I’ll see if I can implement someting like pulp , reposync or sync-repo, but I’d prefer finding something that’s not just limited to syncing yum

 

Regards

 

De : puppet...@googlegroups.com [mailto:puppet...@googlegroups.com] De la part de Daniel Dreier
Envoyé : vendredi 30 juin 2017 20:57
À : puppet...@googlegroups.com
Objet : Re: [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements

 

De : puppet...@googlegroups.com [mailto:puppet...@googlegroups.com] De la part de Daniel Dreier


Envoyé : mercredi 28 juin 2017 20:26
À : puppet...@googlegroups.com
Objet : [Puppet Users] apt, yum, downloads, and rsync infrastructure improvements

 

Today we're making improvements to apt.puppetlabs.com, yum.puppetlabs.com, and downloads.puppetlabs.com. I don't anticipate any user-visible changes, and this notification is just to let folks know so that you can report problems to me.

 

Specifically, we're switching out the CDN backend - it's been running off a server in Linode, and we're switching to an S3 backend, which will be faster and more reliable.

 

If you see any problems with the apt/yum download infrastructure, please let me know.

 

--

Daniel Dreier

Technical Operations Engineer

GPG: BA4379FD

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.



 

--

Daniel Dreier

Technical Operations Engineer

GPG: BA4379FD

--

You received this message because you are subscribed to the Google Groups "Puppet Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAGk8suZaso1PSeQUvKQ3QYKRuPhGUagBCgiOEQJmKR-eGRBFFQ%40mail.gmail.com.

Reply all
Reply to author
Forward
0 new messages