downloads from archives.jenkins-ci.org never finish...

[Aaron Johnson]

Can someone please verify this:

I am unable to download RPMs from archives.jenkins-ci.org, example: http://archives.jenkins-ci.org/redhat-stable/jenkins-1.424.2-1.1.noarch.rpm

The download starts, and goes very slow (less than 5Kbps) and never finishes...

Because of this I am unable to create an internal mirror of RPM packages for Jenkins.

[Richard Mortimer]

Hi,

On 16/01/2015 15:16, Aaron Johnson wrote:
> Can someone please verify this:
>
> I am unable to download RPMs from archives.jenkins-ci.org, example:
> http://archives.jenkins-ci.org/redhat-stable/jenkins-1.424.2-1.1.noarch.rpm
>
> The download starts, and goes very slow (less than 5Kbps) and never
> finishes...

I just tried and see the same behaviour as you. Firefox estimates a 9
hour download time.

I've copied the infrastructure folks on this reply to ensure that they
are aware of the problem

Richard

[Christopher Orr]

On 16/01/15 16:16, Aaron Johnson wrote:
> Can someone please verify this:
>
> I am unable to download RPMs from archives.jenkins-ci.org, example:
> http://archives.jenkins-ci.org/redhat-stable/jenkins-1.424.2-1.1.noarch.rpm
>
> The download starts, and goes very slow (less than 5Kbps) and never
> finishes...

Yup, I can reproduce this from Germany; but Rackspace seems to be doing
fine, and other nearby IP addresses seem speedy.

Since you're downloading a three-year-old package, you're being
redirected from mirrors.j.o to archives.j.o.
I'm just speculating, but maybe there's a data transfer limit on the
archives server which gets activated if there has been too much traffic,
to avoid excess data transfer costs (as has happened in the past).

> Because of this I am unable to create an internal mirror of RPM packages
> for Jenkins.

Out of interest, why do you need to create a mirror of long-outdated
Jenkins packages?

Regards,
Chris

[Aaron Johnson]

Out of interest, why do you need to create a mirror of long-outdated
Jenkins packages?

Regards,
Chris

Personally I don't care to have the old packages, but repodata/filelists.xml.gz contains references for all packages in the repository, not just the 10 newer packages stored on the jenkins-ci mirrors...

In order to make a mirror of the entire repo I must be able to download all packages in the repo metadata.

Right now I am able to download the new RPMs and the metadata, but cannot get a complete list of RPMs due to how the repository is split-up...

I am using Pulp to create the internal mirror and this problem with archives.jenkins-ci.org is causing the download of the repo to fail...

[Aaron Johnson]

If no one cares about the packages hosted from archives.jenkins-ci.org why are they included in the main repo metadata?

Can you just host the archived RPMs from a separate repo and exclude them from the main repo so that users who want to mirror the latest packages can do so without issue?

[sweeney]

When I started at my current employer I found thousands of copies of jenkins.war in /root on on of our Jenkins masters, mostly dated Nov 25 2013.  I have just found and fixed our broken Puppet configuration that must have caused this.  I'm trying to download now (I'm building a test Jenkins instance to test upgrades) and I'm currently seeing about 1KB/s from archives.jenkins-ci.org; I guess I'm paying the price for my idiot predecessor downloading the same file over and over to the tune of 350GB in one day :(

[sweeney]

Actually, I've just realised that wget preserves the mtime of the remote file by default, and furthermore our Puppet infrastruction runs 'puppet agent' at minimum 30 minute intervals.  Since we wget the file to two different locations, the most it could have gotten in any one day would be ~6GB.  So while I can no longer claim to know when these downloads occured, I can confidently state that we were probably downloading up to 96 times a day for the best part of two months at some point *after* Nov 25th 2013.  I wouldn't be surprised if we are being explicitly throttled by domain name at this point. :-/

[Alex Domoradov]

Any updates?

[Alex Domoradov]

I do really need exactly jenkins_1.565.2_all.deb (debian 7). Could anyone share with me?

[R. Tyler Croy]

(replies inline)

On Sat, 23 Jan 2016, Alex Domoradov wrote:

> I do really need exactly jenkins_1.565.2_all.deb (debian 7). Could anyone
> share with me?

archives.jenkins-ci.org is back up and running. It would appear that our
monitoring of this host was insufficient and that while Apache was running, and
listening on port 80, it didn't actually feel like serving any requests (!).


I've corrected this and have been able to download older releases properly.


- R. Tyler Croy

------------------------------------------------------
Code: <https://github.com/rtyler>
Chatter: <https://twitter.com/agentdero>

% gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
------------------------------------------------------