ftp.belnet.be jenkins mirror

[Louis Roebben]

Hi

I just wanted to bring to attention the following.
https://get.jenkins.io/war/2.423/jenkins.war?mirrorstats seems to show
ftp.belnet.be/mirror/jenkins as being out of sync. but the date changed
is being show on the site is today.

Yours truly,

Louis Roebben

[Mark Waite]

As far as I know, the Jenkins project no longer assumes that ftp.belnet.be is a working Jenkins mirror.  I believe there were issues in the past that caused us to record it as "offline", just as serverion was recorded as "offline"

https://get.jenkins.io/war/2.423/jenkins.war?mirrorlist shows the six active mirrors that are included in the Jenkins status page.

Mark Waite

[Damien Duportal]

Hi Louis, thanks for your email!

We disabled the mirror a few month ago (without deleting its reference) as we saw a lot discrepancies back in that time.

However we never tracked the actions properly in the public issue tracker so we can't really be sure why we disabled it.

As such, I've added a comment in https://github.com/jenkins-infra/helpdesk/issues/3136#issuecomment-1722432348 and we'll most probably triage it to our next milestone (next week) so we can re-enable the mirror sync as soon as possible for belnet mirror.

If you have any question or remark, please shoot! Thanks for helping the Jenkins project.

We'll let you know, do not hesitate to comment in this email thread or in the github helpdesk issue.

Damien Duportal

[Damien Duportal]

Hi @Louis

The mirror is now actively scanned and used.

Thanks for raising the topic!

Damien DUPORTAL

[David Raison]

Hi,

Just a quick heads up that this mirror is still causing problems.

See https://community.jenkins.io/t/how-to-hardconfig-the-mirror-to-use/10099 and https://community.jenkins.io/t/issue-while-upgrading-plugins-on-latest-jenkins/9846 for recent reports by other people.

Our CI server, which is hosted in France is made to download from that server and it or the IP range is uses seems to have been blocked by belnet. (I can access the site just fine locally).

The following packages will be upgraded:
  jenkins
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 88.9 MB of archives.
After this operation, 55.3 kB disk space will be freed.
Do you want to continue? [Y/n]
Err:1 https://pkg.jenkins.io/debian binary/ jenkins 2.427
  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Failed to fetch https://ftp.belnet.be/mirror/jenkins/debian/jenkins_2.427_all.deb  Could not connect to ftp.belnet.be:443 (193.190.198.27), connection timed out
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I can understand that someone wants to protect their infrastructure from attacks or the like, but that makes belnet.be no longer a valid mirror and pkg.jenkins.io does not seem to offer any alternatives or fallback options. Meaning I am unable to upgrade jenkins, which puts me (and everyone with the same issue) at risk myself.

Thanks,
David

[Damien Duportal]

Hi @louis....@telenet.be  , could you help us on this topic to check if the IP blocking is on belnet side or on @dra...@gmail.com 's network side?

We have temporarily disabled the Belnet mirror (ref. https://github.com/jenkins-infra/helpdesk/issues/3784) to unblock people, and it would be nice to get a confirmation from Belnet side.

Thanks in advance for your help on the project!

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/7a7ae67b-5f26-440c-9f6b-63119e0e3d2fn%40googlegroups.com.

[Damien Duportal]

For info, we reached the Belnet administrator whom confirmed there was a block list and cleared it.

The mirror is back to our list of enabled mirrors.

[Robin Banbury]

I believe we are being affected by this issue too, even though the block list has been cleared.

Our Jenkins instances are managed using Puppet. We tried an upgrade today and a lot of the plugins failed to download from ftp.belnet.be with the message 'Network is unreachable'.

Is it possible this is still an issue?

Output from curl, for reference:

$ curl -vvv -L https://updates.jenkins.io/download/plugins/h2-api/1.4.199/h2-api.hpi -o h2-api                                                                                                                                                                                                                                                                                                       23 ✘  1s    13:47:54 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 52.202.51.185:443...
* Connected to updates.jenkins.io (52.202.51.185) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3979 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=updates.jenkins.io
*  start date: Oct 20 05:08:42 2023 GMT
*  expire date: Jan 18 05:08:41 2024 GMT
*  subjectAltName: host "updates.jenkins.io" matched cert's "updates.jenkins.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* [HTTP/2] [1] OPENED stream for https://updates.jenkins.io/download/plugins/h2-api/1.4.199/h2-api.hpi
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: updates.jenkins.io]
* [HTTP/2] [1] [:path: /download/plugins/h2-api/1.4.199/h2-api.hpi]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /download/plugins/h2-api/1.4.199/h2-api.hpi HTTP/2
> Host: updates.jenkins.io
> User-Agent: curl/8.4.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [281 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Thu, 23 Nov 2023 13:48:51 GMT
< server: Apache
< location: https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi
< content-length: 240
< content-type: text/html; charset=iso-8859-1
<
* Ignoring the response-body
{ [240 bytes data]
100   240  100   240    0     0    858      0 --:--:-- --:--:-- --:--:--   857
* Connection #0 to host updates.jenkins.io left intact
* Issue another request to this URL: 'https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi'
*   Trying 20.7.178.24:443...
* Connected to get.jenkins.io (20.7.178.24) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4039 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=get.jenkins.io
*  start date: Nov  4 10:52:28 2023 GMT
*  expire date: Feb  2 10:52:27 2024 GMT
*  subjectAltName: host "get.jenkins.io" matched cert's "get.jenkins.io"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://get.jenkins.io/plugins/h2-api/1.4.199/h2-api.hpi
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: get.jenkins.io]
* [HTTP/2] [1] [:path: /plugins/h2-api/1.4.199/h2-api.hpi]
* [HTTP/2] [1] [user-agent: curl/8.4.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /plugins/h2-api/1.4.199/h2-api.hpi HTTP/2
> Host: get.jenkins.io
> User-Agent: curl/8.4.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Thu, 23 Nov 2023 13:48:52 GMT
< content-type: text/html; charset=utf-8
< content-length: 0
< location: https://ftp.belnet.be/mirror/jenkins/plugins/h2-api/1.4.199/h2-api.hpi
< cache-control: private, no-cache
< link: <https://ftp.halifax.rwth-aachen.de/jenkins/plugins/h2-api/1.4.199/h2-api.hpi>; rel=duplicate; pri=1; geo=de
< strict-transport-security: max-age=2592000; includeSubDomains; preload
<
{ [0 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #1 to host get.jenkins.io left intact
* Issue another request to this URL: 'https://ftp.belnet.be/mirror/jenkins/plugins/h2-api/1.4.199/h2-api.hpi'
*   Trying 193.190.198.27:443...
*   Trying [2001:6a8:3c80::27]:443...
* Immediate connect fail for 2001:6a8:3c80::27: Network is unreachable

[Damien Duportal]

Hi Robin,

could you open an issue in https://github.com/jenkins-infra/helpdesk/issues and provides us (privately through the email jenkins-i...@googlegroups.com) your outbound public IP(s) please, so we'll see with Belnet

Damien

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/c303a2d6-eb30-49f3-a548-9181126e88d4n%40googlegroups.com.

[Robin Banbury]

Thanks Damien.

New issue is here: https://github.com/jenkins-infra/helpdesk/issues/3830

I'll send the email with our outbound IPs now.

Cheers,

Robin

On Thursday, 23 November 2023 at 14:16:37 UTC damien....@gmail.com wrote:

Hi Robin,

could you open an issue in https://github.com/jenkins-infra/helpdesk/issues and provides us (privately through the email jenkins-infra-team@googlegroups.com) your outbound public IP(s) please, so we'll see with Belnet

Damien

To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infra+unsubscribe@googlegroups.com.

[Damien Duportal]

Email received, we've contacted the belnet admins.

Le jeu. 23 nov. 2023 à 16:27, 'Robin Banbury' via Jenkins Infrastructure <jenkin...@googlegroups.com> a écrit :

Thanks Damien.

New issue is here: https://github.com/jenkins-infra/helpdesk/issues/3830

I'll send the email with our outbound IPs now.

Cheers,

Robin

On Thursday, 23 November 2023 at 14:16:37 UTC damien....@gmail.com wrote:

Hi Robin,

could you open an issue in https://github.com/jenkins-infra/helpdesk/issues and provides us (privately through the email jenkins-i...@googlegroups.com) your outbound public IP(s) please, so we'll see with Belnet

Damien

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/c303a2d6-eb30-49f3-a548-9181126e88d4n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Jenkins Infrastructure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkins-infr...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/jenkins-infra/36587e45-9071-4274-a07f-21855b37c123n%40googlegroups.com.

[Damien Duportal]

Hi @Robin Banbury ,

The Belnet administrators just answered and they do not have any firewall rules blocking your IPs.

The problme might be somewhere else: can you check on your side that your network don't have any outgoing firewall rules potentially blocking egress HTTP requests?

Also can you try a traceroute as well to see where the requests are blocked?

Tip: you might want to search your infrastructure for the IP "52.202.51.185" which is the Ip associated to both "updates.jenkins.io" and "get.jenkins.io": if you have any sort of "allow-list" it will help you to find it.

Damien, for the Jenkins infrastructure team

[Robin Banbury]

Hi Damien,

Thanks for your response.

I’ve spoken with our IT team - we don’t have any firewalls blocking egress HTTP requests. 

The other European mirror (https://ftp.halifax.rwth-aachen.de/jenkins/) is working fine for us.

That’s essentially how we managed to apply the upgrade - we ran the plugin installation scripts 3 or 4 times, just downloading the remaining plugins each time - eventually, the plugin installation script started pulling the remaining plugins from the other European mirror https://ftp.halifax.rwth-aachen.de/jenkins/ - with this mirror there were no problems downloading the plugins. After a few attempts we were able to get all the plugins we needed and apply the change.

On Monday, we can do some more experiments and run the traceroute test you suggested.

Kind regards,

Robin

Robin Banbury Principal Release Engineer  |  ForgeRock t (+44) 7867 315 568  |  e robin....@forgerock.com web www.forgerock.com

[Damien Duportal]

Thanks for the details.

We are keeping belnet disabled a few days (in case you have need to rebuild your instances on short notice).

Please note that the Aachen University is less closer to you than the Belnet server, hence the selection.

Having the traceroute would help a lot to identify where are your request blocked as I can't reproduce the issue from my AWS, French, Belgium and Azure network zones.

Cheers

Damien

[Damien Duportal]

Hi Robin, any news?

[Robin Banbury]

Hi Damien,

Sorry, I wasn’t able to get back to you before Christmas, and I was on parental leave after.

When the Belnet mirror was removed, we were able to download the Jenkins plugins fine.

Since the Belnet mirror was re-enabled, we haven’t seen the issue happen again, so I think things are ok now.

Kind regards,

Robin

Robin Banbury Principal Release Engineer  |  ForgeRock t (+44) 7867 315 568  |  e robin....@forgerock.com web www.forgerock.com

On 29 Nov 2023, at 07:01, Damien Duportal <damien....@gmail.com> wrote:

Hi Robin, any news?

Le ven. 24 nov. 2023 à 16:37, Damien Duportal <damien....@gmail.com> a écrit :

Thanks for the details.

We are keeping belnet disabled a few days (in case you have need to rebuild your instances on short notice).

Please note that the Aachen University is less closer to you than the Belnet server, hence the selection.

Having the traceroute would help a lot to identify where are your request blocked as I can't reproduce the issue from my AWS, French, Belgium and Azure network zones.

Cheers

Damien

Le ven. 24 nov. 2023 à 16:29, Robin Banbury <robin....@forgerock.com> a écrit :

Hi Damien,

Thanks for your response.

I’ve spoken with our IT team - we don’t have any firewalls blocking egress HTTP requests. 

The other European mirror (https://ftp.halifax.rwth-aachen.de/jenkins/) is working fine for us.

That’s essentially how we managed to apply the upgrade - we ran the plugin installation scripts 3 or 4 times, just downloading the remaining plugins each time - eventually, the plugin installation script started pulling the remaining plugins from the other European mirror https://ftp.halifax.rwth-aachen.de/jenkins/ - with this mirror there were no problems downloading the plugins. After a few attempts we were able to get all the plugins we needed and apply the change.

On Monday, we can do some more experiments and run the traceroute test you suggested.

Kind regards,

Robin

Robin Banbury Principal Release Engineer  |  ForgeRock t (+44) 7867 315 568  |  e robin....@forgerock.com web www.forgerock.com

<PastedGraphic-1.tiff>

CONFIDENTIALITY NOTICE: This email may contain confidential and privileged material for the sole use of the intended recipient(s). Any review, use, distribution or disclosure by others is strictly prohibited.  If you have received this communication in error, please notify the sender immediately by e-mail and delete the message and any file attachments from your computer. Thank you.

[Damien Duportal]

Hi Robin,

Congratulations for the release of a new human ;) Hope you'll have some time to rest!

Thanks for the confirmation, it is really kind of you!

Damien