[#191951] Your support inquiry @jfrog.com

[JFrog Support]

Hello,

Thank you for contacting JFrog Support.

We want to make sure that your ticket is prioritized correctly for follow-up.

We will review the ticket and one of our representatives will get back to you.

You may also find some interesting information on our User Guide.

If there is an urgent/critical issue please reply with the information and we will make sure to prioritize it accordingly.

Best regards,
JFrog Support





ref:_00D20M3v0._500694043Rv:ref

[Damien Duportal]

Hi JFrog Support,

We haven't heard anything from your services about this incident, could you give us news or follow up please?

Thanks

[Damien Duportal]

Hi dear JFrog support,

we haven’t add any feedback or news from your service about the last week outage.

It appears that there seems to be another outage on your service today: the response time are way above 5s on the "ping" URL (https://repo.jenkins-ci.org/api/system/ping) AND we see generally terribly slow download from this artifact repository while there isn’t any information on https://status.jfrog.io/.

Could you help us please?

For the Jenkins Project,

Damien DUPORTAL

[JFrog Support]

Hi Damien,

Thank you for reaching out to JFrog Support.

I have parsed over the logs where I did not find any errors on the Artifactory logs for 13th Jan. Hence, I would request you to share the below information to investigate further.

1) Whether you are seeing slowness while executing a specific build or for all the builds? If it is a specific build, please share the build script.

2) Do let us know the build name and Build ID for which you observed the slowness and time of issue occurrence along with timezone so that we will correlate the logs at our end.

3) Could you please confirm whether this issue is occurring for specific users or all the users? This will rule the network issues.

4) Please let us know when you have started observing this issue?

Looking forward to your response.
 

Best regards,
Sandeep Anikapati
JFrog Support





                                                                                                                                   

ref:_00D20M3v0._500694043Rv:ref

[Damien Duportal]

Hello, thanks for your feedback !

1) Whether you are seeing slowness while executing a specific build or for all the builds? If it is a specific build, please share the build script.

All builds with a maven download are affected, running on AKS US-East, AWS Frankfurt, AWS US-East-2 or locally in Belgium and France.

They haven't any error but they are suffering from really slow download rates, and thus they fail because of the one hour timeout (expected build times of 10 minutes max)

Example on a local maven build of 

https://github.com/jenkinsci/jenkins-infra-test-plugin/ 

 from Belgium:

- everything coming from Apache Central: ~3Mb/s

- everything coming from repo.jenkins-ci.org: less than 5Kb/s

2) Do let us know the build name and Build ID for which you observed the slowness and time of issue occurrence along with timezone so that we will correlate the logs at our end.

Any build on 

https://ci.jenkins.io/, especially those: https://ci.jenkins.io/job/Plugins/ 

3) Could you please confirm whether this issue is occurring for specific users or all the users? This will rule the network issues.

This issue has been encountered by multiple users, cf 

https://github.com/jenkins-infra/helpdesk/issues/2733 

Ex: 

https://github.com/jenkins-infra/helpdesk/issues/2733#issuecomment-1015295048

> It looks like we are experiencing a similar disruption: build take much more than usual to build. While it usually takes max 30 mins, it is still building at 53 minutes. ci.jenkins.io/blue/organizations/jenkins/Plugins%2Fconfig-file-provider-plugin/detail/master/172/pipeline 

4) Please let us know when you have started observing this issue?

The issues seems to happen every morning (European time) since 13th of January 2022, as you can see in our mentioned issue above.

For the Jenkins Infra Team,

[Damien Duportal]

Hi dear JFrog support,

Additionally to our last email, we also have gathered data from our metrics system.

We have a probe that measure the response time of the endpoint https://repo.jenkins-ci.org/api/system/ping (an webpage expecting to answer "OK" in the body with HTTP/200).

You can see in attachment the measure of the past 2 weeks: you can clearly see that there have been answer (no error) but more than 5s in average, which triggers alerts on our monitoring system.

The slow response time are correlated with the slow artifact download rates: the 13th of January, the 17th and today (18th).

[Damien Duportal]

Hi dear support,

Is it possible to have a post-mortem or more information on what happened the 13th, the 17th and the 18th?

For the Jenkins Infra,

Damien DUPORTAL

Le 18 janv. 2022 à 15:06, Damien Duportal <damien....@gmail.com> a écrit :

Hi dear JFrog support,

Additionally to our last email, we also have gathered data from our metrics system.

We have a probe that measure the response time of the endpoint https://repo.jenkins-ci.org/api/system/ping (an webpage expecting to answer "OK" in the body with HTTP/200).

You can see in attachment the measure of the past 2 weeks: you can clearly see that there have been answer (no error) but more than 5s in average, which triggers alerts on our monitoring system.

The slow response time are correlated with the slow artifact download rates: the 13th of January, the 17th and today (18th).

<Capture d’écran 2022-01-18 à 15.02.56.png>

[JFrog Support]

Hi Damien,

Thank you for your response.

I am currently checking internally regarding this behavior, I will get back to you with my findings at the earliest.

[Damien Duportal]

Hi dear JFrog,

it’s now the 4th consecutive day with downloads speed around 4-5 Kb/s in average.

This outage is completely blocking the whole Jenkins project and we definitively need some help from you.

For the Jenkins Project,

Damien DUPORTAL

[JFrog Support]

Hi Damien,

Further checking internally, we made a few changes at our end. can you please check if the performance has improved?

Looking forward to your update.

[Damien Duportal]

Thanks, we are going to test this for the 3 next hours and we shall report back to you as soon as we can gather meaningful results.

[JFrog Support]

Hi Damien,

Thank you for your response.

Sure, we will await your further update on this.

[Damien Duportal]

Hi dear support team,

for now, we do not see any changes in the download speeds : it’s stil really really slow and unusable.

Reproduction:

- Retrieve the (public) project http://github.com/jenkinsci/jenkins-infra-test-plugin 

- Ensure that you have a JDK11 and Maven (3.6+) in your path

- Ensure that the directory "~/.m2/repository" does not exist or is empty (to ensure there is no cache)

- Run the command "mvn clean compile"

Excerpts on my personal machine for instance:

Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven/2.2.0/maven-2.2.0.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven/2.2.0/maven-2.2.0.pom (22 kB at 2.9 kB/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-parent/11/maven-parent-11.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-parent/11/maven-parent-11.pom (32 kB at 4.0 kB/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/apache/5/apache-5.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/apache/5/apache-5.pom (3.8 kB at 452 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-project/2.2.0/maven-project-2.2.0.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-project/2.2.0/maven-project-2.2.0.pom (2.8 kB at 336 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-settings/2.2.0/maven-settings-2.2.0.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-settings/2.2.0/maven-settings-2.2.0.pom (2.2 kB at 253 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-model/2.2.0/maven-model-2.2.0.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/apache/maven/maven-model/2.2.0/maven-model-2.2.0.pom (3.2 kB at 380 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-utils/1.5.15/plexus-utils-1.5.15.pom (6.8 kB at 824 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus/2.0.2/plexus-2.0.2.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus/2.0.2/plexus-2.0.2.pom (12 kB at 1.0 kB/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-interpolation/1.11/plexus-interpolation-1.11.pom (889 B at 95 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-components/1.1.14/plexus-components-1.1.14.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-components/1.1.14/plexus-components-1.1.14.pom (5.8 kB at 600 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-container-default/1.0-alpha-9-stable-1/plexus-container-default-1.0-alpha-9-stable-1.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-container-default/1.0-alpha-9-stable-1/plexus-container-default-1.0-alpha-9-stable-1.pom (3.9 kB at 427 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-containers/1.0.3/plexus-containers-1.0.3.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus-containers/1.0.3/plexus-containers-1.0.3.pom (492 B at 53 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus/1.0.4/plexus-1.0.4.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/codehaus/plexus/plexus/1.0.4/plexus-1.0.4.pom (5.7 kB at 571 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/junit/junit/4.13.1/junit-4.13.1.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/junit/junit/4.13.1/junit-4.13.1.pom (25 kB at 2.7 kB/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.pom (766 B at 82 B/s)
Downloading from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom
Downloaded from repo.jenkins-ci.org: https://repo.jenkins-ci.org/public/org/hamcrest/hamcrest-parent/1.3/hamcrest-parent-1.3.pom (2.0 kB at 215 B/s)

As you can see the downloads are … unusable (less than 1 kb/s).

The build with full download is expected to be around 2-3 minutes usually. It’s been 12 min downloading from https://repo.jenkins-ci.org .

[JFrog Support]

Hi Damien,

Thank you for the update.

We have upgraded your instance, can you monitor now and let us know if there is performance has improved?

[JFrog Support]

Hi Damien,

Can you please let us know if you are still encounter the mentioned slowness?
As I understand, things look much better now.

Looking forward to your confirmation.

Best regards,
Guy Cohen

[Damien Duportal]

Hi dear JFrog support.

We can confirm that is has been fixed, either from local builds and on our build platform.

A change around 14h58 UTC seems to have fix it.

Our issue has been closed on our side: https://github.com/jenkins-infra/helpdesk/issues/2733 .

Many many thanks for your help and work !

I got a last question: did we follow correctly the support procedure, or is there something that we, as end users of your service, could have done better to help you (so we can improve in the future)?

For the Jenkins Infra team

Damien DUPORTAL

[Damien Duportal]

Hello,

it seems that the slowness are back since 08:00 UTC today.

We are currently in a blocked status again, could you look again please? (smells like an automatic configuration delivery ;) )

[JFrog Support]

Hi Damien,

I am currently looking into the issue, I will get back to you with the updates shortly.

Best regards,
Sandeep Anikapati

[JFrog Support]

Hi Damien,

Can you please monitor the behavior now and let us know if you are still encountering the performance issue?

Looking forward to your update.

[Damien Duportal]

Hi !

It seems that the problem is now resolved. Many thanks!

[JFrog Support]

Hi Damien,

Thank you for the confirmation.

I am glad to hear that the issue has been resolved.

Regarding your query on support procedure (asked on Jan 20), I am looping in our service team and your Account Manager who will be happy to assist you further on the queries.