Post-Mortem: ci.jenkins.io's ACI outage

[Damien Duportal]

Hello dear contributors, maintainers and developers!

An outage on ci.jenkins.io happened the 3 and 4th of June 2021.

The post-mortem report is available here: https://hackmd.io/@jenkins-infra/SkeMSOsqd and open to feedback for the next 7 days (either by email or in the report itself).

Any feedback, proposal or comment is welcome of course!

Sorry for the inconvenience,

For the Jenkins-Infra team,

Damien DUPORTAL

[Oleg Nenashev]

Thanks for the post mortem and summary Damien! Much appreciated.

It looks like we are still affected by the issue

[Damien Duportal]

Hello Oleg, do you have more details about the issue still affecting ci.jenkins.io?

At the time of writing these lines, the build queue is empty and we did not receive any alerts about ci.jenkins.io since the 6th of June, so it might be a new kind of issues with ACI agents.

Thanks,

Damien

-- 
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/8S564oYeDbs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d7f7f153-1f31-4734-841f-217944ead4b2n%40googlegroups.com.

[Damien Duportal]

As discussed with Oleg and James on different channels, ci.jenkins.io is currently experiencing random agent disconnection that are failing builds (in particular the long-running builds).

Status is up to date (thanks James!): https://github.com/jenkins-infra/status/pull/32 - status.jenkins.io and a banner has bee added to ci.jenkins.io.

Controller restarts will happen in the next minutes.

Damien