Postmortem [ci.jenkins.io, plugins.jenkins.io, wiki.jenkins.io] - 2019-12-17

[Olblak]

Hi,

Before going into what went wrong, here some context

Yesterday I was working on the Jenkins-infra/azure where multiple things needed to be done

Mainly updating DNS records related to INFRA-1797 including jenkins.io and more other things

https://issues.jenkins-ci.org/browse/INFRA-1797

What I missed while reviewing the changes, is that two months ago I manually update ci.jenkins.io VM size to have 32GB of RAM without changing the terraform code

So yesterday ci.jenkins.io was downsized to 16GB by accident which leads to

* The VM was restarted

* The Jenkins process couldn't start because it didn't have enough memory available

* plugins.jenkins.io stopped working because it depends on https://ci.jenkins.io/job/Infra/job/plugin-site-api/job/generate-data/lastSuccessfulBuild/artifact/plugins.json.gzip

To fix this issue, I updated the terraform code and then re-applied it

The second issue that happened at the same time is due to the way we define our DNS record.

We use a 'hack' in terraform to use loops, Terraform doesn't correctly keep track of the different resources and so when we add/delete DNS record, it also delete and recreate other DNS records, and if for some reasons something goes wrong before the record is re-created, then we just lose that DNS record and this is what happened to wiki.jenkins.io

So what could we do better

* plugins.jenkins.io should generate his data on his own and not having strong dependencies on ci.jenkins.io

  I would be happy to discuss it with someone willing to contribute to that service.

* DNS record, we have to test if the loop mechanism introduces in terraform 0.12  correctly handle the different resources generated based on an array

* wiki.jenkins.io, we should get rid of that service

Cheers

[Oleg Nenashev]

+1 for making plugin site self-sufficient (read as: depends only on update center). Wiki is being slowly migrated, including plugin docs and other foundation documentation, contributions will be much appreciated.

Regarding the outage, it looks like ci.jenkins.io still does not build all components (see changelog PRs for jenkins.io)

_______________________________________________
Jenkins-infra mailing list
Jenkin...@lists.jenkins-ci.org
http://lists.jenkins-ci.org/mailman/listinfo/jenkins-infra

[Olblak]

Regarding the outage, it looks like ci.jenkins.io still does not build all components (see changelog PRs for jenkins.io)

To me, it seems to be working fine, so If you an error feel free to share it

---

gpg --keyserver keys.gnupg.net --recv-key 52210D3D

---

[Marky Jackson]

I would be willing to contribute but the correct access will need to be granted for testing. 

Previous I wanted to onboard and help out but access was limited to much of the infrastructure and I ended up paying for my own infra to test and that became a burden financially.

So if we can figure that out I can more then help given my knowledge in this area.

Thanks kindly.

-- 
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/d6383938-16c3-4e1e-ae8f-b333efc7fb81%40www.fastmail.com.

[Olblak]

I would be willing to contribute but the correct access will need to be granted for testing. 

I don't have the permission to invite any additionnal users on the current azure account at the moment (I am working it), but it's not needed for working on this, Minikube is more than enough.

Timja created a helm chart which can be deployed on minikube, with instruction located here

One of the application settings defines where to fetch data as defined here

Those data are generated on ci.jenkins.io as defined here

Now those data should be generated from the pluginsite instead of ci.j.io but the 'how' depends

* Run the cronjob from a cron resource

* Run the cronjob from the plugin-site-api

---

gpg --keyserver keys.gnupg.net --recv-key 52210D3D

---

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/B04EEDEF-D317-47F1-8F81-90EBAC7AFF0C%40gmail.com.

[Oleg Nenashev]

@Olivier. CI does not seem to be triggered in some cases. See https://github.com/jenkins-infra/jenkins.io/pull/2722 or https://github.com/jenkins-infra/jenkins.io/pull/2720

[Slide]

I had a similar issue that I flagged with the repository-permission-updater. The permissions for jenkinsadmin were removed from the whole jenkins-infra org, so the repo scan was not working correctly. 

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPfivLDOoTa31-ooJFV%2BhsC4RYHt10Q0xKoAsaW-u1fAqSo35g%40mail.gmail.com.

[Olblak]

Alex rises another important change that I did yesterday (not related to the outage)

The irc bot(jenkinsadmin), was using a Github account with owner permission on the jenkins-infra organization witch means admin permission on every repository, so I reduced that permission to only jenkins-infra/jenkins-infra

Apparently that Github account was also used our jenkins instances to synchronize with Github repositories and needed at least write permission so I changed that for every repository that needs to be synced from ci.jenkins.io

That Github account may be used somewhere else that I am not aware, so keep that in mind.

---

gpg --keyserver keys.gnupg.net --recv-key 52210D3D

---

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAPiUgVf2RNMPfk--JrkRMKcHHLAVaOX2-ypbC%2B6N%3DSRkgubQqw%40mail.gmail.com.

[Mark Waite]

Have the ci jobs for jenkins.io started running again? They were offline all day yesterday with a permission error in the scan log

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/651e5dab-d6bf-4e4a-86e2-bd14702949a4%40www.fastmail.com.

[Olblak]

It should, I updated jenkins-admin permission this morning then triggered a repository scan and it was working, so for me it should

---

gpg --keyserver keys.gnupg.net --recv-key 52210D3D

---

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAO49JtF2Eu068%3DenSEs0O6Sr%3D4LphAYXcexcK4pG-Hjq%3DJVpLA%40mail.gmail.com.

[Mark Waite]

Thanks!  They are working.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/313a9535-a8f4-48aa-b84a-af750385b422%40www.fastmail.com.

--

Thanks!

Mark Waite