Hacktoberfest CI Load

[jn...@cloudbees.com]

Hi all,

as Hacktober-fest is coming I may naively expect (hope!) some greater load on the CI infrastructure.

I at least hope to find some time to get the ATH to zero failures at some point.

Is there a plan to increase the capacity (temporarily or not) for this period, or are the auto scaling / instance caps in a reasonable place now?

/James

[Jesse Glick]

On Fri, Sep 10, 2021 at 11:32 AM jn...@cloudbees.com <jn...@cloudbees.com> wrote:

I at least hope to find some time to get the ATH to zero failures at some point.

That would be great!

Is there a plan to increase the capacity (temporarily or not)

or INFRA-1633 

[Gavin Mogan]

ooh maybe use some of our digital ocean credits to expand build capacity

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr1ZvaDVX-MyDgF2CtDD5eNSAcxeMVSr4uk_5RPgCEyt1A%40mail.gmail.com.

[Oleg Nenashev]

It is a good question to discuss with the Infra Team. I have added it to the contributor summit logistics Google Doc, because this is the weekend when the load is expected to spike the most. https://docs.google.com/document/d/1QLWXNG23ui-LvQXth3UREzOvLYgTMSZcv-El0H141a4/edit#heading=h.jhodwcaw6jcl

FTR last year we did not have serious issues with CI, because we did not promote the event too much. In 2019 we had a massive CI outage in the beginning of the event, so James definitely has a point. We are interested to keep our CI stable to provide good contributor experience, whether they participate in Hacktoberfest or not

[Damien Duportal]

Hello there ! We (the Jenkins admin) we'll try to add more capability for the month of October. We'll have to plan but Digital Ocean (:lauve: Gavin) is the first step, along with exteding the current limitation for VMs and containers.

This topic has been added to the upcoming Infra meeting: https://hackmd.io/tYSCkhn7RSu7wREEKH9QIg .

Sorry for the delay of answer from the infra team, we were quite busy with the consequences of the wiki thing.

This email thread will be updated with the conclusions from the team meeting.

Cheers

Damien

[jn...@cloudbees.com]

Hi Damien,

Did this lead to anything?  The CI servers resources seem to be already exhausted with a few PRs to ATH.

/James

[Damien Duportal]

Hello James!

Thanks for the reminder and the bump.

We saw the peak this morning: The ATH is not using container agents (Is it because it uses Docker, a bunch of memory or both?) but VMs of kind "higmem".

It means that the work we’ve done on the Kubernetes agent scaling is not used.

We are working on increasing the limit on VMs, but we hit cloud limitations for now (to be quite transparent, the setup of VM agent is not really fine tuned).

Today shall see the ability to spawn 50 highmem VM in Azure and 30 in AWS EC2 (instead of 15/15 this morning).

Hope it will help.

Damien

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/7jitt4zBZvo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/cd7f6d44-d921-4cb8-ad63-e8b8ad3bdcebn%40googlegroups.com.

[James Nord]

Many thanks Damien!

/James

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CA34BD95-B624-4170-A710-0268D325C698%40gmail.com.