Too many job in ci.jenkins.io

[Rick]

hi,

Does anyone notice that there're too many jobs in ci.jenkins.io. That cause many later PRs can't be built.

Regards,

Rick

--

https://github.com/LinuxSuRen

[Baptiste Mathus]

Each merge generally rebuilds all ongoing PRs. Given the core takes ~2.5h, necessitates 5 or 6 agents IIRC, and there are almost 100 open PRs, this causes quite a spike each time.

We are working on improving the situation to reduce the open PRs, basically review and merge or close.

HTH

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Oliver Gondža]

Just the other day, my path was crossed by a github bot[1] closing idle
PRs. Not that I would be delighted by the experience, but it might be
something to help us with the housekeeping.

[1] https://github.com/probot/stale

On 08/02/2019 08.40, Baptiste Mathus wrote:
> Each merge generally rebuilds all ongoing PRs. Given the core takes
> ~2.5h, necessitates 5 or 6 agents IIRC, and there are almost 100 open
> PRs, this causes quite a spike each time.
>
> We are working on improving the situation to reduce the open PRs,
> basically review and merge or close.
>
> HTH
>
> Le ven. 8 févr. 2019 à 04:11, Rick <linux...@gmail.com

> <mailto:linux...@gmail.com>> a écrit :

>
> hi,
>
> Does anyone notice that there're too many jobs in ci.jenkins.io

> <http://ci.jenkins.io>. That cause many later PRs can't be built.

>
> Regards,
> Rick
>
> --
> https://github.com/LinuxSuRen
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com

> <https://groups.google.com/d/msgid/jenkinsci-dev/CAMM7nTFVb0sCnpa9eAZiYDCwT3c6Ntt49VtG00fCkhXbjt7vTA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to jenkinsci-de...@googlegroups.com

> <mailto:jenkinsci-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7hMRfx93ntJjONwq1T1amziyYjDV%2BkNA%3DgMUzge8aBcQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/jenkinsci-dev/CANWgJS7hMRfx93ntJjONwq1T1amziyYjDV%2BkNA%3DgMUzge8aBcQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.

> For more options, visit https://groups.google.com/d/optout.

--
oliver

[Jesse Glick]

On Fri, Feb 8, 2019 at 2:40 AM Baptiste Mathus <m...@batmat.net> wrote:
> Each merge generally rebuilds all ongoing PRs.

It does not have to be this way.

https://issues.jenkins-ci.org/browse/INFRA-1633

Closing potentially valid, though slow-moving, PRs just to prevent
Jenkins from endlessly rebuilding them seems perverse when the actual
problem can be corrected directly.

[Kohsuke Kawaguchi]

This excessive rebuilding behaviour also causes lots of ops overhead by eating up disk space.

I think we need to change the current behaviour of ci.jenkins.io.

2019年2月8日(金) 6:25 Jesse Glick <jgl...@cloudbees.com>:

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr355LqhOH3rk3n0TnvXk%3D8ykb0vrsXjj_sWdHR3%3Dog7vg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Kohsuke Kawaguchi

[R. Tyler Croy]

(replies inline)

On Fri, 08 Feb 2019, Kohsuke Kawaguchi wrote:

> This excessive rebuilding behaviour also causes lots of ops overhead by
> eating up disk space.

The disk space issues we experience are due to unaddressed scalability tickets
in Jenkins Pipeline, we would be seeing them regardless.

>
> I think we need to change the current behaviour of ci.jenkins.io.

I have been opposed to the change, but I don't consider my voice the final say
here.

My objection is that I fundamentally do not believe the green check mark on a
pull request is accurate unless the pull request is rebuilt and tested against
the latest HEAD of the target branch, even in cases of fast-forwards.


There are 88 pull requests open on Jenkins core right now, and I think it's
absolutely false to characterize anything that hasn't been updated in more than
three to six months as "valid."

I believe Daniel and Oleg have done a great job trying to keep things labeled
and organized, so we can tell that of that 88 we have 27 marked as needing more
review and 21 marked as works in progress, both going back for 2-3 years.


If core alone is causing a denial of service on ci.jenkins.io (it's not), and
we are unwilling to address the root cause, one easy solution is to add a
specific label for core only, which would be a fixed set of agents, allowing
other workloads to bypass the bottleneck.


If the maintainers of Jenkins core wish to change the settings for
core in ci.jenkins.io, I know at least one of them has the privileges to do so.


Personally, I like the probot suggestion from Oliver a lot more. Stale pull
requests are a boat anchor which drags down an open source community, and that
is IMHO the underlying problem worth fixing here.



Cheers
--
GitHub: https://github.com/rtyler

GPG Key ID: 0F2298A980EE31ACCA0A7825E5C92681BEF6CEA2

[Gavin Mogan]

I also like the idea of stalebot, I don't know much about the state of Jenkins core, but stale PRs are frustrating and it's not hard to comment when the warning goes out.

[James Nord]

I agree with Tyler, however...

Jenkins-x addresses the issue of merging something broken by a later commit by not allowing anyone to merge and merging by machine.  it uses Tide (part of prow) to do this and groups any prs together in a merge commit and retests them in a single batch before merging (in the happy case or breaking the merge into smaller units to identify the bad pr).

the branchsource plugin has all the info required and it could (if anyone had the time and inclination) be extended to merge like this.

then you don't need to rebuild everything all the time, just before a potential merge.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20190209011052.GR23906%40grape.brokenco.de.

[Jesse Glick]

On Fri, Feb 8, 2019 at 8:11 PM R. Tyler Croy <ty...@monkeypox.org> wrote:
> If core alone is causing a denial of service on ci.jenkins.io (it's not)

But it is. I have personally had my work obstructed on multiple
occasions by core PR builds (though `git-plugin` has been a lesser
contributor).

> one easy solution is to add a specific label for core only

Though this would make for poor resource utilitization. Baptiste
already suggested using a Jenkins plugin to prioritize plugin builds
over core builds, which would be better than nothing, though it would
not address the wasted cost issue.

James Nord wrote:
> Jenkins-x addresses the issue…

Yes, Tide’s behavior is already mentioned in INFRA-1633. This is
certainly one option, though it would involve nontrivial new feature
development as well as a workflow change.