Build History - 100s/1000s of builds - performance

[Tom Fennelly]

Hi.

I'm interested in hearing how people would feel about a change to how the BuildHistory widget handles build histories where there are 100s or 1000s of builds. As JENKINS-26445 discusses, the performance of this widget can stink if there are a huge number of builds.

Currently if there are more than 40 or so builds, the widget adds a "More..." link to the bottom. The user can press that link and Jenkins will attempt to render ALL of the builds on that Job, even if there are 1000s of them. This seems insane to me. I propose changing it to support a pagination type mechanism where you get "Prev", "Next" and "Top" buttons on the widget and only ever display 40'ish builds at a time.

This would obviously mean that you will only track the top/head of the build history if you are on "page 1". If you move to the next page (e.g. builds 450 down to 400), then you stop tracking the top of the history (similar to how the changelog tracking stops once you move away from the tail). This might seem like the loss of an important feature but as I see it, we can't have it both ways i.e. a responsive BuildHistory widget that can also handle an unlimited number of builds all at once.

Thoughts...

Regards,

Tom.

[Jesse Glick]

On Fri, Jan 23, 2015 at 10:53 AM, Tom Fennelly <tom.fe...@gmail.com> wrote:
> I propose changing it to support a pagination type mechanism where you get "Prev", "Next" and "Top" buttons on the widget

+1 for either pagination or ProgressiveRendering.

[do...@fortysix.ch]

+1 for pagination
I know its not exactly the same thing, but maybe this could also create some ideas on how to solve an issue in the timeline widget: https://issues.jenkins-ci.org/browse/JENKINS-22008
Domi

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2r4eNu4GzZB1aiG20aF8SxOJKmrsoKa51f0JYT2bECiA%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Fri, Jan 23, 2015 at 11:11 AM, do...@fortysix.ch <do...@fortysix.ch> wrote:
> I know its not exactly the same thing, but maybe this could also create some ideas on how to solve an issue in the timeline widget: JENKINS-22008

I think the implementation work would be mostly unrelated.
JENKINS-22008 “just” needs someone to sit down and wrestle with the
JavaScript a little bit to use a lazy iterator.

[Robert Sandell]

What about the "in queue" part of the widget?

I vote for moving it out into a separate widget otherwise tha pagination could get tricky if a build has been added or removed from the queue between page requests.

/B

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3BC%2By2hMAuyC41jf7P5UBfCsf0%2BB0LYuDHunrTnRZ9rw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Robert Sandell

Software Engineer

CloudBees Inc.

[Tom Fennelly]

Hi Bobby.

What do you mean when you talk about moving it out into a separate widget? Move which parts out? Are you saying have 2 widgets, one for in-queue/running builds and one for completed builds?

Perhaps I don't know all the use case but I was thinking (as I said in the original email) that once you move away from the top/head of the build history, then the widget stops tracking the top/head of the build history i.e. doesn't track at all or just tracks the "window" in the build history that they are currently looking at. If you want to track the top/head again, just "page" back to the top (or press the "Top") button. Seems to me like it's not really humanly possible to look at 100s/1000s of builds all at one time anyway.

[Robert Sandell]

Well, the algorithm for the buildhistory widget on the job page is, put simply; list all queued builds then start listing the history (that would include currently running builds iirc).

So if only the history part of the list was paginated, the first page iiuc would have the queue plus X builds and page 2 would have X builds making the height of the witget change between page flips. If the first page only listed "queue length - X" builds how would it handle transitions between pages if a build left the queue and started building between page flips?

And what if you have 100s/1000s of builds of that job in the queue?

I'm not saying don't do it, something needs to be done about that widget, I'm rather lifting some concerns :)

/B

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/0f084c3a-f4a9-4fd1-ad22-578697037f9a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Tom Fennelly]

Okay, thanks Bobby. Let me have a closer look at it to see can I come up with something that might work within a single widget.

[Jesse Glick]

On Mon, Jan 26, 2015 at 5:10 AM, Robert Sandell <rsan...@cloudbees.com> wrote:
> the algorithm for the buildhistory widget on the job page is […]:

> list all queued builds then start listing the history

These are two distinct widgets; the job index page shows both, but a
view page (for example) shows only the queue widget. I think both need
replacing, though the build history widget is a higher priority.

> that would include currently running builds iirc

Yes, build history includes running builds.

[Surya Gaddipati]

Might be slightly off topic but since we are talking about build history. 

I was always confused by why the most important part of the page 'build history' is tucked into a corner instead of being ( literally) front and center. Paginating that small widget in the corner would be awkward at best. 

None of the information on the main page is of much significance if you are using git with many branches in a multiuser environment. 

Here is a screenshot of revamped UI that I am working for dotci, that makes buildhistory the main focus of the page. 

http://i.imgur.com/qm8BqE8.png 

Surya

[Tom Fennelly]

I created PR #1641 around efforts to address the issue of huge build histories. Hopefully there'll be a CI build available soon (from CI PR builder).

It's a WiP but please try it out and let us know what works, doesn't work, is missing etc