[PROPOSAL] Separate build logs from Jenkins master

75 views
Skip to first unread message

Laurence Bordowitz

unread,
Jan 13, 2016, 1:10:53 PM1/13/16
to Jenkins Developers
Just like with abstract artifacts (see VirtualFile, ArtifactManager), I think there should be an abstract log. At Yahoo, have some issues with logs being stored on the master. Since the output stream of the log is wrapped by a remote proxy in StreamBuildListener, every log message on the slave gets transferred to the master. That's a lot of throughput. Also, some of our logs get big. That's a lot of storage.

I propose that task/build listeners be made extensible, so that users can have something other than a StreamBuildListener in a build. Further, the log should no longer be file-based; the Listener provides the input and output stream. This should be easy to serialize in the "remote logging service" perspective: just provide the service configuration, and an API call can be made from the slave to publish to the log, and any authorized user of the log can subscribe. The master can either directly subscribe and pipe the log through the UI, or another service which is authorized to subscribe to the logging service can provide the UI externally.

One of the tougher things about this change is that it will not be compatible with any BuildWrapper and ConsoleLogFilter plugins. Those plugins decorate the OutputStream on master referred to by the slave's proxy stream. Since the plugins are on the master, it forces the slave to send all log lines to the master for decoration. This pattern tightly couples the logging concept with a master-based file. In order to maintain compatibility for those users for whom this is not an issue, I recommend a "high availability mode" feature flag. This flag will enable certain features that improve availability and scalability, and disable those extension points (BuildWrapper, ConsoleLogFilter) which are incompatible with this new flow.

As Jenkins moves towards high availability and scalability, I'm certain there will be more incompatible extension points. I think having that flexibility of choice is good. Those users who aren't hitting that performance ceiling can continue to use Jenkins as-is. Users who are having performance issues can trade-off a bit of sparkle for a more scalable Jenkins. I think that this new logging pattern and feature flag will make that possible.

Thank you,
-- Larry Bordowitz

Andrew Bayer

unread,
Jan 13, 2016, 2:12:40 PM1/13/16
to jenkin...@googlegroups.com
This seems worth discussing for whatever comes after 2.0, for sure. Could you open a JIRA detailing your proposal?

A.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/199146495.3725000.1452708602502.JavaMail.yahoo%40mail.yahoo.com.
For more options, visit https://groups.google.com/d/optout.

Laurence Bordowitz

unread,
Jan 13, 2016, 8:15:10 PM1/13/16
to jenkin...@googlegroups.com

Robert Sandell

unread,
Jan 14, 2016, 7:10:37 AM1/14/16
to jenkin...@googlegroups.com
I think it's more than a little sparkle :)
Some plugins use it as hooks to fullfil parts of their primary feature, like the Build Failure Analyzer [1], some plugins even log ConsoleNotes directly into the log and makes the log look like garbage without Jenkins being there to decode it.
At least I think some more thought needs to be put into how this is mitigated.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/904751963.3804782.1452734061474.JavaMail.yahoo%40mail.yahoo.com.

For more options, visit https://groups.google.com/d/optout.



--
Robert Sandell
Software Engineer
CloudBees Inc.

Laurence Bordowitz

unread,
Jan 15, 2016, 11:35:09 AM1/15/16
to jenkin...@googlegroups.com
The plan is to have this as an optional feature flag. Every user who doesn't opt-in will experience Jenkins as it is today.

I probably shouldn't have used the word "sparkle." These plugins provide critical functionality for many users' builds (they wouldn't be used otherwise!). In our use case, the current logging paradigm creates a networking bottleneck, and we don't strictly need any plugins.

-- Larry Bordowitz


Jesse Glick

unread,
Jan 15, 2016, 11:59:38 AM1/15/16
to Jenkins Dev
On Fri, Jan 15, 2016 at 11:34 AM, 'Laurence Bordowitz' via Jenkins
Developers <jenkin...@googlegroups.com> wrote:
> The plan is to have this as an optional feature flag. Every user who doesn't opt-in will experience Jenkins as it is today.

My counterproposal would be to do any requested processing on the
slave side, avoiding the need for a compromise.

Laurence Bordowitz

unread,
Jan 15, 2016, 1:14:44 PM1/15/16
to jenkin...@googlegroups.com
I think that would be great to do all processing on the slave side! The issue would be how to send the necessary plugins over. There's no serialization requirement for BuildWrappers, and some plugins may require master-only functionality. This would break current plugins. I would not be opposed to a new extension point which executes on the slave-side and returns serializable "state packets" to the master at key points in the build.

-- Larry Bordowitz


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr30qdKMMA0fz%2BRPcSKovg%2BT8q0CikPdHy6p6ZXru7DAfw%40mail.gmail.com.

Jesse Glick

unread,
Jan 15, 2016, 1:21:11 PM1/15/16
to Jenkins Dev
On Fri, Jan 15, 2016 at 1:07 PM, 'Laurence Bordowitz' via Jenkins
Developers <jenkin...@googlegroups.com> wrote:
> The issue would be how to send the necessary plugins over. There's no
> serialization requirement for BuildWrappers, and some plugins may require
> master-only functionality. This would break current plugins.

Well, anything along these lines is going to be incompatible and break
plenty of current plugin releases. The question is just whether the
required changes can be made modest enough that plugin maintainers are
not burdened by them.

Laurence Bordowitz

unread,
Jan 19, 2016, 8:35:54 PM1/19/16
to jenkin...@googlegroups.com
I don't think it can be supported with the current extension points, nor do I think the changes will be modest. Again, I want this to be optional; that is, all these extensions will work with the StreamBuildListener as it is, but using a different build listener choice will NOT use these extensions. I think that should be an option.

-- Larry Bordowitz


--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages