External logs storage for Pipeline/FreeStyle jobs

[kuisathaverat]

Hi,

I've started a plugin to send pipeline logs to an Elasticsearch instance using the described API in JEP-210: External log storage for Pipeline  

It is based on the example implementation pipeline-cloudwatch-logs-plugin

The thing is that it only for pipelines so I have taken a look at JEP-207: External Build Logging support in the Jenkins Core 

but it seems abandoned, or it is what I understand reading the last comment on jenkins#3575 

Is that right? if so what is the best approach to implement a distributed log system for FreeStyle jobs (and others), a ConsoleLogFilter?

--

Un Saludo

Iván Fernández Calvo

[Jesse Glick]

On Mon, May 10, 2021 at 11:11 AM kuisathaverat <kuisat...@gmail.com> wrote:

it [is] only for pipelines so I have taken a look at JEP-207: External Build Logging support in the Jenkins Core 

but it seems abandoned

Yes. Something like that could be resurrected if there is a need, with https://javadoc.jenkins.io/plugin/workflow-api/org/jenkinsci/plugins/workflow/log/package-summary.html reworked to use the core API (there are still some Pipeline-specific aspects such as step delineation).

I am personally less interested in supporting non-Pipeline builds as in addressing the key architectural limitation of current JEP-210 implementations, which is that Stapler and thus core classes involved in log handling use `long` as a cursor rather than an opaque token, making it impossible to offer high-performance log rendering inside Jenkins (“classic”, Blue Ocean, etc.) particularly for running builds.

what is the best approach to implement a distributed log system for FreeStyle jobs (and others), a ConsoleLogFilter?

Several longstanding plugins are able to mirror (non-Pipeline) log output to external systems. Without something like JEP-207 you cannot have the external system be the source of truth: Jenkins will continue streaming the log to a local file, and reading it from there.

[Oleg Nenashev]

Yes, JEP-207 was abandoned after, ehm some reorg at my company. Great to see my former teammates commenting in this thread :D

JEP-207 was indeed abandoned, but there were pull requests submitted to the Jenkins core and reference implementations (see the JEP for links). We had some disagreements with Jesse on APIs, especially w.r.t splitting the Log Storage and the Log Browser extensions (I needed it for Logstash/ELK where there might be multiple browsing implementations). But overall the API work was ready for merge with @Beta annotations IMHO. Same goes to External Log Storage API plugin. Note that JEP-207 was designed not only for AbstractProject-alike jobs, but it also added foundations for external Task Logs which is also a good use-case for centralized log systems.

The work could be definitely finished. As Jesse said, the most complicated part would be ensuring JEP-207 and JEP-210 interoperability without performance loss. Also there would be a major merge conflict if we recover the old PR I'd guess.

If somebody is interested, I am interested to participate as much as my time allows. Spoiler alert: not so much

[Markus Winter]

The logstash plugin can forward logs to for both pipeline and freestyle to elastic search and others.
But it doesn't support to read the logs from there to display in Jenkins, what it forwards is rather heavy weight and for pipeline jobs not the complete output is transferred (all the output from the pipeline core not coming from a step itself is missing, like the first lines with what caused the build and the final line with the result)

There is an implementation for what you want to do at https://github.com/SAP/elasticsearch-logs-plugin (inspired by the cloudwatch logs plugin)

This plugin was mainly implemented to forward the logs of a JenkinsFileRunner from inside a short living Kubernetes pod so it is available outside. It is partially generic as it has an extension point (similar to logstash plugin) and we implemented also forwarding to fluentd (that will then forward to Elastic Search, this has proven to be more robust in our scenario).
It can also read from ES, but performance is rather bad here. Also some plugins still rely on the log being available in the file system, so we decided to have both though this meant to copy code from the pipeline plugins.
It can also forward logs of Freestyle jobs.

The plugin also provides eventing for pipeline steps and transfers much less data than the logstash plugin.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/453bf8e6-30b9-43ce-90c2-b4f4dc7d7f78n%40googlegroups.com.

[Oleg Nenashev]

>  The logstash plugin can forward logs to for both pipeline and freestyle to elastic search and others. But it doesn't support to read the logs from there to display in Jenkins, what it forwards is rather heavy weight and for pipeline jobs not the complete output is transferred (all the output from the pipeline core not coming from a step itself is missing, like the first lines with what caused the build and the final line with the result)

I had a PoC for pluggable log storage and the Logstash plugin in 2016... We started this project but then I had to stop working on that due to the reasons outside my control. FTR https://speakerdeck.com/onenashev/jw2016-hackathon-external-build-logging-prototype-demo

FWIW it is much easier to implement a push-only pluggable log storage than one with visualization. In JEP-207 / JEP-210 Jesse and me designed a seamless integration with runtime visualization support. It ended up being a massive time overhead compared to be a push-only solution.

[Ivan Fernandez Calvo]

I will continue working on my implementation of JEP-210 for pipelines the API is simple to implement and extend https://github.com/kuisathaverat/pipeline-filebeat-logs so far it works fine on a production-grade instance, there are a few details I want to change. For the JEP-207 I think I will not try to use something similar it seems too complex to simply send the logs to external storage, I will simply redirect the logs and that's it, we do not have too many FreeStyle jobs and we can convert them to pipelines.

[Oleg Nenashev]

I agree w.r.t. JEP-207 and the current reference impl are too complex. Initially I created independent LogStorage and LogBrowser extension points right inside Jenkins, with ability to have a Noop and Completed-only log browsers in mind. Jesse was against putting two extensions inside the core. Our biss made a call, and I had to move separate extension points into External Log Storage API Plugin PoC. It complicated JEP-207 a lot IMHO. And yes it has been never delivered later due to other events. Could have landed it in Beta otherwise.

I do not see JEP-207 being merged in the current form, I would prefer to revert the design to the 2 extension point implementation. I also do not see JEP-207 being landed without seamless JEP-210 integration. Developers who target Jenkins Pipelines only (not freestyle jobs, task logs, etc.) should just continue using JEP-210

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Developers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-dev/NEvBAcjrCts/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/6cbaa684-9d4e-4960-8288-d7d967a4eabbn%40googlegroups.com.

[Tony Noble]

Looking to see if I can get a quick answer, before I spend much more time googling the wrong terms.

While accepting PRs to modernize xtrigger-lib, one contributor helpfully sorted the move from using AbstractProjects to BuildableItems, including the code to actually schedule a build of the triggered job, which went from:

abstractProject.scheduleBuild(...)

to

hudson.model.Queue.getInstance().schedule(...)

This all seems to work, except in its handling of parameterised builds - the old method happily scheduled builds with the default parameters set, but the new one leaves them all null.  So the question is:  Is there a guide anywhere on moving between these two in a way that takes care of this, or is there a quick and simple way of detecting whether a build is parameterised, so I can then add the default values to the actions when scheduling it?  So far, I'm not having much luck...

[Jesse Glick]

See https://www.jenkins.io/doc/developer/plugin-development/pipeline-integration/ and use https://javadoc.jenkins.io/jenkins/model/ParameterizedJobMixIn.html