[Essentials] Error Telemetry logging: how to do it (was: Re: Thoughts on sending error telemetry for Jenkins Essentials)

[Baptiste Mathus]

Hello,

(Changing the subject also because I don't think it's strictly necessary for people to read the previous thread to be able to provide help/feedback here).

I've started working on https://issues.jenkins-ci.org/browse/JENKINS-49805, and the tests I've done so far led me to see two possible ways out for this design.

Now we are able to catch the logs we want, we started to discuss how we would ship them to the backend.

For many reasons (the main one being the client is the one securely connected to the backend), it makes sense to push those logs to the [local] evergreen-client. The most obvious way for it is to push those logs using a SocketHandler.

The thing is: SocketHandler is unusable for real life IMO: any loss of connection will not be reconnected. So we need to develop something that *will* reconnect and be more robust for our usage.

I see two possible ways for this:

== Adding that improved SocketHandler into the core

Pros: 

* we can use a simple logging.properties file and configure the logging exclusively from there. No plugin or so involved. 

* And we get early Jenkins startup logs without without stunt.

Cons: means we add that component in the core, or at least in some jar added to its classpath or something (?). Less common usage pattern than a plugin, could it lead to issues? (not saying we couldn't make it work. I'm more talking about "is it worth taking this path")

== Adding that SocketHandler into an "Essentials Jenkins plugin"

Pros:

* maybe a more usual way to change the features Jenkins offers (keeping the core leaner and leaner)

* maybe slightly more usual too to upgrade (though Essentials is designed also to make the core upgrades far more common than it is now too, granted)

Cons:

* I have a PoC for this, but we need to use a bit more convoluted way to get the early logs from the startup reading Jenkins.logRecords before being able to setup our own JUL Handler 

* we'll miss a few very early startup logs which are, IIUC, sent before the core sets up the RingBufferLogHandler to populate Jenkins.logRecords.

WDYT? Any feedback/opinion on the preferred way forward? I've started writing the JEP for this, and that's why I'm asking: I want to make sure to get everything possible with pros & cons of both strategies to move forward.

Thanks!

2018-03-01 1:17 GMT+01:00 R. Tyler Croy <ty...@monkeypox.org>:

I just wanted to make sure I shared an update on this thread. I have filed the
following ticket (https://issues.jenkins-ci.org/browse/JENKINS-49805) to
capture some of the prototyping work necessary here.

I expect we'll have a draft JEP coming up after the prototyping work is done.


Thanks for the ideas and feedback everybody!



Cheers
- R. Tyler Croy

------------------------------------------------------
     Code: <https://github.com/rtyler>
  Chatter: <https://twitter.com/agentdero>
     xmpp: rty...@jabber.org

  % gpg --keyserver keys.gnupg.net --recv-key 1426C7DC3F51E16F
------------------------------------------------------

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/20180301001723.rhhvyfhwpxthmsnx%40blackberry.coupleofllamas.com.

For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Fri, Mar 30, 2018 at 11:15 AM, Baptiste Mathus <m...@batmat.net> wrote:
> SocketHandler is unusable for real life IMO: any loss of connection will not be reconnected.
> So we need to develop something that *will* reconnect and be more robust for our usage.

You seem to be assuming that you must stream logs to the
`evergreen-client` using a local TCP socket, which is not obvious to
me. There are other options, such as a text file or set of files in
the `var` directory, which would have different robustness
characteristics.

> means we add that component in the core, or at least in some jar added
> to its classpath or something (?). Less common usage pattern than a plugin,
> could it lead to issues?

I think what you are looking for is `jenkins-module` packaging: akin
to a plugin, but bundled in `jenkins.war!/WEB-INF/lib/*.jar` so that
its classes are available in the same loader as `jenkins-core.jar`. If
you are patching `jenkinsci/jenkins/war/pom.xml` for this anyway, this
is trivial (just add it to the list of modules); otherwise, there is a
mojo in `maven-hpi-plugin` to create a customized Jenkins WAR file
based on a stock `org.jenkins-ci.main:jenkins-war` release with some
additions and modifications, which would be appropriate for a
distribution like Essentials / Evergreen anyway.

> we need to use a bit more convoluted way to get
> the early logs from the startup reading Jenkins.logRecords before being able
> to setup our own JUL Handler

This is pretty trivial, no? You register your handler, and at that
moment you also copy everything in `Jenkins.logRecords`—done.
Obviously if there were more than ~256 messages prior to this plugin’s
initialization, you will miss some of the earlier ones, but in such a
case it should be obvious that something is very wrong.

> we'll miss a few very early startup logs which are, IIUC, sent before the
> core sets up the RingBufferLogHandler to populate Jenkins.logRecords.

Sure, but if there were any `WARNING`+ messages printed before that
point very early in Jenkins startup, it likely that the instance is
more or less hosed and you will notice this from a standard HTTP
status check anyway.

[Baptiste Mathus]

2018-04-02 15:27 GMT+02:00 Jesse Glick <jgl...@cloudbees.com>:

On Fri, Mar 30, 2018 at 11:15 AM, Baptiste Mathus <m...@batmat.net> wrote:
> SocketHandler is unusable for real life IMO: any loss of connection will not be reconnected.
> So we need to develop something that *will* reconnect and be more robust for our usage.

You seem to be assuming that you must stream logs to the
`evergreen-client` using a local TCP socket, which is not obvious to
me. There are other options, such as a text file or set of files in
the `var` directory, which would have different robustness
characteristics.

Yes, sorry. I should have clarified upfront why I didn't put that here to save iterations for everyone.

So the design/thought process has been the following: writing to rotating log files was my first idea, we would be able to have a fixed disk usage for logs using java.util.logging.FileHandler.limit and java.util.logging.FileHandler.count for instance OOTB.

But Tyler was "not super keen on hitting the file system for completely ephemeral data passed between evergreen-client and jenkins", and asked asked if we couldn't use a unix socket instead.

After a quick search, I concluded that though technically feasible, using unix sockets from Java was not the right thing to do (IIUC, it requires using JNI/JNR, which I still have scars from JNI usage in 2003 :)).

> means we add that component in the core, or at least in some jar added
> to its classpath or something (?). Less common usage pattern than a plugin,
> could it lead to issues?

I think what you are looking for is `jenkins-module` packaging: akin
to a plugin, but bundled in `jenkins.war!/WEB-INF/lib/*.jar` so that
its classes are available in the same loader as `jenkins-core.jar`. If
you are patching `jenkinsci/jenkins/war/pom.xml` for this anyway, this
is trivial (just add it to the list of modules); otherwise, there is a
mojo in `maven-hpi-plugin` to create a customized Jenkins WAR file
based on a stock `org.jenkins-ci.main:jenkins-war` release with some
additions and modifications, which would be appropriate for a
distribution like Essentials / Evergreen anyway.

Right. TBH though I fully agree with your last statement for the next months, I wonder how much this is *immediately* desirable for the next _weeks_. 

Because that would mean to me we right away need to setup a more involved release pipeline on the Jenkins Essentials WAR itself, when we're currently still working hard on making it just usable (and switching to another WAR flavour _could_ be done later).

 

> we need to use a bit more convoluted way to get
> the early logs from the startup reading Jenkins.logRecords before being able
> to setup our own JUL Handler

This is pretty trivial, no? You register your handler, and at that
moment you also copy everything in `Jenkins.logRecords`—done.
Obviously if there were more than ~256 messages prior to this plugin’s
initialization, you will miss some of the earlier ones, but in such a
case it should be obvious that something is very wrong.

Yes. It is trivial. Hence why I said "bit more convoluted", and not "more complex" :-).

And IMO you summarized well the convolution again in the 5 lines above again, which makes it eligible as one of the cons of this strategy :).

 

> we'll miss a few very early startup logs which are, IIUC, sent before the
> core sets up the RingBufferLogHandler to populate Jenkins.logRecords.

Sure, but if there were any `WARNING`+ messages printed before that
point very early in Jenkins startup, it likely that the instance is
more or less hosed and you will notice this from a standard HTTP
status check anyway.

That is right. And for first iterations we plan anyway indeed to only check Jenkins health from an HTTP call, so we'd detect it.

But that is still just another (small) disadvantage of this solution.

 

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr2mbREAVyteyaC0KDNKw33fNdzCBZK14RN374QYUAnGTQ%40mail.gmail.com.

[Baptiste Mathus]

So, to clarify: I would like to keep this thread dedicated to the pros and cons to either write a `jenkins-module` as advised by Jesse, or a Jenkins plugin to host the classes that this will require, whatever the implementation paths we choose. 

For instance, I think I'll also need to write a JSONFormatter for easier consumption/parsing by the evergreen-client, so we definitely need a place to put those.

I'll start or fork a dedicated thread to discuss implementations choices. Or someone feel free to do it if having any feedback to provide.