Adding audit logs to Jenkins

[Matt Sicker]

With the recent release of Log4j Audit [1], I had an ambitious thought: can we add audit logging to Jenkins? Logging is currently handled mostly via the standard Java logging system which has many disadvantages over using a proper logging framework like Log4j 2 or Logback. One important advantage given by Log4j2 not found in SLF4J, Logback, JUL, and others, is the log message API [2] which allows for structured logging. Log4j Audit provides a way to define and catalog audit events (e.g., a build is invoked by user A with other contextual details like request IDs, etc.) along with logging those events. An important difference between an audit log and say a debug log is that an audit log is not disabled at production; in fact, trying to or successfully disabling audit logging should itself create an audit log. Audit logs are extremely important in certain domains, and in order to encourage the use of DevOps and CI/CD in constrained or regulated environments, such a feature can help put to rest auditor concerns. On a more personal level, this also makes it easier to centrally debug your own Jenkins system with better context and history. If I put on my security hat, then this feature also helps admins detect any security breaches in realtime (provided they're monitoring the audit logs).

I did a cursory search for existing efforts and found an old plugin [3] along with an old blog post [4] about how to do this. I'd like to initiate an effort to make audit logging, distributed trace logging, and other logging improvements. This would supersede my previous post about using Log4j as this becomes a superset of the idea.

I'd like to gauge the interest here before spending any time writing my first JEP.

[1]: https://github.com/apache/logging-log4j-audit

[2]: https://logging.apache.org/log4j/2.x/manual/messages.html

[3]: https://wiki.jenkins.io/display/JENKINS/Audit+Trail+Plugin

[4]: https://www.cloudbees.com/blog/best-practices-setting-jenkins-auditing-and-compliance

--

Matt Sicker

Software Engineer, CloudBees

[Jesse Glick]

On Thu, Jun 21, 2018 at 11:05 AM Matt Sicker <msi...@cloudbees.com> wrote:
> e.g., a build is invoked by user A with other contextual details like request IDs, etc.

From this one example it is hard for me to see where you are going in
terms of user-visible features. Can you flesh out the kinds of use
cases you would expect to enable with such an infrastructure? Would
they be roughly comparable to the features of the old Audit Trail
plugin, or going beyond that? Is there any relationship to
JEP-304/308?

[Matt Sicker]

On Thu, Jun 21, 2018 at 2:54 PM Jesse Glick <jgl...@cloudbees.com> wrote:

From this one example it is hard for me to see where you are going in
terms of user-visible features. Can you flesh out the kinds of use
cases you would expect to enable with such an infrastructure? Would
they be roughly comparable to the features of the old Audit Trail
plugin, or going beyond that? Is there any relationship to
JEP-304/308?

I'll need to do more research on existing audit use cases, but yes, the idea would be to try and replicate the existing audit data. I wanted to make sure this isn't an inviable idea first before sinking more time into it. And yes, I think this could be useful for JEP 304 and 308, though I'm not too familiar with Jenkins Essentials.

Some advantages I could see to this system would be integrating audit events from other applications or infrastructure to form a cohesive overview of your system. There's a bit more justification behind Log4j Audit itself on its homepage [1]. Going with Log4j for general logging also opens up several potential logging-related improvements, but that may be outside the scope of this.

[1]: http://logging.apache.org/log4j-audit/latest/

[Matt Sicker]

I've started exploring the data already available in Jenkins APIs and have started a proposal. Here's an intro so far:

Overview

Many corporations are bound to regulatory requirements where certain actions must be restricted to certain people and must be proveable in some fashion to regulators. One way to help close any gaps in security issues or poor role definitions technically is to maintain an immutable audit log of actions performed in a system including information such as who did it, when did it occur, what systems were affected, and any relevant data used in the change. Then an audit can be done by simply checking the audit log to see if anyone is performing unauthorized activity. Automatic security monitoring software can be used with these audit logs to detect security problems as they occur rather than finding out much later. There exist a couple plugins that add some audit information to Jenkins: Job Config History, Audit Trail, and Audit to Database. One key issue with these plugins (of those that are maintained) do not provide a way to centralize these audit logs. Another issue is that by utilizing the underlying java.util.logging framework, these frequent audit events can hit known performance issues with JUL.

Using Apache Log4j Audit, this project will define several base audit events along with configurability to where these audit events will be logged (e.g., syslog, log file, database, etc.). Log4j Audit provides an easy to use API for logging audit events along with an event catalog editor that itself keeps an audit trail in Git. Dynamic audit events can themselves be audited when configured with a Postgres database (or potentially any SQL-compatible RDBMS). As a side effect of making this a core feature of Jenkins, this also opens up the opportunity to standardize on using Log4j 2.x as the standard diagnostic logging API instead of using JUL.

In Scope

There are several event domains that should be covered by this project. These domains include builds/runs, jobs, tasks in those jobs, projects, the build queue, Saveable config files, computers, nodes/agents (static, dynamic, and ephemeral), cloud provisioning, users, and workspaces. The types of audit events for these domains generally revolve around CRUD operations, start/stop information, and other high level events. In each audit event, the user responsible should be included as well as a unique correlation/request ID added to each HTTP request.

Out of Scope

This project is only being scoped around Jenkins. Thus, things like keeping audit logs of Git or SCMs themselves is out of scope. This project does not intend to bundle a database for use with storing logs.

[Matt Sicker]

I also want to add that there are two divergent approaches to implementing this: integrating directly with Jenkins along with guidelines for other plugins to add new events; or implemented as a plugin. The plugin approach would allow for support of older versions of Jenkins, but logging config management gets rather complicated when implemented in this way.

[Jesse Glick]

BTW you should also look at `SecurityListener` and talk with Wadeck
who had some ideas about that.

[Oleg Nenashev]

We have recenrly started the External Logging stories in Cloud Narive SIG. If you plan a fresh new implemebration for a pluggable Audit Logging engine, it may worth taking a look. We will have more updates this week.

Regars8ng the Audit, I believe that integrarion of SecurityListener into Audit Trail could be a good first change. Then we may get an engine for Jenkins events.

Integrating fresh new tools is a valiant goal, but maybe we need to discuss it a bit more before implementing

[Jesse Glick]

On Tue, Jul 24, 2018 at 7:54 PM Oleg Nenashev <o.v.ne...@gmail.com> wrote:
> We have recenrly started the External Logging stories in Cloud Narive SIG. If you plan a fresh new implemebration for a pluggable Audit Logging engine, it may worth taking a look.

I doubt there is much overlap. All audit events of interest would be
sent from the master (since agents are untrusted). If system log
events are being directed somewhere other than stderr of the servlet
container, which they would probably need to be in order to do
meaningful audits of structured data, then fine. We expect to
demonstrate a very simple redirection of JUL logs to Fluentd using a
custom handler emitting JSON, but of course you could do the same for
Log4J, most likely using a standard connector.

[Matt Sicker]

I'm one of the main developers of Log4j, so besides being a bit biased towards using Log4j 2.x in the first place, I've also got an extensive list of reasons to use it, especially over something like JUL.

Also, based on my research from last month, I noticed that the best way to implement this does appear to be in core as the various listeners don't really expose all the info I'd want, and I'd expect audit logging to be more reliable than that.

As for the external logging, had we been using log4j-api or even slf4j-api or commons-logging, then we could've written an Elasticsearch appender for Log4j (or used an existing one) and saved a lot of trouble in the externalized logging epic. I already wrote a Cassandra appender in the past for a similar use case, so it wouldn't be hard to make one for ES. :)

As for a quick argument for log4j-api over slf4j-api: structured log messages. In log4j-api, you can use custom Message classes for more detailed logs. This is how audit logging is implemented actually. If you want structured log messages stored to something like ES, this is fantastic.

--
You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CANfRfr3H%3Dd%3D%3DU81jobX9Fe5tJg2_X-j2hf3qLxUhB%2B9BNL-aBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

> On 25. Jul 2018, at 17:47, Matt Sicker <msi...@cloudbees.com> wrote:
>
> I noticed that the best way to implement this does appear to be in core as the various listeners don't really expose all the info I'd want

Doesn't that just mean the listeners need to improved?

Do you have specific examples?

[Matt Sicker]

On Wed, Jul 25, 2018 at 6:03 PM Daniel Beck <m...@beckweb.net> wrote:

Doesn't that just mean the listeners need to improved?

Either way, that would require core changes.

 

Do you have specific examples?

Not offhand, though when I take a look at this again, I'll write up some examples.

[Matt Sicker]

I've started tasking out this as an epic: https://issues.jenkins-ci.org/browse/JENKINS-54082

[Liam Newman]

Matt, 

Have you considered make a JEP for this effort?  It seems like something that would benefit from having a design document that captures how the design was arrived at and what alternatives were considered.  

-Liam N.

--

You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAEot4ox4soNv29ARgfRrzDZeWq%2Bx7BXA5hk5p39tFX%2BNzrNsBg%40mail.gmail.com.

[Matt Sicker]

I've thought about a JEP, yes. I'm not sure if the scope warrants it yet. I'm currently exploring this as a potential Outreachy mentorship project, so it might be a bit early still.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/CAA0qCNy7WKOi4oJfh_kV5kMeiRFP2y_hAsg%2Bg-TX4uYrd98WJA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jesse Glick]

On Mon, Oct 15, 2018 at 6:00 PM Matt Sicker <msi...@cloudbees.com> wrote:
> I've thought about a JEP, yes. I'm not sure if the scope warrants it yet.

If you have to wonder, the answer is yes. :-)

[Baptiste Mathus]

Heh, I like that. 

Proposed https://github.com/jenkinsci/jep/pull/209 :-)

-- Baptiste 

[Matt Sicker]

On Mon, Oct 15, 2018 at 7:17 PM Jesse Glick <jgl...@cloudbees.com> wrote:

If you have to wonder, the answer is yes. :-)

Simple enough!